A Jury of Your Peers: Quality,
Experience and Ownership in Wikipedia
Aaron Halfaker
Aniket Kittur (CMU)
Robert Kraut (CMU)
John Riedl
University of Minnesota
Presentation Outline
1) Formal peer review and Wikipedia
2) Analysis of review in Wikipedia
3) Results
•
Quality
•
Experience
•
Ownership
4) Conclusions
University of Minnesota
A Jury of Your Peers
Presentation Outline
1) Formal peer review and Wikipedia
2) Analysis of review in Wikipedia
3) Results
•
Quality
•
Experience
•
Ownership
4) Conclusions
University of Minnesota
A Jury of Your Peers
Presentation Outline
1) Formal peer review and Wikipedia
2) Analysis of review in Wikipedia
3) Results
•
Quality
•
Experience
•
Ownership
4) Conclusions
University of Minnesota
A Jury of Your Peers
Presentation Outline
1) Formal peer review and Wikipedia
2) Analysis of review in Wikipedia
3) Results
•
Quality
•
Experience
•
Ownership
4) Conclusions
University of Minnesota
A Jury of Your Peers
Peer Review Systems
●
Peer reviewed conferences (like WikiSym)
●
NSF grant panels
●
Academic journals
●
Art competitions
University of Minnesota
A Jury of Your Peers
Peer Review Systems
●
Peer reviewed conferences (like WikiSym)
●
NSF grant panels
●
Academic journals
●
Art competitions
●
Wikipedia?
University of Minnesota
A Jury of Your Peers
University of Minnesota
A Jury of Your Peers
Traditional Peer Review
University of Minnesota
A Jury of Your Peers
Traditional Peer Review
University of Minnesota
A Jury of Your Peers
Traditional Peer Review
University of Minnesota
A Jury of Your Peers
Traditional Peer Review
University of Minnesota
A Jury of Your Peers
Wikipedia Peer Review
University of Minnesota
A Jury of Your Peers
Wikipedia Peer Review
University of Minnesota
A Jury of Your Peers
Wikipedia Peer Review
University of Minnesota
A Jury of Your Peers
Wikipedia Peer Review
University of Minnesota
A Jury of Your Peers
Wikipedia Peer Review
University of Minnesota
A Jury of Your Peers
Wikipedia
University of Minnesota
Traditional
A Jury of Your Peers
Analysis of Traditional Peer Review
●
Expert Evaluation
●
Goal: What is the quality of a review?
–
●
How many rejections should have been accepted?
Statistical Model
●
Goal: What predicts the outcome of a review?
–
–
Is past acceptance highly correlated with future
acceptance?
Are submissions that are critical of reviewers' previous
work correlated with rejection?
University of Minnesota
A Jury of Your Peers
Our approach: Statistical
●
Can't determine the quality of a review
●
Resilient to researcher bias.
●
●
Allows us to compare the effects of various
metrics in predicting outcome
●
Quality
●
Experience
●
Ownership
Answers: How do quality, experience and
ownership effect the outcome of review—being
reverted?
University of Minnesota
A Jury of Your Peers
Statistical Model
●
Sample
●
Jan. 2008 snapshot of English Wikipedia
●
Observation = revision
●
n = 1.4 million
●
Boolean outcome: was the revision eventually
reverted?
University of Minnesota
A Jury of Your Peers
Hypothetical “Ideal” Model
University of Minnesota
A Jury of Your Peers
Hypothetical “Ideal” Model
University of Minnesota
A Jury of Your Peers
Hypothetical “Ideal” Model
University of Minnesota
A Jury of Your Peers
Hypothetical “Ideal” Model
University of Minnesota
A Jury of Your Peers
Hypothetical “Ideal” Model
University of Minnesota
A Jury of Your Peers
Quality = Word Persistence
Assumption: Words that last are part of a high quality edit.
University of Minnesota
A Jury of Your Peers
Quality = Word Persistence
Application
●
●
Removed Word Persistence: Edits that remove established
words are removing high quality content and are likely to be low
quality changes.
Recent Edit Quality: Editors who have recently contributed
content that lasts several revisions are high quality editors.
Justification
●
Significant correlation between high word persistence
editors and increased in the Wikipedia 1.0 Assessment
rating.
University of Minnesota
A Jury of Your Peers
Hypothetical “Ideal” Model
University of Minnesota
A Jury of Your Peers
Hypothetical “Ideal” Model
University of Minnesota
A Jury of Your Peers
Hypothetical “Ideal” Model
University of Minnesota
A Jury of Your Peers
Stepping on Toes
“Removing words added by editors when they are likely to
notice.”
University of Minnesota
A Jury of Your Peers
Hypothetical “Ideal” Model
University of Minnesota
A Jury of Your Peers
Results
●
Quality
●
Experience
●
Ownership
University of Minnesota
A Jury of Your Peers
Quality
●
●
●
Removed word
persistence
Recent edit quality
Recent success
rate
University of Minnesota
A Jury of Your Peers
Recent quality is a powerful predictor.
●
●
●
Removed word
persistence
Recent edit quality and revert probability
Recent edit quality
Recent success
rate
University of Minnesota
A Jury of Your Peers
Recent quality is a powerful predictor.
●
●
●
Removed word
persistence
Recent edit quality and revert probability
Recent edit quality
Recent success
rate
University of Minnesota
A Jury of Your Peers
Experience
University of Minnesota
A Jury of Your Peers
Editors don't exhibit a learning effect.
Tenure: time since editor's first edit
University of Minnesota
A Jury of Your Peers
Editors don't exhibit a learning effect.
Tenure: time since editor's first edit
Lifetime: time an editor will continue to edit
University of Minnesota
A Jury of Your Peers
Editors don't exhibit a learning effect.
Tenure: time since editor's first edit
Lifetime: time an editor will continue to edit
Tenure
p = 0.37
The predictive
power of lifetime
consumes tenure.
Lifetime
p = < 0.0001
University of Minnesota
A Jury of Your Peers
Ownership
University of Minnesota
A Jury of Your Peers
Ownership has a powerful effect over
which contributions are rejected.
University of Minnesota
A Jury of Your Peers
Ownership has a powerful effect over
which contributions are rejected.
●
University of Minnesota
Very powerful predictor
A Jury of Your Peers
Ownership has a powerful effect over
which contributions are rejected.
Very powerful predictor
● Independent
●
●
●
●
University of Minnesota
A Jury of Your Peers
Experience
Quality of work
Recent success rate
Ownership has a powerful effect over
which contributions are rejected.
Very powerful predictor
● Independent
●
●
●
●
University of Minnesota
A Jury of Your Peers
Experience
Quality of work
Recent success rate
Ownership has a powerful effect over
which contributions are rejected.
Very powerful predictor
● Independent
●
●
●
●
University of Minnesota
A Jury of Your Peers
Experience
Quality of work
Recent success rate
Ownership has a powerful effect over
which contributions are rejected.
Very powerful predictor
● Independent
●
●
●
●
University of Minnesota
A Jury of Your Peers
Experience
Quality of work
Recent success rate
Ownership has a powerful effect over
which contributions are rejected.
Very powerful predictor
● Independent
●
●
●
●
Experience
Quality of work
Recent success rate
If you step on other
editors' toes, you're
likely to be reverted.
University of Minnesota
A Jury of Your Peers
Resulting Model
Quality
University of Minnesota
A Jury of Your Peers
Resulting Model
Quality
●
●
University of Minnesota
A Jury of Your Peers
As hypothesized
Neg. correlated
with revert prob.
Resulting Model
Quality
●
●
As hypothesized
Neg. correlated
with revert prob.
Experience
●
University of Minnesota
A Jury of Your Peers
No effect after
controlling for
lifetime/total edits
Resulting Model
Quality
●
●
As hypothesized
Neg. correlated
with revert prob.
Experience
●
University of Minnesota
A Jury of Your Peers
No effect after
controlling for
lifetime/total edits
Resulting Model
Quality
●
●
As hypothesized
Neg. correlated
with revert prob.
Experience
●
No effect after
controlling for
lifetime/total edits
Ownership
University of Minnesota
A Jury of Your Peers
Resulting Model
Quality
●
●
As hypothesized
Neg. correlated
with revert prob.
Experience
●
No effect after
controlling for
lifetime/total edits
Ownership
●
University of Minnesota
A Jury of Your Peers
Powerful and
independently
correlated with
revert prob.
Contributions/Summary
We constructed a logistic regression model to
estimate the effects of various editor and edit
characteristics on rejection in Wikipedia.
●
Verified word persistence as a robust metric
for perceived quality.
●
Editors do not show a learning effect.
●
Ownership of content has a powerful effect.
University of Minnesota
A Jury of Your Peers
Acknowledgments
●
●
●
My co-authors
●
Niki Kittur - [email protected]
●
Robert Kraut - [email protected]
●
John Riedl - [email protected]
The Wikipedia group in the GroupLens lab
This work has been financially supported by the
National Science Foundation: IIS 05-34420
Aaron Halfaker
[email protected]
Recent quality is a powerful predictor.
●
●
●
Removed word
persistence
Removed word persistence and revert probability
Recent edit quality
Recent success
rate
University of Minnesota
A Jury of Your Peers