Aaron Swartz: I first met Jimbo Wales, the face of Wikipedia, when he came to speak at Stanford. Wales told us about Wikipedia’s history, technology, and culture, but one thing he said stands out. “The idea that a lot of people have of Wikipedia,” he noted, “is that it’s some emergent phenomenon — the wisdom of mobs, swarm intelligence, that sort of thing — thousands and thousands of individual users each adding a little bit of content and out of this emerges a coherent body of work.”† But, he insisted, the truth was rather different: Wikipedia was actually written by “a community … a dedicated group of a few hundred volunteers” where “I know all of them and they all know each other”. Really, “it’s much like any traditional organization.”
The difference, of course, is crucial. Not just for the public, who wants to know how a grand thing like Wikipedia actually gets written, but also for Wales, who wants to know how to run the site. “For me this is really important, because I spend a lot of time listening to those four or five hundred and if … those people were just a bunch of people talking … maybe I can just safely ignore them when setting policy” and instead worry about “the million people writing a sentence each”.
So did the Gang of 500 actually write Wikipedia? Wales decided to run a simple study to find out: he counted who made the most edits to the site. “I expected to find something like an 80-20 rule: 80% of the work being done by 20% of the users, just because that seems to come up a lot. But it’s actually much, much tighter than that: it turns out over 50% of all the edits are done by just .7% of the users … 524 people. … And in fact the most active 2%, which is 1400 people, have done 73.4% of all the edits.” The remaining 25% of edits, he said, were from “people who [are] contributing … a minor change of a fact or a minor spelling fix … or something like that.” ...
[But what if we analyze the amount of text contributed by each person, not just the number of edits? See original for analysis of edit patterns of specific articles, including amount of text added.]
... When you put it all together, the story become clear: an outsider makes one edit to add a chunk of information, then insiders make several edits tweaking and reformatting it. In addition, insiders rack up thousands of edits doing things like changing the name of a category across the entire site — the kind of thing only insiders deeply care about. As a result, insiders account for the vast majority of the edits. But it’s the outsiders who provide nearly all of the content.
And when you think about it, this makes perfect sense. Writing an encyclopedia is hard. To do anywhere near a decent job, you have to know a great deal of information about an incredibly wide variety of subjects. Writing so much text is difficult, but doing all the background research seems impossible.
On the other hand, everyone has a bunch of obscure things that, for one reason or another, they’ve come to know well. So they share them, clicking the edit link and adding a paragraph or two to Wikipedia. At the same time, a small number of people have become particularly involved in Wikipedia itself, learning its policies and special syntax, and spending their time tweaking the contributions of everybody else.
Thursday, January 07, 2010
Wikipedia: emergent phenomenon?
Is Wikipedia a magical aggregator and filter of expertise from millions of different contributors? Or is it more like traditional encyclopedia projects, with a thousand or so core Wikipedians doing most of the work? The distribution of edits (a typical power law) supports the latter interpretation, but a detailed analysis of particular articles shows that important knowledge is injected by individuals who are not part of the core group.
Subscribe to:
Post Comments (Atom)
8 comments:
What I've always wondered is if Wikipedia articles tend to improve (i.e. be more accurate and complete) over time. It seems like many of them improve until a peak then degrade from there. Also, it seems like that pattern is especially pronounced on controversial articles.
Based on my own experience, that's exactly what I always suspected. I only edited three entries ever, on subjects where I am likely to be among ~ top 10^2 experts in the world (and I do wish that people did not touch subjects on which they know little in depth). But then, I am HTML-illiterate and I can't be bothered with a non-GUI interface to "Wiki" the edits. Someone else then took care of the formatting, sectioning and style. Those entries are all noncontroversial and the articles do improve over time (only marginally from certain point).
I also noticed that those insiders frequently start new entries even though they have very limited knowledge of the subject. I suppose this sort of "priming" can be good but as a result many articles on relatively obscure things look like a total joke.
Swartz's research is very old and his conclusions are not shared by more updated researchers. It occurred well before the 2007-2008 bump in the number of new regular editors of Wikipedia.
I highly recommend the work of PARC's Augmented Social Cognition Group as an alternative.
Their work clearly demonstrates that not only are the majority of contributions derived from a core of dedicated contributors, but that the margin between those who contribute casually and those who contribute a great deal is growing.
Steven Walling: Are you saying that an overwhelming majority of the actual Wikipedia expertise resides with Wikipedians? I would actually prefer the alternative that Nanonymous describes, in which real experts contribute their knowledge, which is then cross-checked, cleaned up, etc. by Wikipedian editors.
Your description and Nanonymous' experience could be reconciled if there are a lot of mediocre articles -- that is, the number of entries written by dedicated (but non-expert) Wikipedians dominates the number written (or contributed to) by real experts. I do notice a lot of "stub" entries on specialized, but important, subjects that seem to have been merely cribbed from a generic source.
I think you will get a laugh out of this. It isn't related to wikipedia at all though.
http://online.wsj.com/article/SB126238854939012923.html
Steve:
To ask whether the expertise comes from amateur Wikipedians or professionals is a question based on a fundamental misunderstanding of what Wikipedia is. Wikipedia is a tertiary source that depends on what published sources, not individuals, say. Yes, some poorer articles don't cite their sources well, but no fact in Wikipedia ever rests on personal expertise.
If personal expertise is not a substantial influence on the site, then asking whether the expertise comes from amateur Wikipedians or anonymous experts is pointless. As Cory Doctorow explains, "While the Britannica says, These facts are true, Wikipedia says, It is true that these facts were reported by these sources. The Britannica contains facts, Wikipedia contains facts about facts."
Swartz's research isn't about where the facts within Wikipedia come from, it's about who contributes the majority of valuable textual content - whether it's a core community of editors or a distributed, anonymous crowd. As a Wikipedia editor himself, he knows that, no matter who they are, none of knowledge added to Wikipedia comes directly from the individual making the edits.
Where more contemporary research disagrees with Swartz is in his conclusion that the majority of substantive content is added by anonymous, casual contributors rather than devoted Wikipedians who contribute regularly.
"To ask whether the expertise comes from amateur Wikipedians or professionals is a question based on a fundamental misunderstanding of what Wikipedia is. Wikipedia is a tertiary source that depends on what published sources, not individuals, say."
This is a misunderstanding of how knowledge is actually filtered and aggregated. It requires significant expertise to understand *which* references are important, *which* points are the most salient, *which* claims are correct or well-supported by facts or data. To claim that the process of assembling an encyclopedia does not rely on specialized expertise is completely incorrect. Whose expertise is being used to create the entries? is an extremely important question. Many scientific encyclopedias rely on individual contributors to cover particular topics and the quality of an entry is usually a direct reflection of the quality of the expert. (This is rather obvious to me when I look at Wikipedia, btw. I strongly suspect that crappy articles are often assembled in a low quality way by enthusiastic but non-expert core Wikipedians or outsiders who are not really experts, whereas very good articles have had outside input from real experts.)
If Wikipedia were only about "facts about facts" it would be unreadable and many entries would be 100x longer than they currently are. One cannot report all statements ever made about the big bang, even all the correct ones. An article of finite length has to rely on expert judgements about *which* facts or theories are important enough to be included (as well as which ones are correct to begin with!). This is related to the concept of compression, and many experts believe that AI is more or less equivalent to the ability to compress information. So to generate a compressed representation (i.e., encyclopedia entry) of a body of knowledge requires intelligence. The question is: whose intelligence? That of 1000 Wikipedians, or some larger set, with crucial contributions from outside?
From the reading I've done and my experience as a Wikipedian, the answer to your question is it's both, depending on how you look at it.
Statistical studies I trust show unambiguously that the majority of significant additive edits to Wikipedia are the product of a dedicated community of a few thousand people.
The degree to which those high frequency editors are professionals/experts varies. If you're talking about very technical subjects, maths and sciences, there are more experts. But those are generally exceptions when you look at Wikipedia as a whole. On average Wikipedians are passionate, knowledge-hungry amateurs who write about something to learn as much as to share what they already know.
But regardless of the subject and whether the editor is a professional or not, Wikipedia relies on verification from outside expert source material. If we want to talk about crucial contributions from those who consider themselves outside the community, I would say that's where it's at, rather than in actual edits to the site.
Post a Comment