Wikipedia:Wikipedia Signpost/2012-04-23/Investigative report

It began on 17 April with a misleadingly titled report on the newswise site, "Survey finds most Wikipedia entries contain factual errors". This was reprinted on the same day by the online research news site Science Daily ("Most Wikipedia entries about companies contain factual errors, study finds"). Within days it had gone viral on internet news sites all over the world. The story was picked up by the American ABC news blog ("Wikipedia: public relations people, editors differ over entries"), The Telegraph in the UK ("Six out of 10 Wikipedia business entries contain factual errors"), the Indian edition of NYDailyNews ("Wikipedia entries full of factual errors, says researcher"), and The Register, a British technology news and opinion website ("Let promoters edit clients' Wikipedia entries"). One outlet, the Business2Community, went so far as to announce that "a new study published in the Public Relations Journal shows that a stunning 60 percent of articles about specific companies contained factual errors."

At the centre of the hubbub are a set of research results that their author, Pennsylvania State University's Marcia W. DiStaso, claims "will help establish a baseline of understanding for how public relations professionals work with Wikipedia editors to achieve accuracy in their clients' entries". The study involved a survey of nearly 1300 public relations and communications professionals to analyse how they work with the English Wikipedia. Funded by Penn State's Arthur W. Page Center for Integrity in Public Communication and recently published in Public Relations Journal, the paper goes by the title "Measuring public relations Wikipedia engagement: how bright is the rule?", a play on Jimmy Wales's "bright line" – a reference to the boundary he advocates people with a conflict of interest in a topic should not cross by never editing articles directly.

The results, which have cast a shadow over the English Wikipedia's company articles, have relevance to the ongoing debate about whether paid editing should be officially permitted on Wikipedia. "Public relations professionals have their hands tied," DiStaso told ABC. "They can only make comments on discussion pages suggesting corrections, and wait for the public to reply.” She believes that while waiting for a reply, a company may be caught in a crisis of public image: “In today’s fast-paced society, five days is a long time.”

Scrutinising the 60% claim
The claim that 60% of Wikipedia's articles contain factual inaccuracies, however, doesn't stand up to basic scrutiny. We have bulleted the quotation for ease of reading:
 * "When asked if there are currently factual errors on their company or client’s Wikipedia articles,
 * 32% said that there were (n=406),
 * 25% said that they don’t know (n=310),
 * 22% said no (n=273), and
 * 22% said that their company or client does not have a Wikipedia article (n=271).
 * In other words, 60% of the Wikipedia articles for respondents who were familiar with their company or recent client’s article contained factual errors." (our underlining)

The problem is hidden in the underlined clause. This cleverly allows DiStaso to exclude from the sample the 25% of respondents who ticked "don't know". DiStaso told The Signpost, "if all respondents were familiar with their articles this could go either way – more 'yes' answers would make it a higher % and more 'no' answers would make it lower." But although it's valid to exclude respondents whose companies have no article (the last bullet), excluding the don't knows, which boosts the factual error rate to 60% (406 / [406 + 273]) raises difficult issues. Including the don't knows would yield 41% (406 / [406 + 310 + 273]). This problematic calculation was independently pointed out by Tilman Bayer (HaeB) of Wikimedia's communications team, who has postgraduate degrees in mathematics and co-edits the Wikimedia Research Newsletter. The true percentage is almost certainly not 60.

One of the problems with the way in which the findings have been disseminated is the omission of the critical clause from most news reports. Most news journalists have apparently not grasped that the "60%" claim represents a relatively narrow statistical artefact. They can hardly be blamed when the press release by Dick Jones Communications – the PR company that represents her college – started that ball rolling ("Survey finds majority of Wikipedia entries contain factual errors").

Bayer also identified that two statistical biases are not accounted for in DiStaso's results. "Companies which note errors in the WP article about them," he told us, "would seem more likely to pay PR professionals to devote attention to that article". Then there's participation bias: "PR professionals who are called on to solve such problems," he says, "would seem more likely to feel motivated and informed enough to participate in such a survey than those who have not encountered such problems." And the “call for action” text while soliciting respondents here risks being regarded as a significant contaminating influence on participation.

In addition, a list of categories of errors were presented to respondents, mixing the more political and subjective "Criticisms" with more objective criteria such as dates, board membership, and even spelling in the article text. Indeed, more than one in five respondents ticked "incorrect spelling" as a category of inaccuracy, which suggests the possible scenario in which one US spelling in an article about a British company might be enough to classify a whole article as inaccurate. DiStaso told us, "I suppose it is possible but probably unlikely that [such a spelling inconsistency] would be considered a 'factual error' [but] there could be other words that if misspelled could be considered factual errors such as the misspelling of a product name." We wonder whether this distinction was clear to respondents.

Another potential flaw in the methodology was that respondents were not asked to read the article on their company or client and identify the errors as they saw them; this would have enabled reliable verification of what the perceived errors were, and the extent to which they could be considered to be errors.

Further results
According to Science Daily, DiStaso says "what is surprising ... is that 25 percent of survey respondents indicated they are not familiar with the Wikipedia articles for their company or clients. At some point most, if not all, companies will determine they need to change something in their Wikipedia entries. Without clear, consistent rules from Wikipedia regarding how factual corrections can be made this will be a very difficult learning process for public relations professionals."

A key claim in interpreting the data is that public relations professionals find responses to their talk-page requests for changes to articles either slow or non-existent (nearly a quarter reported no response at all, and 12% said it took "weeks" to get a response). However, the data appear to disregard the size, age, quality, and hit-rates of articles on which the data was based.

The data also led to the claim that PR professionals have little understanding of Wikipedia's rules for editing and the protocol for contacting editors to have facts altered – but just how a lack of understanding interacts with political considerations was not made clear: "Only 35 percent of respondents were able to engage with Wikipedia, either by using its "Talk" pages to converse with editors or through direct editing of a client's entry. Respondents indicated this figure is low partly because some fear media backlash over making edits to clients' entries. ... Twenty-nine percent said their interactions with Wikipedia editors were 'never productive'."

This lack of understanding by PR professionals of how to use Wikipedia's infrastructure, some of it explicitly set up for them, sits oddly with the article's strident argument that Wikipedia's policies needs to be changed. Unusually for a scholastic journal, the key "findings" of the paper are displayed on the download site in a larger-than-life ad-like infographic, which is now appearing elsewhere on the internet.

Facebook lobby group
DiStaso is closely allied with the newly established Facebook group Corporate Representatives for Ethical Wikipedia Engagement (CREWE). One of the aims of the group is to get Jimmy Wales to change his opinion about paid editors directly editing articles.

There have been claims that Wales is or has been a member of CREWE. He told The Signpost in no uncertain terms: "(1) I am not a member of CREWE. I do not approve of their attempt to forcibly change Wikipedia policy by off-site coordination of paid advocates in a facebook group; and (2) I think it's a mistake to give their nonsense any attention whatsoever. Wikipedia is not for sale."

Phil Gomes, who launched CREWE and has played a major part in recruiting its members, responded to Wales's remarks: "CREWE is about exploring where company communicators and Wikipedians can work together towards the mutual objective of accurate entries. The primary outputs of this group have looked at the best ways that we can educate PR people to do right by Wikipedia. ... To characterize this is as an 'attempt to forcibly change Wikipedia policy by off-site coordination of paid advocates in a Facebook group' is inaccurate and a bit exaggerated. No one is forcing anyone to do anything. Our most passionate contributors even describe themselves as Wikipedians, not PR people. Evidently, the "public shaming" approach to past bad PR behavior is not a deterrent. We're trying to be proactive by educating instead. If someone considers any of this to be 'nonsense', then that's a shame."

Andrew Lih, author of The Wikipedia Revolution: How a Bunch of Nobodies Created the World's Greatest Encyclopedia, an associate professor at the University of Southern California Annenberg School of Communication and Journalism, has taken DiStaso's work to task on the CREWE page itself. "I will state the question again, which has been avoided. Can you in good conscience and as a good academic use that report to stand by the words "60% of Wikipedia articles had factual errors" (these are DiStaso's own words to accompany the announcement of the report). Or stand by the PRSA headline: "Survey finds majority of Wikipedia entries contain factual errors" (PRSA's words to announce the report). These two statements must be strongly rejected, or there is no chance to see eye to eye on having PR folks to edit Wikipedia."

Science Daily reports DiStaso as saying that "the status quo can't continue. A high amount of factual errors doesn't work for anyone, especially the public, which relies on Wikipedia for accurate, balanced information. ... If errors are found or if public relations professionals believe content needs to be added or changed, they should refer to the [CREWE] Wikipedia Engagement Flowchart, available on Wikimedia Commons, for guidance on requesting edits." The flowchart, first posted to Commons on 2 April, is due to be finalised by the end of June. The Signpost notes that among other advice, the flowchart says that if an issue raised at the COI noticeboard has not been addressed within 48 hours, "you are now entitled to complain about Wikipedia in any forum you want."

ABC reports Jay Walsh, director of communications for the Wikimedia Foundation, as conceding that DiStaso has a point about the slow responses by Wikipedians to requests for corrections to company articles.