User:Scs/Wiki karma proposal

Lots of people are interested in Wikipedia and other wikis, because of their open, dynamic, collaborative nature. However, lots of people are also worried about Wikipedia, because of the everpresent possibility of vandalism and incorrect information.

Lots of people have made lots of proposals for how to "fix" wikipedia. Some argue for a more formal article review process; some argue for the banning of anonymous editors, or for the establishment of distinct "grades" of editors with different permissions and capabilities. But at the same time, others fear that any such mechanisms could significantly damage the openness that made Wikipedia a success in the first place.

Here is my own proposal -- which at this point is really just an idea, certainly not a proposal, let alone a formal one -- for a "point" system to capture the quality of individual wiki articles and the experience of individual wiki editors.

Though I wrote this and it's currently under my home page, it is just like anything else in Wikipedia in that you can edit it, or discuss it on its talk page. But note that it is not a formal proposal at this point, and that everywhere it uses the word "will", it should be read as "would, if something like this idea is pursued".

This scheme obviously shares some ideas and some nomenclature with other, similar rating systems, such as slashdot's "karma" system and everything2's "XP" system. However, it should not be imagined to be an exact copy of (or even directly analogous to) any of those systems; please think abut and evaluate it on its own merits.

Idea
The idea is to encapsulate two different kinds of "ratings" or "scores":


 * 1. A notion of the "maturity" of an article; articles gradually "solidify" as they are improved over time, and require more determination to edit.


 * 2. A notion of the "experience" or "karma" of an editor. Over time, responsible editors gain "points" which enable them to perform more -- and more significant -- edits.

Notion 1 is somewhat like the hypothetical, emerging notion of an explicitly-stamped "reviewed" or "official" revision in the existing Wikipedia, but it is a continuous and open-ended function, not a binary distinction.

Notion 2 is somewhat like the distinction in some wikis between casual and "approved" or "official" editors, but again, it is a continuous and open-ended function, not a binary distinction.

Notion 2 obviously presupposes registration as an editor. Anonymous editing may still be allowed, but an anonymous editor will always be assumed to have 0 karma or experience.

Your ability to edit an article depends on (1) its maturity, (2) your karma, and (3) the size and scope of the edit you're trying to make. Simply speaking, the more mature an article is and the bigger the change you're trying to make, the more karma you require. However, inexperienced editors might still be able to make small changes -- e.g. obvious typo corrections -- even to "stable" articles.

The "size of the change you're attempting to make" will be computed by an at least slightly sophisticated algorithm. Changes involving more words are obviously bigger than changes involving fewer. Changes involving actual words are bigger than changes involving punctuation, whitespace, or other formatting details. Changes involving deletions may be considered "bigger" than changes involving additions. Certain words will be scored higher: for example, the insertion or deletion of the single word "not" will be scored much higher than other single-word changes.

An individual editor's "karma" or "experience" will depend on many factors.


 * 1. Every edit you make will "cost you some points".  If you have very little experience, not only will you be limited to minor edits, but you will only be able to make a certain number of them per day.  That is, each edit decrements your karma balance by some weighted function of the edit's "size" and the edited article's maturity.


 * 2. Edits you make that survive "review" will gain you some points, and the points you so gain will be significantly more than the points the edit cost you in the first place.  That is, edits "cost" you karma only in the short run -- unless they're contentious, they rapidly gain you karma in the medium and long run.


 * 2a. The notion of "review" does not presuppose some explicit, authoritative editorial review board. Rather, an edit is said to have "passed review" if (a) it is not reverted or massively reedited within some relatively short time window, and quite possibly also if (b) it is positively voted on by readers.  (More on reader feedback below.)


 * 3. An editor's karma will also depend on the quality of that editor's behind-the-scenes interactions with other editors.  Editors who engage in revert wars or are otherwise argumentative or difficult to work with will lose points.  Editors who are able to edit controversial articles diplomatically, and who can nurture the production of articles having the coveted NPOV nature, will gain points. (More on this below.)

Likewise, an article's "maturity" will depend on several factors. Normally, each uncontested edit to an article increases its maturity. Moreover, an edit by an experienced editor bumps the article's maturity level by a higher increment than does an edit by an inexperienced editor. Feedback by readers (see below) can also affect an article's maturity. Articles that are edited very rapidly (multiple times per day), or that have many back-and-forth edits, lose maturity. Articles that go for a long time without being edited may gain maturity. [Are there some contradictions here?]

Finally, it should also be possible for ordinary readers (not just editors) to provide feedback on the quality of the articles they read. This feedback may take several forms:


 * 1. It will be possible for any reader to rate an article, probably on a 5-point scale from 1="needs work" to 5="truly excellent".  Logged-in readers will have "bigger" votes than anonymous readers, but it will be just as possible for average, anonymous readers to rate articles (with of course the necessity of IP-based mechanisms to minimize vote fraud).


 * 2. It will also be possible for a reader to optionally compare two revisions and indicate a preference for one or the other.  This will give more fine-grained feedback on those edits that improve articles versus those that damage them.  (This mechanism is in some ways similar to slashdot's "meta moderation" scheme.)


 * 3. It may also be possible for readers to rate the "notability" of an article, separately from its quality.


 * 4. In a completely passive way, the system may gain some notion of the quality (or at least relevance) of an article by the number of times it is viewed.  (Of course, there's an obvious conundrum here having to do with low-quality but highly controversial and thus oft-viewed articles.)

The most difficult, the most contentious, but perhaps the most important part of this scheme will be the way it allows editors to rate each other. It will be contentious because it will allow editors to vote each other down, and of course contentious editors will want to abuse this ability. It will be difficult because it will have to be carefully tuned to avoid such abuse. But it it important because one of the main points of the exercise is to systematically deprive counterproductive editors of the ability to do substantial damage, so that human administrators and arbitrators don't have to constantly step in and mediate.

The basic premise is that any editor will be able to rate another editor on a 5-point scale from 1="very difficult to work with" to 5="a joy to work with". An editor rated 2 will lose points; an editor rated 4 will gain points. Ratings of 1 and 5 will subtract/add more points than 2 and 4. However, rating another editor in this way will also cost the rating editor some points -- and in fact, it will cost you points regardless of whether you rate the other editor up or down. Moreover, the number of points it costs you will be a function not only of the "strength" of your rating (2,4 versus 1,5), but also of the number of points you already have. For example, giving another editor a "1" rating might cost you 1% of your own points. (And in a similar vein, the number of points added/deducted from the rated editor's score may also be a function of that editor's previous score.) It may sound harsh to "penalize" an editor for rating another editor, but that's not the way to look at it: rating another editor is not free; it costs you something; it's not something to be taken lightly. Imposing a cost on the rating editor in this way is vital in order to prevent editors from ganging up on each other (or from artificially promoting each other).

As mentioned, the precise algorithms and mechanisms underlying editor feedback will be much trickier to design. It will be a challenge to arrange that the system is not unduly vulnerable to the very real possibilities of "gaming" and slandering and other forms of abuse.

At any rate, the system will not be perfect. It will, at least initially, likely be possible for a determined vandal to "game" it, for example by discovering other single-word edits which change meaning as drastically as does the word "not". But the system is obviously expected to evolve over time. (And I tend to suspect that reasonably simple-to-implement mechanisms, such as having longer lists of specific words which are more expensive to insert and/or delete, will by highly effective at catching most forms of abuse.)

Ideally, the scheme would be 100% automatic, requiring no "votes" to approve or disapprove the status of an editor, no formal "review process" to determine the status of an article, and requiring no "administrators" to conduct any votes or reviews, or to perform other overseeing actions. If the article-maturity and editor-karma algorithms were all tuned perfectly, the system would intrinsically reward those articles that were high-quality and those editors that perform high-quality editing. It would quietly marginalize vandals and editors incapable of making high-quality edits. But it would have the same low barriers to entry as the existing Wikipedia. And it would not require everincreasing numbers of administrators and arbitrators.

Practically, this automated scheme will not be sufficient to protect every high-profile, vandalism-prone article, or to quash every determined vandal. However, it should be a huge help in the vast majority of cases, freeing volunteer editors and administrators to concentrate their efforts on actually writing and editing articles, and performing occasional acts of mediation or cleanup, rather than drowning in constant vandal-fighting or debating forever with fanatical POV-pushers and trolls.

Documentation
[In the same way that the documentation for computer software is often usefully written before the software itself is written, it is useful to sketch out some of the words that might be used to describe this system to its users, were the system ever to be adopted.]

The [karma] system is designed to produce high-quality articles and to promote a reasoned, diplomatic, professional editing environment for editors. It is not designed to give editors a scale to judge themselves or each other by; it is not intended that one's own karma score be bragged about or printed on a satin ribbon pinned to one's puffed chest. As Rob Malda has written, "[slashdot's karma] is not your IQ, dick length/cup size, value as a human being, or a score in a video game... It does not cure cancer or grant you a seat on the secret spaceship that will be traveling to Mars when the Krulls return to destroy the planet in 2012.  Karma fluctuates dramatically as users post, moderate, and meta-moderate.  Don't let it bother you.  It's just a number in the database."

The [karma] system is a work-in-progress which continues to evolve to make sure it works well and achieves its goals. It will probably never be perfect, so we need to hear about instances when it has not been achieving the desired result. If you are aware of an article or an editor whose score is too high or too low, please list it in the appropriate section on the _____ page. (Please be aware that resources to investigate these nominations are limited, that well-reasoned arguments receive more attention than irascible ones, that requests to examine a mis-scored article receive more attention if they come from someone who does not have a vested interest in editing that article, that requests to examine a too-low editor score receive more attention if they come from someone who is not that editor, and that requests to examine too-high editor scores usually do not receive any attention at all.)

Our attitude towards the operation and tuning of this feedback system is very similar to google's attitude towards the operation of its search engine. When there is a discrepancy or dispute, only as a last, last resort will we manually adjust the maturity level of an article or the experience level of an editor. Instead, our response will almost always be to fine-tune the automatic scoring mechanisms so that they would have performed better not only in the disputed situation but in all similar situations. Occasionally, we may invent new scoring or feedback mechanisms which are intended to capture and respond to -- automatically, all the time -- those aspects of the situation which the disputants felt they had to bring to our attention for manual arbitration. Again, those new mechanisms will be intended to address not only the current disupute, but all such future disputes. (Naturally, we will be alert to the possibility of "gaming" or abuse of any new feedback mechanism, and indeed of the entire system.)

If you feel that your score as an editor is too low, there are lots of things you can do to change it -- see the page _____ for a long list of suggestions. Please read those suggestions, and really think about them, before or instead of contacting an administrator to complain that you have somehow been unfairly mistreated by the system. Please be especially thoughtful and introspective before claiming that your score is too low not because of things you've done or have control over, but rather because of the conspiratorial actions of your enemies. Our experience -- sorry to be blunt about this -- is that "reasonable" editors don't have these problems. We're sorry if our definition of "reasonable" does not match yours, but this wiki can only work if all of its participants share a definition of "reasonable behavior" which allows them to collaborate on the editing process in a spirit of mutual trust and cooperation. It is simply not possible to accommodate conflict-provoking patterns which end up requiring manual, individual, subjective, personal intervention, arbitration, and resolution.