Talk:Inter-rater reliability

Untitled
Is Joint Probability the same as Simple agreement? Ben 23:53, 9 March 2007 (UTC)

What are the consultation methods for asking a group of experts to reach an agreement that can be tested for inter-rater reliability? --131.170.90.3 06:21, 19 September 2007 (UTC)

Multiple Issues
I added a multiple issues cleanup tag focusing on (a) the need for neutral citations (there are citations which link to a sales page for the cited book); (b) the need for more reliable references, e.g., the first section (Inter-rater reliability) does not have any citations; and (c) the article lacks a logical flow (copy editing needed). If you disagree, let's discuss here first before removing the tag. - Mark D Worthen PsyD  (talk)  09:53, 26 December 2018 (UTC)
 * And the references are a jumbled mess - they should probably be reworked from the beginning.  - Mark D Worthen PsyD   (talk)  10:01, 26 December 2018 (UTC)

Resolving inter-rater disagreements
The page lacks a section on how to resolve discrepancies among many raters. There are at least two ways to solve them: if data are categorical, the most common decision among raters is chosen as the answer; if the number of raters who chose each option is even, then a random process like a coin toss decides. When the data are quantitative, the ratings are averaged to achieve the final score (arithmetic mean). Gandalf 1892 (talk) 22:03, 19 May 2022 (UTC)