User talk:Risker/Audit Panel

Discussion points

 * This is based entirely on Review Board; history of contributions is attached to that page (added for GFDL compliance).
 * Any references to this panel/board reviewing actions of the Arbitration Committee itself have been removed, in order to simplify this proposal and focus on the core issue of tool use. I propose that the community consider a separate process to consider review of the Arbitration Committee, whose function is so different from CheckUser and Oversight that it would be imprudent to group the two notions together.
 * Three or five members? It may be difficult to find five individuals who enjoy broad-based community confidence; we only have a few successful RfBs each year, but would be looking for that level of confidence.
 * I have included authorization for panel members to use the CheckUser tool to recreate CU investigations in order to verify results. This is a standard auditing process, and I am hard pressed to see how they would be able to carry out their assigned duties without being able to verify that the check on Account X revealed information Q, R, and S.
 * I have not specified the process by which the panelists are selected; this should have further input from others.

Risker (talk) 08:18, 23 December 2008 (UTC)


 * In the "Selection" section, I have written that the Arbitration Committee would voice its concerns during the nomination/candidacy process, but would make recommendations for appointment based on the results of the community's selection process.
 * As an individual and as a part of the Arbitration Committee, I would find it very difficult to recommend someone for appointment if I knew that they had a history of privacy breaches, and yet the community selected them despite being aware of this information. I am not quite sure how that kind of problem should be addressed.  Risker (talk) 15:16, 23 December 2008 (UTC)
 * You can't give up your fundamental responsibilities here. This is a delegation of Arbcom's authority to a body that is as independent as it is practical to make it.  You get transparency and accountability by having good people on the panel and paying attention to their sensible recommendations, not by closing your eyes and abdicating your fundamental responsibilities. Thatcher 03:03, 24 December 2008 (UTC)

My thoughts

 * Checkuser limitations: Too specific, broad and general is better.  We don't need to hamstring people for the sake of satisfying someone's need for the appearance of something.  Training exception of course.  But the "only previously logged" is too tightly worded, suppose Smith complains he was checked, and the checkuser said, "I saw something funny when I checked Jones", well the auditor might want to explore that by running checks on different IPs or from a different angle to see what the case looks like.  If a simple admonition like "Auditors are not to perform routine checkusers and only use the tool for investigating matters under review by the panel" is not good enough, then you have the wrong auditors.
 * Read-only access: what is the point of this? The panel might want to broadly solicit opinions on a case or issue a general caution or warning where it seems appropriate. Thatcher 02:59, 24 December 2008 (UTC)


 * The whole #Auditing of CheckUser and Oversight use section is pretty much unnecessary, unless Arbcom really wants that sort of data. I have stats on one month of my activity here, it took a long time to format and collate that data in excel, group into categories, sort, etc.  That's a lot of data to compile from 20+ checkusers.  And on top of that you want random audits?  As the percentage of troubling checks seems to be very small, a random audit will not pick them up unless you have a very large sample size.  Better to just ask the auditors to monitor the logs and let them notice any patterns.  The most likely complaints are going to be things like "Does Smith, who has green sensibilities, run checkusers on purples that he shouldn't be."  But to look at my log, for example, if I am running 1600 logged checks a month and even once a month I check someone who pisses me off, is a random audit going to catch it? Thatcher 03:11, 24 December 2008 (UTC)
 * Those are very detailed stats you developed there, Thatcher; my inclination would be (at least in the early stages) simply to verify total number of checks, by checkuser. This would give the committee an idea of who is very active and who is barely active, to support decisions on increasing the number of checkusers and/or suggesting that some checkusers consider withdrawing. We've lost good checkusers to burnout because of workload before (I count you in that group), and I am pretty sure the committee would want to prevent that in the future.
 * There are a range of less serious audits that could be run once a quarter or even biannually, for example whether all log edits have an edit summary or what percentage of CUs were run in an effort to identify sleeper accounts of a particular vandal; however, I would be inclined to leave decisions on what additional audits need to be done to the panel itself.
 * I've included the term "randomized and targeted reviews of checkuser investigations" to give the panel the greatest leeway in determining exactly what they want or need to audit; if I was a panel member myself (which I will not be), I would be inclined to start with maybe 10-15 reviews of investigations a month, 80% of them targeted and the remainder random. As the panel finds its feet, they need to have some flexibility to try different methods of auditing and reviewing cases; they may find one process more useful than another, or discover a particular issue that requires more rigorous follow-up. At this stage, none of us are entirely certain what is important and what's trivial.  Thanks a lot for your comments.  Risker (talk) 03:55, 24 December 2008 (UTC)
 * (I didn't quit because I ran too many checks, there were other reasons.) I think you need to be careful about binding the panel to any particular schedule or method of analysis.  I found the Adam and Eve case by spotting it in the log one day; I found Smith by doing a detailed analysis of someone.  You're also expanding the scope of the review to giving Arbcom feedback on how many checks there are and whether some checkusers are pulling their weight and some are burning out.  That doesn't have much to do with political checks and overreaching oversight. Thatcher 04:27, 24 December 2008 (UTC)
 * Noted, and my comment amended about burnout. It's not my expectation that the Audit Panel will produce regular audit reports on a set schedule until it figures out what it's doing (hence my use of the term "periodic" instead of "monthly" or "quarterly"), although I would suggest that a baseline "reading" be taken so that they have an idea of volumes of use of each of the tools and who is actually using them. That will give them some ideas on how to proceed. As to the burnout/pulling their weight issue, I see the Audit Panel's role as reporting the numbers, and Arbcom's role as being responsible for personnel decisions and resource allocation, in part based upon the information provided by the Audit Panel. You are correct, in your comments above; it is a direct responsibility of the Arbitration Committee, and should not be dumped off onto some other group. Risker (talk) 05:43, 24 December 2008 (UTC)
 * In an ideal world, the only thing the audit panel will do is say "There were X checks this quarter, (boring spreadsheet broken down by user available here) there were three complaints (as itemized on the notice board) none of which warranted complete investigation." Reporting the numbers will give them something to do. Of course, in the real world...--Tznkai (talk) 06:01, 24 December 2008 (UTC)

A selection model
This has some edges that need to be buffed out, but I figured its one possible way to have a strong community voice in the elections but still handle concerns.


 * Candidates for the audit panel are self nominated.
 * After candidates nominate themselves, there is a one week vetting period. During this time the community is encouraged to submit serious concerns to the Arbitration Committee, who will will review them. By the end of the week, the Committee may initiate vote to exclude certain editors from the election but may not do so at any other time.
 * After the vetting week, there is an election week, with simple support/oppose voting. Candidates with over 80% support are eligible for the panel.
 * Arbitrators are encouraged to vote in elections.
 * Terms are staggered into tranches, and are four months long, candidates must take at least a 2 month break between terms.
 * Up to the top 15 (if there are enough members form the pool for the panel's membership - this is the only pool to draw from for the entire year. Panel members who have completed a term or discover themselves needing a break are returned to the pool, and a new member is immediately installed.

Part of the purpose of electing a pool is to allow the panel to ensure it has a good mix of skills sets, another is to reduce the number of time-to-drama convention machine elections.--Tznkai (talk) 05:52, 24 December 2008 (UTC)
 * With respect Tznkai - do you really want to establish a system which allows 'some of those who will be investigated' ie. Arbcoms with CU hats the first say in candidate selection? Better, I'd have thought would be self selection, election (which incorporates public vetting), then submission to Arbcom for approval. Arbcom should be able to demonstrate and explain any decision to overturn community concensus. --Joopercoopers (talk) 15:58, 24 December 2008 (UTC)
 * Thats exactly the opposite of what we want. If arbcom acts as the final gatekeeper, it opens up the election system to tactical selection by arbcom, and encourages a certain amount of sneaky-ness in withholding important information until the last possible minute (in hopes that the election will go the desired way anyway). Not to mention, imagine the chaos if the ArbCom needs to exclude someone because of a serious issue - and a majority of the arbitrators happened to have voted oppose on that candidate? No matter what the explanation, or how legitimate, it will threaten the entire legitimacy of the election to begin with. The purpose of an ArbCom veto is to allow them to fulfill their fundamental responsibility to use their increased access to important information to safeguard the encyclopedia and the community from accidents. If we accept that veto needs to be available (and I do), the system least vulnerable to drama and corruption is early exclusion - not overturning results.--Tznkai (talk) 17:08, 24 December 2008 (UTC)
 * I assume you mean its "exactly the opposite of what you want" - presumably your're not talking on behalf of the Arbcom here? We currently have a selection process for Arbcom members which picks the top candidates for the posts. (I'll conceed there's been concerns raised about which metrics to use, but that's rather a side issue - we can agree % of votes or the pros less the minuses.) But fundamentally how would Arbcom assert a tactical selection on that basis? In an attempt to head off future drama, I fear you are likely creating more, not that drama-limitation is the be all and end all of political considerations. The cut and thrust of elections rather create it in any event. What do you envision might be revealed at appointment stage that would actually preclude a candidate from assuming the role? Not being 18 years old? Not being willing to identify to the foundation? How would such revelations threaten the legitimacy of the election? If this is the case, why don't we vet Arbcom candidates for similar reasons? The ground rules are clear from the off, and any idiot putting his name forward who's not willing to do such will get short shrift from a community if they've voted for him or her. But the idea that Arbcom will essentially field its prefered candidates at the outset, based on presumably private complaints from god knows who seems like a recipe for disaster, not to mention reinforcing the idea, true or not, that power at wikipedia is stitched up. --Joopercoopers (talk) 17:42, 24 December 2008 (UTC)


 * Speaking as someone who would bear responsibility for such appointments, my greatest concern would be that the "preferred" candidate turns out to have a history of privacy breaches, particularly privacy breaches that have not been disclosed on-wiki so would not be known by the majority of voters; I could not ethically support that person's access to these tools, but would also be ethically bound not to publicise the extent and specifics of any privacy breaches. Hence, the pre-election vetting to remove such candidates. I need to think about this a bit more, but am leaning to advanced vetting rather than having the election and only then identifying who the problematic candidates are. Risker (talk) 18:04, 24 December 2008 (UTC)
 * Risker, isn't a privacy breach that's so secret its not know about on-wiki or anywhere else, actually not a privacy breach? Otherwise, isn't it just stuff that can come out during elections? Elections are after all just public vetting. If user:x can stand up at an election and say user:y revealed my private information and is unsuitable for the job, the community can decide if they believe the claims and user:x can decide if he'd like to provide evidence. Sounds like a fairly rare occurance to me, and, as I argued above, what's the difference between community elections for Arbcom and this post in terms of trust - both get the CU hat in the end - Jimbo has a right to veto in the former case, which he's yet to exercise, and Arbcom has one in the later. I'm struggling to see opposition to this as anything other than an enabler of cronyism - we should trust the community to make the right decisions. --Joopercoopers (talk) 18:50, 24 December 2008 (UTC)
 * If checkuser Xyclon tells user:Redcloak that user:Banjo is a sock-puppet of user:Elan, thats a privacy breach whether the rest of the community knows about it or not. And when the arbitration system was put into place, we naively thought all they had to do was police fairly simple but intractable content disputes, maybe an obsessive edit warrior or two- Arbitration is a different animal now - nor are these processes built for the same purposes.--Tznkai (talk) 19:22, 24 December 2008 (UTC)
 * Hmm ok. So in summary we are saying the people to whom a veto might apply are current and ex checkusers, and those inelligble to take the post if offered? The simplest solution then would be to bar current and ex checkusers and oversighters from application. The possible benefit of checkuser experience on the board will be forfeited but I can't support a selection process where those who might be under investigation enjoy a weighted influence on the process. Can we at least agree this is a problem, if only from a public perception point of view? --Joopercoopers (talk) 19:38, 24 December 2008 (UTC)
 * I'm no longer convinced that no CU/OS/Arb is a good idea for the panel's membership, especially for CU, you need at least one panelist at all times who has proven technical competence to understand the CU process - the rest of the panel just need to be bright with well honed bullshit detectors. As to the rest - we're in that situation anyway, permissions are funneled through the arbcom - but the arbs are democratically selected (mostly) most of which are reform minded, so if they're willing to collectively (and they'd have to do so in concert) manage to exclude anyone but their bestest friends - we're kind of screwed anyway aren't we?--Tznkai (talk) 20:17, 24 December 2008 (UTC)
 * Of course, there's no reason why that person should have voting rights, which should be something conferred through elections - there are other ways to do this - the board could appoint a consultant CU for instance, who would be expected to provide impartial advice but not vote - or they could consult CUs on masse, or the active CU's may elect their representative consultant. Actually we're not "in that situation anyway" - currently Arbcom are appointing CUs (bad idea in my view, CUs should be elected too), we're talking here about appointments to a regulatory body (who will need CU access) - they're different things. --Joopercoopers (talk) 21:18, 24 December 2008 (UTC)
 * I thought of having a tasked CU - but I've been convinced its a bad idea: you get a single non-organic "gatekeeper" (or perhaps keymaster), whose CU access is appointed by arbcom - that a lot of technical data has to get funneled through a single person anyway. Its cumbersome and I think would reduce the independence of the panel more than having former arb/cu/os's organic to it. We're really dealing with three fundamental questions here: what role (if any) does the community have in the selection process, what role ArbCom has, and what role do former CU/OS/Arbs have in it. We agree that the community needs to have a strong voice. I think that the ArbCom needs to have some sort of active mechanism - similar to an election board, canvassers board, or municipality clerk in Meatspace democracy - to ensure that the candidates are genuinely eligible (it might be worthwhile to hammer out some examples of what would be required there). As for former arbs/cu/os, I think the pilot period should be free of former arbs, but we should have at least one former checkuser (or a current checkuser willing to suspend their checkuser privileges while in the panel pool) for the technical competence they bring.
 * I don't want to go into the details because of some WP:BEANS issues, but ArbCom collectively influencing the selection a five member panel to the point where the CUs and OSs get a "free pass" on their excesses is difficult, especially in a pre-election vetting model, and nearly impossible without being noticed. And again - we elected these people, we have to trust them at some point.--Tznkai (talk) 22:27, 24 December 2008 (UTC)

Public reporting
Hi Thatcher, Regarding this, I agree the panel shouldn't be a pillory, but disagree that naming the subjects of investigation amounts to that - especially if they are cleared. Rather than fragment discussion may I suggest the debate and subsequent text of the reporting policy remains at ? Cheers --Joopercoopers (talk) 15:42, 24 December 2008 (UTC)

Split discussion
Risker, if the proposal to make the review board only look at CU and OS sticks (till boxing day say) can I suggest we move these discussion to the talk page over there and abandon over here, to centralise discussion? --Joopercoopers (talk) 21:45, 24 December 2008 (UTC)
 * Yes, I was thinking the same thing, Joopercoopers. In fact, I might do some rearranging tonight, dependent on any on-wiki activity. Risker (talk) 22:13, 24 December 2008 (UTC)