Wikipedia talk:Labels/Edit types/Archive 1

Start with grounded theory
Hey Mdann52. I hope you don't mind me pinging you for this question. You're the only volunteer so far!

So, we're working on a taxonomy on the meta pages: m:Research:Automated classification of edit types/Taxonomy But I'm thinking that we might do a better job if we actually look at a sample of edits and try to describe them. A standard way of identifying categories when you aren't sure what categories exist is grounded theory. Essentially, I imagine that we'd load the Wiki labels system with a small sample of edits and rather than a form full of pre-filled categories that can be selected, we'd just have one big text box in which we'd describe the edit however we thought was appropriate. Afterwards, we can review these descriptions and compare them to our taxonomy.
 * Benefits: We'd make sure our taxonomy reflects reality and the intuition you have when looking at the edit.
 * Drawbacks: We'd increase the workload for you, me, and our future collaborators.

What do you think? --EpochFail (talk &bull; contribs) 16:40, 30 October 2015 (UTC)
 * Ping and   :D --EpochFail  (talk &bull; contribs) 22:19, 4 November 2015 (UTC)


 * Sorry, thought I'd already replied! I think this would be a better way of doing it, yes - I'm sure we could always change if it was too much work. Mdann52 (talk) 08:28, 5 November 2015 (UTC)'
 * OK Mdann52, I'll get something set up. --EpochFail  (talk &bull; contribs) 15:03, 6 November 2015 (UTC)

Recruitment
&, would you be interested in helping us recruit some other Wikipedians? Our volunteer pool is pretty small now which means a lot of work for us and less diversity in judgements. I'll be doing some more reaching out to folks today too. --EpochFail (talk &bull; contribs) 15:05, 6 November 2015 (UTC)

Recruitment from last campaign (WP:Labels/Edit quality)
,, , , , , , , , , , , , , , , , . , , Hey folks. I'm sorry for the mass ping. I figured that this would be less disruptive than posting on everyone's talk pages. Based on your previous work helping us label edit quality (which was a massive success and resulted in much better vandalism detectors), I'd like to invite you to participate in a new labeling campaign. In this one, we'll ask you to help us evaluate edits by the type of change that is made. Our goal is to stand up an automated system that can automatically label edits. See our simple mockup of what this might look like on an article history page. I figure that such a system has quite broad applications.

Unlike the last wiki labels campaign, we need your help before we can get started with labeling. We need to figure out what taxonomy we should apply to edits. Specifically, I'd like you to review/extend our complete taxonomy. If you'd like to help out, please add your name to the list of volunteers and I'll make sure you receive updates when we're ready to get started with work. If you're not interested, I'll not ping you again about this campaign. Thanks! --EpochFail (talk &bull; contribs) 15:31, 6 November 2015 (UTC)

Mini open coding
Hey folks. To get a sense for how to start the open coding process, I'd like to have us look at and describe a very small, random sample of edits together. I gathered a random sample of 20 edits from the last week of October and pasted links below. We'll use these open evaluation of work to extend the taxonomy of edit types and the form that we'll build for wiki labels. --EpochFail (talk &bull; contribs) 16:29, 6 November 2015 (UTC)


 * Instructions: For each edit, describe both the operations performed in the edit and the reason/meaning for the changes in whatever format you think makes sense.


 * Special:Diff/687368718
 * Corrects fact (data-modification) --EpochFail (talk &bull; contribs) 16:36, 6 November 2015 (UTC)
 * Special:Diff/687966014
 * Vandalism (information-modification) --EpochFail (talk &bull; contribs) 16:36, 6 November 2015 (UTC)
 * Special:Diff/687673674
 * Adds information (information-addition), Unsourced removal (information-deletion, named_citation-deletion, wikilink-deletion) --EpochFail (talk &bull; contribs) 16:36, 6 November 2015 (UTC)
 * Special:Diff/687915843
 * Modifies templates and Wikilinks to try and improve page - appears good-faith but inexperienced editor. Mdann52 (talk) 19:22, 6 November 2015 (UTC)
 * Special:Diff/687201047
 * Cleanup, removes whitespace, external link. ~  ONUnicorn (Talk&#124;Contribs) problem solving 18:06, 6 November 2015 (UTC)
 * Special:Diff/687120365
 * Information-modification, wikilinks and adding sources Mdann52 (talk) 19:27, 6 November 2015 (UTC)
 * Special:Diff/687562055
 * Information-addition, updating results. Mdann52 (talk) 19:27, 6 November 2015 (UTC)
 * Special:Diff/687068834
 * Information-modification, correcting name of work. Mdann52 (talk) 19:27, 6 November 2015 (UTC)
 * Special:Diff/687316453
 * Information-modification, updating data. ~  ONUnicorn (Talk&#124;Contribs) problem solving 13:07, 7 November 2015 (UTC)
 * Special:Diff/687398537
 * Information-modification, correcting credits. ~  ONUnicorn (Talk&#124;Contribs) problem solving 13:33, 8 November 2015 (UTC)
 * Special:Diff/688014075
 * Information-addition, adding film to table. ~  ONUnicorn (Talk&#124;Contribs) problem solving 13:33, 8 November 2015 (UTC)
 * Special:Diff/687745257
 * Information-modification; vandalism. ~  ONUnicorn (Talk&#124;Contribs) problem solving 13:33, 8 November 2015 (UTC)
 * Special:Diff/687875461
 * Adds information (information-insertion, wikilink-insertion), Wikitext formatting (whitespace-insertion) --EpochFail (talk &bull; contribs) 14:58, 9 November 2015 (UTC)
 * Special:Diff/687383565
 * Wikitext fix (markup-modification), Manual of Style (punctuation-modification, data-modification) --EpochFail (talk &bull; contribs) 14:58, 9 November 2015 (UTC)
 * Special:Diff/687268700
 * Removes unnecessary specificity (information-modification) --EpochFail (talk &bull; contribs) 14:58, 9 November 2015 (UTC)
 * Special:Diff/687272244
 * Special:Diff/687629542
 * Special:Diff/688084608
 * Special:Diff/687813330
 * Special:Diff/687348108
 * Special:Diff/687899935
 * Special:Diff/687597334
 * Special:Diff/687663640
 * Special:Diff/687412474
 * Special:Diff/687581430
 * Special:Diff/687742528
 * Removes deleted image (image/deletion) ~  ONUnicorn (Talk&#124;Contribs) problem solving 16:42, 6 November 2015 (UTC)

Mocking up a new edit type labeling interface
Hey folks. I've been working with Diyiy to update the form that we'll be using with Wiki labels. We're trying to work out how to organize the work of associating "semantic" meaning about edits with "syntactic" operations. We've mocked up a form interface (see right) and we've been working a set of "semantic" meanings to capture in that form. I'll paste our proposed semantic categories below. Please feel free to boldly make modifications. --EpochFail (talk &bull; contribs) 15:17, 11 November 2015 (UTC)


 * I figured that a little more explanation might be necessary. The mockup on the right shows an interface that allows you to select a semantic "intention" for a change.  Then within each semantic intention box, you can add syntactic operations.  Syntactic operations are "object" and "action" pairs.  Building the labels this way will allow us to associate syntactic changes with the intention of those changes.  An edit can have many intentions -- though only one of each.  An intention can many operations associated with it. --EpochFail  (talk &bull; contribs) 18:26, 12 November 2015 (UTC)

Proposed "semantic" categories

 * Link Disambiguation
 * redirect, repair a dead link


 * Fact Update
 * update numbers, dates, scores, table, etc..
 * correct inaccuracies


 * Formatting
 * change font, fix errors in wikitext code


 * Verifiability
 * add sources, citation


 * Cleanup
 * remove unnecessary information, delete whitespace


 * Vandalism
 * a deliberate attempt to damage articles


 * Neutral Point of View
 * fairly, proportionately and without bias


 * Clarification
 * specification, explanation to existing view


 * Adding Information
 * elaboration, add substantive content, insert point of views, add facts.


 * Copy Editing
 * rephrases, improve grammar, spelling, tone, punctuation

Here, copy editing might be categorized into several different semantic categories. For example, rephrase expresses the sentence/word meaning in an alternative way, replacing the original words with similar ones. Grammar checking focuses more on modifying the spelling, punctuation, tone, etc. Both rephrase and grammar checking correspond to word-level modification, and we are not sure whether they are two different types of editing work, and should be put into copy editing or two separate categories. Please feel free to provide your feedback about this, thanks! :-) Diyiy (talk) 17:27, 15 November 2015 (UTC)

Status update: 2015-11-21
Hey folks. Thanks for your patience. Diyiy and I are working to turn the mockup into a usable part of m:wiki labels and we're making good progress, but it means that we're silent here. We're making sure that we interact well with mw:OOjs_UI, MediaWiki's standard library for form elements. Here's the repo where we are doing the work: https://github.com/halfak/semantic-labeler I suspect that we'll have something ready for testing next week. --EpochFail (talk &bull; contribs) 15:54, 21 November 2015 (UTC)
 * Quick update. We've got a working prototype that is wrapped up in OOjs-UI style code.  We're now working on integration into Wiki Labels.  If you want to see our progress, clone https://github.com/halfak/semantic-labeler and open   in your browser. I'll update again when we have something working in Wiki labels.  Then, we'll be able to start a pilot run.  Thanks for your patience! --EpochFail  (talk &bull; contribs) 00:22, 3 December 2015 (UTC)
 * Quick update again. The code is done.  It was a lot of engineering work, but we now have a flexible, internationalized strategy for gathering a mixture of semantic and syntactic labels.  We're doing a quick pilot run with just the research team so that we can give you folks something cleaner to critique.  If you are impatient and want to see what we are doing, you can request a workset from the "Edit type (0.1k random sample)" campaign right now on WP:Labels.  Otherwise, I'll do another ping here in a few days to let you know that the pilot run is ready and to provide instruction.  Thanks for you patience! --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail  (talk &bull; contribs) 16:32, 22 December 2015 (UTC)

Pilot ready for testing!
Ping:, , , , , , , , , , , ,

We've deployed the new labeling form and a campaign containing a random sample of 100 edits. Please request a workset from the "Edit types (0.1k random sample)" campaign and label 10 edits. If you get an error saying that there are no tasks left, let us know and we'll add more. To label an edit, you'll need to "Select a semantic meaning" from the main drop-down box and add object/action pairs for the syntactic operations related to that change. If you'd like us to make changes to the available Semantic Meanings, Objects or Actions, please edit the Labels/Edit types/Taxonomy boldly and we'll implement the changes for the full run. If you find any edits difficult to label, please post about them here so that we can all review.

Our goal during this pilot is to vet the taxonomy against real data and to test the labeling form to make sure it is working as expected. Once we're done with the pilot, we'll implement changes based on your feedback and deploy the full campaign. Thanks for your patience! This data is going to be super valuable. Thanks for your help. --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail (talk • contribs) 17:30, 29 December 2015 (UTC)


 * We've gotten enough labels to move forward from the pilot. Thanks everyone who contributed.  We'll be looking through the notes that you left as you were working.  Please post any other thoughts or concerns you want us to address in the next iteration.  --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail  (talk • contribs) 17:24, 9 January 2016 (UTC)


 * Mdann52 & ONUnicorn, pinging because you have been the most active here. I want to make sure that we know what you'd like changed in the labeling form before we do the full run.  Could you reply with any concerns/ideas you have that you'd like to see implemented.  We like to deploy the full campaign on Wednesday.  Any notes that you can give us before then would be great. --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail  (talk • contribs) 17:24, 11 January 2016 (UTC)
 * A couple of the diffs were talk or user talk page conversations, and I don't think any of the labels really apply there. Also, one was a page move. ~  ONUnicorn (Talk&#124;Contribs) problem solving 18:21, 11 January 2016 (UTC)
 * Thanks ONUnicorn. Saw those talk page edits.  We'll make sure they are filtered out before we run the larger campaign.  Otherwise, any concerns about the category structure?  Did you have difficulty using it to describe an edit or is there anything missing that you'd like us to include?  --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail  (talk • contribs) 18:30, 11 January 2016 (UTC)
 * It seems somewhat limited; like one shouldn't be able to describe every edit using those descriptors. That said, I can't put a finger on what's missing. I think most edits can be sandwiched into those categories, but they are broad enough that I'm not sure they'd all be helpful.  Also, sometimes parts of one edit can be described with more than one category (for example, copy-editing an existing paragraph and adding another with new info at the same time), but the solution there is to label for the most significant change. Overall, I don't think it's too difficult. ~  ONUnicorn (Talk&#124;Contribs) problem solving 18:36, 11 January 2016 (UTC)
 * Aha! It may not be clear, but you can actually apply multiple Semantic categories.  That way you can label an edit for all of the changes.   Just select a new semantic meaning from the drop-down and click the "+" button again. --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail  (talk • contribs) 18:42, 11 January 2016 (UTC)
 * Ping ONUnicorn :) --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail (talk • contribs) 20:34, 11 January 2016 (UTC)
 * not a lot to add here, other then the non-mainspace edits coming up :) Mdann52 (talk) 09:33, 12 January 2016 (UTC)
 * Will do! Thanks.  Diyi and I are going to work on a few examples edits that we can use to discuss the labels.  I think this will help clarify how multiple semantic meanings can be described.  We should have a post today or tomorrow about that. --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail  (talk &bull; contribs) 16:19, 12 January 2016 (UTC)

Testing out some templates.
Looking at Special:Diff/651810957. I see:

Does that make sense? --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail (talk &bull; contribs) 16:34, 12 January 2016 (UTC) Ping ONUnicorn. Does the above make sense to you or would you label the edit differently? --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail (talk &bull; contribs) 16:35, 12 January 2016 (UTC)
 * Yep, that makes sense. ~  ONUnicorn (Talk&#124;Contribs) problem solving 16:38, 12 January 2016 (UTC)
 * Great! Thanks!  --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail  (talk &bull; contribs) 16:42, 12 January 2016 (UTC)

Examples, Taxonomy, and Discussions
Hey folks. We've been working to make it easier to discuss our current taxonomy and potential modifications. See Labels/Edit_types/Taxonomy and discussion at Wikipedia_talk:Labels/Edit_types/Taxonomy.

I've also been working on a set of examples at Labels/Edit_types/Examples. I'd like to have ~10 representative examples there before we start the full campaign so that we can direct everyone to those examples for training. Right now, there's just two. Please feel free to add your own examples. It would be great if you could target some hard-to-classify edits. --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail (talk &bull; contribs) 15:57, 13 January 2016 (UTC)

Pilot v2 is live!
Ping:

Hey folks! I just loaded a new pilot campaign into the Wiki labels system. This pilot contains 500 revisions so that we can more thoroughly test the taxonomy against real edits. Please see the new "Edit types (0.5k sample)" campaign at Labels and request some worksets. Also please take notes in the labeling form if you struggle to fit an edit into the taxonomy. We'll use your notes to make changes. Happy labeling! --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail (talk &bull; contribs) 17:01, 16 February 2016 (UTC)

Pilot v2 Notes
@   Based on the revisions that I've labeled, I summarized my notes as below:
 * 1)  'Relocation': I do not need how to properly label this one. Currently, I regard 'Relocation' as 'Refactoring' intention, and for the syntactic operations, every operation comes in pair. That is, if the relocated paragraph contains wikilink and body copy, I will have wikilink-insertion, wikilink-deletion, body copy-insertion and body copy-deletion.
 * 2) I found it is helpful to better identity the intentions by taking into account the comments that editors provided to summarize their edits.
 * 3) Why do people add pictures or files? I labeled it as 'Elaboration'.
 * 4) For adding categories, I labeled it as 'Process'. Does this make sense?
 * 5) I'm not sure for formatting (remove punctuation or insert space in some tables), should I label it as Copy Editing or Wikification?
 * 6) Maybe we should pay special attention to 'Information' object in the syntactic operations, if the current edit modifies the meaning of the article, then please select information-modification or information-insertion. If an edit removes some opinions from the article, then it belongs to information-deletion. --Diyiy (talk) 23:15, 18 February 2016 (UTC)
 * I thought adding categories fell more under "Wikification". ~  ONUnicorn (Talk&#124;Contribs) problem solving 17:41, 19 February 2016 (UTC)
 * +1 for Wikification. That's my thought too.  --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail  (talk &bull; contribs) 20:26, 19 February 2016 (UTC)
 * +1 for pictures or file adding going into Elaboration. I agree that special attention needs to be paid to the "Information" object.  Do you think we should break that out of the SemanticOperationsSelector and have it be a separate field?  That might call the appropriate attention to it. --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail  (talk &bull; contribs) 20:45, 19 February 2016 (UTC)
 * @  : I think 'Information' here represents a non-physical object, where all other objects (body copy, template, markup) are all physical objects. Having it in the syntactic objects seems reasonable. But we need to let people know that, for each annotation, we also care about whether such edit changes the meaning or not.


 * Other thoughts:
 * I think it might sense to change Fact Update to Fact Correction. Fact update means, something was correct in the past (not correct in present). and we update it to be correct now. Fact Correction means, something could be either correct or incorrect, and we change it to be correct in present.
 * Body Copy seems a bit too professional, a bit journalistic. Should we consider 'Body Text' here?--Diyiy (talk) 15:52, 22 February 2016 (UTC)

Diyiy, it was not clear from your response... Are you in favor of separating "Information" into a separate field? I'm having a bit of fun imaging what Maxwell would say about information being a physical thing or not. lol I don't see a clear need to change "Fact Update". I don't see the term "Correction" to be relevant in the case that a fact is added -- nothing is being "corrected". I also don't see what the problem is with being professional or how "body text" is more desirable. Could you elaborate? --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail (talk &bull; contribs) 16:08, 22 February 2016 (UTC)
 * Ping Diyiy. :) --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail  (talk &bull; contribs) 15:19, 15 March 2016 (UTC)
 * @: (1) I agree to put information or meaning changes annotation into a separate field; might be a separate box below the current Notes field. However, it might increase the annotation cost; (2) Using either Body Copy or Body Text is okay, as long as we could explain/make 'Body Copy' easy to understand for others; (3) Fact correction could update something from incorrect to correct, from less correct to correct and from none to correct; fact update can deal with something that were correct in the past (incorrect right now), and we update it to be correct in present. Even though there are some slight differences, using fact update is fine. --Diyiy (talk) 15:20, 16 March 2016 (UTC)

Pilot v2 Preliminary Analysis
Up to March 10, 2016, we've had 199 revisions annotated by 15 different annotators. About 60 revisions were annotated by two judges. The most common semantic labels used were copyediting (37), elaboration (33), wikify (26) and factupdate (12) and the most common syntactic labels were bodycopy_insert (24) wikilink_insert (19) bodycopy_modification (18) and bodycopy_deletion (16).

To calculate the agreement between annotations, we divided the number of labels in common for the same revision (i.e., the interaction of labels) by total of distinct labels used on that revision (i.e., their union). Agreement was low, with 22% overlap for semantic labels and 25% for syntactic ones. At this point, we don't know why the agreement is so low. One possibility is that both judges were correct in the labels they applied, but identified different subsets of the edits to a revision. The other is that judges had low agreement because they disagreed about the edits they saw (e.g., someone labeled the same revision as a copyedit and another labeled it as wikify).

In any case, this level of agreement is too low to use as the basis of ground truth for a classifier to automatically label revisions. It suggests that we can't just simply rely upon expert Wikipedians with no training to do the labeling. To increase reliability, we'll need multiple judges per revisions and will also need to train the judges. In addition, the rate of labeling was very slow, with only 199 of 1000 revisions annotated in a month period. We're considering whether we should recruit Wikipedian volunteers more vigorously and provide them with online training. Another possibility is to recruit paid research assistants at a university and provide them training. Robertekraut (talk) 18:42, 11 March 2016 (UTC)
 * I wonder if we could look through a few of these examples of disagreement so that we could evaluate why they happened. It seems like a good idea to include any edits that were particularly hard to classify into the WP:Labels/Edit types/Examples.  It seems like the 78% of edits with two labels and no overlap would be a good place to start. --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail  (talk &bull; contribs) 15:18, 15 March 2016 (UTC)

Annotation Comparison
This part compares annotations given by two users in the pilot 2 study. --Diyiy (talk) 19:39, 15 March 2016 (UTC)

Problems:
 * Some annotators submit Empty annotations, with either Empty semantic labels or Empty syntactic labels or both.
 * Empty submit

OK Reviewing the table here, I can see some trends that are probably due to our lacking instructions or some interface issues. I'll sign after each note so that we can have some sub-discussions.
 * It looks like Aubrey, Stuartyeates, Masssly, The Polish, Epicgenius and Arian Ar didn't submit any labels -- only used the notes field. Did you see the dropdown labeled "Choose likely intention(s) for the edit" when you loaded up the labeling interface? --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail  (talk &bull; contribs) 15:50, 16 March 2016 (UTC)
 * I'm pretty sure I chose labels for each edit(except for two I believe).I Don't know what to tell you!(a bug perhaps?)Let's see what will other users say about this.--Arian Ar (talk) 16:02, 16 March 2016 (UTC)
 * Arian Ar. Aha!  It could be a bug.  What browser & OS are you using?  If you load up the workset again, do you see the labels you submitted earlier? --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail  (talk &bull; contribs) 16:03, 16 March 2016 (UTC)
 * It's fixed!Last time the green add button for the lables wasn't there!I think it was a problem with my browser.I used google chrome v47 but today I updated it and saw the add button!Anyway I fixed my workset and added labels for each edit.(btw I use Windows 7) tnx for informing me!--Arian Ar (talk) 12:51, 17 March 2016 (UTC)
 * Great! Thanks for having another look.  It seems like others are having similar issues, so I'll keep looking into this, but your report is very helpful for finding out what went wrong. --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail  (talk &bull; contribs) 13:42, 17 March 2016 (UTC)
 * I think I used labels for all of the edits, but I'm not sure. Can I go back and look again? epicgenius (talk) 16:15, 16 March 2016 (UTC)
 * epicgenius, seems to be a recurring theme here. When I look into the wikilabels system, it looks like we got your notes, but no semantic/syntactic labels.  You can re-load your past worksets from WP:Labels to have a look, but I'm guessing you won't see past labels there.  Can you tell me what browser/operating system you are using?  --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail  (talk &bull; contribs) 16:54, 16 March 2016 (UTC)
 * OS X El Capitan version 10.11.2. I use either Mozilla Firefox or Google Chrome. Today, I used Firefox, but before, I think I used Chrome. epicgenius (talk) 18:17, 16 March 2016 (UTC)
 * epicgenius, I'll try to replicate the problem. In the meantime, could you try to submit a few more labels to see if the system is currently working? --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail  (talk &bull; contribs) 20:51, 16 March 2016 (UTC)
 * I just submitted another workset, and I labeled all of the 10 edits. The strange thing is, when I went to review my workset again, the notes were still there, but the labels had disappeared. epicgenius (talk) 21:24, 16 March 2016 (UTC)
 * , I just looked through our [//labels.wmflabs.org/campaigns/enwiki/22/?tasks data report] and it seems that these labels weren't saved either. Are you clicking on the "+" button?  I've take some screenshots demonstrating how the "+" button works.  See [//imgur.com/JrbQygl imgur.com/JrbQygl].  --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail  (talk &bull; contribs) 14:32, 17 March 2016 (UTC)
 * Nope. (Oops.) I guess I should go back and fix these. epicgenius (talk) 15:18, 17 March 2016 (UTC)
 * Sorry, User:EpochFail, I've been busy with other things. Mainly real life things. Stuartyeates (talk) 08:39, 17 March 2016 (UTC)
 * , Aha! Our fault!  The UI should be more clear.  I'll think about this and see if there's a nice change that we can make to make this more obvious. --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail  (talk &bull; contribs) 21:26, 17 March 2016 (UTC)
 * It's OK, I think I didn't pay attention to the [+] label, as I am inattentive a lot of the time. epicgenius (talk) 14:23, 18 March 2016 (UTC)
 * Stuartyeates, no worries. I wonder if you have some thoughts on why we got a bunch of blank labels from you though.  It seems that some people have been reporting that they'd been submitting labels, but they haven't been saved.  --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail  (talk &bull; contribs) 13:41, 17 March 2016 (UTC)


 * DarTar notes that "edits adding single items to an existing list seem pretty frequent and it's unclear if list modified is the right way of capturing them" -- e.g. Special:Diff/704750361. My personal opinion is that this is exactly what "list modified" would mean.  But maybe that is ambiguous to others.  Thoughts? --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail  (talk &bull; contribs) 15:50, 16 March 2016 (UTC)
 * Diyiy marked Special:Diff/701852174 as "simplification", but it adds a comment to the lead.  I'd like to call this "clarification" or "elaboration".  Thoughts? --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail  (talk &bull; contribs) 15:50, 16 March 2016 (UTC)
 * I think this is clarification; this might be my mis-selection (will be more careful next time :-) ). --Diyiy (talk) 17:38, 16 March 2016 (UTC)


 * For edits like Special:Diff/701542520, I think we should label these as "wikification" since they are more appropriately situating the article in the category structure of the wiki. But maybe I'm stretching here.  Thoughts?  --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail  (talk &bull; contribs) 15:50, 16 March 2016 (UTC)

Inconsistent Annotation
Here are some inconsistent annotations. --Diyiy (talk) 19:39, 15 March 2016 (UTC)

Some notes on these examples with inline signatures so we can have separate discussion threads.
 * Special:Diff/705204449 changes the color of some table cells, Special:Diff/704437632 looks like a bunch formatting changes to a table, and Special:Diff/702518115 adds some italics to a word. It looks like Kharkiv07 and User:Diyiy applied the "copy editing" label to these edits, but I think "wikify" makes more sense.  Could you share your thoughts re "copy editing"? --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail  (talk &bull; contribs) 16:02, 16 March 2016 (UTC)
 * Special:Diff/705007180 seems to be clearly "clarification" as it makes it clear which bridge in what location is being shown. But I could understand "elaboration" here too as it does add new information.  So, how so we want to handle this?  Should we drop one of the two intentions or should we clarify the distinction?  If we're going to drop one of the two, I'm in favor of dropping "clarification" as it seems to be a sub-type of "elaboration".  --<span style="display:inline-block;padding:0 .25em;border-bottom:2px solid #008;background: #f6f6f6;" title="aka halfak">EpochFail  (talk &bull; contribs) 16:02, 16 March 2016 (UTC)