User:MMiller (WMF)/Suggested edits update Feb 2022

Hello everyone -- we last talked about expanding the deployment of the Growth features back in November. We had run a test in which we gave the features to 25% of new accounts, and gave mentorship to a smaller set of 5% of new accounts. After we went over those results together, it looked like all the community members in the conversation supported increasing the Growth features to go to 100% of newcomers, and people recommended that I assemble an RfC so that the broader community could weigh in on that direction.

I think there are two questions for us to agree on before we put up the RfC:


 * What percent of new accounts should get the Growth features?
 * What percent of new accounts should get mentorship?

To help us have this conversation, I pulled more recent data about how things have been going with the Growth features at the "25 / 5%" state in which they've been for the past few months. They're in the collapsed section below. It's collapsed because the patterns are essentially the same as in our previous look at the data in November, with almost the same revert rates and other breakdowns. The volume of edits is a bit higher, which I believe is due to the annual January bump in editing that we've observed the last few years.

This data was collected from January 1, 2022 to January 31, 2022.

Suggested edits


 * Edits
 * 3,305 suggested edits were completed. During the previous month we looked at (September/October 2021), that number was 1,979.  I think this increase is likely seasonal -- we tend to see increases in editing activity every January.  If we were to extrapolate out to 100% of accounts, we might expect some 13,000 suggested edits.
 * 430 of those 3,305 edits were reverted. That's a revert rate of 13%, which is exactly the revert rate we saw with the previous test.  By comparison, the overall revert rate of newcomer edits during January was 28%, so these suggested edits seem to be of consistently higher quality.
 * Users
 * 823 distinct users completed suggested edits. At 100%, we might expect this to be some 3,300 per month.
 * 61 users completed suggested edits on three or more separate days (7%). These are users who seem to be engaged enough with suggested edits that they come back consistently.
 * 298 users completed three or more suggested edits (36%).
 * 75 users completed ten or more suggested edits (9%).
 * 2.9% of the accounts who got the Growth features ended up making a suggested edit, compared to the previous test where that number was 2.1%.
 * Observations
 * This link filters Recent Changes to just these suggested edits.

Mentorship


 * Edits
 * 206 mentor questions were asked. At 100%, we might expect this to be about 4,000 per month.  This is the same rate from our previous test, in which 214 questions were asked.
 * With 53 mentors signed up, mentors got about 4 questions each during the month-long test. If we kept the same number of mentors and scaled up to 100% of newcomers, we might expect them to get 80 questions per month.
 * 21 of those 214 questions were reverted. That's a revert rate of 8%.
 * Users
 * 183 distinct users asked questions. At 100%, we might expect this to be 3,600 per month.
 * Observations
 * This link filters Recent Changes to just these mentor questions.

What percent of new accounts should get the Growth features?

When we last talked about this, we discussed whether we should bring the 25% number up to 80%, or go all the way to 100%. After thinking it through, I think we should go all the way to 100%. Here's why:


 * The advantage to 80% is that we would be continuously running an experiment on English Wikipedia to monitor the overall impact of the Growth features. 20% of accounts would be in the control group, allowing us to look at which group has improved outcomes.
 * But because we've been running that same experiment in dozens of other wikis over the last couple years, we've repeatedly established that the Growth features have a positive impact on newcomers, and we believe that would extend to English Wikipedia. To keep tabs on the overall impact of the Growth features, our team will continue to retain control groups on a set of smaller Wikipedias who are okay with the potential confusion (Arabic, Spanish, Czech, Bengali, Persian, Turkish, and a few others).
 * And we've learned about a big downside from working with the other communities: it leads to confusion both on- and off-wiki when one out of five accounts doesn't have the features. At editing events and in places on wiki like the Teahouse or in help materials, it makes it harder for experienced editors to recommend that people use the Growth features, since a good portion of users won't have them.

For these reasons, I think that going to 100% will help English Wikipedia smoothly adopt the features with minimal confusion. What do you all think?

What percent of new accounts should get mentorship?

Right now, 5% of new accounts have mentorship available in their newcomer homepage. There are 53 mentors signed up. This is yielding about 4 questions per month per mentor. If we were to increase mentorship to 100% of new accounts, it would mean that with 53 mentors, they would get something like 80 questions per month, which would be about 2 or 3 per day -- which sounds like it would be an unmanageable amount. After discussion with all of you, our team has worked on improvements to mentorship that could help:


 * Mentor dashboard: the mentor dashboard is now available, which helps mentors see who their mentees are and keep track of their activity.
 * Volume control: within the next couple weeks, the mentor dashboard will have a feature that will allow mentors to control how many mentees are assigned to them, essentially choosing between "high", "medium", and "low".
 * Opt-in/out: we're working on an ability for a mentee to opt-out (and back in) to having a mentor.

To that end, I think there are a few paths English Wikipedia could take for mentorship:


 * Sign up more mentors. If we want mentors to get about 2 questions per week at 100% of mentees, we would need something like 500 mentors -- 10 times more than we have now.  That's a really major increase.   Wikipedia talk:Growth Team features
 * Opt-out by default
 * Continue to offer mentorship to only a portion of newcomers