User:Ocaasi/badgegrant


 * Anya Shyrokova (User Anyashy) and Jake Orlowitz (User Ocaasi)
 * Grant contact name: Anya Shyrokova
 * Grant contact username or email: anna.shyrokova@gmail.com
 * Grant contact title (position): User experience and user-generated content expert
 * Project lead name: Anya Shyrokova
 * Project lead username or email: anna.shyrokova@gmail.com
 * Project lead title (position), if any:
 * Full project name: Wikimedia Badges for Community Engagement
 * Amount requested in USD or local currency (USD will be assumed if no other currency is specified): $30,000
 * Provisional target start date: November 1, 2012
 * Provisional completion date: April 30, 2013

Budget breakdown

 * Anya Shyrokova - 20hr/wk * 24 weeks * $25/hr = $12,000
 * Jake Orlowitz - 20h4/wk * 24 weeks * $25/hr = $12,000
 * Consulting for research methodology, statistical analysis, and database support = $2,000
 * Consulting for graphic design = $2,000
 * Travel and incidental expenses = $2,000

Project goal
Our project will attempt to improve how Wikipedia editors learn the technical skills, policies, procedures and social norms necessary to become successful contributors by building, implementing, and testing a digital badge system. We will focus on the Teahouse, a Wikipedia project that provides expert mentoring and peer support for hundreds of new editors every month, and build a badge system to help encourage participation and community buy-in to the project. Wikipedia provides an opportunity for studying the impact of badge systems on voluntary, interest-driven participation in open contributor systems. We intend to surface elements of effective badge systems and recommend badge design principles.

Specifically, our intended goals are to:


 * Improve understanding of the best practices of existing badge, award, and barnstar systems.
 * Create a badge system that builds upon and aims to improve existing recognition, award, and barnstar systems -- particularly, in terms of engaging and guiding new editors, allowing for scaling, and minimizing gaming of the sytem.
 * Pilot the badge system in 2 different areas of English Wikipedia, testing for impact on editor participation and project engagement.
 * Further the Teahouse’s engagement with editors, particularly the demographic of female editors.
 * Collect findings and disseminate information on best practices based on what has been learned.
 * Produce a document of best practices and community responses as to how and when badges are most effective as tools for encouraging acknowledgement and editor retention.

Motivation and rationale
This project proposal is to build badge systems to help encourage participation on Wikimedia projects. Badges could be used as a reward for contributions that may not necessarily be extraordinarily significant from a Wikimedia project’s point of view but may still be meaningful to an individual.

Research has shown that set goals and specific asks can help encourage participation. Rewarding community members for smaller contributions could lead to significant aggregate contribution to a project. Badges could provide community members of all stages, novice to expert, with goals to work toward and feedback about whether their goals were attained. For example, badges could be given out for best new article in a given category, or badges could be given out for X number of tickets cleared on an OTRS queue. Additionally, badges could help signal to new potential participants appropriate points of entry for a project and outline the pathways they could take to grow into more advanced participants.

A badge can be understood as a graphic packaged together with an explanation of how it is earned. It can also be combined with information on who earned the badge and how many badges were awarded. Although each badge represents a discrete accomplishment, badges can be hierarchical with some badges being earned only after certain pre-requisite badges have been earned. Depending on the context, badges can be thought of as simple rewards, as feedback of accomplishment, or as accreditation of skills earned.

A badge system is how badges are displayed to participants and how participants interact with the badges. Designing a badge system means considering the total number of badges to be created, types of badges in the system, types of goals set and accomplishments rewarded, and roles of participants within the system. Badge systems can allow participants to share their badges widely on social media sites or to display locally within the system. Badge systems can also vary based on who does the badge evaluation and assignment -- the wider community, an authority, or an automated system.

Badges v. Barnstars
Although a robust system of barnstars and personal rewards exists to reward “above and beyond” contributions to Wikipedia and its related projects, there is currently no similar way to reward participation for more commonplace skills and tasks. Badges tend to be understood as symbols of accomplishments, identity, and feedback. Barnstars are acknowledgements and “thank you’s” of exceptional work. They are like badges because they recognize accomplishment and can be used to identify contributors that are particularly active on their project. As we see them, barnstars are a type of badge, just named and used in a specific Wikimedia context.

Yet, barnstars don’t accomplish all of the things that could potentially be accomplished with a robust badge system. For example, barnstars don’t tend to acknowledge work that is much less than exceptional. Barnstars don’t acknowledge someone’s first edit or first full article (though these can be great accomplishments for new editors). As pinnacles of achievement barnstars don’t show a path to becoming a better contributor and they poorly recognize intermediate progress. Last, they are not consistent: barnstars may not always be given out regularly to all contributors doing equally exceptional levels of work.

Barnstars could be revamped and redesigned to be better as badges – barnstars could be created for small scale achievements, they could be organized into nested pathways, and they could be more consistently given out. But it also makes sense to preserve barnstars as reward and acknowledgement of exceptional work and to not cheapen or dilute them. To show new editors pathways to success, a separate class of badges could be created to recognize work done on a smaller scale. These badges could be awarded automatically once someone reached a certain level or awarded by people for doing work of a certain level of quality. Additionally, they could be the necessary prerequisites to earning a barnstar. The idea would be to create a badge system that works together with existing barnstars in a way that acknowledges achievement at all levels of engagement.

Design
The aim of this pilot is to build a collection of existing barnstars and additional new badges to serve as symbols of acknowledgement and to monitor their impact on the community. Ideally, badges would provide a mechanism for participants to reward each other’s work, decrease the feeling of discouragement, and help support the editors' success in increasing constructive activity retention. A successful badge pilot would help to establish the process of identifying an appropriate barnstar/badge set and a more straightforward process for rewarding and acknowledging project work.

The badges should directly reflect the work done throughout the encyclopedia and align with participants’ motivations for doing the work. To ensure a good tie to appropriate accomplishments, interviews of project organizers and/or project participants will be coupled with review past qualitative data collected by the project like the Teahouse. Then, community members will be asked to submit relevant badge ideas. A good badge collection would represent acknowledgements for a diverse amount of tasks, reward accomplishments of varying difficulty, and be perceived as meaningful.

The presentation of badges would involve establishing a good badge aesthetic that incorporates the aesthetic of the Teahouse and the chosen barnstars. Badges should also be closely tied with their criteria, titles and descriptions, and should be presented in ways that allow for easy discovery of these meta-data elements.

Project scope and list of activities
Following a review of past badge systems, we will develop multiple iterations of a Wikipedia badge system to groups of editors who visit the Teahouse. In order to examine what types of badge designs work best at motivating editors to learn core editing skills, some of the areas we’d like to explore are:


 * badges representing skills vs. identity roles;
 * badges presented in a sequence vs. as a whole set;
 * community-given vs. self-requested badges;

Additional planning and documentation for this project can be found at: http://en.wikipedia.org/wiki/Wikipedia:Badges.

Project plan

 * Stage 1, Research and preparation
 * Look through past metrics, interviews, and other research to better understand participant motivations and goals on the project
 * Evaluate other award systems
 * Survey participants to gain a baseline measurement for perceived levels of engagement and interest
 * Create a testing framework
 * Create a space to document project progress and findings (develop this page)
 * Explore opportunities for automation (bot, WikiLove, userscript)
 * Outline a badge delivery mechanism
 * Research the conceptual framework behind badges
 * Outline a research plan


 * Stage 2, Brainstorming and community engagement:
 * Engage community and project leaders in brainstorming potential badges for a badge collection.
 * Brainstorm designs for how badges should be presented to participants
 * Ask community members to provide ideas for badges


 * Stage 3, Build
 * Build badges and any associated pages
 * Create any related scripts to enable badge awarding (WikiLove, common.js install script)
 * Finalize research protocol


 * Stage 4, Pilot #1
 * Recruit select participants to start awarding badges on a consistent basis
 * Monitor badge awarding; tweak where necessary


 * Stage 5, Pilot #2 completion
 * Recruit select participants to start awarding badges on a consistent basis
 * Monitor badge awarding; tweak where necessary


 * Stage 6, Measurement and reporting
 * Interview and survey participants
 * Collect and analyze final quantitative metrics
 * Synthesize and report findings and best practices

Research plan

 * Question 1 - How are badges currently being used on Wikipedia? What are some common practices?
 * Methods:
 * Survey overview of common projects
 * Interviews with project organizers


 * Question 2 - Do badges increase editor retention and activity?
 * Methods:
 * Collect user editing behavior over time:
 * Divide users into the following groups:
 * Users who participate in the Teahouse with an invite to participate in the badge program (+TH, -BG)
 * Users who participate in the Teahouse without an invite to participate in badge program (+TH, -BG)
 * Users who get a Teahouse invite and an invite to participate in the badge program but don't participate in the Teahouse (-TH, +BG)
 * Users who get a Teahouse invite but no invite for badge program and don't end up participating in the Teahouse (-TH, -BG)


 * Question 3 - What do we know about people that are using badges?
 * Methods:
 * Track where users are coming from when they respond to the invitation to participate with in the Teahouse with or without badges.
 * Survey demographic questions:
 * Gender
 * Age
 * Motivations for editing
 * Their academic/professional background
 * Articles they like to edit the most (projects they participate in)


 * Question 4 - How are badges perceived by the people involved (hosts and guests)? What are people’s reactions?
 * Methods:
 * Track badge usage rate and nominations
 * Request comments, questions, and suggestions
 * Survey questions about how they felt their badge experience went


 * Question 5 - What aspects of badge design affect the success of the system?
 * Methods:
 * Track badge usage rate and nominations

Non-financial requirements
We will need to coordinate our work with the Teahouse and its leaders. We have built strong relationships with Siko Bouterse, Sarah Stierch, Jonathan Morgan, and Heather Walls; they are interested in using badges to improve the efficacy of the Teahouse on editor retention and engagement and would support our research.

However, we are predominately self-sufficient. Any technical or design help that we will need we are able to source through the Wikimedia community or other contacts.

Fit to strategy
Badges could provide community members of all stages, novice to expert, with goals to work toward and feedback about whether their goals were attained. The main goal of the Wikipedia Badges Fellowship is to see if there is a Wikipedia appropriate badge system design that can help acknowledge the work of editors, be verifiable and help support overall editor retention. Ideally, badges would provide a mechanism for participants to reward each other’s work, decrease the feeling of discouragement, and help support the Teahouse’s success in increasing editor retention. A successful Teahouse badge pilot would help to establish the process of identifying an appropriate barnstar/badge set and a more straightforward process for rewarding and acknowledging project work.

Other benefits
Several projects have looked into potentially using badges as ways to reward and motivate their community members. This project would develop a document of best practices and community responses as to how and when badges are most effective as tools for encouraging acknowledgement and editor retention.

Academic research into badges is currently in the experimental stage. There are not many structured projects where badges have been tested. This would be one of the first. This project would act as proof concept not only for Wikimedia but also for other user generated content environments.

Successful implementation of this program may also be of media interest, at least to the Wikimedia Blog and the Signpost, but also possibly to technology, education, social-media, gamification, and user interface design publications.

Measures of success
Successful implementation would be measured by quantifying and qualifying:
 * Increased editor Retention
 * Increased editor activity
 * Number of people receiving badges
 * Improved editor identification with the Teahouse project
 * Improved editor identification with the goals of the Wikipedia community and Wikimedia movement
 * Improved editor motivation, self- and community-understanding, and confidence

Team members (optional)

 * Anna Shyrokova (User:Anyashy): M.S.I. from the University of Michigan, Interaction Designer at Design for America (Northwestern University), and winner of the 2012 Digital Media and Learning badge-building competition
 * Jake Orlowitz (User:Ocaasi): B.S. in Political Economy from Wesleyan University, 4-year Wikipedia editor with over 20,000 edits, Wikipedia Help channel Operator, creator of The Plain and Simple Guide for Editors, and designer of the learning game The Wikipedia Adventure