Wikipedia:WikiProject External links/Webcitebot2

The WebCiteBOT Replacement Task Force is a sub-group of WikiProject External links. It is a coordinated effort to address the issue of citation templates having dead external links. The number of articles with dead tagged links went from under 10,000 to over 105,000 in 2010.

Overview
Wikipedia relies on verifiable information from reliable sources to ensure that the information it carries is accurate and presented from a neutral point of view. These are very basic needs of the project. Wikipedia uses external links to reliable information to accomplish this task. Unfortunately, the Internet is not stable and links come and go. So this issue needs to be addressed if Wikipedia is to have a succesful future.

WebCiteBOT was an incredible resource that monitored the external link feed of the IRC channel #wikipedia-en-spam. It searched for these links in citation templates and then submitted them to WebCitation.org and updated the templates. Unfortunately the bot died in November 2009 and dead links have been increasing ever since. (see graph above)

Goals
This is yet to be defined. At a minimum, some type of tool should be developed to assist archiving external links in citation templates.
 * 1) The bot or tool should be operated by at least 2 users so that when an operator leaves, we don't have this issue again.
 * 2) A goal of a functional equivalent of WebCiteBOT is probably worthwhile. This means an automated bot that monitors the external link feed of the IRC channel #wikipedia-en-spam and submits them to WebCitation.org and updates citation templates.
 * 3) Another related issue that was discussed over the last year was having the Wikimedia Foundation take over the role of WebCitation.org (for citation links) or more formally collaborate with / support webcitation.org to guarantee sustainability of the service.

Previous attempts
There have been several Wikipedians that have brought this issue before the greater Wikipedian community in order to have this problem addressed. A cash reward has even been offered by one user in order to get this issue fixed. Below is a list of the attempts in chronological order.
 * 13 June 2010 - Bot requests/Archive 37
 * 24 August 2010 - WP:Village pump (policy)/Archive 78
 * 24 August 2010 - Bot requests/Archive 37
 * 23 September 2010 - Administrators' noticeboard/Archive216
 * 27 September 2010 - Bot requests/Archive 38 (loads slowly)
 * 11 November 2010 - Village pump (miscellaneous)/Archive 29
 * 11 November 2010 - Bounty board
 * 20 November 2010 - village pump

Bots possibly in development
A number of our software engineers have been working on the WebCiteBOT issue and may possibly have bots in the development stage. The current status of most of these bots is unknown at this time.

Related bots
Several bots have been developed over the years that have dealt with external link issues. Several of these are no longer operating, but they may provide examples that software engineers can use to help develop a new bot. Please note that some/most of the source code below is not free to use. Therefore, it can only be examined to give ideas for possible solutions to different tasks.

Bot resources
Below are some resources for users wanting to build bots that work with external link issues.
 * AutoWikiBrowser semi-automated editor for Wikipedia
 * Pywikipediabot a basic bot (written in Python) that can be expanded
 * weblinkchecker.py script for Pywikipediabot that finds broken external links

WebCitation.org's position on bots

 * WebCiteBOT's operator, ThaddeusB, was in contact with Gunther Eysenbach of WebCitation.org and said they were supportive of the project.
 * I confirm that WebCite is very supportive. Ideally WebCite want to have the metadata associated with a reference handed over as well, so that statistics like "most cited author on wikipedia" can be calculated and displayed on a dashboard on the WebCite site. Please also hand over the "citing" article URL, e.g. in the refdoi field. WebCite is open source, so changes to the WebCite backend code (including a WebCitebot dashboard on the webcitation.org site) are also possible. To communication with WebCite, please use email only (I do not check my talk page on wikipedia), see http://www.webcitation.org/faq --Eysen (talk) 18:20, 9 February 2011 (UTC)


 * Regarding the submission rate, nn123645 contacted WebCitation.org and they asked for an initial limitation of 1 submission every 5 seconds though that limitation would likely be removed later.