Wikipedia talk:Research help


 * This page is for discussing the deployment of WP:Research help following the proposal to add Research help to article pages, following the process outlined here. Please create a new section to add new feedback.

Comment
Came here after seeing this template added to debut novel. Honestly this doesn't seem like a good idea. If the goal is to have it on every page (which the project page says is the goal), then using templates in article space is a bad idea. It should be in the left hand column under "Interaction" so it doesn't clutter up article space. There are many many things editors should be aware of, that's why there is a "About Wikipeage" page linked to the left, where you will find Using Wikipedia as a Research Tool. -- Green  C  00:02, 21 December 2015 (UTC)
 * Hi, I added that template using my volunteer account in my volunteer time, because I was excited about the page, and piloting it. I shouldn't have done that: we are trying to work systematically with community consensus on the pilot -- and I shouldn't be mixing my work/volunteer time.
 * As for the concept: we are responding to a particular ask/concern of the libraries community: how do we get readers (not editors) who are using Wikipedia for academic or other kinds of research do so with critical awareness. Our theory, is that the page will meet readers best when its concise and it helps them do the thing they are doing in the moment: research. There are particular questions that researchers, and teachers of students doing research, ask in every outreach situation (from library conferences, to GLAM-Wiki events, to WEP recruitment). The questions are designed to meet our ideal audience, which can be split into two buckets: 1) students using Wikipedia in research and 2) people who teach research and literacy (librarians and teachers). That we reach potential new editors, is a secondary feature, and we would welcome links to more learning tools: but something we have learned, is that new users are easy to overwhelm and pages like WP:About, are just that: overwhelming. We want something a researcher can engage with in a very short window of time.
 * We don't plan on the template being a permanent solution -- if consensus is for the link to be in the reference section we are hoping to integrate it into reflist or . Esquivalience  t 02:47, 14 March 2016 (UTC)
 * Brilliant! Thanks, going to add that to the FAQ, Astinson (WMF) (talk) 18:23, 14 March 2016 (UTC)

Research methodology
Taking off my "grumpy old Wikipedian" hat and donning my academic bonnet of a professional researcher, I am curious how this trial is to be evaluated in terms of the success of its mission.

The commentary here suggests this is a trial on 10,000 pages. What is the duration of this trial? I saw some mention of one month. Is that correct or is there some other criteria that ends the trial?
 * Yes, we had intended 1 month -- but we didn't label it as such, because we have seen a number of small failures in maintaining schedules -- so didn't want to overcommit. We planned on withdrawing the template with noinlcudes at 1 month or when a concensus emerged that it was done. Because of delays in implementing it and in our other research, our preferred timeline has become mismatched when we are getting which data.Astinson (WMF) (talk) 00:06, 11 March 2016 (UTC)

After the trial is completed, how is the success or failure of the trial evaluated? I saw a mention of hits on the link to the Research Help page. Is that the criteria? If so, how will hits by people actually in need of this information be distinguished from those who were (like me) curious/irritated/enraged? I saw a comment about number of survey completions, but given the call to action is to "help improve this page", then it suggests that responders may well be people who want more information about the topic or those who have come to this page feeling they understand the topic but think the advice could be improved in some way. It seems neither of these have any real potential as a measure of success.Kerry (talk) 22:14, 10 March 2016 (UTC)
 * We are doing several things at once: collecting the data from source, getting consultant research in digital literacy from both a library practicum student and a couple of outside experts (included, the Rutgers study librarian), and doing some reader walk through UX/UI conversations. We hoped to analyze all these to see if there is a better decision and provide that data to the volunteer community to get wider consensus, rather than one point of failure. We honestly didn't know what the results would look like, because we aren't aware of a similar intervention in the way readers interact with our content (most interventions have been purely design oriented or interface oriented). We also have a body of colloquial knowledge collected from the GLAM, Education and Library outreach communities suggesting this kind of basic literacy is huge barrier to new reader conversion to editors (and is something being validated by Aaron Halfaker's work). Our hope was to examine this as a programmatic intervention -- as part of the Wikipedia Library's domain of research outreach, and generating editors who do research. This means its less rigorous -- because it operates more within our theory of change and programmatic opportunity. That being said, we are trying to collect as much learning as we can, with the connections/support we do have. Astinson (WMF) (talk) 00:04, 11 March 2016 (UTC)

Is it intended to run another Twitter survey? If so, what level of improvement in the responses is desired as a metric of success? And do we believe that the audience of Twitter (generally) or the followers of Wikipedia's twitter account (particularly) are representative of Wikipedia readers generally or of the Wikipedia readers who we think are in need of the information in the Research Help page. FWIW, Twitter has a relatively low following compared to (say) Facebook and with a different demographic. See here for some discussion of demographics. One specific point raised there is that Twitter is much more "male" and Facebook more appealing to "women". Given Wikipedia's somewhat notorious gender gap, I would be very careful about taking input from a predominantly male source for fear of increasing the systemic bias.Kerry (talk) 22:14, 10 March 2016 (UTC)
 * The Communications team was using that survey as a test, in how to engage its audience. I would love to do something similar again as well in other contexts. I do know a few things about our social media audience. From the #1lib1ref campaign we were able to get over 50% women engaging on twitter, so at least the @WikiLibrary handle has a strong female following, and the biggest audiences for both Facebook and Twitter for the Wikipedia handle are in India. For Wikipedia 15, they have done some initial analysis of the comments in both mediums, and also found a wide range of comments about educational use, student research, and "credibility questions". The reading and communications team are also doing broader studies, which haven't been released yet, that suggest non-Western audiences don't have even as much understanding of Wikipedia (waiting for those studies to be published, but I promise they exist). I think of the twitter survey, as one of a constellation of signals that we have a core literacy problem -- that we need to tackle in multiple ways.Astinson (WMF) (talk) 00:04, 11 March 2016 (UTC)

As per the mentioned poster, is it intended to grab the usual "convenient sample" of undergraduate students keen to make their teacher happy by being research subjects (university students are the best studied species on the planet) to be interviewed about whether they have seen these links to the Research Help page and whether they have followed the link and felt better informed because of it or changed their behaviours as a result of it? I note that a sample of 30 students (as used on the poster) are probably statistically unlikely to have even seen the link given it's on 10,000 out of 5M pages. So I think you'd have to survey a lot more readers to find ones that had genuinely seen it in their random browsing rather than construct a survey in which the student keen to please their teacher is shown the link first and then asked about it.Kerry (talk) 22:14, 10 March 2016 (UTC)
 * I agree, we need bigger studies. That something we would love to see, and something TWL wants to encourage among our partners, but we don't have the bandwidth to persue it actively. We do have some research from years ago (for example) which talks about research usage/literacy, but those data points are really old -- I compiled the proposal material using much newer data points. Astinson (WMF) (talk) 00:04, 11 March 2016 (UTC)

I ask these questions because I do believe it is genuinely difficult to judge the success of a Wikipedia-based program aimed at unknown anonymous readers behaving differently in contexts outside of Wikipedia (e.g. students doing better research and writing better assignments) using any kinds of metrics that we can gather from Wikipedia itself. If you look at the 1Lib1Ref program, there are ways you can use Wikipedia metrics to evaluate it -- you can look for the hashtage #1Lib1Ref that they were asked to add in the edit summary and you compare the number/proportion of edits that add a ref tag before/during/after the campaign to see if the campaign made a visible impact and to what extent it may have been sustained after the campaign.Kerry (talk) 22:14, 10 March 2016 (UTC)
 * I would love feedback on our #1lib1ref campaign analysis at meta:The Wikipedia Library/1Lib1Ref/Lessons. It was a small scale program, and operates under much the same theory as this intervention: that the biggest barrier to contribution in our community, is a lack of deep understanding of how editors identify what needs more research and verification.Astinson (WMF) (talk) 00:04, 11 March 2016 (UTC)

So, for this trial, is it possible to measure the rate of click-through of external links in citations? If the campaign is successful, we would expect readers to be more interested in reading citations, which should manifest in higher click-through rates in the references section. So, in theory, we could measure: Kerry (talk) 22:14, 10 March 2016 (UTC)
 * any changes in the rate of click-through to citations in the articles with the Research Help link
 * ... in other articles within the scope of the Wiki Projects participating
 * ... in articles generally
 * Unfortunately, I think this kind of change is going to be almost impossible to measure in the short term. We do have some evidence from the Wikipedia Education program, that students that understand how to edit, are more articulate and self aware about their own research habits (I have a 2 term study of different groups of students I taught at K-State that will be published soon which verifies this)-- and how they use Wikipedia. We have also so seen the rate of DOI referrals increase from 8th to 5th. Moreover, WMF Research and TWL are working to document how referrals work to academic works as part of research on how the HTTPS change effected referral information. Because the TWL team is largely experienced in programatic support, we are approaching this pilot as a programmatic activity, which is more based on our audience needs and seeing if we can generate community support for a theory of change.Astinson (WMF) (talk) 00:04, 11 March 2016 (UTC)

Is such data available? Lifts in the click-through rates of citations would demonstrate a measure of success -- it might even convince this grumpy old Wikipedian :-) Kerry (talk) 22:14, 10 March 2016 (UTC)
 * I did some responses above, in line--- feel free to break them out, or respond there as well, Astinson (WMF) (talk) 00:06, 11 March 2016 (UTC)
 * Thinking a little more about it, we possibly can get a tighter set of data around student behaviour from Wikipedia itself. We have 85000+ IP users which are tagged as being educational institutions (Template:Shared IP edu). So we could add to my list above:


 * any changes in the rate of click-through to citations by educational-institution IP addresses

Is this data obtainable? Kerry (talk) 22:36, 10 March 2016 (UTC)
 * I would love to help someone obtain that data, and I am sure its something that WMF Research would be interested in helping someone acquiring this aggregated anonomously. If you are interested, I would love to connect you -- go ahead and email me: astinson@wikimedia.org, Astinson (WMF) (talk) 00:04, 11 March 2016 (UTC)


 * Who is saying what here ...looks like one person replying to themselves.--Moxy (talk) 02:10, 12 March 2016 (UTC)
 * like in the above section, I responded to the bulleted questions by Kerry, with indented responses directly after the question, rather than have hard to read responses that are mismatched. Astinson (WMF) (talk) 18:22, 14 March 2016 (UTC)
 * While I agree the structure of talk pages isn't helpful here, if you are going to respond in-line in this way, I think you need to copy my signature over to the ends of each chunk before your reponse, so others can clearly distinguish what I said and what you said. As it stands, it looks as though you have written much of what I wrote based on the positioning of the signatures. Kerry (talk) 20:54, 14 March 2016 (UTC)
 * ✅ and will do so with the other section, Astinson (WMF) (talk) 22:19, 14 March 2016 (UTC)

Some important points

 * 1) You should get approval from WP:VP(T) WP:VP before imposing this on 10,633 prominent pages.
 * 2) The name is misleading - the project is mainly about "How to use references"
 * 3) WikiProjects do not "own" articles that are within their scope.

All the best: Rich Farmbrough, 11:24, 22 March 2016 (UTC).


 * Hi Thanks for the feedback. This is a pilot - we sought concensus from the editors most effected by the change and notified village pump about the conversations. We plan to pull down the pilot after it has been on article content for 1 month, to make sure we get enough data and feedback to determine if it continues, and to get sufficient feedback on the page proper to give us a good point of reference. The no consensus at this TFD suggests that there is at least significant enough support for this pilot, though also significant concern about the template, to better assess this and have a full concensus (we plan to have a full scale VPT conversation before any other scaling).
 * As for misleading title: is it? How to use references, and how to check if Wikipedia content is verifiable, is an important part of anyone's research process if they are consulting Wikipedia. Also, our feedback on the survey strongly suggests that the page answers most of those questions, and that students and educators are finding it useful (over half of the survey respondents are related to teaching/universities). If you would like to suggest some other titles, we would be happy to test them. Astinson (WMF) (talk) 17:01, 22 March 2016 (UTC)
 * It would be useful to have a definition of "research" in this context. We have the following (at least):
 * People who publish "research papers"
 * People who are researching for other published material, which may or may not include references
 * People who are doing "research" to show some didact they have learned how to do it.
 * People who are finding out about something "researching it" for a practical purpose
 * People who are adding to their store of knowledge - "researching" for their own understanding
 * It is far from clear which of these is the target audience, or how the message is relevant to other audiences.
 * (Now if WMF wants to trial something like this, why not propose it to the community (and have it put into the sidebar)? Has WMF learnt nothing from Lila's departure?  From Visual Editor?  From Flagged revisions?  From Superprotect?)
 * All the best: Rich Farmbrough, 17:30, 22 March 2016 (UTC).


 * Hi Rich: this fell off my radar in tracking the other conversations, many apologies: "research" for the vast majority of people in the public, means learning and collecting information for synthesizing that information for another audience (whether in school, hobby or as part of a job). In developing this, our ideal audiences were 1)Librarians and educators attuned to this kind of conversation already, 2) folks in educational and research setting which robustly use reference sections and other parts of Wikipedia, and 3) potential contributors and/or new contributors who could imagine themselves as "doing research" -- a skill we need very broadly in the community, because it provides the foundations of core policies . I will say that all of the feedback we received in the survey from readers did not challenge or question this. We believe its sufficiently clear in many contexts, but would be open to a better and/or simpler language.
 * As for the meta question of "hasn't WMF learned". Quite frankly, we are very different teams, working in very different areas, and this was not a software change, so doesn't follow the same kinds of processes as VE, etc: we were doing a programmatic reader-facing experiment relevant too our mission and scope as a program (programatically exploring how to improve the entire Research process on Wikipedia). During the time we were working on this page, two seperate reader surveys and interviews conducted by two seperate teams at the foundation in South Africa and Mexico identified the exact same weakness in reader literacy that we are trying to examine: this isn't just a pet project, but a global issue of readers not understanding how Wikipedia is created, or knowing how much (or if) to trust that process -- and subsequently the content and platform. Moreover, without this knowledge, you are defeating the broader editor pipeline (you need to know how Wikipedia works, before you can edit it). We have pretty substantial research in the Western libraries and education communities to support this as well-- a part of our next step in the process, is to better understand what we didn't present effectively to the community about this. I appreciate the feedback and thorough thinking, and I would greatly appreciate your input/feedback on next steps, once we do a "after action" review of our process. Astinson (WMF) (talk) 23:41, 4 April 2016 (UTC)

What are we trying to solve?
I have read this section four times. It explains the authors' opinions on several matters, with some references, which might provide the context of the problems, but does not itemise the problem or problems to be solved.

All the best: Rich Farmbrough, 14:37, 25 March 2016 (UTC).


 * Hi Rich: we are trying to solve a broad illiteracy amongst our global readers about the process by which our content is created, and to provide tangible language and stories that help readers, esp. educator, librarians and student but also more broadly partially "bought-in" readers, to understand how and why Wikipedia works the way it does, with a wide range of quality in content. For Wikipedia Education Programs, Library outreach, and other GLAM outreach the world over: this is a oft repeated set of knowledge, and often provides the first barrier for creating new editors. We believe that this will help fill the trust gap, by informing readers where they are at in the process of reading and researching new knowledge on Wikipedia. See above and the data in the Proposal page for more information of why we believe this is a key problem for our community and readership. Astinson (WMF) (talk) 23:47, 4 April 2016 (UTC)

Courtesy notice
There has been posted at WP:VPR a proposal to terminate the trial. BethNaught (talk) 22:26, 25 March 2016 (UTC)
 * I pinged everyone. All the best: Rich Farmbrough, 05:09, 26 March 2016 (UTC).


 * Rich, I didn't get your ping. Maybe they don't work when they're inside a box?  WhatamIdoing (talk) 20:35, 1 April 2016 (UTC)
 * Curious. There may be a limit to the number of accounts you can ping from one message. All the best: Rich Farmbrough, 20:41, 1 April 2016 (UTC).

Noinclude
I have noincluded the template just as the WMF staffers said it could be if there was any objection.

I therefore think it unreasonable for said WMF staffers to revert, especially since we have consensus.

Nor is it concomitant with Wiki culture to filibuster, to get the "required" outcome.

All the best: Rich Farmbrough, 01:27, 3 April 2016 (UTC).


 * FYI, none of the staff involved in this project reverted the consensus change, that was Ed17 -- during his personal, volunteer time. He has not been involved in our programmatic work -- its two different parts of the organization. Astinson (WMF) (talk) 18:08, 4 April 2016 (UTC)
 * Yes, Ed has made that clear. And I made clear why I had thought he had his WMF hat on.  It's interesting how people believe that their innate honesty and ability to compartmentalize without being affected by COI, shines through the written word.
 * Do you have any response to sections 15 and 16?
 * All the best: Rich Farmbrough, 18:55, 4 April 2016 (UTC).


 * Sorry for not responding sooner: too many moving projects at once, and lost track of these messages. I hope the above explanations help. I would really like to partner with you to see if we can find a better way to approach and communicate how a project like this might work -- or what the better options might be for approaching the broader problems of our readers not understanding how Wikipedia's content is tied to sources, thus able to use it effectively in research. Astinson (WMF) (talk) 23:49, 4 April 2016 (UTC)

Alternative
Here is a possible alternative
 * Reference help help.png

A box would be nice of course.

All the best: Rich Farmbrough, 19:12, 14 April 2016 (UTC).


 * All the best: Rich Farmbrough, 11:45, 6 May 2016 (UTC).


 * Hey Rich, thanks for the idea. I have been traveling a fair bit recently, so sorry for not responding sooner. We are analyzing the survey and pilot data right now, so I will have a better response to you on the alternative and where this fits into the larger research that is going on right now, in terms of helping our readers understand our content (it appears that several other teams were thinking about similar things at WMF, and we have begun talking to them as well).
 * Personally, and based on the research we are seeing come out of other teams at the Foundation, I don't think adding a link on the left hand side of the page is a good solution (though the way you insert the text right now is very interesting -- and definitely something that I might apply in other contexts). I really want to make sure we are proposing solutions that don't just shuffle the link into part of the interface that we do not have good evidence that its used very frequently. Thanks for staying engaged! Astinson (WMF) (talk) 21:52, 10 May 2016 (UTC)
 * I think the standard "tool-box" is something people are acclimatised to, and therefore ignore. Something like this, standing on its on would draw attention, whilst still maintaining the division between content and instruction.
 * I may propose a weeks trial, if I have the energy.
 * All the best: Rich Farmbrough, 16:54, 1 June 2016 (UTC).


 * Would love your feedback on the evaluation of the outcomes described in the section below. Sorry for not getting the information published sooner, Astinson (WMF) (talk) 14:39, 14 September 2016 (UTC)

Input on Research Help Pilot evaluation
The Wikipedia Library has now posted a report on its spring pilot test of a Research Help portal. As the report outlines, our target audience of readers and new editors generally reacted more positively to the pilot than experienced editors, who raised important critiques for discussion. The report provides more details on the results and some proposed next steps for the project. Your input is welcome on the report talk page.Astinson (WMF) (talk) 14:38, 14 September 2016 (UTC)

what are the facilities for self employment
I need an answer to the question.

DivineG (talk) 07:40, 29 January 2017 (UTC)