User talk:The Transhumanist/PortalTool.js


 * This script is under development, and is not yet functional

= Script's workshop =
 * This is the work area for developing the script and its documentation. Even though this documentation is in user space, feel free to edit it. The talk page portion of this page starts at, below.

Description & instruction manual for PortalTool.js

 * This script is under development, and is not yet functional

When completed, this script will do various things to portals. Initially, it will insert article titles from a category into the template in the selected article section (if there is one).

General approach
(general approach goes here)

More specifically, starting at the beginning...

Desired/completed features

 * Completed features are marked with ✅


 * Menu item: Specify source – prompt user for which section with transclusion template to deal with, prompt user for the source for the chosen section, and insert source tag as a comment into or before the template. Have menu item show up only if one or more of the requisite templates are on the page (if possible).
 * Menu item: Update queques – when clicked, script will check the source tag for each section and replace the existing list with the current list from the indicated source (page or page section). If no source tag exists for a section, the script will prompt the user for the source for the chosen section, and insert source tag as a comment into or before the template, before continuing.
 * Menu item: Update On/Off. When activated, script will initiate Update queues without prompting first.
 * Menu item: Create section – present menu of section types available (won't show WikiMedia as choice if there is already a Wikimedia section.
 * Portal analysis – (automatic when program first accesses the page. Use time stamp to prevent it from running every time page is accessed? Is there a way to tell when the page is left? Also, prevent from running again on a reload from the page, if possible.) Analyzes portal and reports findings to user. Types of sections present, types of sections missing, how many titles in each transclusion template, etc. To give a bird's eye view.
 * Menu item: Add titles – For the sections that use rotating selective transclusion, allow user to add pagenames, even if a source tag is in place (and have those be left in place by the Update queues command). Prompt user for which section with transclusion template to deal with, prompt user for the titles to add. Have menu item show up only if one or more of the requisite templates are on the page (if possible).
 * Portal control console – provides fill-in form through which to modify the portal.
 * Quote section creation (subchoice of "create section")
 * Add quotes – a way to combine user-supplied quotes with program-gathered quotes in portals' quote sections. Use delimiters to surround human editor entries?

categorymembers
Due to the way they are built, categories would be a near ideal place from where to automatically collect page names for Selected article sections.

Unfortunately, Lua can't access the members of a category.

Therefore, we need to find another way to do this.

The api command to list the pages in a category is. See https://en.wikipedia.org/w/api.php?action=help&recursivesubmodules=1&modules=main#query+categorymembers.

There is a user script that does something very similar to what we are after. User:Ais523/catwatch adds new category members to one's watchlist. It seems very likely that Catwatch could be adapted to do this for Selected article sections.

One thing it does is store category names in a js page, formatted in JavaScript as part of the program. I'd like to store the category names in the Selected article sections themselves, and not formatted as JavaScript.

And, of course, rather than add new category members to one's watchlist, we want to add them to the Selected article section as parameters to the transclusion template there.

categorylinks table
It turns out that Catwatch.js accesses the categorylinks table for the data it needs (see discussion below). The author said that it was obsolete, as "MediaWiki has its own category-watchlisting functionality", and you could "use the watchlist table instead".

Script dependencies
= Discussions =


 * This is where the actual talk page starts for User:. Please post your discussion threads below...

What does ListPages.js do?
The title makes me wonder if this might have something we could use, but I can't figure out how to get it to work, or even what it does: User:Dr Brains/ListPages.js.

The doc is no longer around, and didn't leave any instructions behind.

What does it do, and how does it work? &mdash; The Transhumanist  23:45, 4 July 2018 (UTC)
 * Looks like it was intended to get all articles in a category, including all subcategories, and then find each article's interwiki target on the French Wikipedia, and produce a table of articles and their French equivalent. But it is using outdated/deprecated techniques, which would be why you can't get it to work. - Evad37 &#91;talk] 05:31, 6 July 2018 (UTC)
 * Thank you. I'll cross this one off my list.  &mdash; The Transhumanist   07:58, 8 July 2018 (UTC)

Adapting catwatch.js
Dear Ais523,

I'm The Transhumanist, one of the acting curators over at the Portals WikiProject. We've run into a brick wall, and we need your help.

After years of being dormant, the Portals WikiProject was jump started this past April, has grown to almost 100 members, and is in the process of revamping the entire portal system. We are going all out, and have a very active and productive development team. You can see the progress we've made so far in the project's newsletter archive. The excitement level is high.

While many portals will continue to be maintained by hand, we are attempting to build a portal model that is entirely automated, as most of the portals do not have active maintainers. There are currently about 1500 portals, of which about 10% are actively editor-maintained.

So far, we've managed to automate 3 standard portal sections, and have gotten 4 others to semi-automated status. In other words, 3 sections are now virtually maintenance-free. It would be nice if the other ones can be too.

Which brings us to the brick wall...

We have been trying to use Lua (modules) to automate certain portal sections with a method known as selective transclusion. Here is an example of a "Selected item" section using the Transclude random excerpt template:

It displays an excerpt randomly, taken from one of the pages from its internal list, each time the portal page is purged.

The above template requires that the user supply the names of the pages to be transcluded. That makes this only a semi-automated solution. The sections using this rotate material, and therefore don't go stale as fast as a section with a single excerpt, but, adding new article titles by hand is how they are updated, and unfortunately, that method is not scalable.

We've been banging our heads against the brick wall to find a way to pull the names from a category with Lua, so that portals using these templates will be auto-updating (as the category grows, so would the selection in the portal accessing that category), but the generated nature of categories and the limitations of Lua prevent that.

Which means we must find a way other than Lua to generate a list of the pages from a category. Since JavaScript can make a wide range of api calls, that looks like the way we will have to go. (See https://en.wikipedia.org/w/api.php?action=help&recursivesubmodules=1&modules=main#query+categorymembers`). (That's how I found catwatch and you: via the search string "categorymembers").

It seems very likely that Catwatch.js could be adapted to do this for Selected article sections.

One thing catwatch does is store the category names in a js page, formatted in JavaScript to be run as part of the program. Instead, I'd like to store the category names in the Selected article sections themselves, such as in a hidden comment.

And, rather than add new category members to one's watchlist, I want to add them to the Selected article section as parameters to the transclusion template there, just like | Frog | Toad | Salamander | Caecilian

are included in the example section provided above.

Any assistance or guidance you could provide would be most helpful.

I look forward to your reply. Sincerely,  &mdash; The Transhumanist   21:51, 5 July 2018 (UTC)

P.S.: I've pinged TheDJ, because the script mentioned he worked on it, and Evad, as he has been working on this problem and also has familiarity with JS. -TT
 * Replied on User talk:The Transhumanist. --ais523 05:38, 6 July 2018 (UTC)
 * One of the issues with user scripts is that, by definition, they only apply to a single user. So you can't use a user script to do something like extract category entries to be displayed on a page for anonymous users; you could do it so that you saw excerpts yourself, but that would defeat the likely point of the project (to update the portal pages so that everyone sees them).
 * There's another issue with using JavaScript for this, too; being a client-side technology (i.e. it runs in the end user's browser), it runs entirely after all the server-side technology (i.e. things that run on Wikimedia's servers), meaning that you couldn't use JavaScript "live" to produce information that was passed to a Lua module. This applies to all uses of JavaScript (there are two main uses of it in Wikipedia, user scripts and MediaWiki:common.js, although I suspect that you'd have problems convincing people to change the latter for this purpose as not all users will run it; many users have JavaScript disabled by default, and there are possible performance issues too).
 * It seems like the correct solution to the problem, therefore, is not to try to work out the list of category entries when a user views a page (which is what the attempted Lua solution would do, and what you seem to have been assuming the JavaScript solution would do). Rather, what you want to do is automatically update lists of category entries somewhere that a Lua module can see them. That sounds like a job for a bot, rather than a script. I don't have much experience with fully automated bots from the point of view of writing them (I ran a bot in the past but it was basically an account with a very specific set of user scripts); you'd probably want to use an existing bot framework for something like this, and it would probably be worth contacting someone with more experience in those.
 * One other possibility would be to ask the developers to make the task you're aiming for easier. The category membership table is fairly quick to query behind the scenes (and in fact, Special:RandomInCategory exists but I haven't found any way to make its random selection feature accessible from Lua or wikitext). The best/simplest solution to this problem would be for MediaWiki to implement a magic word or ParserFunction which returns a randomly selected page from a category, and given that it would obviously improve the wiki (and use much less resources than a bot or script would!), it seems reasonable that such a request might be accepted. WP:VPT might be a good place to talk about this. --ais523 05:37, 6 July 2018 (UTC)
 * Thank you for your input. Some very good ideas there.
 * My understanding of going through Phabricator is that it is hit and miss and a lot of waiting and wondering. So, while I won't rule out that as an option, I'd like to pursue this avenue of development as well.
 * I wasn't thinking about a viewing script doing its thing (pulling category members) in real time for everyone. What I would like to do is write a program to edit the portal pages, updating them for everyone's benefit. I'm sorry my explanation wasn't clearer.
 * You are right that I want to update lists of category entries somewhere that a Lua module can see them. That is exactly what I was talking about. The place I want those lists is as template parameters right in the portal's wikicode. The templates use Lua, which in turn use the parameters as input. Here is an example with a list already inserted &mdash; read the embedded comment for clarification:
 * The actual section that this example was taken from is at Portal:Amphibians, and includes over fifty frog articles. There is also a toad section, a salamander section, and a caecilian section, each with their own list. Now you can see why we don't want to have to insert the article names manually.
 * I don't mind if the program is a bot, written in JavaScript, or a userscript providing a menu item. It would be easier to develop it as the latter first, to allow for more interactive development and testing (with multiple users) before making a fully automated (bot) version.
 * You mentioned "work out the list of category entries". That's what I need to know how to do. I can handle menu items (see SearchSuite.js), and editing wikicode (see RedlinksRemover.js). It's what goes in between (fetching category members and putting them into a variable) that I'm having trouble with.
 * Therefore, I have some questions for you...
 * How does catwatch fetch the members of a category?
 * How does catwatch process those?
 * How does catwatch insert the result into a page? (I understand that this is not an edit page, but I don't recognize the methods being used, and I would like to grok this program.)
 * Any enlightenment you could provide concerning these mysteries would be very helpful.
 * I look forward to your replies.  &mdash; The Transhumanist   08:49, 6 July 2018 (UTC)
 * Oh, in that case, I have some bad news for you: catwatch doesn't fetch the members of a category. Doing so would be much too inefficient to do on every page load if the category were large. (There are some people who catwatch Category:Living people!) It's fetching only the most recently recategorised member of the category, via an API search of the category links table. (Incidentally, now that MediaWiki has its own category-watchlisting functionality, it would be possible, and probably more efficient, to use the watchlist table instead. catwatch is mostly redundant at this point.) As such, no processing is actually needed; we ask MediaWiki for the most recently recategorised page in the category, then simply display that to the user.
 * For your purposes, you'd probably want to record the time at which the script was last run. You could then query the categorylinks table for pages in the category that were recategorised since your last run of the script. Unfortunately, that would only spot insertions, not removals (as a page that was removed from the category will no longer be in the category); note that catwatch currently can't spot removal of a page from a category either. (Perhaps your script would also better work off a watchlist than a category table! Especially if you use a bot account to update the portals, even if the bot's implemented using user scripts, you could put all the categories in question – and nothing else – onto the bot account's watchlist, meaning that a watchlist query would be a very efficient way to get a list of exactly what you needed to update and on which pages.)
 * Inserting the result into a page is done by editing the page's HTML once it loads. You can use a very similar technique to edit the edit box of a page; simply find the edit box via its ID, then edit its value. Presumably you'd want some comments in the page specifying which bit gets updated by the bot; then it could split the string into three pieces using the comments in question and update just the second piece. --ais523 09:05, 6 July 2018 (UTC)
 * Inserting the result into a page is done by editing the page's HTML once it loads. You can use a very similar technique to edit the edit box of a page; simply find the edit box via its ID, then edit its value. Presumably you'd want some comments in the page specifying which bit gets updated by the bot; then it could split the string into three pieces using the comments in question and update just the second piece. --ais523 09:05, 6 July 2018 (UTC)