User talk:The Transhumanist/PortalTool.js

This is the workshop support page for the user script PortalTool.js. Comments and requests concerning the program are most welcome. Please post discussion threads below the section titled Discussions. Thank you. By the way, the various scripts I have written are listed at the bottom of the page.

This script is under development, and is not yet functional

Script's workshop

This is the work area for developing the script and its documentation. Even though this documentation is in user space, feel free to edit it. The talk page portion of this page starts at #Discussions, below.

Description & instruction manual for PortalTool.js

This script is under development, and is not yet functional

When completed, this script will do various things to portals. Initially, it will insert article titles from a category into the template in the selected article section (if there is one).

Explanatory notes (source code walk-through)

This section explains the source code, in detail. It is for JavaScript programmers, and for those who want to learn how to program in JavaScript. Hopefully, this will enable you to adapt existing source code into new user scripts with greater ease, and perhaps even compose user scripts from scratch.

You can only use so many comments in the source code before you start to choke or bury the programming itself. So, I've put short summaries in the source code, and have provided in-depth explanations here.

My intention is Threefold:

1. to thoroughly document the script so that even relatively new JavaScript programmers can understand what it does and how it works, including the underlying programming conventions. This is so that the components and approaches can be modified, or used again and again elsewhere, with confidence. (I often build scripts by copying and pasting code that I don't fully understand, which often leads to getting stuck). To prevent getting stuck, the notes below include extensive interpretations, explanations, instructions, examples, and links to relevant documentation and tutorials, etc. Hopefully, this will help both you and I grok the source code and the language it is written in (JavaScript).
2. to refresh my memory of exactly how the script works, in case I don't look at the source code for weeks or months.
3. to document my understanding, so that it can be corrected. If you see that I have a misconception about something, please let me know!

In addition to plain vanilla JavaScript code, this script relies heavily on the jQuery library.

If you have any comments or questions, feel free to post them at the bottom of this page under Discussions. Be sure to {{ping}} me when you do.

General approach

(general approach goes here)

More specifically, starting at the beginning...

Aliases

An alias is one string defined to mean another. Another term for "alias" is "shortcut". In the script, the following aliases are used:

$ is the alias for jQuery (the jQuery library)

mw is the alias for mediawiki (the mediawiki library)

These two aliases are set up like this:

( function ( mw, $ ) {}( mediaWiki, jQuery ) );

That also happens to be a "bodyguard function", which is explained in the section below...

Bodyguard function

The bodyguard function assigns an alias for a name within the function, and reserves that alias for that purpose only. For example, if you want "t" to be interpreted only as "transhumanist".

Since the script uses jQuery, we want to defend jQuery's alias, the "$". The bodyguard function makes it so that "$" means only "jQuery" inside the function, even if it means something else outside the function. That is, it prevents other javascript libraries from overwriting the $() shortcut for jQuery within the function. It does this via scoping.

The bodyguard function is used like a wrapper, with the alias-containing source code inside it, typically, wrapping the whole rest of the script. Here's what a jQuery bodyguard function looks like:

1 ( function($) {
2     // you put the body of the script here
3 } ) ( jQuery );

See also: bodyguard function solution.

To extend that to lock in "mw" to mean "mediawiki", use the following (this is what the script uses):

1 ( function(mw, $) {
2     // you put the body of the script here
3 } ) (mediawiki, jQuery);

For the best explanation of the bodyguard function I've found so far, see: Solving "$(document).ready is not a function" and other problems   (Long live Spartacus!)

The ready() event listener/handler

The ready() event listener/handler makes the rest of the script wait until the page (and its DOM) is loaded and ready to be worked on. If the script tries to do its thing before the page is loaded, there won't be anything there for the script to work on (such as with scripts that will have nowhere to place the menu item mw.util.addPortletLink), and the script will fail.

In jQuery, it looks like this: $( document ).ready(function() {});

You can do that in jQuery shorthand, like this:

$().ready( function() {} );

Or even like this:

$(function() {});

The part of the script that is being made to wait goes inside the curly brackets. But you would generally start that on the next line, and put the ending curly bracket, closing parenthesis, and semicolon following that on a line of their own), like this:

1 $(function() {
2     // Body of function (or even the rest of the script) goes here, such as a click handler.
3 });

This is all explained further at the jQuery page for .ready()

For the plain vanilla version see: http://docs.jquery.com/Tutorials:Introducing_$(document).ready()

Only activate for vector skin

Initially each script I write is made to work only on the vector skin, the skin under which I developed it, and by default the only skin for which it is initially tested with. To limit the script to working for vector only, I use the following if control structure:

if ( mw.config.get( 'skin' ) === 'vector' ) {
}

To test it with another skin, remove or comment out the above code from the script.

Deactivation filters

Many scripts are written to work on a particular page or page type, and might have unexpected results if run on some other page. So a deactivation filter is used so the program does not run for the wrong pages.

For example:

    if (document.title.indexOf("Watchlist - Wikipedia") == -1) {
        // use a return statement to end the local function and hence the program's body
        // important: this approach does not work outside of a function
        return;
    }

What this if statement does is checks that the current page is not the one we want, and if that is true, we end the program via a return statement.

What return; does when alone like this (without any parameters), is to end the highest-level function which it is within. And since the body of the program is also within that function, if the if statement isn't true, the program ends.

You could do something similar with a straight if construct without "return;", checking for a page match, but then you'd have to have your whole script body inside the construct, which adds a level of indentation. The more filters, the more levels of indentation. The above approach avoids unnecessary indentation, and makes it easier to keep track of the curly brackets, as the closing bracket isn't way off at the end of the program.

var

This is the reserved word var, which is used to declare variables. A variable is a container you can put a value in. To declare the variable portletlink, write this:

var portletlink

A declared variable has no value, until you assign it one, such as like this:

portletlink = "yo mama";

You can combine declaration and assignment in the same statement, like this:

var portletlink = mw.util.addPortletLink('p-tb', '#', 'Remove red links');

Caveat: if you assign a value to a variable that does not exist, the variable will be created automatically. If it is created outside of a function, it will have global scope. For user scripts used on Wikipedia, having a variable of global scope means the variable may affect other scripts that are running, as the scripts are technically part of the same program, being called via import from a .js page (.js pages are programs). So, be careful. Here are some scope-related resources:

• The Horror of Implicit Globals
• Scope in JavaScript

Task list

Desired/completed features

Completed features are marked with  Done

• Menu item: Specify source – prompt user for which section with transclusion template to deal with, prompt user for the source for the chosen section, and insert source tag as a comment into or before the template. Have menu item show up only if one or more of the requisite templates are on the page (if possible).
• Menu item: Update queques – when clicked, script will check the source tag for each section and replace the existing list with the current list from the indicated source (page or page section). If no source tag exists for a section, the script will prompt the user for the source for the chosen section, and insert source tag as a comment into or before the template, before continuing.
• Menu item: Update On/Off. When activated, script will initiate Update queues without prompting first.
• Menu item: Create section – present menu of section types available (won't show WikiMedia as choice if there is already a Wikimedia section.
• Portal analysis – (automatic when program first accesses the page. Use time stamp to prevent it from running every time page is accessed? Is there a way to tell when the page is left? Also, prevent from running again on a reload from the page, if possible.) Analyzes portal and reports findings to user. Types of sections present, types of sections missing, how many titles in each transclusion template, etc. To give a bird's eye view.
• Menu item: Add titles – For the sections that use rotating selective transclusion, allow user to add pagenames, even if a source tag is in place (and have those be left in place by the Update queues command). Prompt user for which section with transclusion template to deal with, prompt user for the titles to add. Have menu item show up only if one or more of the requisite templates are on the page (if possible).
• Portal control console – provides fill-in form through which to modify the portal.
• Quote section creation (subchoice of "create section")
• Add quotes – a way to combine user-supplied quotes with program-gathered quotes in portals' quote sections. Use delimiters to surround human editor entries?

Development notes for PortalTool.js

categorymembers

Due to the way they are built, categories would be a near ideal place from where to automatically collect page names for Selected article sections.

Unfortunately, Lua can't access the members of a category.

Therefore, we need to find another way to do this.

The api command to list the pages in a category is list=categorymembers. See https://en.wikipedia.org/w/api.php?action=help&recursivesubmodules=1&modules=main#query+categorymembers .

There is a user script that does something very similar to what we are after. User:Ais523/catwatch adds new category members to one's watchlist. It seems very likely that Catwatch could be adapted to do this for Selected article sections.

One thing it does is store category names in a js page, formatted in JavaScript as part of the program. I'd like to store the category names in the Selected article sections themselves, and not formatted as JavaScript.

And, of course, rather than add new category members to one's watchlist, we want to add them to the Selected article section as parameters to the transclusion template there.

categorylinks table

It turns out that Catwatch.js accesses the categorylinks table for the data it needs (see discussion below). The author said that it was obsolete, as "MediaWiki has its own category-watchlisting functionality", and you could "use the watchlist table instead".

Rough rough talk-through

Further information: Pseudocode

Discussions

This is where the actual talk page starts for PortalTool.js. Please post your discussion threads below...

What does ListPages.js do?

The title makes me wonder if this might have something we could use, but I can't figure out how to get it to work, or even what it does: User:Dr Brains/ListPages.js.

The doc is no longer around, and didn't leave any instructions behind.

What does it do, and how does it work?    — The Transhumanist   23:45, 4 July 2018 (UTC)

Looks like it was intended to get all articles in a category, including all subcategories, and then find each article's interwiki target on the French Wikipedia, and produce a table of articles and their French equivalent. But it is using outdated/deprecated techniques, which would be why you can't get it to work. - Evad37 [talk] 05:31, 6 July 2018 (UTC)

Thank you. I'll cross this one off my list.    — The Transhumanist   07:58, 8 July 2018 (UTC)

Adapting catwatch.js

Dear Ais523,

I'm The Transhumanist, one of the acting curators over at the Portals WikiProject.

We've run into a brick wall, and we need your help.

After years of being dormant, the Portals WikiProject was jump started this past April, has grown to almost 100 members, and is in the process of revamping the entire portal system. We are going all out, and have a very active and productive development team. You can see the progress we've made so far in the project's newsletter archive. The excitement level is high.

While many portals will continue to be maintained by hand, we are attempting to build a portal model that is entirely automated, as most of the portals do not have active maintainers. There are currently about 1500 portals, of which about 10% are actively editor-maintained.

So far, we've managed to automate 3 standard portal sections, and have gotten 4 others to semi-automated status. In other words, 3 sections are now virtually maintenance-free. It would be nice if the other ones can be too.

Which brings us to the brick wall...

We have been trying to use Lua (modules) to automate certain portal sections with a method known as selective transclusion. Here is an example of a "Selected item" section using the {{Transclude random excerpt}} template:

Selected amphibian type

Spotted salamander, Ambystoma maculatum

Salamanders are a group of amphibians typically characterized by their lizard-like appearance, with slender bodies, blunt snouts, short limbs projecting at right angles to the body, and the presence of a tail in both larvae and adults. All ten extant salamander families are grouped together under the order Urodela from the group Caudata. Salamander diversity is highest in eastern North America, especially in the Appalachian Mountains; most species are found in the Holarctic realm, with some species present in the Neotropical realm.

Salamanders never have more than four toes on their front legs and five on their rear legs, but some species have fewer digits and others lack hind limbs. Their permeable skin usually makes them reliant on habitats in or near water or other cool, damp places. Some salamander species are fully aquatic throughout their lives, some take to the water intermittently, and others are entirely terrestrial as adults.

This group of amphibians is capable of regenerating lost limbs as well as other damaged parts of their bodies. Researchers hope to reverse engineer the regenerative processes for potential human medical applications, such as brain and spinal cord injury treatment or preventing harmful scarring during heart surgery recovery. The remarkable ability of salamanders to regenerate is not just limited to limbs but extends to vital organs such as the heart, jaw, and parts of the spinal cord, showing their uniqueness compared to different types of vertebrates. ⁤⁤This ability is most remarkable for occurring without any type of scarring. ⁤⁤This has made salamanders an invaluable model organism in scientific research aimed at understanding and achieving regenerative processes for medical advancements in human and animal biology. (Full article...)

It displays an excerpt randomly, taken from one of the pages from its internal list, each time the portal page is purged.

The above template requires that the user supply the names of the pages to be transcluded. That makes this only a semi-automated solution. The sections using this rotate material, and therefore don't go stale as fast as a section with a single excerpt, but, adding new article titles by hand is how they are updated, and unfortunately, that method is not scalable.

We've been banging our heads against the brick wall to find a way to pull the names from a category with Lua, so that portals using these templates will be auto-updating (as the category grows, so would the selection in the portal accessing that category), but the generated nature of categories and the limitations of Lua prevent that.

Which means we must find a way other than Lua to generate a list of the pages from a category. Since JavaScript can make a wide range of api calls, that looks like the way we will have to go. (See https://en.wikipedia.org/w/api.php?action=help&recursivesubmodules=1&modules=main#query+categorymembers`). (That's how I found catwatch and you: via the search string "categorymembers").

It seems very likely that Catwatch.js could be adapted to do this for Selected article sections.

One thing catwatch does is store the category names in a js page, formatted in JavaScript to be run as part of the program. Instead, I'd like to store the category names in the Selected article sections themselves, such as in a hidden comment.

And, rather than add new category members to one's watchlist, I want to add them to the Selected article section as parameters to the transclusion template there, just like

| Frog
| Toad
| Salamander
| Caecilian 

are included in the example section provided above.

Any assistance or guidance you could provide would be most helpful.

I look forward to your reply. Sincerely,    — The Transhumanist   21:51, 5 July 2018 (UTC)

P.S.: I've pinged @TheDJ and Evad37: TheDJ, because the script mentioned he worked on it, and Evad, as he has been working on this problem and also has familiarity with JS. -TT

Replied on User talk:The Transhumanist. --ais523 05:38, 6 July 2018 (UTC)

One of the issues with user scripts is that, by definition, they only apply to a single user. So you can't use a user script to do something like extract category entries to be displayed on a page for anonymous users; you could do it so that you saw excerpts yourself, but that would defeat the likely point of the project (to update the portal pages so that everyone sees them).

There's another issue with using JavaScript for this, too; being a client-side technology (i.e. it runs in the end user's browser), it runs entirely after all the server-side technology (i.e. things that run on Wikimedia's servers), meaning that you couldn't use JavaScript "live" to produce information that was passed to a Lua module. This applies to all uses of JavaScript (there are two main uses of it in Wikipedia, user scripts and MediaWiki:common.js, although I suspect that you'd have problems convincing people to change the latter for this purpose as not all users will run it; many users have JavaScript disabled by default, and there are possible performance issues too).

It seems like the correct solution to the problem, therefore, is not to try to work out the list of category entries when a user views a page (which is what the attempted Lua solution would do, and what you seem to have been assuming the JavaScript solution would do). Rather, what you want to do is automatically update lists of category entries somewhere that a Lua module can see them. That sounds like a job for a bot, rather than a script. I don't have much experience with fully automated bots from the point of view of writing them (I ran a bot in the past but it was basically an account with a very specific set of user scripts); you'd probably want to use an existing bot framework for something like this, and it would probably be worth contacting someone with more experience in those.

One other possibility would be to ask the developers to make the task you're aiming for easier. The category membership table is fairly quick to query behind the scenes (and in fact, Special:RandomInCategory exists but I haven't found any way to make its random selection feature accessible from Lua or wikitext). The best/simplest solution to this problem would be for MediaWiki to implement a magic word or ParserFunction which returns a randomly selected page from a category, and given that it would obviously improve the wiki (and use much less resources than a bot or script would!), it seems reasonable that such a request might be accepted. WP:VPT might be a good place to talk about this. --ais523 05:37, 6 July 2018 (UTC)

Thank you for your input. Some very good ideas there.

My understanding of going through Phabricator is that it is hit and miss and a lot of waiting and wondering. So, while I won't rule out that as an option, I'd like to pursue this avenue of development as well.

I wasn't thinking about a viewing script doing its thing (pulling category members) in real time for everyone. What I would like to do is write a program to edit the portal pages, updating them for everyone's benefit. I'm sorry my explanation wasn't clearer.

You are right that I want to update lists of category entries somewhere that a Lua module can see them. That is exactly what I was talking about. The place I want those lists is as template parameters right in the portal's wikicode. The templates use Lua, which in turn use the parameters as input. Here is an example with a list already inserted — read the embedded comment for clarification:

{{Box-header|title=Selected frog article|EDIT=yes|noedit=yes}}
{{Transclude random excerpt| paragraphs=1-3 | files=1 | more=
 | <!-- The script I want would list the category members below like this: -->
 | Archaeobatrachia
 | Mesobatrachia
 | Neobatrachia
 | Alsodidae
 | Alytidae
 | Amietia
 }}
{{Box-footer}}

The actual section that this example was taken from is at Portal:Amphibians, and includes over fifty frog articles. There is also a toad section, a salamander section, and a caecilian section, each with their own list. Now you can see why we don't want to have to insert the article names manually.

I don't mind if the program is a bot, written in JavaScript, or a userscript providing a menu item. It would be easier to develop it as the latter first, to allow for more interactive development and testing (with multiple users) before making a fully automated (bot) version.

You mentioned "work out the list of category entries". That's what I need to know how to do. I can handle menu items (see SearchSuite.js), and editing wikicode (see RedlinksRemover.js). It's what goes in between (fetching category members and putting them into a variable) that I'm having trouble with.

Therefore, I have some questions for you...

How does catwatch fetch the members of a category?

How does catwatch process those?

How does catwatch insert the result into a page? (I understand that this is not an edit page, but I don't recognize the methods being used, and I would like to grok this program.)

Any enlightenment you could provide concerning these mysteries would be very helpful.

I look forward to your replies.    — The Transhumanist   08:49, 6 July 2018 (UTC)

Oh, in that case, I have some bad news for you: catwatch doesn't fetch the members of a category. Doing so would be much too inefficient to do on every page load if the category were large. (There are some people who catwatch Category:Living people!) It's fetching only the most recently recategorised member of the category, via an API search of the category links table. (Incidentally, now that MediaWiki has its own category-watchlisting functionality, it would be possible, and probably more efficient, to use the watchlist table instead. catwatch is mostly redundant at this point.) As such, no processing is actually needed; we ask MediaWiki for the most recently recategorised page in the category, then simply display that to the user.

For your purposes, you'd probably want to record the time at which the script was last run. You could then query the categorylinks table for pages in the category that were recategorised since your last run of the script. Unfortunately, that would only spot insertions, not removals (as a page that was removed from the category will no longer be in the category); note that catwatch currently can't spot removal of a page from a category either. (Perhaps your script would also better work off a watchlist than a category table! Especially if you use a bot account to update the portals, even if the bot's implemented using user scripts, you could put all the categories in question – and nothing else – onto the bot account's watchlist, meaning that a watchlist query would be a very efficient way to get a list of exactly what you needed to update and on which pages.)

Inserting the result into a page is done by editing the page's HTML once it loads. You can use a very similar technique to edit the edit box of a page; simply find the edit box via its ID, then edit its value. Presumably you'd want some comments in the page specifying which bit gets updated by the bot; then it could split the string into three pieces using the comments in question and update just the second piece. --ais523 09:05, 6 July 2018 (UTC)