mwn

mwn

mwn is a modern MediaWiki bot framework in NodeJS, orginally adapted from mwbot.

Development status: Unstable. Versioning: while mwn is in version 0, changes may be made to the public interface with a change in the minor version number.

Documentation given below is incomplete. There are a number of additional classes such as bot.title, bot.wikitext, bot.page, etc that provide useful functionality but aren't documented. You can learn about these by looking through the source code.

Amongst the major highlights are batchOperation and seriesBatchOperation which allow you run a large number of tasks with control over concurrency and sleep time between tasks. Failing actions can be automatically retried.

Setup

To install, run npm install mwn.

Or obtain the latest development copy:

git clone https://github.com/siddharthvp/mwn.git
cd mwn
npm install		# install dependencies

mwn uses JSON with formatversion 2 by default; formatversion 2 is an improved JSON output format introduced in MediaWiki in 2015.

Node version

mwn is written with Node.js 13 in hand. While everything may still work in older versions of Node, you can consider upgrading to Node.js 13. If your bot is hosted on Toolforge, you can install the latest node.js in your home directory, using:

npm install npm@latest     # update npm first to the latest version
npm install n
export N_PREFIX=~
./node_modules/n/bin/n latest
export PATH=~/bin:$PATH

Check that your .profile or .bashrc file includes the line PATH="$HOME/bin:$PATH", so that the path includes your home directory every time you open the shell.

MediaWiki version

mwn is written for and tested on the latest version of MediaWiki used on WMF wikis.

Set up a bot password or OAuth credentials

mwn supports authentication via both BotPasswords and via OAuth. Use of OAuth is recommended as it does away the need for separate API requests for logging in, and is also a bit more secure.

Bot passwords may be a bit easier to set up. To generate one, go to the wiki's Special:BotPasswords page.

If you're migrating from mwbot, note that:

• edit in mwbot is different from edit in mwn. You want to use save instead.
• If you were using the default formatversion=1 output format, set formatversion: 1 in the config options.

Documentation

Importing mwn:

In JavaScript:

const {mwn} = require('mwn');

Note: Prior to mwn v0.8.0, import was via const mwn = require('mwn');

In TypeScript:

import {mwn} from 'mwn';

Create a new bot instance:

const bot = new mwn({
    apiUrl: 'https://en.wikipedia.org/w/api.php',
    username: 'YourBotUsername',
    password: 'YourBotPassword'
});
await bot.login();

Or to use OAuth:

const bot = new mwn({
    apiUrl: 'https://en.wikipedia.org/w/api.php',
	oauth_consumer_token: "16_DIGIT_ALPHANUMERIC_KEY",
	oauth_consumer_secret: "20_DIGIT_ALPHANUMERIC_KEY",
	oauth_access_token: "16_DIGIT_ALPHANUMERIC_KEY",
	oauth_access_secret: "20_DIGIT_ALPHANUMERIC_KEY"
});
bot.initOAuth(); // does not involve an API call
// Any errors in authentication will surface when the first actual API call is made

A more complete constructor syntax:

const bot = new mwn({
    apiUrl: 'https://en.wikipedia.org/w/api.php',
    username: 'YourBotUsername',
    password: 'YourBotPassword',

    userAgent: 'myCoolToolName 1.0 ([[link to bot user page or tool documentation]])',
    defaultParams: {
        assert: 'user' // API parameter to ensure we're logged in 
    }
});

Set default parameters to be sent to be included in every API request:

bot.setDefaultParams({
	assert: 'bot',
	maxlag: 4 // mwn default is 5
});

Set bot options. The default values for each is specified below:

bot.setOptions({
	silent: false, // suppress messages (except error messages)
	maxlagPause: 5000, // pause for 5000 milliseconds (5 seconds) on maxlag error.
	maxlagMaxRetries: 3, // attempt to retry a request failing due to maxlag upto 3 times
	apiUrl: null // set the API URL, can also be set by a bot.setApiUrl
});

Maxlag: The default maxlag parameter used by mwn is 5 seconds. Requests failing due to maxlag will be automatically retried after pausing for a duration specified by maxlagPause (default 5 seconds). A maximum of maxlagMaxRetries will take place (default 3).

Fetch an CSRF token required for most write operations.

bot.getCsrfToken();

The token, once obtained is stored in the bot state so that it can be reused any number of times.

If an action fails due to an expired or missing token, the action will be automatically retried after fetching a new token.

For convenience, you can log in and get the edit token together as:

bot.loginGetToken();

If your bot doesn't need to log in, you can simply set the API url using:

bot.setApiUrl('https://en.wikipedia.org/w/api.php');

Set your user agent (required for WMF wikis):

bot.setUserAgent('myCoolToolName v1.0 ([[w:en:User:Example]])/mwn');

Edit a page. Edit conflicts are raised as errors.

bot.edit('Page title', rev => {
	// rev.content gives the revision text
	// rev.timestamp gives the revision timestamp

	var text = rev.content.replace(/foo/g, 'bar');

	return {  // return parameters needed for [[mw:API:Edit]]
		text: text,
		summary: 'replacing foo with bar',
		minor: true
	};

});

Save a page with the given content without loading it first. Simpler verion of edit. Does not offer any edit conflict detection.

bot.save('Page title', 'Page content', 'Edit summary');

Create a new page.

bot.create('Page title', 'Page content', 'Edit summary');

Post a new section to a talk page:

bot.newSection('Page title', 'New section header', 'Section content', additionalOptions);

Read the contents of a page:

bot.read('Page title');

Read a page along with metadata:

bot.read('Page title', {
	rvprop: ['content', 'timestamp', 'user', 'comment']
});

Read multiple pages using a single API call:

bot.read(['Page 1', 'Page 2', 'Page 3']).then(pages => {
	// pages[0], pages[1], pages[2]
});

Delete a page:

bot.delete('Page title', 'deletion log summary', additionalOptions);

Restore all deleted versions:

bot.undelete('Page title', 'log summary', additionalOptions);

Move a page along with its subpages:

bot.move('Old page title', 'New page title', 'move summary', {
	movesubpages: true,
	movetalk: true
});

Parse wikitext (see API:Parse for additionalOptions)

bot.parseWikitext('Input wikitext', additonalOptions);

Parse the contents of a given page

bot.parseTitle('Page name', additionalOptions);

Upload a file from your system to the wiki:

bot.upload('File title', '/path/to/file', 'comment', customParams);

Direct calls

request(query)

Directly query the API. See mw:API for options. You can create and test your queries in the API sandbox. Be sure to set formatversion: 2 in the options for format=json!

Example: get all images used on the article Foo

bot.request({
	"action": "query",
	"prop": "images",
	"titles": "Foo"
}).then(data => {
	return data.query.pages[0].images.map(im => im.title);
});

Bulk processing methods

continuedQuery(query, maxCallsLimit)

Send an API query, and continue re-sending it with the continue parameters received in the response, until there are no more results (or till maxCalls limit is reached). The return value is a promise resolved with the array of responses to individual API calls.

bot.continousQuery(apiQueryObject, maxCalls=10)

Example: get a list of all active users on the wiki using continuedQuery (using API:Allusers):

bot.continuedQuery({
	"action": "query",
	"list": "allusers",
	"auactiveusers": 1,
	"aulimit": "max"
}, 40).then(jsons => {
	return jsons.reduce((activeusers, json) => {
		return activeusers.concat(json.query.allusers.map(user => user.name));
	}, []);
});

A simpler way is to use bot.continuedQueryGen which is an asynchronous generator.

var activeusers = [];
for await (let json of bot.continuedQueryGen({
    "action": "query",
    "list": "allusers",
    "auactiveusers": 1,
    "aulimit": "max"
})) {
    let users = json.query.allusers.map(user => user.name);
    activeusers = activeusers.concat(users);
}

Use of continuedQueryGen is recommended for several reasons:

• If there are a large number of calls involved, continuedQuery will fetch the results of all the API calls before it begins to do anything with the results. continuedQueryGen gets the result of each API call and processes them one at a time.
• Since making multiple API calls can take some time. you can add logging statements to know the number of the API calls that have gone through.

massQuery(query, nameOfBatchField, batchSize)

MediaWiki sets a limit of 500 (50 for non-bots) on the number of pages that can be queried in a single API call. To query more than that, the massQuery function can be used, which splits the page list into batches of 500 and sends individual queries and returns a promise resolved with the array of all individual API call responses.

Example: get the protection status of a large number of pages:

bot.massQuery({
	"action": "query",
	"format": "json",
	"prop": "info",
	"titles": ['Page1', 'Page2', ... , 'Page1300'],  // array of page names
	"inprop": "protection"
}) // 2nd parameter is taken as 'titles' by default
.then(jsons => {
	// jsons is the array of individual JSON responses.
});

The 3rd parameter hasApiHighLimit is set true by default. If you get the API error 'toomanyvalues' (or similar), your account doesn't have the required user right, so set the parameter as false.

Any errors in the individual API calls will not cause the entire massQuery to fail, but the data at the array index corresponding to that API call will be error object.

batchOperation(pageList, workerFunction, concurrency)

Perform asynchronous tasks (involving API usage) over a number of pages (or other arbitrary items). batchOperation uses a default concurrency of 5. Customise this according to how expensive the API operation is. Higher concurrency limits could lead to more frequent API errors.

The workerFunction must return a promise.

bot.batchOperation(pageList, (page, idx) => {
	// do something with each page
	// the index of the page in pageList is available as the 2nd argument

	// return a promise in the end
}, 5, 2); // set the concurrency as the third parameter, number of retries as 4th parameter

seriesBatchOperation(pageList, workerFunction, sleepDuration)

Perform asynchronous tasks (involving API usage) over a number of pages one after the other, with a sleep duration between each task (default 5 seconds)

The workerFunction must return a promise.

bot.seriesBatchOperation(pageList, (page, idx) => {
	// do something with each page
	// the index of the page in pageList is available as the 2nd argument

	// return a promise in the end
}, 5000, 2); // set the sleep duration in milliseconds as the third parameter, max number of retries for each action is set as the 4th parameter

Note that seriesBatchOperation with delay=0 is same as batchOperation with concurrency=1.