render-client

Render client

The render client is our successor to Chromeshot. Its main purpose is to render PNG screenshots of charts and also export SVG and PDF versions.

⏩ Click here for a reference of all the available tasks and their parameters.

With Chromeshot the instances tried to form a P2P network and continously enter an election process about who can do the next job. Here we're going to switch to a simpler client-server model, in which the render clients (RC) do the work that the render server (RS) tells them to do, basically.

See also:

• render-server
• datawrapper-orm/jobs
• datawrapper-crons

Render client lifecycle

The lifecycle of a render client (RC) looks like this:

Startup sequence

• RC starts
• RC connects to render server (RS)

Entering IDLE phase

• RC tells RS that it is IDLE
• RC waits for RS to send a new job id

RC receives a job id from RS

• RC pulls full job details (including task list) from database
• RC sets job status to processing
• RC performs the tasks in order
• RC sets job status to done (or failed)
• --> entering IDLE phase

Jobs

A job is a ordered list of tasks. Jobs are usually related to one chart and are created/initiated by one user. Jobs are stored in the export_job database table.

Jobs can be created directly by a user through the JSON API (e.g. by calling the endpoint for exporting a chart as PDF) or through clicking the "Export as ..." buttons in Datawrapper. However, jobs can also be created indirectly by a background process (e.g. to take a screenshot after a user edited a chart).

Jobs have a status that describes where they are in their lifecycle. A job begins its life being queued, waiting for a client to start to process it. When this is done the status turns to in progress. When the processing is complete the status is either done or failed

Jobs have different priorities and are executed from highest to lowest priority. Jobs directly triggered by users have higher priority than jobs scheduled automatically.

Tasks

Tasks are the things the render client knows how to do. A set of tasks describes a job. Example tasks are:

• Create a screenshot of a URL and save as temporary file
• Permanently store a temporary file to S3
• Invalidate a set of URLs on Cloudflare or Cloudfront
• Add 20px white border around a temporary image
• Copy a temporary file to a permanent file

Each task is further specified by a set of parameters. The screenshot task accepts as parameters the URL to screenshot, the screenshot size, the scale factor, etc.

Reference

Here's a complete list of all the available tasks:

border

Add a colored border around images.

// Parameters
{
    "image": "input.png",   // input file name
    "color": "#cc2200",     // border color
    "padding": 20,          // px border width
    "out": "output.png"     // output file name (can be same as input)
}

cloudflare

Invalidate a list of URLs on CloudFlare. The idea is to allow immediante invalidation of urls after an S3 upload, so that users don't see the old version anymore from CF cache.

// Parameters
{
    "urls": ["https://img.datawrapper.de/12345/plain.png"]
}

Note that you need to include the following section in the config.js for this to work:

// in config.js
{
    cloudflare: {
        zone_id: '',
        auth_email: '',
        auth_key: '',
    }
}

cloudfront

Invalidate a list of URLs on CloudFront. The idea is to allow immediante invalidation of urls after an S3 upload, so that users don't see the old version anymore from CF cache.

// Parameters
{
    "urls": ["https://img.datawrapper.de/12345/plain.png"]
}

file

Copies a temporary internal file to a permanent file. This is intended to be used in local development setups where you want screenshots to end up on your local disk rather than a S3 bucket

// parameters
{
    "file": "input.pdf",                    // the internal file name
    "out": "/path/to/my/files/output.pdf"   // the permanent file
}

pdf

Creates a PDF export from a chart and stores it to a temporary file. Note that the chart_id is not part of the parameters because it's already an attribute of the task itself.

// parameters
{
    "width": 7,         // width, in "unit" (default px)
    "height": 5.6,      // height, in "unit" (default px)
    "unit": "cm",       // unit, can be cm,pt or px (default px)
    "scale": 1.2,       // scale factor
    "mode": "cmyk",      // the PDF color mode (cmyk|rgb)
    "plain": true,      // if plain is true, the header and footer are excluded
    "out": "export.pdf" // the temporary file name
}

png

Create one or multiple PNG screenshots of a chart. Note that the chart_id is not part of the parameters because it's already an attribute of the task itself.

// parameters
{
    "sizes": [{
        "width": 600,       // chart width
        "height": 500,      // chart height (use "auto" to use a charts fixed height)
        "zoom": 2,          // zoom factor to increase resolution
        "plain": false,     // if plain is true, the header and footer are excluded
        "out": "export-600.png"  // temporary output file name
    }]
}

s3

Copies a temporary file into a permanent path on a S3 bucket.

// parameters
{
    "file": "export.pdf",           // the temporary file name
    "bucket": "my-s3-bucket",       // the bucket, make sure you have access
    "path": "can/be/any/path.pdf",  // the path on the bucket
    "acl": "public-read"            // optional ACL, default 'public-read'
}

You need to make sure you have configures S3 access either using ~/.aws/credentials or by providing the access keys in the config.js:

// in config.js
{
    s3: {
        access_key_id: 'AKIAXXXXXXXXXXXX',
        secret_access_key: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
    }
}

svg

Creates a SVG export from a chart and stores it to a temporary file. Note that the chart_id is not part of the parameters because it's already an attribute of the task itself.

// parameters
{
    "width": 7,         // width, in "unit" (default px)
    "height": 5.6,      // height, in "unit" (default px)
    "plain": true,      // if plain is true, the header and footer are excluded
    "out": "export.pdf" // the temporary file name
}

Installing the render client

prerequisites

sudo apt install graphicsmagick
npm install