@automationcloud/engine

Automation Engine

Engine powers Automation scripts model and execution.

Quick Start Guide

This section describes the easiest way to run a script created in Autopilot using high-level abstraction called LocalRobot.

The code looks roughly as follows:

import { LocalRobot } from '@automationcloud/engine';

// Robot instance is initialized asynchronously
const robot = await Robot.create({
    script: '/path/to/my/script.json',
    chromePath: '/path/to/chromium/executable',
});

// Run the script and remember promise so we can wait till it finishes
const finish = robot.run({
    url: 'https://store-to-scrape.com',
    // other inputs ...
});

// Wait for outputs emitted by script
const [products, deliveryOptions] = await job.waitForOutputs('products', 'deliveryOptions');

// Wait until script finishes successfully
await finish;

Please refer to source docs for more info.

Embedding Engine

@automation/engine is a library which can be used as part of any Node.js application.

The application obviously needs to be architected to accomodate running the scripts. This includes not only the code setup, but also the surrounding environment (e.g. a runtime capable of running Chrome/Chromium, non-readonly filesystem, resource requests, etc).

This section describes embedding conceptually without going into too much details, which will be application-specific. For actual examples it's a good idea to look at the test setup (which mimics a minimal setup for running scripts and their parts) as well as the worker which is an example of a full-scale application.

Container setup

There are two essential components required to run an automation script:

• Engine — an Inversify container which provides runtime services to scripts;
• Script instance read from JSON, connected to engine.

Engine is typically configured as a singleton at application startup time.

Note: there's nothing in the code that forces the Engine being a singleton, so it's totally possible to create multiple engines and run multiple scripts in parallel. However, given the fact that scripts have an absolute control over the environment they run in (e.g. Chrome, File System, running servers on local network ports, etc) this will most likely result in script-level conflicts further down the line.

If the host application uses Inversify, it is advisable to merge the two containers so that the two can transparently access each other's components.

class App {
    container = new Container();

    constructor() {
        const engine = new Engine();
        const sharedContainer = Container.merge([this.container, engine.container]);
        engine.container = sharedContainer;
        this.container = sharedContainer;
        // Now both app and engine have shared containers, so you can bind/rebind stuff in either one
    }
}

Runtime setup

There are a couple of essential script runtime features that need to be configured using the following services:

• FlowService defines how to obtain the inputs requested by a script and where to put the outputs emitted by a script
• ReporterService defines how to deal with screenshots, HTML snapshots and events generated during script execution
• ApiRequest is used to send the requests to Automation Cloud (at very least to fetch the extensions required for script execution)
• Logger is used to log messages and can be configured to use a different logging destination and/or format.

Chrome Setup

Local setup requires Chromium executable to be available, so that the Robot can launch it with appropriate CLI flags (the most notable one is --remote-debugging-port which allows connecting to Chromium via Chrome DevTools Protocol).

Note: It is strongly recommended to avoid using your regular Chrome browser for performing automations. Doing so may cause data loss (Engine automatically cleans up browsing data to avoid polluted state) and may otherwise compromise your browsing data due to using unsafe CLI arguments. Additionally, your automation scripts will assume full control over browser lifecycle, which is simply not convenient.

Use the links from Chrome Image Repo to obtain the latest supported version of Chromium. Or, if you feel adventurous, just download a snapshot you prefer from https://commondatastorage.googleapis.com/chromium-browser-snapshots.

Running Chrome as child process

Chrome/Chromium are usually fine with being run as a child process. ChromeLauncher can be used to do that, but it's advisable to wrap it in a service.

Note: running non-headless Chrome in a container requires a graphical system setup (for example VNC).

Please refer to ChromeService from worker for an example.

Creating a script runner

A typical application will contain some kind of "Script runner" service with async run(script): Promise<void> method.

Here are a few tips for creating a decent script runner:

• Create and use some sort of shared state service where relevant state/context can be stored and shared across components.
  • Store things like executionId or a current script in it, so that things like loggers have an easy access to it.
  • Avoid dependencies in shared state, instead have other components depend on it.
  • Make sure the state is restored and does not leak between consequtive runs.
• Decide on when Chrome should be instantiated; it can be done either during application initialisation or immediately before running the script.
  • In either case make sure that Chrome is consistently available before running the script.
  • Chrome can crash, so if you run multiple scripts in the same runtime make sure to restart it if the engine is unable to connect.
• Use try { prepareAndRunScript } catch (err) { handleError } finally { finalize } pattern. The handleError and finalize should never throw, so make sure to wrap those two in try/catch and log the corresponding errors. This is especially crucial for running multiple scripts in the same runtime.
• Make sure BrowserService.connect is called before running the script.