circuitscan-pipeline-runner

circuitscan-pipeline-runner

Pipeline packages must have this package as a production dependency:

yarn add circuitscan-pipeline-runner

Pipeline default export

The default export async function from the main file specified in package.json on your pipeline package will be invoked.

export default async function(event, { status })

Arguments

event

Includes a payload object key which is the input from the user's CLI.

status

Includes a synchronous log(msg, data) method for sending status updates to the user's CLI and website frontend build output.

status.log('Reticulating Splines...', { splineLen: 123 });

Return value

It should return the pkgName string on success or throw an error on failure. This package name should have a suffix appended by the uniqueName() function.

During its operation, the pipeline package should upload a file at build/<pkgName>/info.json (using uploadLargeFileToS3()) containing the necessary data to display the information about the circuit. Other build artifacts should also go in this directory.

Standard library

Also includes a standard library for interacting with build artifacts and relaying compiler status.

Function Name	Arguments	Note
uniqueName	prefix (string)	Add a suffix that contains an adjective, color, and animal to make the prefix unique
async execPromise	cmd (string) options (object, optional)	Node.js fs.exec promisified
monitorProcessMemory	processName (string) timeout (number) callback (function)	Callback invoked at interval, returns function that can be invoked to stop monitoring
async downloadBinaryFile	url (string) outputPath (string)	Download file from HTTPS using streams
async uploadLargeFileToS3	keyName (string) filePath (string) logger (object, optional)	Upload filePath to the artifact S3 bucket using streams. If specified, logger must have a log(msg) method.
async zipDirectory	sourceDir (string) outPath (string) logger (object, optional)	Create a zip file of a directory. If specified, logger must have a log(msg) method.
mkdirpSync	targetDir (string)	Recreation of shell command mkdir -p
async deleteS3Keys	keys (Array[string])	Delete some files in the BLOB_BUCKET from S3
class MockStatusReporter	none	For testing, logs are available as array on logs property

Barebones Example

import {writeFileSync} from 'node:fs';
import {join} from 'node:path';
import {tmpdir} from 'node:os';
import {
    execPromise,
    monitorProcessMemory,
    uniqueName,
    uploadLargeFileToS3,
} from 'circuitscan-pipeline-runner';

export default async function(event, { status }) {
    // TODO ... input validation ...

    const pkgName = uniqueName(circuitName);
    const dirPkg = join(tmpdir(), pkgName);

    const compilePromise = execPromise(`gcc foo.c`);
    const cancelMemoryMonitor = monitorProcessMemory(
      'gcc',
      10000,
      memoryUsage => {
        status.log(`Compiler memory usage`, { memoryUsage });
      }
    );
    await compilePromise;
    cancelMemoryMonitor();

    writeFileSync(join(dirPkg, 'info.json'), JSON.stringify({
      requestId: event.payload.requestId,
      type: 'mypipeline',
      importantProperty: 123,
      foo: 'bar',
    }, null, 2));
    await uploadLargeFileToS3(`build/${pkgName}/info.json`, join(dirPkg, 'info.json'));

    return pkgName;
}

See Also

• Circuitscan Circom Pipeline repository

License

MIT