archimedes-jobs

Archimedes

Archimedes is an abstract distributed job engine on AWS.

SDK use

Import SDK

const Archimedes = require("archimedes-jobs")("<NAME_OF_LAMBDA_FUNCTION>");

import archimedes from "archimedes-jobs";
const Archimedes = archimedes("<NAME_OF_LAMBDA_FUNCTION>");

Create job

Archimedes.create({
    jobType: "<JOB_TYPE>",
    inputData: {
        //<all json-like input data the job needs to know>
    },
    inputFiles: {
        //<all file-like input data like urls to s3 the job needs to know>
    },
    estimations: {
        //<all job meta metrics the system needs to know in order to triage job execution>
    }
}).then(job => {
    console.log(job);
}).catch(err => {
    console.log(err);
});

Run job

Archimedes.signal({
    jobId: job.id,
    state: "DISPATCH"
}).then().catch();    

Handle job

Payload for job execution will be:

{
  "id": "<JOB_ID>"
}

Then use Archimedes to get all job data including inputData and inputFiles:

let job = await Archimedes.get({
    jobId: "<JOB_ID>"
});

Then signal that we are executing this:

job = Archimedes.signal({
    jobId: job.id,
    state: "EXECUTE"
})

This will in particular now add a currentExecution object to the job object:

    let jobExecution = job.currentExecution; /* {
        executionId: 'XXX',
        createdAt: 'YYY',
        updatedAt: 'ZZZ',
        metrics: {},
        progress: 0,
        state: 'EXECUTE',
        failureReason: '',
        finishedAt: null
    } */

While you're executing you might want to periodically upgrade metrics (e.g. max cpu load, max memory usage etc.) as well as progress. This can be of use two-fold:

• your own application might query this data to show job progress to the user
• Archimedes is scanning jobs in the background for non-progression to make sure to kill and reschedule stuck jobs

To update progress and/or metrics:

    Archimedes.updateExecution({
        jobId: "XXX",
        executionId: jobExecution.executionId,
        progress: 0.5,
        metrics: {
            cpu: 1234,
            memory: 5678
        }
    })

Once you're done with the job, you can store your outputData and outputFiles like so:

    Archimedes.update({
        jobId: "XXX",
        outputData: {
            ...
        },
        outputFiles: {
            ...
        }
    })

And you transition the job to success or failure like so:

    Archimedes.signalExecution({
        jobId: "XXX",
        executionId: jobExecution.executionId,
        state: "SUCCESS" // "FAILURE"
     // failureReason: "XXXX" 
    })