@openfn/cli

@openfn/cli

This package contains a new devtools CLI for running and deploying OpenFn jobs.

The CLI includes:

• A secure runtime for executing OpenFn jobs and workflows
• A compiler for making OpenFn jobs runnable
• Configurable logging output
• Auto-installation of language adaptors
• Support for the adaptors monorepo
• Deployment of workflows to OpenFn (and Lightning)

Getting Started

• Installation
• Updating
• Migrating from devtools
• Basic Usage
• Advanced Usage
• Deploying Workflows
• Logging
  • Structured/JSON logging
• Workflows
• Compilation
• Contributing
  • Usage from this repo
  • Installing globally
  • Repo Directory
  • Contribution changes

Installation

To install:

npm install -g @openfn/cli

Make sure everything works by running the built-in test job:

openfn test

Check the version:

openfn -v

Get help:

openfn help

Updating

You should be able to install a new version straight on top of your current installation:

npm install -g @openfn/cli

If this fails, try uninstalling the current version first:

npm uninstall -g @openfn/cli

And then re-installing.

Migrating from devtools

If you're coming to the CLI from the old openfn devtools, here are a couple of key points to be aware of:

• The CLI has a shorter, sleeker syntax, so your command should be much shorter
• The CLI will automatically install adaptors for you (with full version control)

Basic Usage

You're probably here to run jobs (expressions) or workflows, which the CLI makes easy:

openfn path/to/workflow.json
openfn path/to/job.js -ia adaptor-name

If running a single job, you MUST specify which adaptor to use.

Pass the -i flag to auto-install any required adaptors (it's safe to do this redundantly, although the run will be a little slower).

When the finished, the CLI will write the resulting state to disk. By default the CLI will create an output.json next to the job file. You can pass a path to output by passing -o path/to/output.json and state by adding -s path/to/state.json. You can use -S and -O to pass state through stdin and return the output through stdout.

The CLI maintains a repo for auto-installed adaptors. Run openfn repo list to see where the repo is, and what's in it. Set the OPENFN_REPO_DIR env var to specify the repo folder. When autoinstalling, the CLI will check to see if a matching version is found in the repo. openfn repo clean will remove all adaptors from the repo. The repo also includes any documentation and metadata built with the CLI.

You can specify adaptors with a shorthand (http) or use the full package name (@openfn/language-http). You can add a specific version like http@2.0.0. You can pass a path to a locally installed adaptor like http=/repo/openfn/adaptors/my-http-build.

If you have the adaptors monorepo set up on your machine, you can also run adaptors straight from the local build. Pass the -m <path> flag to load from the monorepo. You can also set the monorepo location by setting the OPENFN_ADAPTORS_REPO env var to a valid path. After that just include -m to load from the monorepo. Remember that adaptors will be loaded from the BUILT package in dist, so remember to build an adaptor before running!

You can pass --log info to get more feedback about what's happening, or --log debug for more details than you could ever use.

Advanced Usage

The CLI has a number of commands (the first argument after openfn)

• execute - run a job
• compile - compile a job to a .js file
• docs - show documentation for an adaptor function
• repo - manage the repo of installed modules
• docgen - generate JSON documentation for an adaptor based on its typescript

If no command is specified, execute will run.

To get more information about a command, including usage examples, run openfn <command> help, ie, openfn compile help.

Deploying Workflows

⚠️ This feature is still in active development. Expect breaking changes.

The CLI can deploy workflows to OpenFn.org and instances of Lightning.

In order to deploy a workflow, you need the follow:

• A project file written in YAML
• A config file (or env vars) with your OpenFn credentials

Example project file:

---
name: my-new-project
workflows:
  workflow-one:
    name: My New Workflow
    jobs:
      job-a:
        name: My First Job
        enabled: true # default
        adaptor: @openfn/language-http@latest
        body: |
          alterState(state => {
            console.log("Hello world!");
            return state;
          });
      job-b:
        name: My Second Job
        adaptor: @openfn/language-common@latest
        body: |
          alterState(state => {
            console.log("Hello world!");
            return state;
          });
    triggers:
      trigger-one:
        type: webhook # default
    edges:
      webhook->job-a:
        source_trigger: trigger-one
        target_job: job-a
      job-a->job-b:
        source_job: job-a
        target_job: job-b

Example config file:

{
  // Required, can be overridden or set with `OPENFN_API_KEY` env var
  "apiKey": "***",

  // Optional: can be set using the -p, defaults to project.yaml
  "specPath": "project.yaml",

  // Optional: can be set using -s, defaults to .state.json
  "statePath": ".state.json",

  // Optional: defaults to OpenFn.org's API, can be overridden or set with
  // `OPENFN_ENDPOINT` env var
  "endpoint": "https://app.openfn.org"
}

Environment Variables

You can also set the following environment variables to avoid using a config file:

• OPENFN_API_KEY - your OpenFn/Lightning API key
• OPENFN_ENDPOINT - the endpoint to deploy to (defaults to OpenFn.org)

Using the CLI

OPENFN_API_KEY="***" \
openfn deploy

# [CLI] ♦ Changes:
#  {
# + ... diff
# - ... diff
#  }
#
# ? Deploy? yes
# [CLI] ♦ Deployed.

Flags and Options

• -p, --project-path <path> - path to the project file (defaults to project.yaml)
• -s, --state-path <path> - path to the state file (defaults to .state.json)
• -c, --config, --config-path - path to the config file (defaults to .config.json)
• --no-confirm - skip the confirmation prompt

Logging

The CLI is actually a collection of packages, each of which will log with slightly different rules. To help understand where logs are coming from, each package prints a namespace or prefix at the start of its log.

• [CLI] - the CLI itself, responsible for parsing and validating user input, reading and writing to disk, and executing the correct functionality.
• [CMP] - the Compiler will parse openfn jobs into executable Javascript, changing your code
• [R/T] - the Runtime executes your job code in a secure sandboxed environment, one operation at a time
• [JOB] - the actual job code that your wrote. Any console.log statements in your job will appear under this namespace.

The CLI will log information at three different levels of verbosity: default, info and debug (none is also supported).

To set the log level, pass --log info into your command. You can configure this for individual packages, ie --log cmp=debug will run the compiler with debug logging but leave everything else at default. To control multiple components, use comma-seperated values, ie, --log debug,r/t=none,job=info

Note that, unless explicitly overriden, jobs will always report at debug verbosity (meaning job logging will always be shown).

If something unexpected happens during a command, your first step should be to re-run with info-level logging.

default logging is designed to give high-level feedback about what you absolutely need to know. It will show any errors or warnings, as well as high-level reporting about what the command has actually done.

info level logging is suitable for most developers. It is more verbose than default but still aims to provide high-level information about a command. It includes version numbers, key paths, and simple reporting about how the compiler changes your code (see below).

debug level logging is highly verbose and aims to tell you everything that's going on under-the hood. This is aimed mostly at CLI/runtime developers and can be very useful for debugging problems.

Structred/JSON logging

By default all logs will be printed as human-readable strings.

For a more structured output, you can emit logs as JSON objects with level, name and message properties:

{ level: 'info', name: 'CLI', message: ['Loaded adaptor'] }

Pass --log-json to the CLI to do this. You can also set the OPENFN_LOG_JSON env var (and use --no-log-json to disable).

Workflows

As of v0.0.35 the CLI supports running workflows as well as jobs.

A workflow is in execution plan for running several jobs in a sequence. It is defined as a JSON structure.

To see an example workflow, run the test command with openfn test.

A workflow has a structure like this (better documentation is coming soon):

{
  "start": "a", // optionally specify the start node (defaults to jobs[0])
  "jobs": [
    {
      "id": "a",
      "expression": "fn((state) => state)", // code or a path
      "adaptor": "@openfn/language-common@1.75", // specifiy the adaptor to use (version optional)
      "data": {}, // optionally pre-populate the data object (this will be overriden by keys in in previous state)
      "configuration": {}, // Use this to pass credentials
      "next": {
        // This object defines which jobs to call next
        // All edges returning true will run
        // If there are no next edges, the workflow will end
        "b": true,
        "c": {
          "condition": "!state.error" // Not that this is an expression, not a function
        }
      }
    }
  ]
}

Compilation

The CLI will attempt to compile your job code into normalized Javascript. It will do a number of things to make your code robust and portable:

• The language adaptor will be imported into the file
• The adaptor's execute function will be exported form the file
• All top level operations will be added to an array
• That array will be made the default export of the file

The result of this is a lightweight, modern JS source file. It can be executed in any runtime environment: just execute each function in the exported array.

The CLI uses openfn's own runtime to execute jobs in a safe environment.

All jobs which work against @openfn/core will work in the new CLI and runtime environment (note: although this is a work in progress and we are actively looking for help to test this!).

If you want to see how the compiler is changing your job, run openfn compile path/to/job -a <adaptor> to return the compiled code to stdout. Add -o path/to/output.js to save the result to disk.

Contributing

First of all, thanks for helping! You're contributing to a digital public good that will always be free and open source and aimed at serving innovative NGOs, governments, and social impact organizations the world over! You rock. heart

To get this started, you'll want to clone this repo.

You also need to install pnpm.

Usage from this repo

You can run the cli straight from source with pnpm

$ pnpm openfn path/to/job.js
$ pnpm openfn -h

See test/execute.test.ts for more usage examples

Installing globally

To install the CLI globally from the build in repo:

$ npm install -g .

Note that this will install the built source from dist

Repo Directory

The CLI will save and load adaptors from an arbitrary folder on your system.

You should set the OPENFN_REPO_DIR env var to something sensible.

In ~/.bashrc (or whatever you use), add:

export OPENFN_REPO_DIR=~/repo/openfn/cli-repo

To run adaptors straight from the adaptors monorepo:

export OPENFN_ADAPTORS_REPO=~/repo/openfn/adaptors

Contributing changes

Open a PR at https://github.com/openfn/kit. Include a changeset and a description of your change.

See the root readme for more details about changests,