@api3/airnode-feed

airnode-feed

A service for storing and accessing signed data.

Airnode feed is a Node.js service, dockerized and deployable on any cloud provider or hostable on premise. It is continuously running two core loops:

• Fetch beacon data - Each triggers.signedApiUpdates entry defines a group of templates. Airnode feed makes a template request to the API specified in the OIS to get the template data. Airnode feed's wallet is used to sign the responses and these are then saved to in-memory storage.
• Push signed beacon data to signed API - For each triggers.signedApiUpdates, periodically checks the in-memory storage and pushes the signed data to the configured API.

Local development

The Airnode feed needs a configuration in order to run. The config folder contains example configuration which uses:

• Nodary as the data provider, from which the data is fetched.
• Signed API running on http://localhost:8090 where the data is pushed.

To start the the Airnode feed in dev mode run the following:

• cp config/airnode-feed.example.json config/airnode-feed.json - To copy the Airnode feed configuration from the example. Note, the airnode-feed.json file is ignored by git. If you are using Docker Desktop, you need to change the URL from localhost to host.docker.internal. For example:

  "url": "http://host.docker.internal:8090"
  

• cp config/secrets.example.env config/secrets.env - To copy the secrets.env needed for the configuration. This file is also ignored by git.

• Set the NODARY_API_KEY inside the secrets file. Ask someone from development team for the key.

• cp .env.example .env - To copy the example environment variables. Optionally change the defaults.

• pnpm run dev - To run the Airnode feed. This step assumes already running signed API as specified in the airnode-feed.json configuration.

Testing

To run the tests:

pnpm run test
# or to run test only from a specific files (path substring search)
pnpm run test schema
# or to enable logger (by default the logger is disabled by jest.setup.js).
LOGGER_ENABLED=true pnpm run test

Docker instructions

You can use shorthands from package.json. To understand how the docker image is built, read the Dockerfile.

pnpm run docker:build
pnpm run docker:run

Configuration

Airnode feed can be configured via a combination of environment variables and configuration files.

Environment variables

Logging needs to be initialized prior the configuration files are loaded. This is done via environment variables. All of the environment variables are optional and or set with default values for convenience.

Example:

# Defines a logger suitable for production.
LOGGER_ENABLED=true
LOG_COLORIZE=false
LOG_FORMAT=json
LOG_LEVEL=info

or

# Defines a logger suitable for local development or testing.
LOGGER_ENABLED=true
LOG_COLORIZE=false
LOG_FORMAT=json
LOG_LEVEL=info

LOGGER_ENABLED (optional)

Enables or disables logging. Options:

• true - Enables logging.
• false - Disables logging.

Default: true.

LOG_FORMAT (optional)

The format of the log output. Options:

• json - Specifies JSON log format. This is suitable when running in production and streaming logs to other services.
• pretty - Logs are formatted in a human-friendly "pretty" way. Ideal, when running the service locally and in development.

Default: json.

LOG_COLORIZE (optional)

Enables or disables colors in the log output. Options:

• true - Enables colors in the log output. The output has special color setting characters that are parseable by CLI. Recommended when running locally and in development.
• false - Disables colors in the log output. Recommended for production.

Default: false.

LOG_LEVEL (optional)

Defines the minimum level of logs. Logs with smaller level (severity) will be silenced. Options:

• debug - Enables all logs.
• info - Enables logs with level info, warn and error.
• warn - Enables logs with level warn and error.
• error - Enables logs with level error.

Default: info.

Configuration files

Airnode feed needs two configuration files, airnode-feed.json and secrets.env. All expressions of a form ${SECRET_NAME} are referring to values from secrets and are interpolated inside the airnode-feed.json at runtime. You are advised to put sensitive information inside secrets.

You can also refer to the example configuration.

templates

Configuration for the template requests. Each template request is defined by a templateId and a template object. For example:

// Defines a single template.
"templates": {
  "0xcc35bd1800c06c12856a87311dd95bfcbb3add875844021d59a929d79f3c99bd": {
    "endpointId": "0x3528e42b017a5fbf9d2993a2df04efc3ed474357575065a111b054ddf9de2acc",
    "parameters": [{ "type": "string32", "name": "name", "value": "WTI/USD" }]
  }
}

The template ID hash is derived from the template object. You can derive the ID using ethers library:

ethers.utils.keccak256(ethers.utils.defaultAbiCoder.encode(['string', 'string'], [oisTitle, endpointName]));

templates[<TEMPLATE_ID>]

Configuration for the template object with ID <TEMPLATE_ID>.

endpointId

The ID of the endpoint to which the template request is made.

parameters

The parameters of the template request. Refer to Airnode ABI specification for details.

parameters[n]

Defines one of the parameters of the template request.

type

Refer to Airnode ABI available types.

name

The name of the parameter.

value

The value of the parameter.

endpoints

Configuration for the endpoints. Each endpoint is defined by an endpointId and an endpoint object. For example:

"endpoints": {
  // Defines a single endpoint pointing to the OIS with title "Nodary" and endpoint named "feed".
  "0x3528e42b017a5fbf9d2993a2df04efc3ed474357575065a111b054ddf9de2acc": {
    "endpointName": "feed",
    "oisTitle": "Nodary"
  }
}

The endpoint ID hash is derived from the endpoint object. You can derive the ID using ethers library:

ethers.utils.keccak256(ethers.utils.defaultAbiCoder.encode(['string', 'string'], [oisTitle, endpointName]));

endpoints[<ENDPOINT_ID>]

Configuration for the endpoint object with ID <ENDPOINT_ID>.

endpointName

The name of the endpoint.

oisTitle

The title of the OIS to which the endpoint belongs.

triggers.signedApiUpdates

Configuration for the signed API update triggers. There can be multiple triggers, each specifying a different update configuration.

For example:

"triggers": {
  // Defines a single trigger.
  "signedApiUpdates": [
    {
      // The data is pushed to the signed API named "localhost".
      "signedApiName": "localhost",
      // The data is fetched for the templates with the template IDs specified below.
      "templateIds": [
        "0xcc35bd1800c06c12856a87311dd95bfcbb3add875844021d59a929d79f3c99bd",
        "0x086130c54864b2129f8ac6d8d7ab819fa8181bbe676e35047b1bca4c31d51c66",
        "0x1d65c1f1e127a41cebd2339f823d0290322c63f3044380cbac105db8e522ebb9"
      ],
      // The template data is fetched every 5 seconds.
      "fetchInterval": 5,
      // The data remains in in-memory storage for at least 30 seconds before it can be pushed to the signed API.
      "updateDelay": 30
    }
  ]
}

triggers.signedApiUpdates[n]

Configuration for one of the signed API update triggers. Airnode feed periodically pushes the data to the signed API. The period is 2.5 seconds.

Airnode feed only makes a single template request independently of the number of template IDs specified. This is to reduce the number of data provider calls. This implies that all of the templates in the trigger must use the same endpoint and parameters. You can use OIS processing to remove the parameters before making the request (using pre-processing) and later get the corresponding template value based on the endpoint parameters (using-processing). Refer to the example configuration for details.

signedApiName

The name of the signed API to which the data is pushed.

templateIds

The IDs of the templates for which the data is fetched, signed and pushed.

fetchInterval

The interval in seconds between two consecutive fetches of the template data.

updateDelay

The minimum delay in seconds before the data can be pushed to signed API.

signedApis

Configuration for the signed APIs. Each signed API is defined by a signedApiName and a signedApi object. For example:

// Defines a single signed API that uses AUTH_TOKEN secret as Bearer token when pushing signed data to signed API.
"signedApis": [
  {
    "name": "localhost",
    "url": "http://localhost:8090",
    "authToken": "${AUTH_TOKEN}"
  }
]

signedApis[n]

Configuration for one of the signed APIs.

name

The name of the signed API.

url

The URL of the signed API.

authToken

The authentication token used to authenticate with the signed API. It is recommended to interpolate this value from secrets.

If the signed API does not require authentication, set this value to null.

ois

Configuration for the OISes.

ois[n]

Refer to the OIS documentation.

apiCredentials

Refer to Airnode's API credentials.

nodeSettings

Contains general deployment parameters of the Airnode feed.

nodeVersion

The version of the Airnode feed. The version specified in the config must match the version of the Airnode feed at deployment time.

airnodeWalletMnemonic

Mnemonic for the airnode wallet used to sign the template responses. It is recommended to interpolate this value from secrets. For example:

// The mnemonic is interpolated from the "WALLET_MNEMONIC" secret.
"airnodeWalletMnemonic": "${WALLET_MNEMONIC}"

stage

An identifier of the deployment stage. This is used to distinguish between different deployments of Airnode feed, for example dev, staging or production. The stage value can have 256 characters at maximum and can only include lowercase alphanumeric characters and hyphens.

Deployment

To deploy Airnode feed on AWS you can use the Cloud Formation template created by the API integrations team. The template can be found in the private api-integrations repository here.

To deploy on premise you can use the Docker image by reading the instructions below.

Run Airnode feed with Docker

To run the Airnode feed docker image you need to:

• Mount config folder to /app/config. The folder should contain the airnode-feed.json and secrets.env files.
• Pass the -it --init flags to the docker run command. This is needed to ensure the docker is stopped gracefully. See this for details.
• Specify the --env-file with the path to the .env file containing the ENV configuration.
• Optionally, pass the --rm flag to remove the container after it is stopped.
• If running on Linux, you need to use the --network host to access the host network. This has no effect for Docker Desktop.
• Lastly, if you are using Docker Desktop and you want to access the host machine, you need to change the host URL from localhost to host.docker.internal in the configuration files.

For example:

# Assuming the current folder contains the "config" folder and ".env" file.
docker run -it --init --volume $(pwd)/config:/app/config --env-file .env --rm api3/airnode-feed:latest