@rsksmart/rif-marketplace-cache

RIF Marketplace Cache

API server that caches different metrics from blockchain across RIF services

Warning: This project is in alpha state. There might (and most probably will) be changes in the future to its API and working. Also, no guarantees can be made about its stability, efficiency, and security at this stage.

Lead Maintainer

Nazar Duchak

See what "Lead Maintainer" means here.

Table of Contents

• Introduction
• Supported services
  • Storage
  • RNS
  • Rates
  • Confirmations
• Configuration
  • Environment variables overview
  • Database
  • Blockchain
  • Logging
• Usage
  • Commands
• Internal architecture
• Contribute
  • Development
• License

Introduction

This is a backend service that caches different metrics from blockchain for the RIF services. It is build using FeathersJS and hence inherits lot of its features and properties.

It supports both REST access and SocketIO WS connectivity. It is recommended to use Feathers Client for consuming services of this project.

For querying the endpoints remember that you have available Querying schema by FeathersJS.

Supported services

Storage

For RIF Storage there is cached information about Pinning Contract and current Storage Offers.

Offers endpoint

GET: /storage/v0/offers

Returns JSON that represents currently active and available Offers that customers can use to contract their pinning requests.

It has the following schema:

[
  {
    "provider": "string", // Hash, serves as ID
    "peerId": "string?", // Optional PeerId of the Provider's node
    "totalCapacity": "number",
    "utilizedCapacity": "number",
    "acceptedCurrencies": "string[]", // ['rbtc', 'rif']
    "availableCapacity": "number",
    "avgBillingPrice": "number", // on a monthly basis
    "totalStakedUSD": "number",
    "createdAt": "Date",
    "updatedAt": "Date",
    "plans": [
      {
        "id": "number",
        "period": "number",
        "amount": "number",
        "offerId": "string",
        "createdAt": "Date",
        "updatedAt": "Date"
      }
    ],
    "agreements": [
      {
        "numberOfPrepaidPeriods": "number",
        "periodsSinceLastPayout": "number",
        "toBePayedOut": "number",
        "hasSufficientFunds": "boolean",
        "agreementReference": "string",
        "dataReference": "string",
        "consumer": "string",
        "size": "number",
        "isActive": "boolean", // False when agreement is stopped
        "tokenAddress": "string", // billing plan token address
        "billingPeriod": "number",
        "billingPrice": "number",
        "availableFunds": "number",
        "lastPayout": "Date",
        "offerId": "string"
      }
    ]
  }
]

Agreements endpoint

GET: /storage/v0/agreements

Returns JSON that represent currently active Agreements.

It has the following schema:

[
   {
     "numberOfPrepaidPeriods": "number",
     "periodsSinceLastPayout": "number",
     "toBePayedOut": "number",
     "hasSufficientFunds": "boolean",
     "agreementReference": "string",
     "dataReference": "string",
     "consumer": "string",
     "size": "number",
     "isActive": "boolean", // False when agreement is stopped
     "tokenAddress": "string", // billing plan token address
     "billingPeriod": "number",
     "billingPrice": "number",
     "availableFunds": "number",
     "lastPayout": "Date",
     "offerId": "string"
   }
]

Stakes endpoint

GET: /storage/v0/stakes'

Returns JSON that represent all stakes for each account.

It has the following schema:

[
   {
     "total": "number",
     "symbol": "string", // token symbol
     "token": "string", // token address
     "account": "string",
   }
]

Average Billing Price endpoint

GET: /storage/v0/avgBillingPrice

Returns min/max average billing price per month converted to USD.

It has the following schema:

{
  min: 'number',
  max: 'number'
}

Available Capacity endpoint

GET: /storage/v0/availableCapacity

Returns min/max available capacity comparing all available offers.

It has the following schema:

{
  min: 'number',
  max: 'number'
}

Rates

API that caches conversion rates currently for RBTC and RIF Token.

GET: /rates/v0/

Returns (example)

[
  {
    "token": "rif",
    "usd": 0.053091,
    "eur": 0.04936646,
    "btc": 0.00000703,
    "ars": 3.52,
    "cny": 0.375896,
    "krw": 65.56,
    "jpy": 5.72,
    "createdAt": "2020-04-24T07:50:06.340Z",
    "updatedAt": "2020-04-24T09:13:31.370Z"
  },
  {
    "token": "rbtc",
    "usd": 7436.63,
    "eur": 6914.89,
    "btc": 0.98511161,
    "ars": 492732,
    "cny": 52653,
    "krw": 9183042,
    "jpy": 800942,
    "createdAt": "2020-04-24T07:50:06.367Z",
    "updatedAt": "2020-04-24T09:13:31.390Z"
  }
]

RNS

API that caches several parts of the RNS and Domain Sail contracts. Supported routes are:

• /rns/v0/offers - List active Domain Offers
• /rns/v0/:ownerAddress/sold - List sold domains for specific :ownerAddress
• /rns/v0/:ownerAddress/domains - List domains that are owned by the :ownerAddress

Confirmations

API for getting information about confirmation status of transactions related to Marketplace.

Confirmed events (eq. confirmations >= targetConfirmaiton) are retained for a configured amount of blocks, in order for the consumer of this API has enough time to detect that confirmation happened. See waitBlockCountBeforeConfirmationRemoved in Config interface

GET: /confirmations

Returns (example)

[
  {
    "event": "ExpirationChanged",
    "transactionHash": "0x78a9f1912e8bf590e30a43f919f405924c168f55dad6efd298e8bc4ccfd4438d",
    "confirmations": 1,
    "targetConfirmation": 2
  },
  ...
]

There are also two events emitted for WebSocket connections.

newConfirmation event

Emitted when new confirmation was available. It is emitted with an object that has same structured as presented above for the GET /confirmations route. confirmations can be higher then targetConfirmation. Once event reaches confirmations >= targetConfirmations once, then it is emitted one last time and from then point on ignored.

{
  "event": "ExpirationChanged",
  "transactionHash": "0x78a9f1912e8bf590e30a43f919f405924c168f55dad6efd298e8bc4ccfd4438d",
  "confirmations": 1,
  "targetConfirmation": 2
}

invalidConfirmation

Emitted when a block with the transaction was dropped from blockchain and hence won't become ever confirmed. It is emitted with the object containing property transactionHash that contains the hash of the transaction dropped out.

{
  "transactionHash": "0x78a9f1912e8bf590e30a43f919f405924c168f55dad6efd298e8bc4ccfd4438d"
}

Configuration

Required reading: node-config docs

There are several ways how to configure this application:

• Using JSON file
• Using Environmental variables
• Using CLI parameters

To run this caching server there is minimum configuration needed, which is supported with all the configuration ways mentioned above:

• Database connection
• Blockchain connection

For general overview of complete configuration options see Config interface that describe configuration object's properties. If you need advanced configuration you can build your own JSON configuration file and load that either using the --config CLI parameter or using environment variable RIFM_CONFIG.

Environment variables overview

• RIFM_DATA_DIR (string/path): directory where all the persistent data will be placed by default
• RIFM_JWT_SECRET (string): Secret using for JTW tokens (not used for auth purpose)
• RIFM_PORT (number): port on which the server should listen to
• RIFM_DB (string): database connection URI
• RIFM_PROVIDER (string): blockchain connection URI
• RIF Communication settings:
  • RIFM_COMMS_STRATEGY (api, libp2p) - Defines the strategy for communication
  • RIFM_COMMS_LISTEN (array) - Defines an array of multiaddress that the Pinner's libp2p node will listen on. Same as libp2p config's address.listen property.
  • RIFM_COMMS_BOOTSTRAP_ENABLED (true/false) - Defines if bootstrap should be used. Same as libp2p config's bootstrap.enabled property.
  • RIFM_COMMS_BOOTSTRAP_LIST (array) - Defines an array of multiaddress that the Pinner's libp2p node will use to bootstrap its connectivity. Same as libp2p config's bootstrap.list property.
• CORS settings (see more on expressjs documentation):
  • RIFM_CORS_ORIGIN (boolean | string | regexp): Configures the Access-Control-Allow-Origin CORS header
  • RIFM_CORS_METHODS (string) Configures the Access-Control-Allow-Methods CORS header
• Storage related:
  • RIFM_STORAGE_CONTRACT_ADDR (string): address of deployed storage contract
  • RIFM_STORAGE_STARTING_BLOCK (number | string): block from which the caching service should process events
• RNS related:
  • Owner contract:
    • RIFM_RNS_OWNER_CONTRACT_ADDR (string): address of deployed storage contract
    • RIFM_RNS_OWNER_STARTING_BLOCK (number | string): block from which the caching service should process events
  • Reverse contract:
    • RIFM_RNS_REVERSE_CONTRACT_ADDR (string): address of deployed storage contract
    • RIFM_RNS_REVERSE_STARTING_BLOCK (number | string): block from which the caching service should process events
  • Placement contract:
    • RIFM_RNS_REVERSE_CONTRACT_ADDR (string): address of deployed storage contract
    • RIFM_RNS_REVERSE_STARTING_BLOCK (number | string): block from which the caching service should process events
• Logging related (see bellow):
  • LOG_LEVEL (string)
  • LOG_FILTER (string)
  • LOG_PATH (string)
  • LOG_NO_COLORS (boolean) - if set the output won't be colorized

Database

Supported DB engine is only SQLite. You can configure the location of the DB file using either CLI flag --db or environment variable RIFM_DB which should be in format sqlite://<path to the file>.

Blockchain

Connection to RSK blockchain is needed, therefore you have to configure provider so the caching server can listen for the events. You can configure this connection either using CLI flag --provideror environment variable RIFM_PROVIDER.

Preferable it should be web-socket enabled connection.

Logging

There is support for extensive logging inside of the application. By default the logs are outputted to stdout.

You can configure logging using configs placed in config/, using CLI parameters or environment variables. Configuration is placed in property log which supports following properties:

• log.level (string; ENV: LOG_LEVEL) - sets minimal logging level that will be output to logs. Default: 'info'
• log.filter (string; ENV: LOG_FILTER) - sets filtering based on components. See bellow for syntax. Default: '*'
• log.path (string; ENV: LOG_PATH) - sets path to log file where logs will be written to. Default: undefined

Filter syntax

Best to explain with examples:

• watcher: log only watcher service entries
• -db: log everything except db's service entries
• watch*: log entries of services that starts with watch
• -watch*: log entries of every service except those which starts with watch
• watcher, db: log entries of only services watcher and db.
• watcher*, -watcher:fs: log every entry of watcher only except of those starting with watcher:fs.

Usage

$ npm install -g @rsksmart/rif-marketplace-cache

// Connection to your database
$ export RIFM_DB=postgres://user:pass@localhost/db

// Database migrations
$ rif-marketplace-cache db-migration --up

// Connection to your blockchain provider
$ export RIFM_PROVIDER=ws://localhost:8545

// Prefetch all the data from the network
$ rif-marketplace-cache precache all

// Start the server
$ rif-marketplace-cache start --port 8000

// Start the server listening for testnet configuration
$ NODE_ENV=rsktestnet rif-marketplace-cache start --port 8000

For some more details on how to deploy this server please see Deployment guide.

Commands

• rif-marketplace-cache db-migration
• rif-marketplace-cache precache [SERVICE]
• rif-marketplace-cache purge [SERVICE]
• rif-marketplace-cache start

rif-marketplace-cache db-migration

synchronize database schema

USAGE
  $ rif-marketplace-cache db-migration

OPTIONS
  -d, --down                           Undo db migrations
  -d, --generate=generate              Generate migrations using template [--generate=migration_name]
  -m, --migration=migration            Migration file
  -t, --to=to                          Migrate to
  -u, --up                             Migrate DB
  --config=config                      path to JSON config file to load
  --db=db                              database connection URI
  --log=error|warn|info|verbose|debug  [default: warn] what level of information to log
  --log-filter=log-filter              what components should be logged (+-, chars allowed)
  --log-path=log-path                  log to file, default is STDOUT

EXAMPLES
  $ rif-marketplace-cache db --up
  $ rif-marketplace-cache db --down
  $ rif-marketplace-cache db --up --to 0-test
  $ rif-marketplace-cache --up --migrations 01-test --migrations 02-test
  $ rif-marketplace-cache --up --db ./test.sqlite --to 09-test
  $ rif-marketplace-cache --down --db ./test.sqlite --to 09-test
  $ rif-pinning db --generate my_first_migration

rif-marketplace-cache precache [SERVICE]

precache past data for a service

USAGE
  $ rif-marketplace-cache precache [SERVICE]

OPTIONS
  --config=config              path to JSON config file to load
  --log=error|warn|info|debug  [default: error] what level of information to log
  --log-filter=log-filter      what components should be logged (+-, chars allowed)
  --log-path=log-path          log to file, default is STDOUT

DESCRIPTION
  Command will fetch data from blockchain and process them prior turning on the API server.
  Currently supported services:
    - all
    - storage
    - rns
    - rates

EXAMPLES
  $ rif-marketplace-cache precache all
  $ rif-marketplace-cache precache storage rns

rif-marketplace-cache purge [SERVICE]

purge cached data

USAGE
  $ rif-marketplace-cache purge [SERVICE]

OPTIONS
  --config=config              path to JSON config file to load
  --log=error|warn|info|debug  [default: error] what level of information to log
  --log-filter=log-filter      what components should be logged (+-, chars allowed)
  --log-path=log-path          log to file, default is STDOUT

DESCRIPTION
  Can purge all data or for specific service.
  Currently supported services:
    - all
    - storage
    - rns
    - rates

EXAMPLES
  $ rif-marketplace-cache purge all
  $ rif-marketplace-cache purge storage rns

rif-marketplace-cache start

start the caching server

USAGE
  $ rif-marketplace-cache start

OPTIONS
  -d, --disable=disable        disable specific service
  -e, --enable=enable          enable specific service
  -p, --port=port              port to attach the server to
  --config=config              path to JSON config file to load
  --db=db                      database connection URI
  --log=error|warn|info|debug  [default: error] what level of information to log
  --log-filter=log-filter      what components should be logged (+-, chars allowed)
  --log-path=log-path          log to file, default is STDOUT
  --provider=provider          blockchain provider connection URI

DESCRIPTION
  Currently supported services:
    - storage
    - rns
    - rates

EXAMPLE
  $ rif-marketplace-cache start --disable service1 --disable service2 --enable service3

Internal architecture

The server mainly consists of three parts:

• HTTP read-only API - it exposes the cached data to external world using FeathersJS and Feathers Sequalize
• Database layer - for storing the cached data using Sequelize and Sequelize-TypeScript
• Blockchain listener - for listening on events on blockchain and caching them.

This caching server has support for caching multiple services. All service related function should be placed in service specific folder in /src, like for example /src/storage.

For implementing new service, it is recommended to have knowledge of the libraries used and study already implemented services. For listening on blockchain events use the listeners in /src/blockchain/events module.

Contribute

There are some ways you can make this module better:

• Consult our open issues and take on one of them
• Help our tests reach 100% coverage!

Development

Some tips for development:

• for complete development environment please see https://github.com/rsksmart/rif-marketplace-dev
• to use the CLI commands from the cloned repo use npm run bin -- <cmd> <...args/flags>

Troubleshooting

You might have some problems during development, here are few pointers about what could be wrong.

#### No events from Blockchain:

• make sure that the ABIs in Cache match contracts deployed on your network (Ganache, Testnet, etc). The same version of contracts has to be everywhere.
• check if confirmations are enabled (when ran in VERBOSE mode, you should see the whole Config logged). If they are, make sure you emit empty blocks to get enough confirmations.
• run Cache with log level DEBUG (you can filter out database logging with log filter -db) and see if events are incoming to Cache
• if updating contract's version, make sure that topics of the contract's Events matches!

CLI issues

• if you created new command, but you are getting message that the command does not exist, check if in root of the project is oclif.manifest.json, if so delete it. (It is cached information about available commands and their help pages)

License

MIT