tardis-machine

tardis-machine

Locally installable high level API server & caching layer for tardis.dev API - historical tick-level crypto markets data replay service.

Provides easy to use streaming HTTP and WebSocket endpoints that allow replaying supported historical crypto markets WebSocket data feeds from any past point in time.

Check out tardis-client as well if you're using Node.js.

HTTP endpoint

Accessible via /replay?exchange=<EXCHANGE>&from=<FROM_DATE>&to=<TO_DATE>&filters=<FILTERS>

Allows replaying historical crypto exchange WebSocket data feed via streaming HTTP response (new line delimited JSON). Returns up to 400 000 messages per second (depending on the machine set-up and local cache ratio).

Example requests:

• http://localhost:8000/replay?exchange=bitmex&from=2019-05-01&to=2019-05-02&filters=[{"channel":"trade"}] - returns all trades that happened on BitMEX on 1st of May 2019 in ndjson format.
• http://localhost:8000/replay?exchange=bitmex&from=2019-04-01&to=2019-04-02&filters=[{"channel":"trade","symbols":["XBTUSD","ETHUSD"]},{"channel":"orderBookL2","symbols":["XBTUSD"]}] - returns all trades that happened on BitMEX on 1st of April 2019 for XBTUSD and ETHUSD as well as orderbookL2 snapshot and updates for XBTUSD in ndjson format.

Available query string params params:

name	type	default value	description
exchange	string	-	requested exchange name. Check out allowed echanges
from	string	-	replay period start date (UTC) - eg: 2019-04-05 or 2019-05-05T00:00:00.000Z
to	string	-	replay period end date (UTC) - eg: 2019-04-05 or 2019-05-05T00:00:00.000Z
filters (optional)	Url encoded JSON string with {channel:string, symbols?: string[]}[] structure	undefined	optional filters for requested data feed. Check out allowed channels for each exchange, up to date list of accepted symbols can be found via https://tardis.dev/api/v1/exchanges/<EXCHANGE_NAME> API call

Example response snippet:

{"localTimestamp":"2019-05-01T00:09:42.2760012Z","message":{"table":"orderBookL2","action":"update","data":[{"symbol":"XBTUSD","id":8799473750,"side":"Buy","size":2333935}]}}
{"localTimestamp":"2019-05-01T00:09:42.2932826Z","message":{"table":"orderBookL2","action":"update","data":[{"symbol":"XBTUSD","id":8799474250,"side":"Buy","size":227485}]}}
{"localTimestamp":"2019-05-01T00:09:42.4249304Z","message":{"table":"trade","action":"insert","data":[{"timestamp":"2019-05-01T00:09:42.407Z","symbol":"XBTUSD","side":"Buy","size":1500,"price":5263,"tickDirection":"ZeroPlusTick","trdMatchID":"29d7de7f-27b6-9574-48d1-3ee9874831cc","grossValue":28501500,"homeNotional":0.285015,"foreignNotional":1500}]}}
{"localTimestamp":"2019-05-01T00:09:42.4249403Z","message":{"table":"orderBookL2","action":"update","data":[{"symbol":"XBTUSD","id":8799473700,"side":"Sell","size":454261}]}}
{"localTimestamp":"2019-05-01T00:09:42.4583155Z","message":{"table":"orderBookL2","action":"update","data":[{"symbol":"XBTUSD","id":8799473750,"side":"Buy","size":2333838},{"symbol":"XBTUSD","id":8799473800,"side":"Buy","size":547746}]}}

Response structure (single line):

localTimestamp is a date when message has been received in ISO 8601 format with 100 nano second resolution.

message is and JSON object/array with exactly the same structure as provided by particular exchange.

Check out tests to see how such messages stream can be easily consumed using Node.js streams and async iterators.

WebSocket endpoint

Accessible via /ws-replay/?exchange=<EXCHANGE>&from=<FROM_DATE>&to=<TO_DATE>

Exchanges & various 3rd party data providers WebSocket APIs allows subscribing only real-time data feeds and there is no way to subscribe to and "replay" market from any point in the past. Using tardis-machine that is no longer the case.

Example:

Example below shows how to subscribe to BitMEX historical data feed (from 2019-06-01 to 2019-06-02) to trade:XBTUSD and orderBookL2:XBTUSD channels and replay such stream as if it was a real-time one. Subscribe request messages format as well as received messages format is exactly the same as in native exchanges WebSocket APIs.

In many cases such websocket historical data feeds can be consumed using already available WebSocket clients for various exchanges. Imagine having single 'data pipeline' for real-time trading and backtesting.

const ws = new WebSocket('ws://localhost:8000/ws-replay?exchange=bitmex&from=2019-06-01&to=2019-06-02')

ws.onmessage = message => {
  console.log(message)
}

ws.onopen = () => {
  const subscribePayload = {
    op: 'subscribe',
    args: ['trade:XBTUSD', 'orderBookL2:XBTUSD']
  }

  ws.send(JSON.stringify(subscribePayload))
}

Check out tests for more examples.

Available query string params:

In contrast to HTTP API for WebSocket API filters aren't provided explicitly as those are created based on subscribe messages over WebSocket received in first 5 seconds of the connection.

name	type	default value	description
exchange	string	-	requested exchange name. Currently supported: bitmex, coinbase, deribit, cryptofacilities, bitstamp, okex
from	string	-	replay period start date (UTC) - eg: 2019-04-05 or 2019-05-05T00:00:00.000Z
to	string	-	replay period start date (UTC) - eg: 2019-04-05 or 2019-05-05T00:00:00.000Z

Installation

• npx _{(requires Node.js v12 installed on host machine)}

  That will start tardis-machine server running on port 8000 by default (port can be changes --port)

  npx tardis-machine --api-key=YOUR_API_KEY
  

  If you'd like to test it out on sample data (first full day of each month) simply run the command without --api-key option.

  npx tardis-machine
  

  Run npx tardis-machine --help to see all available options (setting custom cache dir etc.)

• npm _{(requires Node.js v12 installed on host machine)}

  Installs tardis-machine globally.

  npm install -g tardis-machine
  
  tardis-machine --api-key=YOUR_API_KEY
  
  

  If you'd like to test it out on sample data (first full day of each month) simply run the command without --api-key option.

  tardis-machine
  

  Run tardis-machine --help to see all available options (setting custom cache dir etc.)

• Docker

  docker run -v ./host-cache-dir:/.cache -p 8000:8000 -e "TM_API-KEY=YOUR_API_KEY" -d tardisdev/tardis-machine
  

  Command above will pull and run latest version of tardisdev/tardis-machine image. Local proxy will be available on host via 8000 port (eg http://localhost:8000/replay/...). It will also pass YOUR_API_KEY for accessing the data (otherwise only first day of each month is accessible). Finally it will use ./host-cache-dir as persistent volume (bind mount) cache dir (otherwise each docker restart will result in loosing local cache)

  In order to debug issues/see detailed logs one can run:

  docker run -v ./host-cache-dir:/.cache -p 8000:8000 -e "TM_API-KEY=YOUR_API_KEY" -e="DEBUG=tardis*" -d tardisdev/tardis-machine
  

  Setting DEBUG=tardis* wil cause to print various useful debug logs to stdout.

  If using volumes causes issues (Windows?) it's perfectly fine to run docker without it (with the caveat of potentially poor local cache ratio after container restart)

  ### running without persitent local cache
  docker run -p 8000:8000 -e "TM_API-KEY=YOUR_API_KEY" -d tardisdev/tardis-machine
  

  tardisdev/tardis-machine docker image can be used in K8S setup as well (just set up proper env variables and persisten volumes in analogue way).

FAQ

Order book snapshots

Order book snapshots are available at the beginning of the day ( 00:00 UTC). (TODO: that requires more info)

How to debug it if something went wrong?

This lib uses debug package for verbose logging and debugging purposes that can be enabled via DEBUG environment variable set to tardis*.

License

MPL-2.0