serum-vial

serum-vial: real-time WS market data API for Serum

Why?

We all know that Serum DEX is awesome, but since it's a new ecosystem, some tooling around it may not be so convenient and productive especially from centralized exchanges APIs users perspective. Serum-vial which is a real-time WebSocket market data API server for Serum DEX hopes to alleviate some of those issues by offering:

• familiar experience for centralized exchanges APIs users

  • WebSocket API with Pub/Sub flow - subscribe to selected channels and markets and receive real-time data as easy to parse JSON messages that can be consumed from any language supporting WebSocket protocol

  • incremental L2 order book updates - instead of decoding Serum market asks and bids accounts for each account change in order to detect order book changes, receive initial L2 snapshot and incremental updates as JSON messages real-time over WebSocket connection

  • tick-by-tick trades - instead of decoding eventQueue account data which can be large (>1MB) and in practice it's hard to consume real-time directly from Solana RPC node due to it's size, receive individual trade messages real-time over WebSocket connection

  • real-time L3 data - receive the most granular updates on individual order level, opens, changes, fills and cancellations for each order Serum DEX handles

• decreased load and bandwidth consumption for Solana RPC nodes hosts - by providing real-time market data API via serum-vial server instead of RPC node directly, hosts can decrease substantially both CPU load and bandwidth requirements as only serum-vial will be direct consumer of RPC API when it comes to market data accounts changes and will efficiently normalize and broadcast small JSON messages to all connected clients

What about placing/cancelling orders endpoints?

Serum-vial provides real-time market data only and does not include endpoints for placing/canceling or tracking own orders as that requires handling private keys which is currently out of scope of this project.

Both serum-rest-server and @project-serum/serum provide such functionality and are recommended alternatives.

Getting started

Run the code snippet below in the browser Dev Tools directly or in Node.js (requires installation of ws lib, see).

// connect to hosted demo server
const ws = new WebSocket('wss://serum-vial.tardis.dev/v1/ws')
// if connecting to serum-vial server running locally
// const ws = new WebSocket('ws://localhost:8000/v1/ws')

ws.onmessage = (message) => {
  console.log(JSON.parse(message.data))
}

ws.onopen = () => {
  // subscribe both to L2 and L3 real-time channels
  const subscribeL2 = {
    op: 'subscribe',
    channel: 'level2',
    markets: ['BTC/USDC']
  }

  const subscribeL3 = {
    op: 'subscribe',
    channel: 'level3',
    markets: ['BTC/USDC']
  }

  ws.send(JSON.stringify(subscribeL2))
  ws.send(JSON.stringify(subscribeL3))
}

Demo

Demo of Serum DEX UI backed by serum-vial WebSocket API for trade and order book data is available at:

serum-dex.tardis.dev

Since by default serum-vial uses confirmed commitment level for getting accounts notification from RPC node, it may sometimes feel slightly lagging when it comes to order book updates vs default DEX UI which uses recent/processed commitment. Trade data is provided faster since by default DEX UI is pooling eventQueue account data on interval due to it's size (> 1MB), and serum-vial uses real-time eventQueue account notification as a source for trade messages which aren't delayed by pooling interval time.

Installation

npx _{(requires Node.js >= 15 and git installed on host machine)}

Installs and starts serum-vial server running on port 8000.

npx serum-vial

If you'd like to switch to different Solana RPC node endpoint, change port or run with debug logs enabled, just add one of the available CLI options.

npx serum-vial --endpoint https://solana-api.projectserum.com --log-level debug --port 8080

Alternatively you can install serum-vial globally.

npm install -g serum-vial
serum-vial

CLI options

name	default	description
port	8000	Port to bind server on
endpoint	https://solana-api.projectserum.com	Solana RPC node endpoint that serum-vial uses as a data source
log-level	info	Log level, available options: debug, info, warn and error
minions-count	1	Minions worker threads count that are responsible for broadcasting normalized WS messages to connected clients
commitment	confirmed	Solana commitment level to use when communicating with RPC node, available options: confirmed and processed
markets-json	@project-serum/serum markets.json file, but only non depreciated markets	path to custom market.json definition file if one wants to run serum-vial for custom markets

Run `npx serum-vial --help` to see all available startup options.

Docker

Pulls and runs latest version of tardisdev/serum-vial Docker Image on port 8000.

docker run -p 8000:8000 -d tardisdev/serum-vial:latest

If you'd like to switch to different Solana RPC node endpoint, change port or run with debug logs enabled, just specify those via one of the available env variables.

docker run -p 8000:8000 -e "SV_LOG_LEVEL=debug" -d tardisdev/serum-vial:latest

ENV Variables

name	default	description
SV_PORT	8000	Port to bind server on
SV_ENDPOINT	https://solana-api.projectserum.com	Solana RPC node endpoint that serum-vial uses as a data source
SV_LOG_LEVEL	info	Log level, available options: debug, info, warn and error
SV_MINIONS_COUNT	1	Minions worker threads count that are responsible for broadcasting normalized WS messages to connected clients
SV_COMMITMENT	confirmed	Solana commitment level to use when communicating with RPC node, available options: confirmed and processed
SV_MARKETS_JSON	@project-serum/serum markets.json file, but only non depreciated markets	path to custom market.json definition file if one wants to run serum-vial for custom markets

SSL/TLS Support

Serum-vial supports SSL/TLS but it's not enabled by default. In order to enable it you need to set CERT_FILE_NAME env var pointing to the certificate file and KEY_FILE_NAME pointing to private key of that certificate.

WebSocket API

WebSocket API provides real-time market data feeds of Serum DEX and uses a bidirectional protocol which encodes all messages as JSON objects.

Every message has a type field that is determining it's data type so it can be handled appropriately.

Each WebSocket client is required to actively send native WebSocket pings to the server with interval less than 30 seconds, otherwise connection may be dropped due to inactivity.

All messages timestamps are returned in ISO 8601 format, for example: "2021-03-23T17:03:03.994Z".

Endpoint URL

ws://localhost:8000/v1/ws

(assuming serum-vial runs locally on default port without SSL enabled)

Subscribing to data feeds

To begin receiving real-time market data feed messages, you must first send a subscribe message to the server indicating which channels and markets to receive.

If you want to unsubscribe from channel and markets, send an unsubscribe message. The structure is equivalent to subscribe messages except op field which should be set to "op": "unsubscribe".

const ws = new WebSocket('ws://localhost:8000/v1/ws')

ws.onopen = () => {
  const subscribeL2 = {
    op: 'subscribe',
    channel: 'level2',
    markets: ['BTC/USDC']
  }

  ws.send(JSON.stringify(subscribeL2))
}

Subscribe/unsubscribe message format

{
  "op": "subscribe" | "unsubscribe",
  "channel": "level3" | "level2" | "level1" | "trades",
  "markets": string[]
}

Sample subscribe message

{
  "op": "subscribe",
  "channel": "level2",
  "markets": ["BTC/USDC"]
}

Subscription confirmation message format

Once a subscribe (or unsubscribe) message is received by the server, it will respond with a subscribed (or unsubscribed) confirmation message or error if received message was invalid.

{
"type": "subscribed" | "unsubscribed",
"channel": "level3" | "level2" | "level1" | "trades",
"markets": string[],
"timestamp": string
}

Sample subscribed confirmation message

{
  "type": "subscribed",
  "channel": "level2",
  "markets": ["BTC/USDC"],
  "timestamp": "2021-03-23T17:06:30.010Z"
}

Error message format

Error message is returned for invalid subscribe/unsubscribe messages - no existing market, invalid channel name etc.

{
  "type": "error",
  "message": "string,
  "timestamp": "string
}

Sample error message

{
  "type": "error",
  "message": "Invalid channel provided: 'levels1'.",
  "timestamp": "2021-03-23T17:13:31.010Z"
}

Available channels & corresponding message types

TODO: Subscribing to channel results in providing messages with various types

• trades

  • recent_trades
  • trade

• level1

  • recent_trades
  • trade
  • quote

• level2

  • l2snapshot
  • l2update
  • recent_trades
  • trade

• level3

  • l3snapshot
  • open
  • fill
  • change
  • done

Available markets

Data messages

recent_trades

TODO: returned from oldest to newest

{
  "type": "recent_trades",
  "market": string,
  "trades": Trade[],
  "timestamp": string
}

Sample recent_trades message

trade

HTTP API

GET /markets

Accepts no params and returns non depreciated Serum markets.

Sample request & response

http://localhost:8000/v1/markets

[
 {
    "symbol": "BTC/USDT",
    "deprecated": false,
    "address": "EXnGBBSamqzd3uxEdRLUiYzjJkTwQyorAaFXdfteuGXe",
    "programId": "EUqojwWA2rd19FZrzeBncJsm38Jm1hEhE3zsmX3bRc2o",
    "tickSize": 0.1,
    "minOrderSize": 0.0001,
    "supportsReferralFees": true,
    "supportsSrmFeeDiscounts": true
  },
  {
    "symbol": "BTC/USDC",
    "deprecated": false,
    "address": "5LgJphS6D5zXwUVPU7eCryDBkyta3AidrJ5vjNU6BcGW",
    "programId": "EUqojwWA2rd19FZrzeBncJsm38Jm1hEhE3zsmX3bRc2o",
    "tickSize": 0.1,
    "minOrderSize": 0.0001,
    "supportsReferralFees": true,
    "supportsSrmFeeDiscounts": true
  }
  ...
]

Architecture