@aresrpg/sui-checkpoint-reader

@aresrpg/sui-checkpoint-reader

Process sui checkpoints with ease

This library allows building custom Sui indexers in JavaScript. It reads checkpoint files from both remote and local sources and processes them efficiently.

It also allows building a local leveldb from Sui's formal snapshots, which you can then process to initialize your indexer without the need to read checkpoints history for past epochs

Note: This is a work in progress and might not be suited for production usage.

🚀 Getting Started

First, install the library:

npm install @aresrpg/sui-checkpoint-reader

Here's a basic example of how to use the library:

import { read_checkpoints } from '@aresrpg/sui-checkpoint-reader/reader'

async function get_remote_checkpoint(num) {
  const response = await fetch(`https://checkpoints.testnet.sui.io/${num}.chk`)
  const buffer = await response.arrayBuffer()
  return buffer
}

async function process_checkpoint(checkpoint, index) {
  console.log('[indexer] process_checkpoint:', index)
}

const known_types = {}

await read_checkpoints({
  from: 1, // Start processing from checkpoint 1
  to: Infinity, // Continue processing indefinitely
  get_remote_checkpoint, // Function to fetch checkpoint data
  concurrent_downloads: 25, // Number of concurrent downloads allowed
  known_types, // BCS types you want to parse
  checkpoints_folder: '', // Local folder for checkpoint files (nesting is supported at 1 level)
  cleanup_checkpoints: false, // Clean up processed checkpoint files
  process_checkpoint, // Function to process each checkpoint
})

And here is how to get your leveldb from the formal snapshot

import { download_and_store_objects } from '@aresrpg/sui-checkpoint-reader/snapshot'

// this will start download the formal snapshot of the specified epoch
// and save any objects recognized by your known types into leveldb (key: id, value: object)
await download_and_store_objects({
  network: 'testnet',
  epoch: 440, // the epoch of the snapshot
  known_types, // same types as for checkpoint reading
  save_objects: false, // if you want to also save the .obj files locally under /epoch_X/*.obj
  start_bucket = 1, // optional start files 1_1.obj
  start_part = 1,
  db_folder = './sui-formal-objects', // the leveldb folder
})

// and here you can iterate easily on your leveldb, note that you can use anything else to read those
for await (const [db, object] of read_snapshot_objects('./sui-formal-objects')) {
  // process all objects here, for example use db.get(object.contents.kiosk_id)
  // to save something like { object, object_kiosk } in your storage of choice
  console.dir(object, { depth: Infinity })
}

⚙️ How It Works

The reader supports two modes: remote-only and hybrid (more powerful) mode.

Hybrid Mode (with checkpoints_folder specified)

1. Initial Check: The reader will first check existing checkpoint files and determine what needs to be downloaded to fill the gap between from and the lowest present checkpoint file.
2. Gap Filling: If from is lower than the lowest checkpoint file, missing checkpoints will be downloaded up to the lower checkpoint file.
3. Sequential Processing: The reader will process checkpoints sequentially as they become available.
4. Stopping Remote Reader: Once all checkpoints up to the lowest file (or the to parameter) are downloaded, the remote reader will stop.
5. Local File Watching: Local checkpoint files will be watched and processed as they become available.

Remote-Only Mode (without checkpoints_folder)

In this mode, the reader will rely solely on remote checkpoint files:

• Fetching Checkpoints: Checkpoints are fetched from the specified remote source.
• Sequential Processing: Checkpoints are processed in sequential order as they are downloaded.
• Efficient Downloads: The reader handles concurrent downloads up to the specified limit.

Example Usage for Remote-Only Mode

import { read_checkpoints } from '@aresrpg/sui-checkpoint-reader'

async function get_remote_checkpoint(num) {
  const response = await fetch(`https://checkpoints.testnet.sui.io/${num}.chk`)
  const buffer = await response.arrayBuffer()
  return buffer
}

async function process_checkpoint(checkpoint, index) {
  console.log('[indexer] process_checkpoint:', index)
}

const known_types = {}

await read_checkpoints({
  from: 1,
  get_remote_checkpoint,
  concurrent_downloads: 25,
  known_types,
  process_checkpoint,
})

🧩 BCS Types

To parse specific objects within the checkpoints, you can provide the BCS definitions directly.

Generating BCS Types

You can generate the necessary types using the https://github.com/Sceat/sui-bcs. Here is an example of how to generate them:

npx sui-bcs gen --package <package_address> --file <output_file> --network <network_name>

The network_name defaults to testnet if not provided. Not that you can also generate 0x2 types as needed

Example Usage

Here's an example of how you can define and use known BCS types:

// npx sui-bcs gen --package 0x68e06aee7966648c7b3258f02d5c7da2cf9b035727d6e9167c90e662f20e0406 --file aresrpg-bcs.js --network testnet
import { read_checkpoints } from '@aresrpg/sui-checkpoint-reader'

import aresrpg_bcs from './aresrpg-bcs.js'

const ARESRPG =
  '0x68e06aee7966648c7b3258f02d5c7da2cf9b035727d6e9167c90e662f20e0406'

const known_types = {
  [ARESRPG]: aresrpg_bcs,
}

This configuration allows the checkpoint reader to correctly parse and process the objects within the checkpoints.

Note that 0x1 & 0x2 are built in

📖 API Reference

read_checkpoints(options)

Options:

• from: Start processing from this checkpoint number (default: 1).
• to: Stop processing once this checkpoint number is reached (default: Infinity).
• get_remote_checkpoint: Function to fetch a checkpoint array buffer from a number.
• concurrent_downloads: Number of concurrent downloads allowed while catching up (default: 25).
• known_types: Generated BCS for types you want to parse.
• checkpoints_folder: Local folder where your Sui node is dumping the checkpoint files (default: '').
• cleanup_checkpoints: Option to delete already processed checkpoint files (default: false).
• process_checkpoint: Function to process a checkpoint.

Contributing 🛠️

Contributions are welcome! Please open an issue or submit a pull request with any improvements or bug fixes.

License 📄

This project is licensed under the ISC License.