@solflare-wallet/utl-aggregator

Unified Token List Aggregator

The Aggregator is a script that generates solana token list JSON based on user specified list of provider sources. By changing the provider source list in the aggregator config one can fine tune the output (explained below), and choose which providers are trusted, and also enables filtering out tokens (for example exclude LP-tokens which could be consumed from other sources). Running this script periodically will ensure that generated UTL is up-to-date. Generated JSON can be hosted on CDN or imported in DB to be exposed through API.

The UTL generated through the aggregation process should be considered as a common source of truth for verified tokens across wallets and dApps.

Installation

npm i @solflare/utl-aggregator

Usage

const fs = require('fs')
const {
    Generator,
    CoinGeckoProvider,
    LegacyTokenProvider,
    ChainId,
} = require('utl-aggregator')

async function init() {
    const generator = new Generator([
        new CoinGeckoProvider(
            null, 
            'https://solana-api.projectserum.com',
            {
                throttle: 200,
                throttleCoinGecko: 65 * 1000,
                batchAccountsInfo: 200,
                batchCoinGecko: 25,
            }
        ),
        new LegacyTokenProvider(
            'https://cdn.jsdelivr.net/gh/solana-labs/token-list@main/src/tokens/solana.tokenlist.json',
            'https://solana-api.projectserum.com', 
            {
                throttle: 1000,
                batchAccountsInfo: 200,
                batchSignatures: 200,
                batchTokenHolders: 1,
            }
        ),
    ])

    const tokenMap = await generator.generateTokenList(ChainId.MAINNET)

    fs.writeFile(
        './solana-tokenlist.json',
        JSON.stringify(tokenMap),
        'utf8',
        function (err) {
            if (err) {
                return console.log(err)
            }

            console.log('The file was saved!')
        }
    )

    console.log('UTL Completed')
}

init()

Token List Providers

Providers are listed in an aggregator. If for example mint/token A is in both CoinGecko and Orca list, only one instance/data will be kept for the final token list, and this is determined based on whether CoinGecko or Orca is positioned higher in the list. If Orca is above CoinGecko, mint A from Orca will be kept, and CoinGecko's mint A will be ignored.

Built-in provider sources will be the Pruned Legacy Token List and CoinGecko. CoinGecko has high barrier of entry for tokens, and is generally excellent when it comes to maintaining token list (since it's their job and business to do so). Legacy token list will be pruned (remove invalid mints, filtering by holders, last activity, LP tokens, scam tokens; this processed was described in Telegram chat) and transformed into the new standardized format.

External Provider sources (Orca, Raydium, Saber, etc..) can host and maintain their own list of verified tokens, that aggregator can use when generating unified token list. Each external provider will have to expose endpoint with a list of tokens they view as verified. This list will be in standardize format (which will include if token is LP-token, etc).

Base external provider repo so any project (Orca, Raydium, Saber..) can host and expose their own verified token list with little developer effort. This allows them to serve as trusted providers for other.

Our Goal

We want to provide every community member a same base source of truth generated by Token List Aggregator and this will provide base verified token list. Anyone can run this without any infrastructure or cost.

Everything after that is only building on top of that, so Token List API is extension, and Token List SDK is extension on top of that. Every step is making things more efficient and optimised.

Everyone can choose what they want to use, host and consume depending on their needs and requirements.