superkv

SUPERKV - Supercharge Clouflare KeyValue store (experimental)

Store Cloudflare KV with more control and power.

Features

• Create complex queries within KV :) - KVTable
• Upload KeyValue bigger the 25MB limit - KVBig
• Manage namespaces (create, rename, delete) - KVNamespaceApi
• Interact with CF KV API v4 - KVApi
• Insert / delete KeyValue in bulk - KVBatch
• You can use it outside of workers and manage KV in your own server.
• Trying to be subrequest efficient (always using bulk insert for prefix keys instead of kv.put...) avoid hitting the 50 subrequest to fast.

Why / Limits / Warning

• I made this because I want more control with KV storage.
• If you have ideas or you know how to optimize/fix things please help :)
• KV uses ascending lexicographical order. You have to use ULID to sort descending https://www.npmjs.com/package/ulidx
• I have no prior knowledge in making a tool for data management and this is the best I could come up with.

TODO

• Local KV api emulator

How to use it

For more example check test folder

Simple usage

import { KVNamespaceApi } from 'superkv'
import { ulid } from 'ulid'

interface User {
  username: string
  points: number
  active: boolean
  timestamp: number
}

const userTableDefinition = {
  name: 'users',
  prefix: {
    'username': {
      keyValue: (metadata) => metadata.username
    },
    '>500points': {
      filter: (metadata) => metadata.points > 500,
    },
    'isActive': {
      sortValue: (metadata) => metadata.active ? 'true' : 'false',
    },
    'timestamp_desc': {
      sortValue: (metadata) => ulid(timestamp),
    },
  }
} as KVTableDefinition<User>

const main = async () => {
  const accountId = '', authToken = ''
  const kvNamespaceApi = new KVNamespaceApi({ accountId }, { authToken })
  const namespace = await kvNamespaceApi.createOrGetNamespace('worker_namespace')
  const kvApi = kvNamespaceApi.useKVApi()
  const kvUsers = kvApi.useKVTable(userTableDefinition)

  // Get user by key
  const user = await kvUsers.getMetadata('{key}')

  // Get user by username
  const user = await kvUsers.getMetadata('{username}'. 'username')

  // Get users with lease 500 points
  const prefix = kvUsers.createPrefixKey('>500points')
  const list = await kvUsers.list({ prefix })

  // Get active users
  const prefix = kvUsers.createPrefixKey('isActive', 'true')
  const list = await kvUsers.list({ prefix })

  // Get latest users
  const prefix = kvUsers.createPrefixKey('timestamp_desc')
  const list = await kvUsers.list({ prefix })
}

main()

KVNamespaceApi

Initialize

import { KVNamespaceApi } from 'superkv'
const accountId = '', authToken = ''
const kvNamespaceApi = new KVNamespaceApi({ accountId }, { authToken })

Create a new namespace

const res = await kvNamespaceApi.createNamespace('{title}')
// { success, errors, messages }
const namespace = res.result
// { id, title, supports_url_encoding }

Rename namespace

const res = await kvNamespaceApi.renameNamespace('{namespaceId}', '{title}')

Remove namespace

const res = await kvNamespaceApi.removeNamespace('{namespaceId}')

List namespaces

const res = await kvNamespaceApi.listNamespaces()
const res = await kvNamespaceApi.listNamespaces({ direction: 'desc', order: 'id', page: 0, per_page: 1 })

Create or get namespace

const namespace = await kvNamespaceApi.createOrGetNamespace('{namespaceTitle}')

Reset or get namespace

I usually use this for testing and want to purge data every run

const namespace = await kvNamespaceApi.resetAndGetNamespace('{namespaceTitle}')

KVApi

Initialize

const namespace = await kvNamespaceApi.createOrGetNamespace('{namespaceTitle}')
const kvApi = kvNamespaceApi.useKVApi(namespace.id)
// or
const kvApi = new KVApi({ kvNamespaceApi, namespaceId: namespace.id })

Read keyValue

const res = await kvApi.readKeyValuePair('{key}')
// { success, errors, messages }
const value = res.result

Write keyValue

const res = await kvApi.writeKeyValuePair('{key}', '{value}')
const res = await kvApi.writeKeyValuePair('{key}', '{value}', metadata, { expiration: 1578435000, expiration_ttl: 300 })

Write multiple keyValue

const res = await kvApi.writeMultipleKeyValuePairs([
  { key: '{key}', value: '{value}' }
  { key: '{key}', value: '{value}', metadata: {} },
  { key: '{key}', value: '{value}', metadata: {}, expiration: 1578435000, expiration_ttl: 300 }
])

Delete keyValue

const res = await kvApi.deleteKeyValuePair('{key}')

Delete multiple keyValue

const res = await kvApi.deleteMultipleKeyValuePairs(['{key}', '{key}'])

List keyValues

const res = await kvApi.listKeys({ prefix: '', limit: 10, cursor: '' })
const result = res.result
// { name, expiration, metadata }[]
const result_info = res.result_info
// { count, cursor }

KVTable

Initialize

const tableDefinition = { name, prefix, ... }
const kvUser = kvApi.useKVTable<Metadata>(tableDefinition)
// or
const kvUser = new KVTable<Metadata>({ kvApi, tableDefinition })

Set user

const user = { username: 'fred', emai: '[e-mail]', ... }
await kvUsers.set('{key}', user)

Del user

await kvUsers.del('{key}')

Get user

await kvUsers.del('{key}')

List users

const list = await kvUsers.list()
const result = list.result
// { key: name, expiration, metadata }[]
const cursor = list.cursor

// or list by prefix
const prefix = kvUsers.createPrefixKey('{prefixName}')
const list = await kvUsers.list({ prefix })

Iterator

const prefix = kvUsers.createPrefixKey('{prefixName}')
const it = kvUsers.iterator(prefix)
let result = await it.next()
while (!result.done) {
  const list = result.value
}

Update prefix

await kvUsers.updatePrefix()

Delete prefix

await kvUsers.delPrefix('{prefixName}')

KVBatch

Initialize

const kvBatch = kvApi.useKVBig()
// or
const kvBatch = new KVBig({ kvApi })

Set keyValue in bulk

for (const user of users) {
    await kvBatch.set('{key}', metadata, '{value}')
}
await kvBatch.finish()

Delete keyValue in bulk

for (const user of users) {
    await kvBatch.del('{key}')
}
await kvBatch.finish()

KVBig

Initialize

const kvBig = kvApi.useKVBig()
// or
const kvBig = new KVBig({ kvApi })

Write kv file from readStream

import { open } from 'fs/promises'

const fd = await open('./data.txt', 'r')
await kvBig.set('{key}', fd.createReadStream())

Get KV file to writeStream

import { open } from 'fs/promises'

const fd = await open('./data.txt', 'a')
await kvBig.get('{key}', fd.createWriteStream())

Previous package

https://github.com/g45t345rt/cf-kvprefix

cf-kvprefix is deprecated in a favor of superkv

• Only works in workers
• Does not use any bulk operations
• Can't update or add new prefix
• Is not subrequest efficient
• Can't upload KeyValue bigger than 25MB