@keyv/bigmap

@keyv/bigmap

Bigmap for Keyv

Features

• Based on the Map interface and uses the same API.
• Lightweight with no dependencies.
• Scales to past the 17 million key limit of a regular Map.
• Uses a hash djb2Hash for fast key lookups.
• Ability to use your own hash function.
• Built in Typescript and Generics for type safety.
• Used in @cacheable/memory for scalable in-memory caching.
• Maintained regularly with a focus on performance and reliability.

Table of Contents

• Features
• Installation
• Overview
• Basic Usage
• Custom Store Size
• Custom Hash Function
• Iteration
  • For...of Loop
  • forEach
  • Keys, Values, and Entries
• Advanced Features
  • Type Safety with Generics
  • Large-Scale Data
• Using with Keyv
  • createKeyv
  • With Custom Options
  • Type Safety
• Integration with Keyv Ecosystem
• API
  • Constructor
  • Properties
  • Methods
    • set
    • get
    • has
    • delete
    • clear
    • forEach
    • keys
    • values
    • entries
    • Symbol.iterator
    • getStore
    • getStoreMap
    • initStore
• Types
• StoreHashFunction
• defaultHashFunction(key, storeSize)
• Contributing
• License

Installation

npm install --save keyv @keyv/bigmap

Overview

BigMap is a scalable Map implementation that overcomes JavaScript's built-in Map limit of approximately 17 million entries. It uses a distributed hash approach with multiple internal Map instances.

Basic Usage

import { BigMap } from '@keyv/bigmap';

// Create a new BigMap
const bigMap = new BigMap<string, number>();

// Set values
bigMap.set('key1', 100);
bigMap.set('key2', 200);

// Get values
const value = bigMap.get('key1'); // 100

// Check if key exists
bigMap.has('key1'); // true

// Delete a key
bigMap.delete('key1'); // true

// Get size
console.log(bigMap.size); // 1

// Clear all entries
bigMap.clear();

Custom Store Size

By default, BigMap uses 4 internal Map instances. You can configure this:

const bigMap = new BigMap<string, number>({ storeSize: 10 });

Note: Changing the storeSize after initialization will clear all entries.

Custom Hash Function

Provide your own hash function for key distribution:

const customHashFunction = (key: string, storeSize: number) => {
  return key.length % storeSize;
};

const bigMap = new BigMap<string, string>({
  storeHashFunction: customHashFunction
});

Using Hashery for Hash Functions

Hashery is a powerful hashing library that provides multiple hash algorithms. You can use it for better key distribution and it is available as an export:

import { BigMap, Hashery } from '@keyv/bigmap';

const hashery = new Hashery();

// Using Hashery's toNumberSync for deterministic key distribution
const bigMap = new BigMap<string, string>({
  storeHashFunction: (key: string, storeSize: number) => {
    return hashery.toNumberSync(key, { min: 0, max: storeSize - 1 });
  }
});

// You can also use different algorithms
const hasheryFnv1 = new Hashery({ defaultAlgorithmSync: 'fnv1' });

const bigMapWithFnv1 = new BigMap<string, string>({
  storeHashFunction: (key: string, storeSize: number) => {
    return hasheryFnv1.toNumberSync(key, { min: 0, max: storeSize - 1 });
  }
});

Hashery supports multiple synchronous hash algorithms:

• djb2 - Fast hash function (default)
• fnv1 - Excellent distribution for hash tables
• murmer - MurmurHash algorithm
• crc32 - Cyclic Redundancy Check

Iteration

BigMap supports all standard Map iteration methods:

For...of Loop

const bigMap = new BigMap<string, number>();
bigMap.set('a', 1);
bigMap.set('b', 2);

for (const [key, value] of bigMap) {
  console.log(key, value);
}

forEach

bigMap.forEach((value, key) => {
  console.log(key, value);
});

// With custom context
const context = { sum: 0 };
bigMap.forEach(function(value) {
  this.sum += value;
}, context);

Keys, Values, and Entries

// Iterate over keys
for (const key of bigMap.keys()) {
  console.log(key);
}

// Iterate over values
for (const value of bigMap.values()) {
  console.log(value);
}

// Iterate over entries
for (const [key, value] of bigMap.entries()) {
  console.log(key, value);
}

Advanced Features

Type Safety with Generics

interface User {
  id: number;
  name: string;
}

const userMap = new BigMap<string, User>();
userMap.set('user1', { id: 1, name: 'Alice' });

Large-Scale Data

BigMap is designed to handle millions of entries:

const bigMap = new BigMap<string, number>({ storeSize: 16 });

// Add 20+ million entries without hitting Map limits
for (let i = 0; i < 20000000; i++) {
  bigMap.set(`key${i}`, i);
}

console.log(bigMap.size); // 20000000

Using with Keyv

BigMap can be used as a storage adapter for Keyv, providing a scalable in-memory store with TTL support.

createKeyv

The createKeyv function creates a Keyv instance with BigMap as the storage adapter.

Parameters:

• options (optional): BigMap configuration options
  • storeSize (number): Number of internal Map instances. Default: 4
  • storeHashFunction (StoreHashFunction): Custom hash function for key distribution

Returns: Keyv instance with BigMap adapter

Example:

import { createKeyv } from '@keyv/bigmap';

// Basic usage
const keyv = createKeyv();

// Set with TTL (in milliseconds)
await keyv.set('user:123', { name: 'Alice', age: 30 }, 60000); // Expires in 60 seconds

// Get value
const user = await keyv.get('user:123');
console.log(user); // { name: 'Alice', age: 30 }

// Check if key exists
const exists = await keyv.has('user:123');

// Delete key
await keyv.delete('user:123');

// Clear all keys
await keyv.clear();

With Custom Options

import { createKeyv } from '@keyv/bigmap';

// Create with custom store size for better performance with millions of keys
const keyv = createKeyv({ storeSize: 16 });

// With custom hash function
const keyv = createKeyv({
  storeSize: 8,
  storeHashFunction: (key, storeSize) => {
    // Custom distribution logic
    return key.length % storeSize;
  }
});

Type Safety

import { createKeyv } from '@keyv/bigmap';

interface Product {
  id: string;
  name: string;
  price: number;
}

const keyv = createKeyv<string, Product>();

await keyv.set('product:1', {
  id: '1',
  name: 'Laptop',
  price: 999
});

const product = await keyv.get<Product>('product:1');

Integration with Keyv Ecosystem

BigMap works seamlessly with the Keyv ecosystem:

import { createKeyv } from '@keyv/bigmap';

const cache = createKeyv({ storeSize: 16 });

// Use with namespaces
const users = cache.namespace('users');
const products = cache.namespace('products');

await users.set('123', { name: 'Alice' });
await products.set('456', { name: 'Laptop' });

// Iterate over keys
for await (const [key, value] of cache.iterator()) {
  console.log(key, value);
}

API

Constructor

new BigMap<K, V>(options?)

Creates a new BigMap instance.

Parameters:

• options (optional): Configuration options
  • storeSize (number): Number of internal Map instances to use. Default: 4. Must be at least 1.
  • storeHashFunction (StoreHashFunction): Custom hash function for key distribution. Default: defaultHashFunction

Example:

const bigMap = new BigMap<string, number>();
const customBigMap = new BigMap<string, number>({
  storeSize: 10,
  storeHashFunction: (key, storeSize) => key.length % storeSize
});

Properties

Property	Type	Access	Description
size	number	Read-only	Gets the total number of entries in the BigMap.
storeSize	number	Read/Write	Gets or sets the number of internal Map instances. Note: Setting this will clear all entries. Default: 4
storeHashFunction	StoreHashFunction | undefined	Read/Write	Gets or sets the hash function used for key distribution.
store	Array<Map<K, V>>	Read-only	Gets the internal array of Map instances.

Examples:

const bigMap = new BigMap<string, number>();

// size property
bigMap.set('key1', 100);
console.log(bigMap.size); // 1

// storeSize property
console.log(bigMap.storeSize); // 4 (default)
bigMap.storeSize = 8; // Changes size and clears all entries

// storeHashFunction property
bigMap.storeHashFunction = (key, storeSize) => key.length % storeSize;

// store property
console.log(bigMap.store.length); // 8

Methods

set

Sets the value for a key in the map.

Parameters:

• key (K): The key to set
• value (V): The value to associate with the key

Returns: Map<K, V> - The internal Map instance where the key was stored

Example:

bigMap.set('user123', { name: 'Alice' });

get

Gets the value associated with a key.

Parameters:

• key (K): The key to retrieve

Returns: V | undefined - The value, or undefined if not found

Example:

const value = bigMap.get('user123');

has

Checks if a key exists in the map.

Parameters:

• key (K): The key to check

Returns: boolean - True if the key exists, false otherwise

Example:

if (bigMap.has('user123')) {
  console.log('User exists');
}

delete

Deletes a key-value pair from the map.

Parameters:

• key (K): The key to delete

Returns: boolean - True if the entry was deleted, false if the key was not found

Example:

const deleted = bigMap.delete('user123');

clear

Removes all entries from the map.

Returns: void

Example:

bigMap.clear();
console.log(bigMap.size); // 0

forEach

Executes a provided function once for each key-value pair.

Parameters:

• callbackfn (function): Function to execute for each entry
  • value (V): The value of the current entry
  • key (K): The key of the current entry
  • map (Map<K, V>): The BigMap instance
• thisArg (optional): Value to use as this when executing the callback

Returns: void

Example:

bigMap.forEach((value, key) => {
  console.log(`${key}: ${value}`);
});

// With custom context
const context = { total: 0 };
bigMap.forEach(function(value) {
  this.total += value;
}, context);

keys

Returns an iterator of all keys in the map.

Returns: IterableIterator<K>

Example:

for (const key of bigMap.keys()) {
  console.log(key);
}

values

Returns an iterator of all values in the map.

Returns: IterableIterator<V>

Example:

for (const value of bigMap.values()) {
  console.log(value);
}

entries

Returns an iterator of all key-value pairs in the map.

Returns: IterableIterator<[K, V]>

Example:

for (const [key, value] of bigMap.entries()) {
  console.log(key, value);
}

Symbol.iterator

Returns an iterator for the map (same as entries()). Enables for...of loops.

Returns: IterableIterator<[K, V]>

Example:

for (const [key, value] of bigMap) {
  console.log(key, value);
}

getStore

Gets the internal Map instance for a specific key.

Parameters:

• key (K): The key to find the store for

Returns: Map<K, V> - The internal Map instance

Example:

const store = bigMap.getStore('user123');

getStoreMap

Gets the internal Map instance at a specific index.

Parameters:

• index (number): The index of the Map to retrieve (0 to storeSize - 1)

Returns: Map<K, V> - The Map at the specified index

Throws: Error if index is out of bounds

Example:

const firstMap = bigMap.getStoreMap(0);

initStore

Initializes the internal store with empty Map instances. Called automatically during construction.

Returns: void

Types

StoreHashFunction

Type definition for custom hash functions.

type StoreHashFunction = (key: string, storeSize: number) => number;

Parameters:

• key (string): The key to hash (converted to string)
• storeSize (number): The number of stores (adjusted for zero-based index)

Returns: number - The index of the store to use (0 to storeSize - 1)

defaultHashFunction

The default hash function using DJB2 algorithm from Hashery:

Example:

import { defaultHashFunction } from '@keyv/bigmap';

const index = defaultHashFunction('myKey', 4);

djb2Hash

DJB2 hash algorithm implementation.

Parameters:

• string (string): The string to hash
• min (number): Minimum value. Default: 0
• max (number): Maximum value. Default: 10

Returns: number - Hash value within the specified range

Example:

import { djb2Hash } from '@keyv/bigmap';

const hash = djb2Hash('myKey', 0, 10);

Contributing

Please see our contributing guide.

License

MIT © Jared Wray