Security News
The Push to Ban Ransom Payments Is Gaining Momentum
Ransomware costs victims an estimated $30 billion per year and has gotten so out of control that global support for banning payments is gaining momentum.
scattered-store
Advanced tools
Readme
Dead simple key-value store for large datasets in Node.js.
I would draw the line of sanity around 10M items in store, and max size of one item around 10MB. However only your disk size and used file system are real limitations.
Scattered-store borrows idea for storing data from Git Objects. Let's say we have code:
var store = scatteredStore.create('my_store'); // Name of directory where to store data
store.set('abc', 'Hello World!'); // key: 'abc', value: 'Hello World!'
The code above, when run will store data in file:
/my_store/a9/993e364706816aba3e25717850c26c9cd0d89d
And the algorithm went as follows:
abc
was hashed with sha1 to: a9993e364706816aba3e25717850c26c9cd0d89d
a9
) became the name of directory where the entry ended up.993e364706816aba3e25717850c26c9cd0d89d
) became the name of file where data Hello World!
has been stored.So every entry is stored in separate file, and all files are scattered across maximum of 256 directories (two hex characters) to overcome limit of files per one directory. That's why it's called scattered-store.
Every entry is stored in separate file what means...
Every entry is stored in separate file what means...
npm install scattered-store
var scatteredStore = require('scattered-store');
var store = scatteredStore.create('path/to/my/store', function (err) {
// This is optional callback function so you can know
// when the initialization is done.
if (err) {
// Oops! Something went wrong.
} else {
// Initialization done!
}
});
// You don't have to wait for initialization to end before calling API methods.
// All calls will be queued and delayed automatically.
store.set('abc', 'Hello World!')
.then(function () {
return store.get('abc');
});
.then(function (value) {
console.log(value); // Hello World!
})
As key only strings can be used. Value could be everything what can be serialized to JSON and any binary data (passed as Buffer). JSON deserialization also automatically turns ISO notation strings into Date objects.
Stores given value
on given key
. String, Object, Array and Buffer are supported as value
.
Returns: promise
store.set('abc', 'Hello World!')
.then(function () {
// Value has been stored!
});
Returns value stored on given key
. If given key
doesn't exist null
is returned.
Returns: promise
store.get('abc')
.then(function (value) {
console.log(value); // Hello World!
});
As keys
accepts array of key
strings, and returns all values for those keys.
Returns: readable stream
var stream = store.getMany(["abc", "xyz"]);
stream.on('readable', function () {
var entry = stream.read();
console.log(entry);
// Every returned entry object has structure: { key: "abc", value: "Hello World!" }
// Order of items returned through stream can't be guaranteed!
});
stream.on('end', function () {
// All entries you asked for had been delivered.
});
Returns all data stored in database through stream (one by one).
Returns: readable stream
var stream = store.getAll();
stream.on('readable', function () {
var entry = stream.read();
console.log(entry);
// Every returned entry object has structure: { key: "abc", value: "Hello World!" }
// Order of items returned through stream can't be guaranteed!
});
stream.on('end', function () {
// Everything there was in the database has been delivered.
});
Deletes entry stored on given key
.
Returns: promise
store.delete('abc')
.then(function () {
// Value has been deleted from database!
});
Hook to know when all queued tasks has been executed and store is idle. Useful e.g. if you want to terminate the process, and want to make sure no dataloss will occur.
Returns: promise
store.whenIdle()
.then(function () {
// Idle now.
});
npm run benchmark
Here are results of this test on MacBook Pro with SSD. Tested with 10K, 100K and 1M items in store.
Testing scattered-store performance: 10000 items, 25KB each, 0.3GB combined.
set 1737 items/s
get 4170 items/s
getMany 7018 items/s
getAll 6817 items/s
delete 4073 items/s
Testing scattered-store performance: 100000 items, 25KB each, 2.5GB combined.
set 1684 items/s
get 3926 items/s
getMany 6671 items/s
getAll 6644 items/s
delete 3733 items/s
Testing scattered-store performance: 1000000 items, 25KB each, 25.0GB combined.
set 1132 items/s
get 1259 items/s
getMany 5139 items/s
getAll 3574 items/s
delete 1348 items/s
FAQs
Key-value store for large datasets
We found that scattered-store demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Ransomware costs victims an estimated $30 billion per year and has gotten so out of control that global support for banning payments is gaining momentum.
Application Security
New SEC disclosure rules aim to enforce timely cyber incident reporting, but fear of job loss and inadequate resources lead to significant underreporting.
Security News
The Python Software Foundation has secured a 5-year sponsorship from Fastly that supports PSF's activities and events, most notably the security and reliability of the Python Package Index (PyPI).