lazy-database

lazy_db

• Free software: MIT license
• Documentation: https://lazy-db.readthedocs.io.

A lazily loaded key:value db intended for use with large datasets that are too big to be loaded into memory. The database supports integers, strings, lists of integers, bytes, and dictionaries. This database is meant to strike a good balance of retrieval/insertion speed and memory usage. This database best fits a scenario where each key has a lot of data stored under it. Scenarios where values are under 100 bytes in size this database is not very well suited for.

Install with pip install lazy-database

Example usage:

from lazy_db import LazyDb

# Simple example usage
db = LazyDb("test.lazy")
db.write("test_value", "value")
print(db.read("test_value"))  # prints "value"
db.close()

# Or use a with statement to insure the database file is closed cleanly and avoid having to call db.close() on your own
with LazyDb("test2.lazy") as db:
    db.write("test_value", "value")
    print(db.read("test_value"))

How it works

File layout

All text in database files are encoded in utf-8 format. Each database has a json string at the start of the file that denotes the database's settings, who's end is marked with a NUL byte (00 in hex)

Each database entry is appended at the end of the file and is laid out in this format:

Name	Size (bytes)	Purpose
NUL	1	Marks the start of the entry. When the initial headers index, the starting byte of each entry is checked for this NUL byte to be sure the database hasn't been corrupted. This is the beginning to what is considered the "header" for the entry (NUL bytes carry a hex value of 0x00)
Key type	1	Marks if the key is an integer or a string.
Key	any	The key for the database entry.
NUL	1	Marks the end of the key. This is necessary since string keys don't have a set size.
Content length	content_int_size	An integer (little endian) depicting the length of the content (including the content type). Defaults to 4 bytes long. This is the end to what is considered the "header" for the entry
Content type	1	Marks if the content is a string, int, int list, dict, or bytes.
Content	Content length	Stores the content

Content type labels

Name	Hex type value	Type description
String	0x01	A utf-8 string
Int	0x02	An integer
Dict	0x03	A dictionary (internally stored as a utf-8 json string)
Int list	0x04	A list of integers. Max integer size is defined by int_list_size (default: 4 bytes)
Bytes	0x05	A bytes object

The algorithm

When loading a database, all entry headers are scanned for their key value and lengths. This allows for values to be retrieved very quickly without having to load the content of every entry, at the cost of having to store the key and content length in memory though. This approach makes the database best for cases where your database will be storing a lot of data in each key that you can't afford to store in memory, however you can afford to store the name values and lengths of each element in memory.