@thi.ng/block-fs
Advanced tools
[!NOTE] This is one of 206 standalone projects, maintained as part of the @thi.ng/umbrella monorepo and anti-framework.
🚀 Please help me to work full-time on these projects by sponsoring me on GitHub. Thank you! ❤️
Customizable block-based storage, adapters & file system layer.
All block types and block storage is fully interface-based. Blocks can have any power-of-two size (up to limits defined by the runtime/host). The following storage providers are included:
As the name indicates, block storage providers only support block-based read/write/delete access to arbitrary binary data (all ops are async). For file-based storage, blocks are created lazily/dynamically.
The package also provides an hierarchical filesystem layer with pluggable storage providers and other customizable aspects. The default implementation supports:
The API provides functions to:
The filesystem stores a bitfield of block allocations in the first N blocks. The number of blocks used depends on configured block size and the max. number of blocks in the storage backend.
Blocks can be reserved for custom purposes by calling
.allocateBlocks().
The root directory starts in block N, directly after the block allocation table. Directories grow dynamically and form a linked list of blocks once more entries are added than can be stored in a single block. Block linkage is stored in the first few bytes of each block.
The filesystem supports fully customizable formats to define directory
entries/sizes. The default Entry
implementation
requires 64 bytes.
Files are stored as linked lists of blocks, with the first few bytes of each block reserved for linkage and number of data bytes in the block.
The number of bytes effectively available for data depends on the configured block size and the max. number of blocks in the storage backend. For example, a max. block count of 65536 and a block size of 256 bytes only requires a two bytes for linkage and a third byte for storing the number of data bytes used in the block. Hence, in this configuration 253 bytes per block are available for data.
The following diagram shows a block which links to block ID 0x1234 and uses the full 253 (0xfd in hex) bytes of data available:
The last block of a file uses a special sentinel marker to indicate that no other blocks follow. This sentinel value again depends on the configured max. block count, and in this example is 0xffff. This example block only stores 64 (0x40 in hex) bytes of data, with the remainder zeroed out.
The package includes a mult-command CLI app with the following operations:
The convert command is used to bundle an entire file tree from the host system
into a single binary blob based on BlockFS with configured block size. The
file tree MUST fit into the RAM available to bun (or node).
Once bundled, the binary blob can then be used together with
MemoryBlockStorage
and BlockFS for
other purposes (e.g. distributed with your web app to provide a virtual
filesystem). Also see API example further
below.
Example usage to bundle the source directory of this package:
npx @thi.ng/block-fs convert -o dummy.dat packages/block-fs/src/
# [INFO] blockfs: number of files: 11
# [INFO] blockfs: number of directories: 2
# [INFO] blockfs: total file size: 40341
# [INFO] blockfs: number of blocks: 56
# [INFO] blockfs: writing file: dummy.dat
General usage:
npx @thi.ng/block-fs convert --help
█ █ █ │
██ █ │
█ █ █ █ █ █ █ █ │ @thi.ng/block-fs 0.4.0
█ █ █ █ █ █ █ █ █ │ Block-based storage & file system layer
█ │
█ █ │
Usage: blockfs <cmd> [opts] input [...]
blockfs <cmd> --help
Available commands:
convert : Convert file tree into single BlockFS blob
list : List file tree of a BlockFS blob
Flags:
-q, --quiet Disable logging
-v, --verbose Display extra logging information
Main:
-bs BYTES, --block-size BYTES Block size (default: 1024)
-i EXT, --exclude EXT [multiple] File exclusion regexp
-i EXT, --include EXT [multiple] File inclusion regexp
-n INT, --num-blocks INT Number of blocks (multiple of 8)
-o STR, --out STR [required] Output file path
The list command is used to list the files & directories stored in a binary blob created via the convert command. Several output options (e.g. tree-like output) are supported.
npx @thi.ng/block-fs list dummy.dat
# /api.ts
# /cli.ts
# /directory.ts
# /entry.ts
# /fs.ts
# /index.ts
# /lock.ts
# /storage
# /storage/astorage.ts
# /storage/file.ts
# /storage/memory.ts
# /utils.ts
npx @thi.ng/block-fs list --tree dummy.dat
# ├── api.ts
# ├── cli.ts
# ├── directory.ts
# ├── entry.ts
# ├── fs.ts
# ├── index.ts
# ├── lock.ts
# ├── storage
# │ ├── astorage.ts
# │ ├── file.ts
# │ └── memory.ts
# └── utils.ts
# display file sizes & modification times
npx @thi.ng/block-fs list --tree --all dummy.dat
# ├── api.ts 2204 2025-04-02T10:22:55.573Z
# ├── cli.ts 6799 2025-04-02T18:07:58.895Z
# ├── directory.ts 3994 2025-04-02T13:47:00.108Z
# ├── entry.ts 4130 2025-04-02T10:22:55.574Z
# ├── fs.ts 16377 2025-04-02T13:46:36.608Z
# ├── index.ts 317 2025-04-01T21:38:08.232Z
# ├── lock.ts 1501 2025-04-01T21:38:08.232Z
# ├── storage 2025-04-02T18:33:47.389Z
# │ ├── astorage.ts 1205 2025-04-02T10:22:55.574Z
# │ ├── file.ts 1780 2025-04-02T14:25:12.461Z
# │ └── memory.ts 1802 2025-04-02T14:26:02.163Z
# └── utils.ts 418 2025-04-02T10:22:55.574Z
General usage:
npx @thi.ng/block-fs list --help
Usage: blockfs <cmd> [opts] input
Flags:
-a, --all Display all attribs
-t, --tree List files as tree
-v, --verbose Display extra process information
-m, --with-mtime Display modified times
-s, --with-size Display file sizes
Main:
-bs BYTES, --block-size BYTES Block size (default: 1024)
ALPHA - bleeding edge / work-in-progress
Search or submit any issues for this package
yarn add @thi.ng/block-fs
ESM import:
import * as bf from "@thi.ng/block-fs";
Browser ESM import:
<script type="module" src="https://esm.run/@thi.ng/block-fs"></script>
For Node.js REPL:
const bf = await import("@thi.ng/block-fs");
Package sizes (brotli'd, pre-treeshake): ESM: 4.50 KB
Note: @thi.ng/api is in most cases a type-only import (not used at runtime)
import { BlockFS, MemoryBlockStorage } from "@thi.ng/block-fs";
// create in-memory storage (64KB)
const storage = new MemoryBlockStorage({ numBlocks: 0x100, blockSize: 0x100 });
// create & initialize filesystem for given storage
const fs = new BlockFS(storage);
await fs.init();
// write a new text file (utf-8)
console.log(
await fs.writeFile("/hello/world.txt", "Hello, world!\n".repeat(20))
);
// { start: 3, end: 4, size: 280 }
// append to file (or auto-create it if missing)
console.log(
await fs.appendFile("/hello/world.txt", "Goodbye, world!\n".repeat(20))
);
// { start: 3, end: 5, size: 600 }
// read file as text
console.log(await fs.readText("/hello/world.txt"));
// Hello, world!
// Hello, world!
// Hello, world!
// ...
// Goodbye, world!
// Goodbye, world!
// Goodbye, world!
// ...
// write binary file, missing intermediate directories will be auto-created
console.log(
await fs.writeFile(
"/deeply/nested/paths/are-ok",
new Uint8Array([1, 2, 3, 4])
)
);
// { start: 9, end: 9, size: 4 }
// read back...
console.log(await fs.readFile("/deeply/nested/paths/are-ok"));
// Uint8Array(4) [ 1, 2, 3, 4 ]
// iterate all files & directory entries in root dir
for await (let entry of fs.root.tree()) {
// entry.path is absolute path
// entry.size is always a bigint
// entry.ctime/mtime is UNIX epoch
console.log(entry.path, entry.size, new Date(entry.ctime));
}
// /hello 0n 2025-04-01T20:18:55.916Z
// /hello/world.txt 600n 2025-04-01T20:18:55.916Z
// /deeply 0n 2025-04-01T20:18:55.919Z
// /deeply/nested 0n 2025-04-01T20:18:55.919Z
// /deeply/nested/paths 0n 2025-04-01T20:18:55.919Z
// /deeply/nested/paths/are-ok 4n 2025-04-01T20:18:55.919Z
This example shows how to use a binary blob created via the CLI blockfs convert command as a virtual file
system...
import { BlockFS, MemoryBlockStorage } from "@thi.ng/block-fs";
// load binary blob
const response = await fetch("./blocks.dat");
const buffer = await response.arrayBuffer();
// wrap as block storage
const storage = new MemoryBlockStorage({
buffer,
blockSize: 1024,
numBlocks: buffer.byteLength / 1024
});
// wrap as file system
const fs = new BlockFS(storage);
// list all entries (recursive)
for await(let f of fs.root.tree()) {
console.log(f.path);
}
// list all entries in a directory
const dir = (await fs.entryForPath("/path/to/dir")).directory;
for await (let f of dir) {
console.log(f.path);
}
// load an image as blob URL (MIME type is inferred automatically)
const img = new Image();
img.src = await fs.readAsObjectURL("/assets/test.jpg");
If this project contributes to an academic publication, please cite it as:
@misc{thing-block-fs,
title = "@thi.ng/block-fs",
author = "Karsten Schmidt",
note = "https://thi.ng/block-fs",
year = 2024
}
© 2024 - 2025 Karsten Schmidt // Apache License 2.0
FAQs
Customizable block-based storage, adapters & file system layer
The npm package @thi.ng/block-fs receives a total of 63 weekly downloads. As such, @thi.ng/block-fs popularity was classified as not popular.
We found that @thi.ng/block-fs demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Socket uncovered npm malware campaign mimicking popular Node.js libraries and packages from other ecosystems; packages steal data and execute remote code.
Research
Socket's research uncovers three dangerous Go modules that contain obfuscated disk-wiping malware, threatening complete data loss.
Research
Socket uncovers malicious packages on PyPI using Gmail's SMTP protocol for command and control (C2) to exfiltrate data and execute commands.