@mohamedhabibwork/storagekit

storagekit

Unified TypeScript file management across multiple storage providers — with strongly-typed native provider options preserved instead of flattened into a lowest-common-denominator API.

import { createStorage } from "@mohamedhabibwork/storagekit";

const storage = await createStorage({
  type: "s3",
  bucket: "uploads",
  region: "eu-central-1",
});

await storage.upload("users/100/avatar.jpg", fileStream, {
  contentType: "image/jpeg",
  // normalized, works on every provider
  cacheControl: "public,max-age=31536000",
  // real AWS options, typed per storage type
  native: {
    StorageClass: "INTELLIGENT_TIERING",
    ServerSideEncryption: "AES256",
  },
});

Change type: 's3' to type: 'azure' and TypeScript now offers Azure-native options (tier: 'Cool', SAS conditions, …) — and rejects AWS ones.

Providers

Provider	Entrypoint	SDK
Local filesystem	storagekit/local	Node built-ins only
AWS S3	storagekit/s3	@aws-sdk/client-s3, @aws-sdk/lib-storage, @aws-sdk/s3-request-presigner
MinIO	storagekit/minio	minio
Azure Blob Storage	storagekit/azure	@azure/storage-blob
Oracle OCI Object Storage	storagekit/oracle	oci-objectstorage, oci-common
RustFS (S3-compatible, Rust)	storagekit/rustfs	@aws-sdk/client-s3, @aws-sdk/lib-storage, @aws-sdk/s3-request-presigner
Cloudflare R2	storagekit/r2	@aws-sdk/client-s3, @aws-sdk/lib-storage, @aws-sdk/s3-request-presigner
Google Cloud Storage	storagekit/gcs	@google-cloud/storage

All provider SDKs are optional peer dependencies. Import only the entrypoint you use; when a driver is loaded without its SDK installed you get a clear StorageInvalidConfigError with the exact npm install command instead of a module crash.

RustFS ships no first-party JS SDK — it speaks the AWS S3 API, so the storagekit/rustfs driver talks to it with the official AWS SDK v3 and bakes in RustFS' server defaults (region: 'us-east-1', forcePathStyle: true). native() returns the same S3Client as the S3 driver; the provider field on results is 'rustfs'. See docs/rustfs.md.

Cloudflare R2 uses the same AWS SDK v3 packages through its S3-compatible API. The dedicated storagekit/r2 driver derives R2's standard account endpoint from accountId, defaults its region to 'auto', and reports versioning: false. See docs/r2.md.

Google Cloud Storage is driven through the official @google-cloud/storage SDK. Authentication uses Application Default Credentials by default (auto-detected on Cloud Run / GKE / GCE / Cloud Functions), or keyFilename / credentials off-cloud. The driver uses GCS resumable uploads, V4 signed URLs, and exposes every FileMetadata field on native. See docs/gcs.md.

Requires Node.js ≥ 20. Ships dual ESM + CJS with TypeScript declarations. Browser usage is not a goal: server-side credentials and filesystem access are first-class here.

Runtime support

Runtime	Status	Notes
Node.js ≥ 20	fully supported	primary target; CI matrix on 20 + 22
Bun ≥ 1.1	fully supported	smoke-tested on every build (CI job)
Deno ≥ 2.0	fully supported	via npm compatibility; smoke-tested on every build (CI job)

The runtime smoke (scripts/runtime-smoke.mjs) exercises the local driver, streams, listings, error normalization and the custom-driver registry on all three runtimes. Browser usage is not a goal: server-side credentials and filesystem access are first-class here.

Documentation

Rendered site: mohamedhabibwork.github.io/storagekit — auto-deployed from docs/ on every push to main.

Full per-driver guides live in docs/ (also shipped in the npm tarball):

Guide	Contents
Site: mohamedhabibwork.github.io/storagekit	rendered MkDocs + Material site (search, dark mode, copy-button)
docs/local.md	config, permissions, traversal protection, symlinks, native options
docs/s3.md	AWS SDK v3, multipart tuning, storage classes/KMS, presigned URLs, LocalStack testing
docs/minio.md	native MinIO client, metadata bags, copy preconditions, presigned URLs
docs/azure.md	auth routes, access tiers, SAS generation, versioning, Azurite testing
docs/oracle.md	OCI auth providers, native multipart, PARs, cross-region copy
docs/rustfs.md	RustFS endpoints, defaults (us-east-1, path-style), AWS SDK v3 mapping, local dev server
docs/r2.md	Cloudflare R2 account/jurisdiction endpoints, auto region, S3-native options, presigned URLs
docs/gcs.md	Google Cloud Storage driver, ADC auth, fake-gcs-server emulator, V4 signed URLs, CMEK
docs/custom-drivers.md	full StorageDriver reference, registry semantics, contract testing, built-in fake driver for tests, correctness checklist
docs/uploads.md	framework upload recipes: multer/Express/NestJS/Koa, Fastify, Hono, Next.js, Elysia, Bun/Deno, formidable, busboy, GraphQL Upload, validation & serving back
docs/examples.md	end-to-end apps: upload service, streaming backup pipeline, fake-driver testing

The design rule

Common operations share one API. Provider-specific behavior stays under the strongly-typed native key, based on the configured storage type.

{
  // ── normalized (same everywhere) ──
  contentType: 'image/jpeg',
  metadata: { userId: '100' },
  overwrite: false,
  multipart: { partSize: 10 * 1024 * 1024, concurrency: 4 },
  signal: controller.signal,

  // ── provider-specific (typed by the storage type) ──
  native: {
    // Storage<'s3'>: PutObjectCommandInput fields (StorageClass, ACL, SSEKMSKeyId, …)
    // Storage<'azure'>: { tier: 'Cool', conditions, tags, blobHTTPHeaders, … }
    // Storage<'minio'>: { metaData: { … } }
    // Storage<'oracle'>: { storageTier: 'Archive', ifMatch, opcMeta, … }
    // Storage<'local'>: { mode: 0o600 }
  },
}

native is merged last: it overrides the common equivalents when both are given.

Quick start (every provider)

import { createStorage } from "@mohamedhabibwork/storagekit";

// Local
const local = await createStorage({
  type: "local",
  root: "./storage/app",
  baseUrl: "https://cdn.example.com", // optional, powers getUrl()
});

// AWS S3 (credentials resolve through the normal AWS chain when omitted)
const s3 = await createStorage({
  type: "s3",
  bucket: "my-files",
  region: "eu-west-1",
});

// MinIO
const minio = await createStorage({
  type: "minio",
  bucket: "uploads",
  endPoint: "localhost",
  port: 9000,
  useSSL: false,
  accessKey: process.env.MINIO_ACCESS_KEY,
  secretKey: process.env.MINIO_SECRET_KEY,
});

// Azure Blob (connection string, shared key, TokenCredential, or inject clients)
const azure = await createStorage({
  type: "azure",
  container: "uploads",
  connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING,
});

// Oracle — native auth providers, never access-key style
const oracle = await createStorage({
  type: "oracle",
  namespaceName: "mynamespace",
  bucketName: "mybucket",
  region: "eu-frankfurt-1",
  auth: { type: "instance-principals" },
});

You can also inject pre-built native clients for dependency injection: client (S3 / MinIO / Oracle), serviceClient / containerClient (Azure), or authProvider (Oracle).

Core operations

// upload — Buffer, string, Uint8Array, ArrayBuffer, Blob or Node stream
await storage.upload("users/1/avatar.jpg", buffer, {
  contentType: "image/jpeg",
});

// upload from a stream — never buffered into memory
await storage.upload("videos/movie.mp4", readableStream, {
  multipart: { enabled: true, partSize: 10 * 1024 * 1024, concurrency: 4 },
});

// download — stream-first
const download = await storage.download("documents/report.pdf");
download.stream.pipe(response); // Node Readable
const text = await download.text(); // or .buffer() / .json()
download.contentType; // normalized metadata
download.etag;
download.contentLength;
download.metadata;

// existence & metadata (native HEAD calls, never full downloads)
await storage.exists("users/1/avatar.jpg");
const stat = await storage.stat("users/1/avatar.jpg"); // size, etag, contentType, …

// delete (idempotent — deleting a missing object is a no-op)
await storage.delete("users/1/avatar.jpg");
await storage.deleteMany(["a.jpg", "b.jpg", "c.jpg"]); // per-path outcome report

// list — one level (default) or recursive, with opaque cursors
const page = await storage.list({ prefix: "users/100/", limit: 100 });
page.files; // StorageFile[] (path, size, etag, lastModified, …)
page.directories; // ['users/100/sub/'] — trailing slash, one level
page.cursor; // pass back via { cursor } to continue
page.hasMore;

// async iteration for very large buckets — pagination handled for you
for await (const file of storage.iterate("uploads/")) {
  console.log(file.path);
}

// copy (server-side) & move (copy + delete; rename on local)
await storage.copy("temp/image.jpg", "images/image.jpg");
await storage.move("temp/file.pdf", "documents/file.pdf");

Framework uploads (multer, Fastify, Hono, formidable, …)

storagekit plugs straight into the upload middleware of any framework — the file streams from the request into your storage, with safe UUID keys derived from the client filename. No adapter imports its framework, so multer, fastify, hono and formidable all stay optional peers.

// Express / NestJS / koa-multer — a real multer storage engine
import multer from "multer";
import { createMulterStorage } from "@mohamedhabibwork/storagekit/adapters/express";

const upload = multer({ storage: createMulterStorage(storage, { directory: "uploads" }) });
app.post("/upload", upload.single("avatar"), (req, res) => {
  res.json({ key: req.file!.key, etag: req.file!.etag, name: req.file!.originalname });
});

// Fastify (@fastify/multipart), Hono / Next.js / Bun / Deno (web File),
// formidable — see docs/uploads.md for each recipe.
import { saveFastifyFile } from "@mohamedhabibwork/storagekit/adapters/fastify";
import { saveWebFile } from "@mohamedhabibwork/storagekit/uploads";

Or describe the upload as an intent

UploadIntent is a plain object — where the bytes come from and how to store them — that you can build anywhere (HTTP handler, queue consumer, CLI) and run against any storage. Web File/Blob, middleware-shaped files and bare bytes all work as the source.

import { defineUploadIntent, saveUploadIntent } from "@mohamedhabibwork/storagekit/uploads";

const intent = defineUploadIntent({
  source: file, // web File | middleware file | Buffer | stream
  directory: "uploads",
  metadata: { tenantId: "tenant-1" },
});
const saved = await saveUploadIntent(storage, intent); // = saveUpload under the hood

See docs/uploads.md for the full contract.

saveUpload() is the universal intake: give it { body, originalName, mimeType } from any middleware and it resolves a traversal-safe key, derives contentType + metadata and streams the body to any driver.

URLs

// public URL — never performs network requests
await storage.getUrl("images/logo.png");

// signed URLs — read (default) / write / delete
await storage.getSignedUrl("documents/private.pdf", { expiresIn: 3600 });
await storage.getSignedUrl("upload-target.bin", {
  action: "write",
  expiresIn: 900,
  native: { ResponseContentDisposition: 'attachment; filename="invoice.pdf"' }, // S3
});

• S3 / MinIO: presigned URLs with full native options.
• Azure: SAS tokens from shared-key credentials or connection strings. TokenCredential-based signing needs user delegation keys — use nativeRequest() with getUserDelegationKey.
• Local: unsupported — serve files through your app (throw StorageUnsupportedOperationError).
• Oracle: no presigned URLs. The storagekit/oracle entrypoint adds a provider-specific createPreauthenticatedRequest() because PARs have a fundamentally different lifecycle (persistent server-side resources).

Expiry is validated: between 1 second and 7 days (expiresIn).

Escaping the abstraction

// the real native client, typed per provider
const client = storage.native(); // S3Client | ContainerClient | Minio.Client | ObjectStorageClient | LocalNativeClient

// any SDK operation the package does not wrap
const info = await storage.nativeRequest((c) => c.someAdvancedSdkCall());

// Azure versioned downloads, Oracle PARs, S3 Select, … all stay reachable

Multiple disks

import { createStorageManager } from "storagekit";

const disks = createStorageManager({
  default: "uploads",
  disks: {
    uploads: { type: "s3", bucket: "uploads" },
    backup: {
      type: "azure",
      container: "backup",
      accountUrl: "https://acct.blob.core.windows.net",
    },
    temp: { type: "local", root: "./storage/temp" },
  },
});

await disks.disk("uploads").upload("a.txt", "…"); // Storage<'s3'>
await disks.disk("backup").upload("a.txt", "…"); // Storage<'azure'>
await disks.disk("temp").delete("a.txt");
await disks.warmup(); // create every configured disk up front // Storage<'local'>

Cross-storage copy

import { copyBetween } from "storagekit";

await copyBetween(sourceStorage, "docs/file.pdf", destinationStorage, "archive/file.pdf", {
  concurrency: 4,
  onProgress: (bytes, total) => console.log(`${bytes}/${total ?? "?"}`),
});

Streams source → destination; the file is never fully buffered. For copies inside one provider prefer storage.copy() (server-side).

Errors

import {
  StorageError,
  StorageNotFoundError,
  StoragePermissionError,
  StorageConflictError,
  StorageInvalidConfigError,
  StorageNetworkError,
  StorageQuotaError,
  StorageUnsupportedOperationError,
  StorageInvalidPathError,
} from "storagekit";

try {
  await storage.download(path);
} catch (error) {
  if (error instanceof StorageNotFoundError) {
    // handle missing object
  }
  error.provider; // 's3' | 'local' | …
  error.operation; // 'download'
  error.path;
  error.code; // provider-native code when available
  error.cause; // the original SDK error
}

Provider errors are normalized into the classes above while the original error stays on cause. Deleting missing objects is idempotent everywhere.

Prefixes

const storage = await createStorage({
  type: "s3",
  bucket: "application",
  prefix: "production/",
});

await storage.upload("users/avatar.jpg", file);
// stored as production/users/avatar.jpg

Prefix behavior is identical across providers — every operation (upload, download, list, copy, URLs, …) applies and strips it consistently. Local paths additionally can never escape the configured root (../../etc/passwd throws StorageInvalidPathError).

Hooks & observability

const storage = await createStorage(config, {
  hooks: {
    beforeUpload: (ctx) => log.debug('uploading', ctx.path),
    afterUpload: (ctx) => metrics.count('upload'),
    uploadError: (ctx) => alert(ctx.error),
    beforeDelete: (ctx) => audit(ctx.path),
  },
  onOperation: (event) => {
    event.provider; event.operation; event.duration; event.success;
  },
});

// subscribe later
const off = storage.on('operation', (event) => { … });
off(); // unsubscribe

Hooks receive sanitized contexts — credentials and signed URLs never appear.

Versioning & encryption

Provider semantics differ and are not hidden: download/stat/delete accept a normalized versionId (S3, Azure, MinIO, Oracle), while native controls (S3 ServerSideEncryption: 'aws:kms', Azure access tiers, OCI storageTier) remain available through native.

Custom drivers

Register your own storage type against the same unified API. Implement the StorageDriver interface, register it, and createStorage resolves it by its type string — native slots are typed unknown for custom types.

import {
  createStorage,
  registerStorageDriver,
  defineDriver,
  type StorageDriver,
} from "@mohamedhabibwork/storagekit";

const driver = defineDriver({
  type: "memory",
  async upload(path, body, options) {
    /* … */ return { path, provider: "memory" };
  },
  async download(path) {
    /* → { stream, buffer(), text(), json() } */
  },
  async delete(path) {},
  async deleteMany(paths, options) {
    /* → { deleted, failed } */
  },
  async exists(path) {
    return false;
  },
  async stat(path) {
    /* → FileStat */
  },
  async list(options) {
    /* → { files, directories, cursor, hasMore } */
  },
  async copy(source, destination, options) {},
  async move(source, destination, options) {},
  async getUrl(path) {
    return "";
  },
  async getSignedUrl(path, options) {
    throw new Error("unsupported");
  },
  native() {
    return store;
  },
  nativeRequest(fn) {
    return fn(store);
  },
  capabilities() {
    return {
      signedUrls: false,
      multipartUpload: true,
      serverSideCopy: true,
      versioning: false,
      metadata: true,
      directories: false,
      bulkDelete: false,
    };
  },
});

registerStorageDriver("memory", (config) => driver);

const storage = await createStorage({ type: "memory" /* custom config fields */ });

Registration is global to the process, cannot collide with builtin types, and an optional async ready() method on the driver is awaited by the factory. Validate your driver against the published contract suite (below).

Testing your own driver

The shared contract suite is published so custom drivers can prove they behave like the built-ins:

import { defineDriverContractTests } from "@mohamedhabibwork/storagekit/testing";

defineDriverContractTests({
  name: "my-driver",
  createStorage: () => createMyStorage(config),
  capabilities: { signedUrls: true },
});

Fake storage for your tests

storagekit/testing/fake ships an in-memory driver (FakeStorageDriver, type 'fake') that implements the full unified API — buffers, strings and stream uploads, downloads with buffer()/text()/json(), listing with pagination, copy/move, capabilities, URLs — with no SDKs, no network and no I/O. It even passes the same contract suite as the real providers, so swapping it in doesn't change what your code may assume:

import { createFakeStorage } from "@mohamedhabibwork/storagekit/testing/fake";

const storage = await createFakeStorage({
  baseUrl: "https://cdn.test", // powers getUrl() + fake signed URLs
  signedUrls: true, // advertise + serve deterministic fake signed URLs
  initialFiles: { "seeded/a.txt": "seed text" }, // preload the store
  latencyMs: 25, // simulate provider latency
});

await storage.upload("uploads/a.txt", "hello");
await (await storage.download("uploads/a.txt")).text(); // 'hello'

Extras for driving test scenarios (on the driver — construct FakeStorageDriver directly or reach it via storage.native()):

• seed(entries) — preload files after construction (any upload body).
• reset() — drop every file and queued failure between tests.
• failOnce(operation, error?) — make the next upload / download / delete / deleteMany / exists / stat / list / copy / move throw, once; clearFailures() cancels queued failures.
• files — the live Map<path, { data, contentType, metadata, lastModified }>

The module has no vitest dependency, so it works in any runner (vitest, node:test, Bun, Deno). To resolve the fake through config-driven code paths, register it like a custom driver:

import { registerStorageDriver } from "@mohamedhabibwork/storagekit";
import { FakeStorageDriver } from "@mohamedhabibwork/storagekit/testing/fake";

registerStorageDriver("fake", (config) => new FakeStorageDriver(config as never));
const storage = await createStorage({ type: "fake" });

See docs/custom-drivers.md for details.

Development

npm install
npm test              # unit + contract + mocked-OCI tests
npm run test:types    # type-level assertions
npm run typecheck     # strict tsc
npm run build         # dual ESM/CJS + d.ts via tsup

Cloud integration tests are env-gated (see tests/integrations/*.ts): LocalStack for S3, minio/minio Docker for MinIO, Azurite for Azure, and opt-in live OCI (OCI_INTEGRATION_TESTS=true).

Security notes

• Credentials, connection strings, SAS tokens and signed URLs are never logged or embedded in normalized error messages.
• Local driver paths are validated against traversal and escape the root with StorageInvalidPathError.
• Signed URL lifetimes are bounded (1 s – 7 days).
• Zero runtime dependencies: every provider SDK is an optional peer dependency, so installing storagekit adds nothing to your supply chain beyond what you already chose to install.
• Releases are published from GitHub Actions with npm provenance — a verifiable, signed link between the published tarball and this repository's build.
• Continuous supply-chain analysis by Socket: the two informational alerts on the package itself — Filesystem access (the local driver is built on fs/fs/promises) and URL strings (hardcoded provider endpoint domains such as amazonaws.com, core.windows.net, oraclecloud.com) — are inherent to what a storage library does, not vulnerabilities. Alerts you may see under the "dependencies" tab belong to the dev-time installs of the optional peer SDKs (mostly Oracle's SDK tree) and are never installed by consumers of this package.

Use with AI (llms.txt)

This repo ships an llms.txt — a curated, LLM-readable map of the API, semantics, and docs, written so coding assistants get it right the first time.

• Cursor / Claude Code / Copilot: open llms.txt or paste the raw text into your rules file (CLAUDE.md, .cursorrules, AGENTS.md).
• ChatGPT / Custom GPTs / Perplexity: add the raw URL — https://raw.githubusercontent.com/mohamedhabibwork/storagekit/main/llms.txt
• Offline / agents in CI: llms.txt, the README, and every guide in docs/ ship inside the npm tarball, so agents can read them straight from node_modules/@mohamedhabibwork/storagekit/.
• Contributing to this repo: AGENTS.md documents layout, commands, and conventions for coding agents.

License

MIT

Logging with loggerkit

Managers accept an optional logger (any object with debug/info/warn/error), so a @mohamedhabibwork/loggerkit Logger plugs in directly with no extra dependency:

import { createLogger } from "@mohamedhabibwork/loggerkit";
import { createStorageManager } from "@mohamedhabibwork/storagekit";

const manager = createStorageManager({ ...config, logger: createLogger({ name: "storage" }) });

Provider creation and close events are logged at debug; creation failures at error.