You're Invited:Meet the Socket Team at RSAC and BSidesSF 2026, March 23–26.RSVP →
Socket
Book a DemoSign in
Socket
Book a DemoSign in

D-MemFS

Package Overview
Dependencies
Maintainers
1
Versions
5
Alerts
File Explorer

Advanced tools

License
Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

D-MemFS - pypi Package Compare versions

Comparing version
0.4.0
 to
0.4.1
+8
-3
D_MemFS.egg-info/PKG-INFO
Metadata-Version: 2.4
Name: D-MemFS
Version: 0.4.0
Version: 0.4.1
Summary: In-process virtual filesystem with hard quota for Python

@@ -80,2 +80,6 @@ Author: D

### SQLite Shared In-Memory DB Auto-Persistence
Combine SQLite's shared-cache in-memory databases (`mode=memory&cache=shared`) with D-MemFS. This allows multiple concurrent connections to share a single live database, while automatically serializing its state to D-MemFS when the last connection closes and restoring it upon the next connection. Ideal for dynamic applications and ETL pipelines.
* 📝 **Tutorial:** [`examples/sqlite_shared_store.md`](examples/sqlite_shared_store.md)
### Multi-threaded Data Staging (ETL)

@@ -417,4 +421,5 @@ Use D-MemFS as a volatile, high-speed staging area for ETL pipelines. It features built-in, thread-safe file locking, ensuring safe concurrent data processing.

- [Architecture Spec v14](https://github.com/nightmarewalker/D-MemFS/blob/main/docs/design/spec_v14.md) — MemoryGuard-integrated architecture spec
- [Detailed Design Spec v2](https://github.com/nightmarewalker/D-MemFS/blob/main/docs/design/DetailedDesignSpec_v2.md) — component-level design and rationale
- [Test Design Spec v2](https://github.com/nightmarewalker/D-MemFS/blob/main/docs/design/DetailedDesignSpec_test_v2.md) — test case table and pseudocode
- [Architecture Spec v15](https://github.com/nightmarewalker/D-MemFS/blob/main/docs/design/spec_v15.md) — MemoryGuard-integrated architecture spec
- [Detailed Design Spec v3](https://github.com/nightmarewalker/D-MemFS/blob/main/docs/design/DetailedDesignSpec_v3.md) — component-level design and rationale
- [Test Design Spec v3](https://github.com/nightmarewalker/D-MemFS/blob/main/docs/design/DetailedDesignSpec_test_v3.md) — test case table and pseudocode

@@ -421,0 +426,0 @@ > These documents are written in Japanese and serve as internal design references.

+1
-1
dmemfs/__init__.py

@@ -39,2 +39,2 @@ from typing import TYPE_CHECKING

]
__version__ = "0.4.0"
__version__ = "0.4.1"
+13
-19
dmemfs/_archive.py

@@ -92,2 +92,8 @@ """dmemfs/_archive.py — Pluggable archive extraction adapters.

Path normalization is the caller's responsibility.
Notes
-----
This is an :func:`~abc.abstractmethod`. Every concrete adapter subclass
**must** implement this method. Do not yield directory entries; only
yield file members.
"""

@@ -227,6 +233,3 @@ ...

names = [c.__name__ for c in _adapters]
raise ValueError(
f"No registered adapter can handle the given source. "
f"Tried adapters: {names}"
)
raise ValueError(f"No registered adapter can handle the given source. Tried adapters: {names}")

@@ -256,11 +259,7 @@

if mfs.is_dir(dest_path):
raise IsADirectoryError(
f"Cannot extract file over existing directory: {dest_path!r}"
)
raise IsADirectoryError(f"Cannot extract file over existing directory: {dest_path!r}")
# exists() and not is_dir() ≡ is_file()
if mfs.is_file(dest_path):
if on_conflict == "raise":
raise FileExistsError(
f"File already exists in MFS: {dest_path!r}"
)
raise FileExistsError(f"File already exists in MFS: {dest_path!r}")
warnings.warn(

@@ -360,4 +359,3 @@ f"Existing MFS file {dest_path!r} will be overwritten.",

raise FileExistsError(
f"Duplicate path in archive (after sanitization): "
f"{dest_path!r}"
f"Duplicate path in archive (after sanitization): {dest_path!r}"
)

@@ -476,4 +474,3 @@ warnings.warn(

raise FileExistsError(
f"Duplicate path in archive (after sanitization): "
f"{dest_path!r}"
f"Duplicate path in archive (after sanitization): {dest_path!r}"
)

@@ -497,4 +494,3 @@ if on_conflict == "skip":

raise IsADirectoryError(
f"Cannot extract file over existing directory: "
f"{dest_path!r}"
f"Cannot extract file over existing directory: {dest_path!r}"
)

@@ -504,5 +500,3 @@ if mfs.is_file(dest_path):

if on_conflict == "raise":
raise FileExistsError(
f"File already exists in MFS: {dest_path!r}"
)
raise FileExistsError(f"File already exists in MFS: {dest_path!r}")
if on_conflict == "skip":

@@ -509,0 +503,0 @@ continue

+75
-0
dmemfs/_async.py

@@ -23,29 +23,39 @@ """Async wrapper around MemoryFileSystem.

async def read(self, size: int = -1) -> bytes:
"""Read bytes. See :meth:`MemoryFileHandle.read`."""
return await asyncio.to_thread(self._h.read, size)
async def write(self, data: bytes) -> int:
"""Write bytes. See :meth:`MemoryFileHandle.write`."""
return await asyncio.to_thread(self._h.write, data)
async def seek(self, offset: int, whence: int = 0) -> int:
"""Set stream position. See :meth:`MemoryFileHandle.seek`."""
return await asyncio.to_thread(self._h.seek, offset, whence)
async def tell(self) -> int:
"""Return current stream position. See :meth:`MemoryFileHandle.tell`."""
return await asyncio.to_thread(self._h.tell)
async def truncate(self, size: int | None = None) -> int:
"""Resize the file. See :meth:`MemoryFileHandle.truncate`."""
return await asyncio.to_thread(self._h.truncate, size)
async def flush(self) -> None:
"""Flush the write buffer (no-op for MFS). See :meth:`MemoryFileHandle.flush`."""
await asyncio.to_thread(self._h.flush)
async def readable(self) -> bool:
"""Return ``True`` if the handle supports reading."""
return await asyncio.to_thread(self._h.readable)
async def writable(self) -> bool:
"""Return ``True`` if the handle supports writing."""
return await asyncio.to_thread(self._h.writable)
async def seekable(self) -> bool:
"""Return ``True`` (MFS handles are always seekable)."""
return await asyncio.to_thread(self._h.seekable)
async def close(self) -> None:
"""Release the file lock. See :meth:`MemoryFileHandle.close`."""
await asyncio.to_thread(self._h.close)

@@ -79,2 +89,24 @@

) -> None:
"""
Parameters
----------
max_quota : int
See :class:`MemoryFileSystem`.
chunk_overhead_override : int | None
See :class:`MemoryFileSystem`.
promotion_hard_limit : int | None
See :class:`MemoryFileSystem`.
max_nodes : int | None
See :class:`MemoryFileSystem`.
default_storage : str
See :class:`MemoryFileSystem`.
default_lock_timeout : float | None
See :class:`MemoryFileSystem`. Default for async is ``30.0``.
memory_guard : str
See :class:`MemoryFileSystem`.
memory_guard_action : str
See :class:`MemoryFileSystem`.
memory_guard_interval : float
See :class:`MemoryFileSystem`.
"""
self._sync = MemoryFileSystem(

@@ -99,2 +131,16 @@ max_quota=max_quota,

) -> AsyncMemoryFileHandle:
"""Open a file and return an :class:`AsyncMemoryFileHandle`.
All parameters and exceptions are identical to
:meth:`MemoryFileSystem.open`; the operation is offloaded to a thread
via :func:`asyncio.to_thread`.
Returns
-------
AsyncMemoryFileHandle
Use with ``async with`` to ensure the handle is closed::
async with await mfs.open("/f.bin", "wb") as f:
await f.write(b"data")
"""
h = await asyncio.to_thread(self._sync.open, path, mode, preallocate, lock_timeout)

@@ -104,56 +150,85 @@ return AsyncMemoryFileHandle(h)

async def mkdir(self, path: str, exist_ok: bool = False) -> None:
"""Create a directory. See :meth:`MemoryFileSystem.mkdir`."""
await asyncio.to_thread(self._sync.mkdir, path, exist_ok)
async def rename(self, src: str, dst: str) -> None:
"""Rename or move a file or directory. See :meth:`MemoryFileSystem.rename`."""
await asyncio.to_thread(self._sync.rename, src, dst)
async def move(self, src: str, dst: str) -> None:
"""Move a file or directory, creating parents as needed. See :meth:`MemoryFileSystem.move`."""
await asyncio.to_thread(self._sync.move, src, dst)
async def remove(self, path: str) -> None:
"""Delete a single file. See :meth:`MemoryFileSystem.remove`."""
await asyncio.to_thread(self._sync.remove, path)
async def rmtree(self, path: str) -> None:
"""Recursively delete a directory. See :meth:`MemoryFileSystem.rmtree`."""
await asyncio.to_thread(self._sync.rmtree, path)
async def listdir(self, path: str) -> list[str]:
"""Return entry names in a directory. See :meth:`MemoryFileSystem.listdir`."""
return await asyncio.to_thread(self._sync.listdir, path)
async def exists(self, path: str) -> bool:
"""Return ``True`` if *path* exists. See :meth:`MemoryFileSystem.exists`."""
return await asyncio.to_thread(self._sync.exists, path)
async def is_dir(self, path: str) -> bool:
"""Return ``True`` if *path* is a directory. See :meth:`MemoryFileSystem.is_dir`."""
return await asyncio.to_thread(self._sync.is_dir, path)
async def is_file(self, path: str) -> bool:
"""Return ``True`` if *path* is a regular file. See :meth:`MemoryFileSystem.is_file`."""
return await asyncio.to_thread(self._sync.is_file, path)
async def stat(self, path: str) -> MFSStatResult:
"""Return metadata for *path*. See :meth:`MemoryFileSystem.stat`."""
return await asyncio.to_thread(self._sync.stat, path)
async def stats(self) -> MFSStats:
"""Return aggregate filesystem statistics. See :meth:`MemoryFileSystem.stats`."""
return await asyncio.to_thread(self._sync.stats)
async def get_size(self, path: str) -> int:
"""Return the size of a file in bytes. See :meth:`MemoryFileSystem.get_size`."""
return await asyncio.to_thread(self._sync.get_size, path)
async def export_as_bytesio(self, path: str, max_size: int | None = None) -> io.BytesIO:
"""Export file contents as ``BytesIO``. See :meth:`MemoryFileSystem.export_as_bytesio`."""
return await asyncio.to_thread(self._sync.export_as_bytesio, path, max_size)
async def export_tree(self, prefix: str = "/", only_dirty: bool = False) -> dict[str, bytes]:
"""Export files as a dict. See :meth:`MemoryFileSystem.export_tree`."""
return await asyncio.to_thread(self._sync.export_tree, prefix, only_dirty)
async def import_tree(self, tree: dict[str, bytes]) -> None:
"""Import a dict of files atomically. See :meth:`MemoryFileSystem.import_tree`."""
await asyncio.to_thread(self._sync.import_tree, tree)
async def copy(self, src: str, dst: str) -> None:
"""Copy a single file. See :meth:`MemoryFileSystem.copy`."""
await asyncio.to_thread(self._sync.copy, src, dst)
async def copy_tree(self, src: str, dst: str) -> None:
"""Deep-copy a directory subtree. See :meth:`MemoryFileSystem.copy_tree`."""
await asyncio.to_thread(self._sync.copy_tree, src, dst)
async def walk(self, path: str = "/") -> list[tuple[str, list[str], list[str]]]:
"""Walk the directory tree and return results as a list.
Unlike the synchronous :meth:`MemoryFileSystem.walk` generator, this
method collects all results into a list before returning (the
generator cannot be safely iterated across thread boundaries).
Returns
-------
list[tuple[str, list[str], list[str]]]
Same structure as :func:`os.walk` top-down results.
"""
return await asyncio.to_thread(lambda: list(self._sync.walk(path)))
async def glob(self, pattern: str) -> list[str]:
"""Return paths matching *pattern*. See :meth:`MemoryFileSystem.glob`."""
return await asyncio.to_thread(self._sync.glob, pattern)
+20
-5
dmemfs/_exceptions.py
class MFSQuotaExceededError(OSError):
"""Raised when the quota limit is exceeded. Subclass of OSError."""
def __init__(self, requested: int, available: int) -> None:
"""
Parameters
----------
requested : int
Number of bytes that were requested.
available : int
Number of bytes that were available at the time of the request.
"""
self.requested = requested
self.available = available
super().__init__(
f"MFS quota exceeded: requested {requested} bytes, "
f"only {available} bytes available."
f"MFS quota exceeded: requested {requested} bytes, only {available} bytes available."
)

@@ -14,8 +22,15 @@

"""Raised when the node count limit is exceeded. Subclass of MFSQuotaExceededError."""
def __init__(self, current: int, limit: int) -> None:
"""
Parameters
----------
current : int
Number of nodes present when the limit was hit.
limit : int
Configured ``max_nodes`` value.
"""
self.current = current
self.limit = limit
super().__init__(requested=current + 1, available=limit - current)
self.args = (
f"MFS node limit exceeded: current {current} nodes, limit is {limit}.",
)
self.args = (f"MFS node limit exceeded: current {current} nodes, limit is {limit}.",)
+21
-5
dmemfs/_file.py

@@ -28,3 +28,5 @@ import bisect

@abstractmethod
def read_at(self, offset: int, size: int) -> bytes: ...
def read_at(self, offset: int, size: int) -> bytes:
"""Read *size* bytes starting at *offset*. Returns ``b""`` if out of range."""
...

@@ -38,4 +40,12 @@ @abstractmethod

memory_guard: "MemoryGuard | None" = None,
) -> "tuple[int, RandomAccessMemoryFile | None, int]": ...
) -> "tuple[int, RandomAccessMemoryFile | None, int]":
"""Write *data* at *offset*, charging quota.
Returns ``(written, promoted_file_or_None, old_quota_size)``.
If the storage promotes itself (Sequential → RandomAccess), the new
storage object is returned as the second element and the caller must
swap ``FileNode.storage``.
"""
...
@abstractmethod

@@ -47,9 +57,15 @@ def truncate(

memory_guard: "MemoryGuard | None" = None,
) -> None: ...
) -> None:
"""Resize to *size* bytes, releasing or reserving quota as needed."""
...
@abstractmethod
def get_size(self) -> int: ...
def get_size(self) -> int:
"""Return the logical file size in bytes."""
...
@abstractmethod
def get_quota_usage(self) -> int: ...
def get_quota_usage(self) -> int:
"""Return the bytes currently charged to quota (data + overhead)."""
...

@@ -56,0 +72,0 @@ def _bulk_load(self, data: bytes) -> None:

+607
-11
dmemfs/_fs.py

@@ -69,2 +69,90 @@ from __future__ import annotations

class MemoryFileSystem:
"""In-process volatile virtual filesystem with hard quota enforcement.
Provides a POSIX-style hierarchical filesystem entirely within a Python
process. All data is stored in RAM; the filesystem ceases to exist when
the process exits.
Key properties
--------------
- **Hard quota**: every write is rejected *before* it occurs if it would
exceed ``max_quota``. The process is never killed by OOM due to MFS
writes.
- **Zero external dependencies**: uses only the Python 3.11+ standard
library.
- **Thread-safe**: concurrent reads are permitted; writes are exclusive.
Lock acquisition order is always ``_global_lock`` → ``FileNode._rw_lock``
to prevent deadlocks.
- **Binary-only modes**: ``open()`` accepts ``rb``, ``wb``, ``ab``,
``r+b``, and ``xb`` only. Text mode raises ``ValueError``.
Parameters
----------
max_quota : int
Maximum total memory budget in bytes that MFS will allocate for file
data. Default is 256 MiB. Writes that would exceed this limit raise
:exc:`MFSQuotaExceededError` before any data is written.
chunk_overhead_override : int | None
Override the per-chunk overhead estimate used for quota accounting in
:class:`SequentialMemoryFile`. ``None`` uses the compiled-in default
(128 bytes). Increase if quota is unexpectedly exceeded on your
platform.
promotion_hard_limit : int | None
Maximum file size in bytes at which a ``SequentialMemoryFile`` may be
automatically promoted to ``RandomAccessMemoryFile``. ``None`` uses
the default of 512 MiB. Set to ``0`` to disable promotion entirely
when combined with ``default_storage="sequential"``.
max_nodes : int | None
Maximum total number of nodes (files + directories) the filesystem
may contain. ``None`` (default) means unlimited. Exceeding this
limit raises :exc:`MFSNodeLimitExceededError`.
default_storage : str
Storage backend used when creating new files. One of:
* ``"auto"`` *(default)* — starts as ``SequentialMemoryFile`` and
promotes to ``RandomAccessMemoryFile`` on the first non-append
write.
* ``"sequential"`` — always ``SequentialMemoryFile``; promotion is
disabled and random-access writes raise ``io.UnsupportedOperation``.
* ``"random_access"`` — always ``RandomAccessMemoryFile``.
default_lock_timeout : float | None
Default timeout in seconds applied to every ``open()`` call that does
not provide an explicit ``lock_timeout``. ``None`` means block
indefinitely (no timeout). Default is ``30.0``.
memory_guard : str
Physical-memory guard strategy. One of:
* ``"none"`` *(default)* — no physical-memory checks.
* ``"init"`` — check available RAM once at construction time.
* ``"per_write"`` — re-check available RAM before every write
(rate-limited by ``memory_guard_interval``).
memory_guard_action : str
Action taken when the guard detects low physical memory. One of
``"warn"`` *(default, emits* ``ResourceWarning`` *)* or ``"raise"``
*(raises* ``MemoryError`` *)*.
memory_guard_interval : float
Minimum seconds between physical-memory checks for ``"per_write"``
mode. Default is ``1.0``.
Raises
------
ValueError
If ``default_storage`` is not one of the accepted values.
Examples
--------
Basic usage::
from dmemfs import MemoryFileSystem
mfs = MemoryFileSystem(max_quota=64 * 1024 * 1024) # 64 MiB
mfs.mkdir("/work")
with mfs.open("/work/data.bin", "wb") as f:
f.write(b"hello")
with mfs.open("/work/data.bin", "rb") as f:
assert f.read() == b"hello"
"""
def __init__(

@@ -174,2 +262,63 @@ self,

) -> MemoryFileHandle:
"""Open a file and return a :class:`MemoryFileHandle`.
Parameters
----------
path : str
Absolute path to the file inside MFS.
mode : str
Binary open mode. Accepted values:
* ``"rb"`` — read-only; raises :exc:`FileNotFoundError` if absent.
Acquires a **shared** read lock (multiple concurrent readers allowed).
* ``"wb"`` — write-only; creates the file if absent, truncates to
zero if it exists. Acquires an **exclusive** write lock.
* ``"ab"`` — append; creates if absent. Every ``write()`` call
automatically moves the cursor to EOF before writing.
Acquires an exclusive write lock.
* ``"r+b"`` — read/write; raises :exc:`FileNotFoundError` if absent.
Does **not** truncate. Acquires an exclusive write lock.
* ``"xb"`` — exclusive create; raises :exc:`FileExistsError` if the
file already exists. Acquires an exclusive write lock.
Text modes (``"r"``, ``"w"``, etc.) are not supported and raise
:exc:`ValueError`. Use :class:`MFSTextHandle` for text I/O.
preallocate : int
If positive, reserve this many bytes of quota immediately and
fill the file with null bytes up to that size. Subsequent writes
within the pre-allocated region do not consume additional quota.
Default is ``0`` (no pre-allocation).
lock_timeout : float | None
Seconds to wait for the file lock. Overrides
``default_lock_timeout`` for this call only. ``None`` inherits the
instance default (``30.0`` by default). ``0.0`` is non-blocking.
Returns
-------
MemoryFileHandle
An open file handle. **Must** be closed via ``with`` statement or
an explicit ``close()`` call to release the lock.
Raises
------
ValueError
*mode* is not one of the five accepted binary modes.
FileNotFoundError
``"rb"`` or ``"r+b"`` and *path* does not exist; or the parent
directory does not exist when creating a new file.
FileExistsError
``"xb"`` and *path* already exists.
IsADirectoryError
*path* refers to a directory.
BlockingIOError
The file lock could not be acquired within *lock_timeout* seconds.
MFSQuotaExceededError
*preallocate* would exceed the quota.
Notes
-----
The lock is held for the **entire** lifetime of the handle (from
``open()`` to ``close()``). In multi-threaded code keep handle scopes
short and always use the ``with`` statement.
"""
valid_modes = {"rb", "wb", "ab", "r+b", "xb"}

@@ -261,2 +410,22 @@ if mode not in valid_modes:

def mkdir(self, path: str, exist_ok: bool = False) -> None:
"""Create a directory, including all missing intermediate directories.
Behaves like :func:`os.makedirs` with ``parents=True``: every missing
ancestor is created automatically. This differs from :func:`os.mkdir`,
which raises :exc:`FileNotFoundError` when a parent is absent.
Parameters
----------
path : str
Absolute path of the directory to create.
exist_ok : bool
If ``True``, do nothing when *path* already exists as a directory.
If ``False`` (default), raise :exc:`FileExistsError` in that case.
Raises
------
FileExistsError
*path* already exists and *exist_ok* is ``False``, or *path*
already exists as a **file** (regardless of *exist_ok*).
"""
npath = self._np(path)

@@ -296,2 +465,36 @@ with self._global_lock:

def rename(self, src: str, dst: str) -> None:
"""Rename or move a file or directory.
Updates the parent node's child mapping; no data is copied.
Both files and directories complete in O(depth) time.
Parameters
----------
src : str
Existing path.
dst : str
New path. The parent directory of *dst* must already exist.
Raises
------
FileNotFoundError
*src* does not exist, or the parent directory of *dst* does not
exist.
FileExistsError
*dst* already exists.
BlockingIOError
*src* (or any file beneath it if *src* is a directory) is
currently open.
ValueError
*src* is the root directory ``"/"``.
Notes
-----
Unlike :meth:`move`, this method does **not** create missing
intermediate directories for *dst*. Use :meth:`move` when automatic
parent creation is desired.
Timestamps are not updated (metadata-only operation, consistent with
POSIX ``rename(2)`` semantics for ``mtime``).
"""
nsrc = self._np(src)

@@ -322,2 +525,25 @@ ndst = self._np(dst)

def move(self, src: str, dst: str) -> None:
"""Move a file or directory, creating intermediate directories as needed.
Identical to :meth:`rename` except that missing parent directories of
*dst* are created automatically.
Parameters
----------
src : str
Existing path.
dst : str
Destination path. Missing parent directories are created.
Raises
------
FileNotFoundError
*src* does not exist.
FileExistsError
*dst* already exists.
BlockingIOError
*src* (or any file beneath it) is currently open.
ValueError
*src* is the root directory ``"/"``.
"""
nsrc = self._np(src)

@@ -359,2 +585,18 @@ ndst = self._np(dst)

def remove(self, path: str) -> None:
"""Delete a single file.
Parameters
----------
path : str
Absolute path to the file to delete.
Raises
------
FileNotFoundError
*path* does not exist.
IsADirectoryError
*path* is a directory. Use :meth:`rmtree` for directories.
BlockingIOError
The file is currently open.
"""
npath = self._np(path)

@@ -378,2 +620,22 @@ with self._global_lock:

def rmtree(self, path: str) -> None:
"""Recursively delete a directory and all its contents.
All quota consumed by files in the subtree is released atomically.
Parameters
----------
path : str
Absolute path to the directory to delete.
Raises
------
FileNotFoundError
*path* does not exist.
NotADirectoryError
*path* is a file. Use :meth:`remove` for files.
BlockingIOError
Any file within the subtree is currently open.
ValueError
*path* is the root directory ``"/"``.
"""
npath = self._np(path)

@@ -415,2 +677,33 @@ if npath == "/":

def listdir(self, path: str) -> list[str]:
"""Return the names of entries directly inside *path*.
Returns bare names only (not full paths), matching the behaviour of
:func:`os.listdir`.
Parameters
----------
path : str
Absolute path to a directory.
Returns
-------
list[str]
Entry names (files and subdirectories) in unspecified order.
Raises
------
FileNotFoundError
*path* does not exist.
NotADirectoryError
*path* is a file.
Examples
--------
::
mfs.mkdir("/a/b")
with mfs.open("/a/b/f.txt", "wb") as f:
f.write(b"x")
mfs.listdir("/a/b") # → ["f.txt"]
"""
npath = self._np(path)

@@ -426,2 +719,15 @@ with self._global_lock:

def exists(self, path: str) -> bool:
"""Return ``True`` if *path* exists (file or directory).
Never raises; returns ``False`` for invalid paths.
Parameters
----------
path : str
Absolute path to check.
Returns
-------
bool
"""
try:

@@ -435,2 +741,15 @@ npath = self._np(path)

def is_dir(self, path: str) -> bool:
"""Return ``True`` if *path* exists and is a directory.
Never raises; returns ``False`` for invalid paths or non-directories.
Parameters
----------
path : str
Absolute path to check.
Returns
-------
bool
"""
try:

@@ -445,2 +764,15 @@ npath = self._np(path)

def is_file(self, path: str) -> bool:
"""Return ``True`` if *path* exists and is a regular file.
Never raises; returns ``False`` for invalid paths or directories.
Parameters
----------
path : str
Absolute path to check.
Returns
-------
bool
"""
try:

@@ -454,2 +786,29 @@ npath = self._np(path)

def stat(self, path: str) -> MFSStatResult:
"""Return metadata for the file or directory at *path*.
Parameters
----------
path : str
Absolute path to a file or directory.
Returns
-------
MFSStatResult
A :class:`~dmemfs.MFSStatResult` TypedDict with the following
fields:
* ``size`` (*int*) — file size in bytes; ``0`` for directories.
* ``created_at`` (*float*) — creation timestamp
(:func:`time.time` format).
* ``modified_at`` (*float*) — last-modification timestamp.
* ``generation`` (*int*) — monotonically increasing change
counter; ``0`` means the file has not been written since
creation or last ``import_tree``.
* ``is_dir`` (*bool*) — ``True`` for directories.
Raises
------
FileNotFoundError
*path* does not exist.
"""
npath = self._np(path)

@@ -477,2 +836,21 @@ with self._global_lock:

def stats(self) -> MFSStats:
"""Return an aggregate usage snapshot of the entire filesystem.
Returns
-------
MFSStats
A :class:`~dmemfs.MFSStats` TypedDict with the following fields:
* ``used_bytes`` (*int*) — quota bytes currently consumed.
* ``quota_bytes`` (*int*) — configured maximum quota.
* ``free_bytes`` (*int*) — remaining quota
(``quota_bytes - used_bytes``).
* ``file_count`` (*int*) — total number of files.
* ``dir_count`` (*int*) — total number of directories.
* ``chunk_count`` (*int*) — total chunks held by
``SequentialMemoryFile`` instances (promoted files are not
counted).
* ``overhead_per_chunk_estimate`` (*int*) — per-chunk overhead
used for quota accounting.
"""
with self._global_lock:

@@ -501,2 +879,21 @@ file_count = 0

def get_size(self, path: str) -> int:
"""Return the size of a file in bytes.
Parameters
----------
path : str
Absolute path to a file.
Returns
-------
int
File size in bytes.
Raises
------
FileNotFoundError
*path* does not exist.
IsADirectoryError
*path* is a directory.
"""
npath = self._np(path)

@@ -512,7 +909,35 @@ with self._global_lock:

def export_as_bytesio(self, path: str, max_size: int | None = None) -> io.BytesIO:
"""Export file contents as a BytesIO object.
"""Export a file's contents as a :class:`io.BytesIO` object.
Note: The returned BytesIO object is outside quota management.
Exporting large files may consume significant process memory
beyond the configured quota limit.
The returned object is a **deep copy** of the file data at the moment
of the call. Subsequent writes to the MFS file do not affect the
returned ``BytesIO``.
Parameters
----------
path : str
Absolute path to the file to export.
max_size : int | None
If given, raise :exc:`ValueError` before allocating memory when
the file exceeds this size. Use to guard against unexpectedly
large exports consuming heap memory outside quota control.
Returns
-------
io.BytesIO
A new ``BytesIO`` positioned at offset 0.
Raises
------
FileNotFoundError
*path* does not exist.
IsADirectoryError
*path* is a directory.
ValueError
The file size exceeds *max_size*.
Notes
-----
The returned ``BytesIO`` is **outside quota management**. Large files
may cause significant process-memory growth beyond ``max_quota``.
"""

@@ -538,2 +963,23 @@ npath = self._np(path)

def export_tree(self, prefix: str = "/", only_dirty: bool = False) -> dict[str, bytes]:
"""Export all files under *prefix* as a ``dict``.
Convenience wrapper around :meth:`iter_export_tree` that materialises
the entire result into memory at once. For large filesystems prefer
:meth:`iter_export_tree` to avoid peak-memory doubling.
Parameters
----------
prefix : str
Only files whose path starts with *prefix* are exported.
Default is ``"/"`` (all files).
only_dirty : bool
If ``True``, only include files whose ``generation`` counter is
greater than zero (modified since creation or last
``import_tree``).
Returns
-------
dict[str, bytes]
Mapping of absolute path → file bytes.
"""
return dict(self.iter_export_tree(prefix=prefix, only_dirty=only_dirty))

@@ -544,2 +990,21 @@

) -> Iterator[tuple[str, bytes]]:
"""Lazily yield ``(path, data)`` pairs for all files under *prefix*.
The set of paths is snapshotted at the start of iteration under the
global lock. Individual file data is read under per-file read locks.
Files deleted between snapshot time and read time are silently skipped
(weak consistency).
Parameters
----------
prefix : str
Path prefix filter. Default ``"/"`` exports all files.
only_dirty : bool
If ``True``, skip files with ``generation == 0``.
Yields
------
tuple[str, bytes]
``(absolute_path, file_bytes)`` for each matched file.
"""
nprefix = self._np(prefix)

@@ -575,2 +1040,31 @@ with self._global_lock:

def import_tree(self, tree: dict[str, bytes]) -> None:
"""Import a dict of files into MFS atomically (All-or-Nothing).
If any error occurs during import (quota exceeded, memory allocation
failure, etc.) the entire operation is rolled back and MFS is left in
its pre-call state.
Parameters
----------
tree : dict[str, bytes]
Mapping of absolute path → file bytes. Missing parent directories
are created automatically. Existing files at the same paths are
replaced.
Raises
------
BlockingIOError
A target path has an open handle.
MFSQuotaExceededError
The net quota increase would exceed the available quota. MFS is
rolled back.
MemoryError
The OS could not allocate memory for one of the files. MFS is
rolled back.
Notes
-----
``generation`` is reset to ``0`` for all imported files (they are
treated as a clean baseline).
"""
if not tree:

@@ -689,2 +1183,27 @@ return

def copy(self, src: str, dst: str) -> None:
"""Copy a single file (deep copy of byte content).
The destination file is a completely independent node with new
timestamps. Equivalent to POSIX ``cp src dst``.
Parameters
----------
src : str
Source file path.
dst : str
Destination file path. Must not already exist.
The parent directory must exist.
Raises
------
FileNotFoundError
*src* does not exist, or the parent directory of *dst* does not
exist.
FileExistsError
*dst* already exists.
IsADirectoryError
*src* is a directory.
MFSQuotaExceededError
Insufficient quota to store the copied data.
"""
nsrc = self._np(src)

@@ -716,2 +1235,28 @@ ndst = self._np(dst)

def copy_tree(self, src: str, dst: str) -> None:
"""Deep-copy an entire directory subtree.
All files are copied as independent byte buffers; no reference sharing
(copy-on-write) is performed. Quota for the entire subtree is
pre-checked before any node is created; the operation is all-or-nothing
with respect to quota.
Parameters
----------
src : str
Source directory path.
dst : str
Destination directory path. Must not already exist.
The parent directory must exist.
Raises
------
FileNotFoundError
*src* does not exist, or the parent of *dst* does not exist.
NotADirectoryError
*src* is not a directory.
FileExistsError
*dst* already exists.
MFSQuotaExceededError
Combined size of all files in the subtree exceeds available quota.
"""
nsrc = self._np(src)

@@ -775,9 +1320,33 @@ ndst = self._np(dst)

def walk(self, path: str = "/") -> Iterator[tuple[str, list[str], list[str]]]:
"""Recursively walk the directory tree (top-down).
"""Recursively walk the directory tree top-down.
.. warning::
Thread Safety (Weak Consistency):
walk() does not hold _global_lock across iterations.
Structural changes by other threads may cause inconsistencies.
Deleted entries are skipped (no crash).
Mirrors the behaviour of :func:`os.walk` for top-down traversal.
Parameters
----------
path : str
Root of the walk. Default ``"/"`` walks the entire filesystem.
Yields
------
tuple[str, list[str], list[str]]
``(dirpath, dirnames, filenames)`` for each directory visited,
where *dirnames* are the names of immediate subdirectories and
*filenames* are the names of immediate files.
Raises
------
FileNotFoundError
*path* does not exist.
NotADirectoryError
*path* is a file.
Warnings
--------
**Weak consistency**: the global lock is *not* held between
``yield`` statements. Concurrent structural changes may cause
inconsistencies (e.g. a yielded subdirectory name may no longer
exist by the time it is visited). Deleted entries are skipped
silently; no exception is raised. For a fully consistent snapshot
use :meth:`export_tree`.
"""

@@ -818,3 +1387,30 @@ npath = self._np(path)

Supports `*` (single dir), `**` (recursive), `?`, `[seq]`.
Parameters
----------
pattern : str
Glob pattern with POSIX-style separators. Supported wildcards:
* ``*`` — matches any sequence of characters within a single
path component (does **not** match ``/``).
* ``**`` — matches zero or more path components (recursive).
* ``?`` — matches exactly one character (not ``/``).
* ``[seq]``, ``[!seq]`` — character classes.
Returns
-------
list[str]
Sorted list of absolute paths that match *pattern*. May be empty.
Examples
--------
::
mfs.glob("/src/**/*.py") # all .py files under /src
mfs.glob("/src/*.py") # .py files directly under /src only
Notes
-----
Applies **weak consistency** (same as :meth:`walk`): concurrent
structural changes during matching may cause incomplete results but
will not raise exceptions.
"""

@@ -821,0 +1417,0 @@ pattern = pattern.replace("\\", "/")

+154
-0
dmemfs/_handle.py

@@ -13,2 +13,19 @@ from __future__ import annotations

class MemoryFileHandle(io.RawIOBase):
"""Binary file handle returned by :meth:`MemoryFileSystem.open`.
Wraps a :class:`FileNode` and provides standard stream semantics over
MFS in-memory storage. Inherits :class:`io.RawIOBase`.
The underlying file lock (read or write) is held for the **entire**
lifetime of the handle — from :meth:`MemoryFileSystem.open` until
:meth:`close`. Always use the ``with`` statement to ensure timely
release::
with mfs.open("/data.bin", "rb") as f:
data = f.read()
Do **not** instantiate this class directly; always obtain it through
:meth:`MemoryFileSystem.open`.
"""
def __init__(

@@ -44,2 +61,23 @@ self,

def read(self, size: int = -1) -> bytes:
"""Read and return up to *size* bytes.
Parameters
----------
size : int
Maximum bytes to read. ``-1`` (default) reads to EOF.
Returns
-------
bytes
The data read. Returns ``b""`` when the cursor is already at or
past EOF.
Raises
------
ValueError
The handle is closed.
io.UnsupportedOperation
The handle was opened in a write-only mode
(``"wb"``, ``"ab"``, or ``"xb"``).
"""
self._assert_open()

@@ -61,2 +99,33 @@ self._assert_readable()

def write(self, data: Any) -> int:
"""Write *data* to the file and return the number of bytes written.
For ``"ab"`` mode, the cursor is automatically moved to EOF before
each write (POSIX append semantics).
Quota is charged only for the net increase in file size.
Overwriting existing bytes (cursor + len(data) ≤ current size) does
not consume additional quota.
Parameters
----------
data : bytes | bytearray | memoryview
Data to write.
Returns
-------
int
Number of bytes written (always ``len(data)`` unless the storage
raises).
Raises
------
TypeError
*data* is not a bytes-like object.
ValueError
The handle is closed.
io.UnsupportedOperation
The handle was opened in read-only mode (``"rb"``).
MFSQuotaExceededError
Writing *data* would exceed the filesystem quota.
"""
self._assert_open()

@@ -91,2 +160,30 @@ self._assert_writable()

def seek(self, offset: int, whence: int = 0) -> int:
"""Set the stream position and return the new absolute position.
Parameters
----------
offset : int
Byte offset relative to the position indicated by *whence*.
whence : int
Positioning anchor:
* ``0`` / ``io.SEEK_SET`` — relative to file start;
*offset* must be ≥ 0.
* ``1`` / ``io.SEEK_CUR`` — relative to current position.
* ``2`` / ``io.SEEK_END`` — relative to EOF;
*offset* must be ≤ 0 (seeking *past* EOF is not supported).
Returns
-------
int
New absolute stream position.
Raises
------
ValueError
*whence* is not 0, 1, or 2; or the resulting position is
negative; or a positive *offset* is used with ``SEEK_END``.
ValueError
The handle is closed.
"""
self._assert_open()

@@ -114,2 +211,14 @@ if whence == 0:

def tell(self) -> int:
"""Return the current stream position.
Returns
-------
int
Current byte offset from the beginning of the file.
Raises
------
ValueError
The handle is closed.
"""
self._assert_open()

@@ -119,2 +228,28 @@ return self._cursor

def truncate(self, size: int | None = None) -> int:
"""Resize the file to exactly *size* bytes and return the new size.
If *size* is larger than the current file size, the file is extended
with null bytes (POSIX semantics). If *size* is smaller, excess data
is discarded and the freed quota is released immediately.
Parameters
----------
size : int | None
Target size in bytes. ``None`` uses the current cursor position.
Must be ≥ 0.
Returns
-------
int
The new file size.
Raises
------
ValueError
The handle is closed, or *size* is negative.
io.UnsupportedOperation
The handle was opened in read-only mode (``"rb"``).
MFSQuotaExceededError
Extending the file would exceed the filesystem quota.
"""
self._assert_open()

@@ -151,2 +286,12 @@ self._assert_writable()

def close(self) -> None:
"""Flush and release the file lock.
Subsequent operations on a closed handle raise :exc:`ValueError`.
Calling ``close()`` on an already-closed handle is a no-op.
Notes
-----
Prefer the ``with`` statement over explicit ``close()`` calls to
guarantee cleanup even when exceptions occur.
"""
if self.closed or self._is_closed:

@@ -169,2 +314,11 @@ return

def __del__(self) -> None:
"""Fallback cleanup: emit ``ResourceWarning`` and close if not already closed.
This is a **last-resort safety net**. It is invoked by the garbage
collector and its timing is not guaranteed (especially outside CPython
or when circular references exist).
Do not rely on ``__del__`` for deterministic lock release. Always use
the ``with`` statement.
"""
if not getattr(self, "_is_closed", True) and not getattr(self, "closed", True):

@@ -171,0 +325,0 @@ warnings.warn(

+52
-0
dmemfs/_lock.py

@@ -35,2 +35,18 @@ import threading

def acquire_read(self, timeout: float | None = None) -> None:
"""Acquire a shared read lock.
Multiple threads may hold the read lock simultaneously. Blocks
while a writer holds the lock.
Parameters
----------
timeout : float | None
Maximum seconds to wait. ``None`` blocks indefinitely.
``0.0`` is non-blocking.
Raises
------
BlockingIOError
The lock could not be acquired within *timeout* seconds.
"""
deadline = _calc_deadline(timeout)

@@ -47,2 +63,9 @@ with self._condition:

def release_read(self) -> None:
"""Release a previously acquired read lock.
Raises
------
RuntimeError
Called without a matching :meth:`acquire_read`.
"""
with self._condition:

@@ -56,2 +79,23 @@ if self._read_count <= 0:

def acquire_write(self, timeout: float | None = None) -> None:
"""Acquire an exclusive write lock.
Blocks until all readers and any current writer have released.
Parameters
----------
timeout : float | None
Maximum seconds to wait. ``None`` blocks indefinitely.
``0.0`` is non-blocking.
Raises
------
BlockingIOError
The lock could not be acquired within *timeout* seconds.
Notes
-----
There is **no fairness mechanism**. Continuous reader activity may
starve a waiting writer indefinitely. Use *timeout* to bound the
wait in high-contention scenarios.
"""
deadline = _calc_deadline(timeout)

@@ -68,2 +112,9 @@ with self._condition:

def release_write(self) -> None:
"""Release a previously acquired write lock.
Raises
------
RuntimeError
Called without a matching :meth:`acquire_write`.
"""
with self._condition:

@@ -77,3 +128,4 @@ if not self._write_held:

def is_locked(self) -> bool:
"""``True`` if any reader or a writer currently holds the lock."""
with self._condition:
return self._write_held or self._read_count > 0
+35
-0
dmemfs/_quota.py

@@ -9,2 +9,9 @@ import threading

class QuotaManager:
"""Thread-safe quota accounting for a single :class:`MemoryFileSystem`.
Acts as the "central bank" for memory allocation: every write
*reserves* quota before data is stored; on failure or rollback the
reservation is released.
"""
def __init__(self, max_quota: int) -> None:

@@ -17,2 +24,19 @@ self._max_quota: int = max_quota

def reserve(self, size: int) -> Iterator[None]:
"""Context manager that reserves *size* bytes of quota.
On entry, checks whether *size* bytes are available and increments
the used counter atomically. On normal exit the reservation is
**confirmed** (counter stays incremented). On exception the
reservation is **rolled back** (counter is decremented).
Parameters
----------
size : int
Bytes to reserve. Non-positive values are silently ignored.
Raises
------
MFSQuotaExceededError
*size* bytes are not available at the time of the call.
"""
if size <= 0:

@@ -34,2 +58,10 @@ yield

def release(self, size: int) -> None:
"""Release *size* previously reserved bytes back to the quota pool.
Parameters
----------
size : int
Bytes to release. Non-positive values are silently ignored.
The used counter never goes below zero.
"""
if size <= 0:

@@ -63,2 +95,3 @@ return

def used(self) -> int:
"""Currently consumed bytes (thread-safe snapshot)."""
with self._lock:

@@ -69,2 +102,3 @@ return self._used

def free(self) -> int:
"""Available bytes remaining (``maximum - used``, thread-safe snapshot)."""
with self._lock:

@@ -75,2 +109,3 @@ return self._max_quota - self._used

def maximum(self) -> int:
"""Configured quota ceiling in bytes."""
return self._max_quota
+16
-0
dmemfs/_text.py

@@ -83,2 +83,7 @@ """MFSTextHandle: bufferless text I/O helper.

Maximum number of characters to read. ``-1`` reads everything.
Returns
-------
str
Decoded text. Returns ``""`` at EOF.
"""

@@ -137,2 +142,7 @@ if size < 0:

Maximum number of characters to read (``-1`` means unlimited).
Returns
-------
str
One line including the terminating newline, or ``""`` at EOF.
"""

@@ -169,6 +179,12 @@ chars: list[str] = []

def __enter__(self) -> MFSTextHandle:
"""Return self to support ``with MFSTextHandle(...) as th:`` usage."""
return self
def __exit__(self, *args: object) -> None:
"""Exit the context. Does **not** close the underlying binary handle.
The binary handle's lifecycle is managed by the enclosing
``with mfs.open(...)`` block.
"""
# Closing the handle is the responsibility of the caller's with mfs.open(...) block
pass
+43
-0
dmemfs/_typing.py

@@ -5,2 +5,25 @@ from typing import TypedDict

class MFSStats(TypedDict):
"""Aggregate filesystem usage statistics returned by :meth:`MemoryFileSystem.stats`.
Fields
------
used_bytes : int
Total quota bytes currently consumed by file data and chunk
overhead.
quota_bytes : int
Configured maximum quota (``max_quota`` passed to the constructor).
free_bytes : int
Remaining quota (``quota_bytes - used_bytes``).
file_count : int
Number of file nodes in the filesystem.
dir_count : int
Number of directory nodes (including the root).
chunk_count : int
Total number of chunks held by :class:`SequentialMemoryFile`
instances. Files that have been promoted to
:class:`RandomAccessMemoryFile` are not counted.
overhead_per_chunk_estimate : int
Per-chunk overhead value used for quota accounting.
"""
used_bytes: int

@@ -16,2 +39,22 @@ quota_bytes: int

class MFSStatResult(TypedDict):
"""Per-path metadata returned by :meth:`MemoryFileSystem.stat`.
Fields
------
size : int
File size in bytes. Always ``0`` for directories.
created_at : float
Creation timestamp in the same format as :func:`time.time`.
modified_at : float
Timestamp of the last data modification (``write()`` or
``truncate()``). Equal to ``created_at`` for files that have
not been written to.
generation : int
Monotonically increasing change counter. ``0`` means the file
has not been written since creation or the last
:meth:`~MemoryFileSystem.import_tree` call.
is_dir : bool
``True`` for directories, ``False`` for files.
"""
size: int

@@ -18,0 +61,0 @@ created_at: float

+8
-3
PKG-INFO
Metadata-Version: 2.4
Name: D-MemFS
Version: 0.4.0
Version: 0.4.1
Summary: In-process virtual filesystem with hard quota for Python

@@ -80,2 +80,6 @@ Author: D

### SQLite Shared In-Memory DB Auto-Persistence
Combine SQLite's shared-cache in-memory databases (`mode=memory&cache=shared`) with D-MemFS. This allows multiple concurrent connections to share a single live database, while automatically serializing its state to D-MemFS when the last connection closes and restoring it upon the next connection. Ideal for dynamic applications and ETL pipelines.
* 📝 **Tutorial:** [`examples/sqlite_shared_store.md`](examples/sqlite_shared_store.md)
### Multi-threaded Data Staging (ETL)

@@ -417,4 +421,5 @@ Use D-MemFS as a volatile, high-speed staging area for ETL pipelines. It features built-in, thread-safe file locking, ensuring safe concurrent data processing.

- [Architecture Spec v14](https://github.com/nightmarewalker/D-MemFS/blob/main/docs/design/spec_v14.md) — MemoryGuard-integrated architecture spec
- [Detailed Design Spec v2](https://github.com/nightmarewalker/D-MemFS/blob/main/docs/design/DetailedDesignSpec_v2.md) — component-level design and rationale
- [Test Design Spec v2](https://github.com/nightmarewalker/D-MemFS/blob/main/docs/design/DetailedDesignSpec_test_v2.md) — test case table and pseudocode
- [Architecture Spec v15](https://github.com/nightmarewalker/D-MemFS/blob/main/docs/design/spec_v15.md) — MemoryGuard-integrated architecture spec
- [Detailed Design Spec v3](https://github.com/nightmarewalker/D-MemFS/blob/main/docs/design/DetailedDesignSpec_v3.md) — component-level design and rationale
- [Test Design Spec v3](https://github.com/nightmarewalker/D-MemFS/blob/main/docs/design/DetailedDesignSpec_test_v3.md) — test case table and pseudocode

@@ -421,0 +426,0 @@ > These documents are written in Japanese and serve as internal design references.

+1
-1
pyproject.toml

@@ -7,3 +7,3 @@ [build-system]

name = "D-MemFS"
version = "0.4.0"
version = "0.4.1"
description = "In-process virtual filesystem with hard quota for Python"

@@ -10,0 +10,0 @@ readme = "README.md"

+7
-2
README.md

@@ -57,2 +57,6 @@ # D-MemFS

### SQLite Shared In-Memory DB Auto-Persistence
Combine SQLite's shared-cache in-memory databases (`mode=memory&cache=shared`) with D-MemFS. This allows multiple concurrent connections to share a single live database, while automatically serializing its state to D-MemFS when the last connection closes and restoring it upon the next connection. Ideal for dynamic applications and ETL pipelines.
* 📝 **Tutorial:** [`examples/sqlite_shared_store.md`](examples/sqlite_shared_store.md)
### Multi-threaded Data Staging (ETL)

@@ -394,4 +398,5 @@ Use D-MemFS as a volatile, high-speed staging area for ETL pipelines. It features built-in, thread-safe file locking, ensuring safe concurrent data processing.

- [Architecture Spec v14](https://github.com/nightmarewalker/D-MemFS/blob/main/docs/design/spec_v14.md) — MemoryGuard-integrated architecture spec
- [Detailed Design Spec v2](https://github.com/nightmarewalker/D-MemFS/blob/main/docs/design/DetailedDesignSpec_v2.md) — component-level design and rationale
- [Test Design Spec v2](https://github.com/nightmarewalker/D-MemFS/blob/main/docs/design/DetailedDesignSpec_test_v2.md) — test case table and pseudocode
- [Architecture Spec v15](https://github.com/nightmarewalker/D-MemFS/blob/main/docs/design/spec_v15.md) — MemoryGuard-integrated architecture spec
- [Detailed Design Spec v3](https://github.com/nightmarewalker/D-MemFS/blob/main/docs/design/DetailedDesignSpec_v3.md) — component-level design and rationale
- [Test Design Spec v3](https://github.com/nightmarewalker/D-MemFS/blob/main/docs/design/DetailedDesignSpec_test_v3.md) — test case table and pseudocode

@@ -398,0 +403,0 @@ > These documents are written in Japanese and serve as internal design references.