morefs

morefs

|PyPI| |Status| |Python Version| |License|

|Tests| |Codecov| |pre-commit| |Black|

.. |PyPI| image:: https://img.shields.io/pypi/v/morefs.svg :target: https://pypi.org/project/morefs/ :alt: PyPI .. |Status| image:: https://img.shields.io/pypi/status/morefs.svg :target: https://pypi.org/project/morefs/ :alt: Status .. |Python Version| image:: https://img.shields.io/pypi/pyversions/morefs :target: https://pypi.org/project/morefs :alt: Python Version .. |License| image:: https://img.shields.io/pypi/l/morefs :target: https://opensource.org/licenses/Apache-2.0 :alt: License .. |Tests| image:: https://github.com/iterative/morefs/workflows/Tests/badge.svg :target: https://github.com/iterative/morefs/actions?workflow=Tests :alt: Tests .. |Codecov| image:: https://codecov.io/gh/iterative/morefs/branch/main/graph/badge.svg :target: https://app.codecov.io/gh/iterative/morefs :alt: Codecov .. |pre-commit| image:: https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white :target: https://github.com/pre-commit/pre-commit :alt: pre-commit .. |Black| image:: https://img.shields.io/badge/code%20style-black-000000.svg :target: https://github.com/psf/black :alt: Black

Features

morefs provides standalone fsspec-based filesystems like:

• AsyncLocalFileSystem that provides async implementation of LocalFileSystem.
• In-memory filesystems DictFileSystem built on nested dictionaries and MemFS built on tries, and are much faster than fsspec's MemoryFileSystem.
• OverlayFileSystem that allows to overlay multiple fsspec-based filesystems.

Installation

You can install morefs via pip_ from PyPI_:

.. code:: console

$ pip install morefs

You might need to install with extras for some filesystems:

.. code:: console

$ pip install morefs[asynclocal] # for installing aiofile dependency for AsyncLocalFileSystem $ pip install morefs[memfs] # for installing pygtrie dependency for MemFS

Usage

AsyncLocalFileSystem

Extended version of ``LocalFileSystem`` that also provides async methods.

.. code:: python

    import asyncio
    from morefs.asyn_local import AsyncLocalFileSystem

    async def main():
        fs = AsyncLocalFileSystem(auto_mkdir=False)

        async with fs.open_async("foo", mode="w") as f:
            await f.write("foobar")

        content = await fs._cat("foo")
        print(content)
        print(fs.cat("foo"))  # you can still use sync methods

    asyncio.run(main())


DictFS
~~~~~~

DictFS is a nested dictionary-based, in-memory filesystem
and acts more like a real LocalFileSystem.

.. code:: python

    from morefs.dict import filesystem

    fs = DictFS()


MemFS
~~~~~

MemFS is a trie-based in-memory filesystem, and acts like a bucket storage.

.. code:: python

    from morefs.memory import MemFS

    fs = MemFS()


OverlayFileSystem
~~~~~~~~~~~~~~~~~

.. code:: python

    from morefs.overlay import OverlayFileSystem

    # use localfilesystem for write, overlay all filesystems for read
    fs = OverlayFileSystem(file={"auto_mkdir": True}, s3={"anon": True})
    # or you can pass filesystem instances directly
    # as variable positional arguments or with keyword argument `filesystems=[]`
    fs = OverlayFileSystem(LocalFileSystem(), s3={"anon": True})


Contributing
------------

Contributions are very welcome.
To learn more, see the `Contributor Guide`_.


License
-------

Distributed under the terms of the `Apache 2.0 license`_,
*morefs* is free and open source software.


Issues
------

If you encounter any problems,
please `file an issue`_ along with a detailed description.


.. _Apache 2.0 license: https://opensource.org/licenses/Apache-2.0
.. _PyPI: https://pypi.org/
.. _file an issue: https://github.com/iterative/morefs/issues
.. _pip: https://pip.pypa.io/
.. github-only
.. _Contributor Guide: CONTRIBUTING.rst