hybridcontents

HybridContents

The HybridContentManager was originally created by Quantopian as part of pgcontents; however, the usage of HybridContentsManager was restricted to the compatibility requirements of pgcontents. These restrictions included postgres dependencies and no support for the latest notebook version (>6).

At Viaduct we used pgcontents exclusively for the HybridContentsManager and wanted to extend its functionality, so we created this fork hybridcontents.

See related pgcontents issues:

• https://github.com/quantopian/pgcontents/issues/66
• https://github.com/quantopian/pgcontents/issues/50
• https://github.com/quantopian/pgcontents/issues/28

Getting Started

Prerequisites:

• A Python installation with Jupyter Notebook >= 4.0.

Installation:

pip

pip install hybridcontents

Anaconda

conda install -c viaduct hybridcontents

conda-forge

See instructions here

Features

• Mix and match different content managers for different directories
• Easily move files between different content managers (i.e local files to s3 backed manager)
• Path validation to keep consistent naming scheme and/or prevent illegal characters

Usage

For a detailed example see, hybrid_manager_example.py

The following code snippet creates a HybridContentsManager with two directories with different content managers.

c = get_config()

c.NotebookApp.contents_manager_class = HybridContentsManager

c.HybridContentsManager.manager_classes = {
    # NOTE: LargFileManager only exists in notebook > 5
    # If using notebook < 5, use FileContentManager instead
    "": LargeFileManager,
    "shared": S3ContentsManager
}

# Each item will be passed to the constructor of the appropriate content manager.
c.HybridContentsManager.manager_kwargs = {
    # Args for root LargeFileManager
    "": {
        "root_dir": read_only_dir
    },
    # Args for the shared S3ContentsManager directory
    "shared": {
        "access_key_id": ...,
        "secret_access_key": ...,
        "endpoint_url":  ...,
        "bucket": ...,
        "prefix": ...
    },
}

def only_allow_notebooks(path):
  return path.endswith('.ipynb')

# Only allow notebook files to be stored in S3
c.HybridContentsManager.path_validators = {
    "shared": only_allow_notebooks
}

Testing

To run unit tests,

tox

This will run all unit tests for python versions 2.7, 3.6, 3.7 and jupyter notebook versions 4, 5, and 6.

Publishing a Release

• Create a new release on Github
• Update the version in setup.py
• Run ./scripts/pip_publish.sh
• Update the version meta.yaml
• Update the sha256 in meta.yaml
• Run ./scripts/anaconda_publish.sh
• Update on Conda Forge