Huge News!Announcing our $40M Series B led by Abstract Ventures.Learn More

mabel

Advanced tools

Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

mabel

Python Data Libraries


Maintainers
1
overlapping arrows

mabel is a Data Engineering platform designed to run in serverless environments.

mabel just runs when you need it, scaling to zero, making it efficient and ideal for deployments to platforms like Kubernetes, GCP Cloud Run, AWS Fargate and Knative.

License Status Regression Suite codecov Static Analysis PyPI Latest Release Maintainability Rating Security Rating mabel Downloads Code style: black commit_freq last_commit PyPI Latest Release FOSSA Status

Focus on What Matters

We've built mabel to enable Data Analysts to write complex data engineering tasks quickly and easily, so they could get on with doing what they do best.

from mabel import Reader

data = Reader(dataset="test_data")
print(data.count())

Key Features

  • On-the-fly compression
  • Low-memory requirements, even with terabytes of data
  • Indexing and partitioning of data for fast reads
  • Cursors for tracking reading position between processes
  • Partial SQL DQL (Data Query Language) support
  • Schema and data_expectations validation

Installation

From PyPI (recommended)

pip install --upgrade mabel

From GitHub

pip install --upgrade git+https://github.com/mabel-dev/mabel

Guides

How to Read Data

Dependencies

  • orjson for JSON (de)serialization
  • orso for data Schemas
  • zstandard for real-time on disk compression
  • LZ4 for real-time in memory compression

There are a number of optional dependencies which are usually only required for specific features and functionality. These are listed in tests/requirements.txt.

Integrations

mabel comes with adapters for the following data services:

Service
GCP StorageGoogle Cloud Storage
MinIoMinIO
AWS S3AWS S3
AzureAzure Blob Storage
LocalLocal Storage

Mabel is extensible with adapters for other data services as required.

Deployment and Execution

mabel supports running on a range of platforms, including:

Platform
DockerDocker
KubernetesKubernetes
WindowsWindows (Notice1)
LinuxLinux (Notice2)
LinuxMac (Notice3)

Notice1 - Some non-core features are not available on Windows.
Notice2 - Tested on Debian (WSL) and Ubuntu.
Notice3 - Tested on Apple Silicon Macs.

How Can I Contribute?

All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.

If you have a suggestion for an improvement or a bug, raise a ticket or start a discussion.

Want to help build mabel? See the contribution guidance.

License

Apache 2.0

FOSSA Status

FAQs


Did you know?

Socket

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

Related posts