vortex-data
Advanced tools
Python bindings for Vortex, an Apache Arrow-compatible toolkit for working with compressed array data.
Join the community on Slack! | Documentation | Performance Benchmarks
Vortex is a next-generation columnar file format and toolkit designed for high-performance data processing. It is the fastest and most extensible format for building data systems backed by object storage. It provides:
Blazing Fast Performance
Extensible Architecture
Open Source, Neutral Governance
Integrations
🟢 Development Status: Library APIs may change from version to version, but we now consider the file format stable. From release 0.36.0, all future releases of Vortex should maintain backwards compatibility of the file format (i.e., be able to read files written by any earlier version >= 0.36.0).
Vortex strictly separates logical and physical concerns:
All features are exported through the main vortex crate.
cargo add vortex
uv add vortex-data
For browsing the structure of Vortex files, you can use the vx command-line tool.
# Install latest release
cargo install vortex-tui --locked
# Or build from source
cargo install --path vortex-tui --locked
# Usage
vx browse <file>
# Optional but recommended dependencies
brew install flatbuffers protobuf # For .fbs and .proto files
brew install duckdb # For benchmarks
# Install Rust toolchain
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# or
brew install rustup
# Initialize submodules
git submodule update --init --recursive
# Setup dependencies with uv
uv sync --all-packages
For optimal performance, we suggest using MiMalloc:
#[global_allocator]
static GLOBAL_ALLOC: MiMalloc = MiMalloc;
Licensed under the Apache License, Version 2.0.
Vortex is an independent open-source project and not controlled by any single company. The Vortex Project is a sub-project of the Linux Foundation Projects. The governance model is documented in CONTRIBUTING.md and is subject to the terms of the Technical Charter.
Please do read CONTRIBUTING.md before you contribute.
If you discover a security vulnerability, please email vuln-report@vortex.dev.
Copyright © Vortex a Series of LF Projects, LLC. For terms of use, trademark policy, and other project policies please see https://lfprojects.org
The Vortex project benefits enormously from groundbreaking work from the academic & open-source communities.
FAQs
Python bindings for Vortex, an Apache Arrow-compatible toolkit for working with compressed array data.
We found that vortex-data demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Socket CEO Feross Aboukhadijeh joins Insecure Agents to discuss CVE remediation and why supply chain attacks require a different security approach.
Security News
Tailwind Labs laid off 75% of its engineering team after revenue dropped 80%, as LLMs redirect traffic away from documentation where developers discover paid products.
Security News
The planned feature introduces a review step before releases go live, following the Shai-Hulud attacks and a rocky migration off classic tokens that disrupted maintainer workflows.