![Quix - React to data, fast](./images/quixstreams-banner.png)
![X](https://img.shields.io/twitter/url?label=X&style=social&url=https%3A%2F%2Ftwitter.com%2Fquix_io)
100% Python Stream Processing for Apache Kafka
Quix Streams is a cloud-native library for processing data in Kafka using pure Python. It’s designed to give you the power of a distributed system in a lightweight library by combining Kafka's low-level scalability and resiliency features with an easy-to-use Python interface (to ease newcomers to stream processing).
It has the following benefits:
- Streaming DataFrame API (similar to pandas DataFrame) for tabular data transformations.
- Custom stateful operations via a state object.
- Custom reducing and aggregating over tumbling and hopping time windows.
- Exactly-once processing semantics via Kafka transactions.
- Pure Python with no need for a server-side engine.
Use Quix Streams to build simple Kafka producer/consumer applications or leverage stream processing to build complex event-driven systems, real-time data pipelines and AI/ML products.
Getting Started 🏄
Install Quix Streams
python -m pip install quixstreams
Requirements
Python 3.8+, Apache Kafka 0.10+
See requirements.txt for the full list of requirements
Documentation
Quix Streams Docs
Example
Here's an example of how to process data from a Kafka Topic with Quix Streams:
from quixstreams import Application
app = Application(
broker_address="localhost:9092",
)
temperature_topic = app.topic("temperature-celsius", value_deserializer="json")
alerts_topic = app.topic("temperature-alerts", value_serializer="json")
sdf = app.dataframe(topic=temperature_topic)
sdf = sdf.apply(lambda value: {"temperature_F": (value["temperature"] * 9/5) + 32})
sdf = sdf[sdf["temperature_F"] > 150]
sdf = sdf.to_topic(alerts_topic)
app.run(sdf)
Tutorials
To see Quix Streams in action, check out the Quickstart and Tutorials in the docs:
Key Concepts
There are two primary objects:
StreamingDataFrame
- a predefined declarative pipeline to process and transform incoming messages.Application
- to manage the Kafka-related setup, teardown and message lifecycle (consuming, committing). It processes each message with the dataframe you provide for it to run.
Under the hood, the Application
will:
- Consume and deserialize messages.
- Process them with your
StreamingDataFrame
. - Produce it to the output topic.
- Automatically checkpoint processed messages and state for resiliency.
- Scale using Kafka's built-in consumer groups mechanism.
Deployment
You can run Quix Streams pipelines anywhere Python is installed.
Deploy to your own infrastructure or to Quix Cloud on AWS, Azure, GCP or on-premise for a fully managed platform.
You'll get self-service DevOps, CI/CD and monitoring, all built with best in class engineering practices learned from Formula 1 Racing.
Please see the Connecting to Quix Cloud page
to learn how to use Quix Streams and Quix Cloud together.
Roadmap 📍
This library is being actively developed by a full-time team.
Here are some of the planned improvements:
Get Involved 🤝
- Please use GitHub issues to report bugs and suggest new features.
- Join the Quix Community on Slack, a vibrant group of Kafka Python developers, data engineers and newcomers to Apache Kafka, who are learning and leveraging Quix Streams for real-time data processing.
- Watch and subscribe to @QuixStreams on YouTube for code-along tutorials from scratch and interesting community highlights.
- Follow us on X and LinkedIn where we share our latest tutorials, forthcoming community events and the occasional meme.
- If you have any questions or feedback - write to us at support@quix.io!
License 📗
Quix Streams is licensed under the Apache 2.0 license.
View a copy of the License file here.