Python stream processing made simple

Pure Python. No JVM. No wrappers. No cross-language debugging. Use Streaming DataFrames and the whole Python ecosystem to develop stream processing pipelines in fewer lines of code.

Process data in Kafka using pure Python:

Copy

from quixstreams import Application
app = Application(broker_address="localhost:9092")
input_topic = app.topic("cardata")
output_topic = app.topic("speed-hopping-windows")
# Converts JSON to a Streaming DataFrame (SDF) tabular format
sdf = app.dataframe(input_topic)
# Calculate hopping window of 1s with 200ms steps
sdf = sdf.apply(lambda row: row["Speed"]) \
      .hopping_window(1000, 200).mean().final() 
# Send rows from SDF back to the output topic as JSON messages
sdf = sdf.to_topic(output_topic)
if __name__ == "__main__":
   app.run(sdf)

Learn more about windowing in Docs

Copy

from quixstreams import Application
app = Application(broker_address="localhost:9092")
input_topic = app.topic("cardata")
output_topic = app.topic("cardata-hardbraking")
# Converts JSON to a Streaming DataFrame (SDF) tabular format
sdf = app.dataframe(input_topic)
# Filter only windows where average brake force exceeded 50%
sdf = sdf[sdf["Brake"] > 0.5]
# Send rows from SDF back to the output topic as JSON messages
sdf = sdf.to_topic(output_topic)
if __name__ == "__main__":
   app.run(sdf)

Learn more about filtering in Docs

Copy

from quixstreams import Application
app = Application(broker_address="localhost:9092")
input_topic = app.topic("cardata")
output_topic = app.topic("cardata-with-features")
# Converts JSON to a Streaming DataFrame (SDF) tabular format
sdf = app.dataframe(input_topic)
# Project world positions columns to new column with 3 scalars
sdf["WorldPosition"] = sdf.apply(lambda row: {
    "X": row["Motion_WorldPositionX"],
    "Y": row["Motion_WorldPositionY"],
    "Z": row["Motion_WorldPositionZ"],
# Derive new column based on source one.
sdf["SpeedMs"] = sdf["Speed"] / 3.6
# Send rows from SDF back to the output topic as JSON messages
sdf = sdf.to_topic(output_topic)
if __name__ == "__main__":
   app.run(sdf)

Learn more about projection in Docs

Get started

Quix Streams repo

Build streaming data pipelines easily

Pure Python

No JVM. No wrappers. No cross-language debugging or complex abstraction layers. Use any package from the entire Python ecosystem.

DataFrame API

Treat data streams as continuously updating tables. Ideal for transitioning projects from Pandas or PySpark.

Flexible

No JAR files. Quix leverages Docker to simplify dependency management for stream processing pipelines.

Stateful operators

Use built-in hopping and tumbling window functions to build stateful calculations with fewer lines of code.

Scalable

Designed for efficient scaling, Quix leverages Kafka and Kubernetes to provide data partitioning, consumer groups, state management, and replication.

Fault tolerant

Guarantees reliable data delivery and robust failure recovery through data replication, service replication, changelogs and checkpointing.

Get started

Quix Streams repo