AIO Python Kafka Client with Tansu on MinIO S3 Storage

This repository showcases the Python asyncio client for kafka using tansu with a S3 compatible storage engine using MinIO.

Prerequisites:

• docker, using compose.yaml which runs tansu and MinIO
• just, a handy way to save and run project-specific commands
• uv, an extremely fast Python package and project manager used to run the examples

justfile contains recipes to run MinIO, create the S3 bucket, and run Tansu.

Once you have the prerequisites installed, clone this repository and start everything up with:

git clone [email protected]:tansu-io/example-aiokafka.git
cd example-aiokafka

MinIO and Tansu

The up recipe will wait for MinIO to startup, create a storage bucket, and start tansu:

just up

Topic Creation

In this example we will create a customer topic with 3 partitions:

just customer-topic-create

Consumer

As the topic has three partitions, up to three consumers can be allocated to them. To begin, start with a single consumer, send some messages and observe the partition shown on the message (the first numeric field after the topic name). Then add another consumer and subsequently kill one or more to observe the group rebalancing.

just consumer

Producer

Produce a single message to the customer topic:

just producer