Welcome to Kira-Kafka, lightweight, and educational distributed log system written in pure C. This project mimics the core architecture of Apache Kafka — covering topics, partitions, message brokers, disk-backed logs, zero-copy reads, memory-mapped I/O, thread-to-core mapping, and log compression.
🔬 Designed for learning, systems programming enthusiasts, and OS-level Kafka internals.
| Kafka Feature | Kira-Kafka Implementation |
|---|---|
| Topic & Partitioning | Topic structure with round-robin partitioning |
| Disk-Based Log Storage | Append-only .log files for each partition |
| Zero-Copy Reads | sendfile() syscall for NIC-efficient file transfer |
| Memory-Mapped I/O | mmap() for reading logs as byte arrays |
| Thread-Core Affinity | pthread_setaffinity_np() for log/consumer separation |
| Flush Daemon (Durability) | Background fsync() log flush thread |
| Load Balancing | Per-thread round-robin dispatch to partitions |
| Open Source Friendly | Community-driven, well-documented, modular structure |
KIRA-KAFKA/src
├── appendLog/
│ ├── include/ # Headers for append-only log
│ ├── src/ # C source files for disk logging
│ ├── temp/ # Temporary log data (buffered writes)
│ ├── kafka_appendLog # Binary output
│ └── main.c # Log writer entry point
│
├── mmap/
│ ├── include/ # Headers for memory-mapped I/O
│ ├── src/ # mmap-based log readers
│ ├── kafka_mmap # Compiled output binary
│ ├── main.c # mmap reader demo
│ └── Makefile # Build script for mmap module
│
├── partitioning/
│ ├── include/ # Headers for topic/partition logic
│ ├── src/ # C source files for round-robin dispatch
│ ├── kafka_partitioning # Compiled output binary
│ ├── main.c # Partitioning logic demo
│ └── Makefile # Build script for partitioning module
│
├── threadCore/
│ ├── include/ # Headers for thread-core mapping
│ ├── src/ # Source files for CPU affinity logic
│ ├── kafka_threadCore # Compiled binary
│ ├── main.c # Thread-core mapping demo
│ └── Makefile # Build script for threadCore module
│
├── zeroCopy/
│ ├── include/ # Headers for zero-copy I/O
│ ├── src/ # sendfile() based implementation
│ ├── kafka_zeroCopy # Output binary for zero-copy demo
│ ├── main.c # Zero-copy file transfer example
│ └── Makefile # Build script for zeroCopy module
│
├── Makefile # Optional root Makefile to build all modules
└── README.md # Full project documentation
cd (required directory)
make
make run
Start a discussion here: 👉 GitHub Discussions
Check the Wiki for: Kira-Kafka Wiki
The Kira-Kafka Wiki contains detailed, developer-friendly documentation of the core concepts and how they map to Kafka's design.
| Feature | Wiki Page Link |
|---|---|
| 📦 Topics & Partitioning | Topic & Partitioning |
| 🧾 Append-Only Logs | Append-Only Logging |
| 📤 Zero-Copy Reads | Zero-Copy with sendfile() |
| 🗃 Memory-Mapped Reads | Memory-Mapped Files |
| 🔄 Log Flusher | Flush Daemon & Durability |
| 🧵 Thread-to-Core Mapping | Thread Affinity |
💡 Each page includes C code explanations, OS-level syscalls used, and comparisons to Kafka’s internal handling.
- Go to the Wiki tab
- Click "New Page"
- Add your content and submit
Architecture diagrams OS-level optimization explanations How Kafka uses these techniques at scale
We ❤️ contributors!
To contribute:
- Fork the repository
- Clone it locally
- Create a new branch for your changes
- Implement your feature or bug fix
- Submit a Pull Request (PR) with a clear message and screenshot if applicable
git clone https://github.com/mery-top/kira-kafka.git
cd kira-kafka
git checkout -b feature/my-improvement
MIT License © Meerthika