Async Apache Kafka client library in Nim
- Producer with configurable acks, compression, and batching (linger.ms style)
- Idempotent and transactional producer (KIP-98 / KIP-360 epoch bump)
- Consumer with group coordination, automatic rebalancing, and offset management
- KIP-320 log-truncation detection (OffsetForLeaderEpoch) on leader changes
- Compression: Gzip, Snappy, LZ4, Zstd
- TLS/SSL support, with trust anchors loaded from PEM
- SASL: PLAIN and SCRAM-SHA-256 / SCRAM-SHA-512, over plaintext or TLS
- RecordBatch v2 format with CRC32C validation
- API version negotiation with brokers
- Partition assignment strategies: Range and RoundRobin
- Nim >= 2.2.8
- Linux
- chronos only
The client targets chronos's single-threaded scheduler: a handle is bound to the event loop that built it, and its state carries no cross-thread synchronization. Drive it from one thread, or serialize access yourself; a multi-threaded program keeps a client per thread.
Within that loop the concurrency contract is per handle, and each symbol's
docstring is the authority: a Cluster is meant to be shared by the producers
and consumers on the loop and closed after them, a KafkaProducer is safe to
share between coroutines, while a KafkaConsumer must be driven from one.
Metrics hooks and rebalance callbacks run inline on that loop, so keep them
short, non-blocking, and free of calls back into the client.
The supported API is the explicit export list of pkg/async_kafka. Its symbols,
signatures, public fields, and documented behavior follow Semantic Versioning:
- incompatible changes require a major release;
- compatible API additions require a minor release;
- compatible fixes require a patch release.
Full-path imports such as pkg/async_kafka/connection,
pkg/async_kafka/broker, pkg/async_kafka/consumer/group, and
pkg/async_kafka/protocol/primitives are unstable advanced/in-repo escape
hatches. Their * declarations, field visibility, helper procs, generated
codecs, and transitive re-exports may change in any release and are not covered
by the package's SemVer contract.
The programs in examples/ are complete and type-checked in CI.
They default to the broker from docker-compose.yml, take its address from
KAFKA_BROKERS, and are compiled and run from the repository root:
nim c -r examples/transactions.nimEach file's header comment names what it needs — topics, a CA, a credential.
import pkg/chronos
import pkg/async_kafka
proc main() {.async.} =
let cluster = newCluster(@["localhost:9092"])
let producer = newKafkaProducer(cluster, acks = 1)
let produced = await producer.send(
topic = "my-topic",
value = toBytes("hello kafka"),
)
echo "Produced to partition ", produced.partition, " at offset ", produced.offset
await producer.close()
await cluster.close()
waitFor main()Batching, compression, and the partition strategies, with the fire-and-forget
form of send, are in
examples/producer_settings.nim.
import pkg/chronos
import pkg/async_kafka
proc main() {.async.} =
var config = initConsumerConfig(
bootstrapBrokers = @["localhost:9092"],
groupId = "my-group",
topics = @["my-topic"],
)
config.offsetResetPolicy = orpEarliest
let consumer = newKafkaConsumer(config)
await consumer.start()
let records = await consumer.pollBatch(maxRecords = 100, timeout = 1.seconds)
for rec in records:
echo "Got message at offset ", rec.offset
await consumer.close()
waitFor main()A manual commit during a rebalance raises KafkaCommitSkippedError from
commitSync (or hands it to the commitAsync callback). It is non-fatal: catch
it and continue polling. examples/consumer.nim runs
that loop and adds pause, resume, and seek; the assignment strategy and the
rebalance callbacks are in
examples/consumer_group.nim.
transactionalId switches a producer into transactional mode. A
consume-process-produce loop commits the consumed offsets inside the same
transaction; examples/transactions.nim is the
complete program.
Commit only after every send and sendOffsetsToTransaction in the
transaction has resolved. A failed send then leaves the transaction
abort-only, and a cancelled one either still goes out with the commit or leaves
it abort-only too, so a commit that succeeds never silently lacks the record.
The exception is KafkaRecordRejectedError (a record over maxBatchBytes): the
client refused that record before it touched the transaction, so the loop may
skip it and still commit.
commitTransaction and abortTransaction close the transaction as soon as they
are called: a send or sendOffsetsToTransaction made after that is refused
and does not affect them. A commit that fails or is cancelled leaves the
transaction closed: abort it, or retry the commit, which is refused when the
failure left the transaction abort-only. A commit called after
abortTransaction is refused, even if that abort failed or was cancelled; a
commit already waiting when the abort is called goes ahead first: if it
succeeds, the abort then fails, and if it fails, the abort proceeds.
Certificate verification is ON by default; skipVerify = true disables it
(dev / self-signed setups only — never in production). A private or self-signed
CA is supplied as PEM-encoded certificates, and newTrustAnchorStoreFromPem
builds the trust anchors from them.
examples/tls.nim builds them from a PEM file and connects
under them; it targets the local broker's SASL_SSL listener, which offers PLAIN
only, so it configures that mechanism as well — the transport and the mechanism
are independent knobs for any other listener.
The same tlsConfig is accepted by newKafkaProducer(bootstrapBrokers, ...)
and by ConsumerConfig, where it configures the Cluster those constructors
build; the verification details (the dNSName-only SAN check, the trust-anchor
fallback) are on TlsConfig.
newSaslScram(user, password, scramSha256 | scramSha512) and
newSaslPlain(user, password) build the mechanisms, which are passed where the
Cluster is built; examples/sasl.nim runs the handshake
end to end. PLAIN sends the password in the clear, so it is refused unless the
connection is TLS with verification on; allowPlaintext = true opts out for a
local or trusted network. Credentials must be printable ASCII (SASLprep is not
implemented).
newCluster(..., saslMechanisms = @[...]) is the multi-mechanism form; the
convenience constructors take a single mechanism for the Cluster they build.
A metadata response can redirect the client to any address the peer can reach,
and under TLS that peer can present a certificate valid for the address it
chose. TLS cannot bound that on its own, so the allowlist is a
cluster-level policy, checked before any socket is opened — plaintext
connections included. examples/host_allowlist.nim
builds one and connects under it.
Rules come from allowHost, allowCidr, and allowPrivateHosts, combined by
newHostAllowlist: a rule names an address, an advertised name is resolved with
every answer checked, the check fails closed, and omitting the parameter permits
every advertised address. allowPrivateHosts covers RFC1918 and loopback
(127/8 and ::1/128), excluding link-local and IPv6 unique-local addresses;
combine it with allowCidr for other IPv6 ranges. The convenience constructors
accept the same rules for the Cluster they build (HostAllowlist).
newCluster(...) creates a Cluster; newKafkaProducer(cluster, ...) and
newKafkaConsumer(config, cluster) borrow it and never close it, so the
caller closes what it created, last.
examples/ownership.nim closes both kinds of handle.
The convenience constructors — newKafkaProducer(bootstrapBrokers, ...) and
newKafkaConsumer(config) — create a Cluster and own it, so their close()
releases it; ownsCluster reports which case you are in. TLS, SASL, and the host
allowlist belong to the Cluster, and a consumer built on an injected one rejects
a config.tlsConfig, config.saslMechanism, or config.hostAllowlist rather
than dialing without them.
A producer built with an errorHandler that returns eaDeadLetter diverts
records it fails to deliver to a dead-letter topic, sent through a second
producer — the destination comes from setDeadLetterQueue, and without it the
send resolves as pstDeadLetterFailed instead. A diverted send reports the
outcome through ProduceResult.status instead of raising, and a transactional
producer never diverts: the failure surfaces so the transaction cannot commit
without the record.
examples/dead_letter_queue.nim wires the two
together. deadLetterOnPermanent(), alwaysDeadLetter(), and alwaysDrop() are
the built-in handlers; an ErrorHandler can choose per FailedRecord.
Diagnostics go through std/logging, and the client never installs a
handler. std/logging starts with no handlers, so out of the box every line
is discarded — enabling logs is the application's job, and
examples/observability.nim installs a console
handler. What is emitted, and at which level:
debug— per-attempt detail: each metadata candidate that failed and why, stale leader epochs that were rejected, coordinator cache invalidations and lookups, and the bookkeeping a cancelled close swallows.info— connection and membership lifecycle: reconnects after failures, the SASL candidate a broker rejected before the next candidate is tried, coordinator resolution, a subscribed topic gaining partitions, and a successful group join.warn— a metadata refresh or coordinator lookup failed on every candidate, a handshake failed, or an acquire/release pairing was violated. The pairing warn names the call site (producer.flushBatch.txn, ...) so the caller can be found without a debugger. Offset resets, corrupt batches, and truncated logs land here too.error— a background loop gives up: the fetcher stops on a permanent error or on a response that exceeded the decompression budget, and the consumer reports that it will not restart it.
The level split is the contract; the individual lines above are the current emit sites, so a line can move between levels or be added in a release.
A failed metadata refresh, coordinator lookup, or request that can go to any broker raises an error that names the most diagnostic candidate failure — endpoint, broker error code, and the broker's own message:
failed to fetch metadata from any broker (cause: 10.0.0.7:9094: SASL_AUTHENTICATION_FAILED (58): SASL auth failed for SCRAM-SHA-256: ...)
so a wrong password is distinguishable from an unreachable port without enabling debug logs.
A handler that raises cannot break the client: the failure is absorbed (Defects still propagate). The library's lines carry no request id; correlate them with the metric stream and the broker/topic/partition named in the message.
MetricsHook is an optional callback that receives every internal event
(producer.send.retry, consumer.fetch.error, ...) with its tags and value.
Pass one to newCluster and newKafkaProducer, and set
ConsumerConfig.metricsHook for a consumer; a nil hook makes each emit a
no-op, and
examples/observability.nim forwards the stream
to a callback.
The full vocabulary — every event, its value semantics, and the tag keys it
carries, and how to group by error_code and reason — is in
docs/metrics.md.
A wire-level wrap of the per-partition sequence number against a real broker
(~2^31 records on one producer-id) is not exercised in CI, so deployments that
sustain a single producer-id for billions of records should validate the broker's
wrap acceptance in their own environment. The mechanism is documented on
newKafkaProducer (enableIdempotence).
Idle-partition state is reclaimed; the idempotent producer's sequence number is
kept for the life of the producer-id. See KafkaProducer.
Each broker connection's in-flight window is fixed at 5 requests and shared by
every partition that broker leads, so produce parallelism to one broker can stay
below what the per-partition maxInFlightRequests suggests. Ordering and
idempotence are unaffected, and the connection window is not user-tunable.
Dynamic membership only (KIP-345) and eager rebalancing only (KIP-429): a
restarting member rejoins as a new member rather than resuming its assignment,
and every rebalance revokes all currently assigned partitions. See
KafkaConsumer.
A broker advertised by host name is resolved synchronously, which stalls the
event loop for the lookup. See newCluster.
SCRAM-SHA-256 / SHA-512 are implemented but RFC 5929 channel binding
(tls-server-end-point) is not. Run SCRAM over TLS for transport
confidentiality; without channel binding the SCRAM exchange itself
cannot detect TLS-layer MITM. Other SASL mechanisms beyond PLAIN and
SCRAM (e.g. OAUTHBEARER, GSSAPI) are not implemented.
sessionLifetimeMs is ignored, so a broker with connections.max.reauth.ms > 0
closes the connection at the session boundary and the next request reconnects —
transient rather than fatal, but a periodic reconnection latency. See
newCluster.
Records in a legacy batch are dropped and the cursor advances by one offset past
baseOffset; a warn log and the consumer.batch.legacy_skipped metric (tagged
with topic / partition) report the gap. This only affects clusters that still
hold segments written in the pre-0.11 message format. See poll.
chronos hands an AsyncLock to the next waiter before that waiter resumes, so a
cancellation landing in between leaves the lock held by a coroutine that unwinds
without releasing it. The broker connect path, the cluster metadata refresh, and
the consumer's offset commit each serialize behind such a lock, and a wedged one
hangs every later call on that object. In practice these paths are cancelled only
on shutdown, where the object is being torn down anyway, and the consumer's final
commit in close() bounds its wait, so close cannot hang on it.
The integration suite runs against Apache Kafka 3.7.0. Versions are negotiated per broker through ApiVersions, within these ranges:
| Message | Versions | flexibleSince |
|---|---|---|
| ApiVersionsResponse | v0–v1 | — |
| MetadataRequest / Response | v1–v12 | v9 |
| FetchRequest / Response | v4–v11 | — |
| ListOffsetsRequest / Response | v1–v2 | — |
| OffsetForLeaderEpochRequest / Response | v2–v3 | — |
| ProduceRequest / Response | v3–v7 | — |
| OffsetFetchRequest / Response | v1–v5 | — |
| OffsetCommitRequest / Response | v2–v8 | v8 |
| FindCoordinatorRequest / Response | v0–v1 | — |
| JoinGroupRequest / Response | v1–v2 | — |
| SyncGroupRequest / Response | v0–v1 | — |
| LeaveGroupRequest / Response | v0–v1 | — |
| HeartbeatRequest / Response | v0–v1 | — |
| InitProducerIdRequest / Response | v0–v3 | v2 |
| EndTxnRequest / Response | v0–v3 | v3 |
| AddPartitionsToTxn / AddOffsetsToTxn / TxnOffsetCommit | v0–v3 | v3 |
| SaslHandshakeRequest / Response | v1 | — |
| SaslAuthenticateRequest / Response | v1 | — |
| Record / RecordHeader / RecordBatch (envelope: v2) | — | — |
How these codecs are generated from one compile-time schema, and how a message
is added, is in docs/codegen.md.
MIT — see LICENSE. The licenses of the C sources the compression
and TLS dependencies compile into a build are reproduced in THIRD-PARTY-NOTICES.md.