Feature request: pluggable / pure-Rust compression codecs (lz4_flex, ruzstd) to avoid C sys-crates #420

Summary

Please consider supporting pure-Rust compression backends (or making the compression codecs pluggable), as an alternative to the current C-backed lz4-sys / zstd-sys.

Current behavior

The compression feature bundles four codecs and is all-or-nothing:

compression = ["lz4", "flate2", "zstd", "snap"]

• lz4 → lz4-sys (C, liblz4)
• zstd → zstd-sys (C, libzstd)
• flate2 → miniz_oxide (Rust) ✅
• snap → Rust ✅

So consumers that need lz4 or zstd (the common broker defaults) unavoidably link two C libraries, even though pulsar-rs is otherwise a pure-Rust client and offers pure-Rust runtimes (tokio-rustls-runtime-*) for TLS.

Motivation

We hit a production memory issue that traces directly to the C codecs. Under a large backlog, ~90 consumers decompress concurrently; the C malloc calls (invisible to a prefixed jemalloc build) went to glibc, whose per-thread arenas bloated RSS to ~15Gi and were never returned to the OS. Pure-Rust codecs would route those allocations through the application's configured Rust allocator, making them observable and reclaimable.

Beyond our specific case, pure-Rust codecs also help:

• musl / fully-static builds (no C toolchain / cross-compile pain for liblz4/libzstd)
• supply-chain / audit surface (fewer -sys crates and system libs)
• build simplicity (no cmake/cc for these codecs)

Proposed options (any one would help)

1. Per-codec pure-Rust feature flags, e.g. lz4-rust (via lz4_flex) and zstd-rust (via ruzstd), mutually exclusive with the existing C-backed features.
2. A trait-based codec registry so applications can supply their own compress/decompress implementations.
3. At minimum, split compression into per-codec features so users can pull in only what they need.

Notes / caveats

• ruzstd is decompression-focused; a producer path may still need a Rust zstd encoder (or keep the C encoder). For consumer-only workloads, decompress-only is sufficient.
• Happy to help test or contribute a PR if there's agreement on the preferred approach (option 1 seems least invasive).

Thanks for maintaining pulsar-rs!