Is your feature request related to a problem? Please describe.
Backgroud: In the scene of log collecting, a go application will collecting logs from many different business sides, and produce to individual topics in Pulsar. So a go application have to create many topic producers, and each may have many partition producers.
Problem: The implementation of the PartitionProducer each holds an CompressionProvider, which eventually holds an instance of the zstdprovider(for pure go context, this may use the github.com/klauspost/compress/zstd).
The zstdprovider will each occupy a large byte slice for storing the history data to improve the compress ratio. And this will only release when the partition producer close. But the partition producer will never close except for the topic producer to be closed.
So, this will lead to the slow(or quick) increasing of the memory of the Go application, which may eventually lead to the frequent OOM of the Go application. It just seems like the leaking of the memory in the partition strategy of RoundRobin( and other partition strategies, which may change the producing partition with the time).
Describe the solution you'd like
1# Just reduce the Compression Provider according to the CompressionLevel will solve this problem.