test: count min sketch serialization test

[proost]

Add cross language compatibility test code.

There is limitation of compatibility test for the CM sketch.

internal hash seeds are randomly generated all the implementations(Java, C++, Go), So we can't expect interoperability correctly. random generation algorithms differs from language by language.

So CM sketch cross language compatibility test just check that each binary format is same.

[proost]

@leerho
CM sketch has limitation for cross language compatibility. Is intended? Or do we have to introduce same random algorithm for all the languages?

[tisonkun]

In Rust, we use murmur3 to create the hash_seeds:

fn make_hash_seeds(seed: u64, num_hashes: u8) -> Vec<u64> {
    let mut seeds = Vec::with_capacity(num_hashes as usize);
    for i in 0..num_hashes {
        // Derive per-row hash seeds deterministically from the sketch seed.
        let mut hasher = MurmurHash3X64128::with_seed(seed);
        hasher.write(&u64::from(i).to_le_bytes());
        let (h1, _) = hasher.finish128();
        seeds.push(h1);
    }
    seeds
}

But anyway a CountMinSketch snapshot is self-described to recover and continue to use?

[proost]

@tisonkun

But anyway a CountMinSketch snapshot is self-described to recover and continue to use?

Yes, If use it's own language. So i generate CM sketch in go and deserialized from rust, It constructed well. But we can't guarantee that estimation of value is within range.

[tisonkun]

But we can't guarantee that estimation of value is within range.

Sorry, I don't dive into CMSketch deeply. Could you elaborate this point a bit? Or provide a concrete example to demonstrate the issue.

[proost]

@tisonkun

Each implementation derives the per-row Murmur3 hash seeds from the user-provided sketch seed using its language-specific random number generator.

Java:

final Random rand = new Random(seed);
for (int i = 0; i < numHashes; i++) {
  hashSeeds_[i] = rand.nextLong();
}

C++:

std::default_random_engine rng(_seed);
std::uniform_int_distribution<uint64_t> extra_hash_seeds(
    0, std::numeric_limits<uint64_t>::max());

for (uint64_t i = 0; i < num_hashes; ++i) {
  hash_seeds.push_back(extra_hash_seeds(rng) + _seed);
}

Go:

rng := rand.New(rand.NewSource(seed))
hashSeeds := make([]int64, numHashes)
for i := range int(numHashes) {
    hashSeeds[i] = int64(rng.Int()) + seed
}

The serialized form contains the counter array and a hash of the user-provided seed, but it does not contain the generated per-row hashSeeds.

During deserialization, each implementation reconstructs hashSeeds from the provided seed using its own random number generator. Because these random number generators do not produce the same sequence across Java, C++, and Go, the reconstructed Murmur3 seeds can differ even when the same input seed is provided.

For example:

1. A C++ sketch updates item x, placing its counts into buckets selected using the C++-generated hashSeeds.
2. Go successfully deserializes the C++ counter array.
3. Go regenerates different hashSeeds.
4. Calling GetEstimate(x) in Go may inspect different buckets from those updated by C++.

Therefore, the binary format can be deserialized across languages, but we cannot currently guarantee that queries, subsequent updates, or merges behave correctly across language boundaries.