Decide on the first module to port

[emmatyping]

We need to choose what the first module we migrate to Rust will be. This will be the only module we port in 3.16. I see three leading options (but please suggest other options if they seem compelling!):

base64

Base64 is a very commonly used algorithm that is also relatively simple. We could use https://docs.rs/base64/latest/base64/ which supports multiple alphabets and padding behaviors which would make implementation much easier, and performs significantly better than the _binascii C implementation. We could also use https://docs.rs/base64-simd/latest/base64_simd/ which would be even faster.

zlib

zlib is used in web compression, zip files (notably including wheels!), gzip files, and many other places. We could use https://github.com/trifectatechfoundation/zlib-rs which is faster than zlib, and about as fast as zlib-ng. The output might differ compared to zlib as the implementation is based on zlib-ng.

bz2

The bz2 module is not used nearly as much as the other two, but that can be a blessing if we want to reduce risks. We could use https://github.com/trifectatechfoundation/libbzip2-rs, which is a safe bzip2 implementation which is also 5-10% faster. I think the benefit here would be a performance and safety gain. The output should also be identical to the existing ubiquitous C library which may not be the case for zlib.

[Qelxiros]

Hi! I'm new here, but not to Rust or Python. I think base64 might be the best option to start with for a couple reasons. First, it's widely used, so lots of people have experience using it and can easily test the rust implementation. Second, the base64_simd crate looks super straightforward to use, so it's a low(zero?)-effort performance benefit that'll be immediately apparent in profiling.

[hugovk]

zlib is used in web compression, zip files (notably including wheels!), gzip files, and many other places.

Anything on pip scale will have a huge impact!

[emmatyping]

Hi! I'm new here, but not to Rust or Python.

@Qelxiros welcome!

Second, the base64_simd crate looks super straightforward to use, so it's a low(zero?)-effort performance benefit that'll be immediately apparent in profiling.

One thing to keep in mind for the base64 module is that while the vast majority of use cases are standard base64 computation, it also supports base16, base32, and base85. We could skip those or implement those ourselves, but I think they should probably be considered when making this decision.

---

zlib is used in web compression, zip files (notably including wheels!), gzip files, and many other places.

Anything on pip scale will have a huge impact!

A blessing and a curse :)

I believe deflate decompression is a bottleneck, or at least a large portion of the time, in pip installation, so moving everyone to zlib-rs would make wheel installs with pip significantly faster if I'm not mistaken.

[Qelxiros]

Second, the base64_simd crate looks super straightforward to use, so it's a low(zero?)-effort performance benefit that'll be immediately apparent in profiling.

One thing to keep in mind for the base64 module is that while the vast majority of use cases are standard base64 computation, it also supports base16, base32, and base85. We could skip those or implement those ourselves, but I think they should probably be considered when making this decision.

Thanks for the extra context! It looks like https://github.com/Nugine/simd includes both base32 and hex with simd support, but I'm not sure about base85. Is it possible to have a split of some sort (either via Rust FFI or CPython's build system) where the common cases use Rust for SIMD and base85 keeps its C implementation?

[egorsmkv]

I would suggest json module because it's more popular nowadays in many areas of applied programming by everyday user