libcrypt.so.1 being phased out?

[dvarrazzo]

Hello,

Just received psycopg/psycopg2#912, according to which libcrypt.so.1 was deprecated on Fedora 30. This sounds like manylinux wheel packages (both 1 and 2010) are incompatible with the platform.

Is there anything I can, or should, do? Would including libcrypt (maybe 2) in the wheel package be a good idea?

[njsmith]

In case anyone else is thrown by the name: this isn't libcrypto, which comes from openssl, but rather libcrypt, which is part of glibc. This appears to be one of the rare cases where glibc is intentionally choosing to break backwards compatibility.

Practically speaking, if some systems have libcrypt.so.1 and some have libcrypt.so.2, then manylinux wheels can't assume that either exists. I guess that means that they should be dropped from auditwheel's library whitelist, and that they should be vendored into any wheels that need them. Normally I'd hesitate to vendor part of glibc and ship it to a system that might be using a different glibc, but hopefully the libcrypt functionality is sufficiently self-contained that it will work out okay...?

@zackw might be able to advise us further.

[zackw]

It seems Fedora has gone further with this transition than I thought any Linux distribution would seriously consider doing, in the relatively short time it's been since the new "libxcrypt" became a thing. There are actually three versions of the library in play:

A. libcrypt.so.1 built from GNU libc's source code. Hasn't changed much in years and, in particular, hasn't been keeping up with advances in password hashing. About two years ago, me and a couple other glibc developers decided the glibc development process was too conservative and slow for this library, and we began a process of splitting it out.

B. libcrypt.so.1 built from besser82/libxcrypt. This has support for newer password hashing algorithms (bcrypt, scrypt, etc), and maintains compatibility with binaries built against glibc's libcrypt, but binaries built against its headers will not be compatible with glibc's libcrypt, and you cannot compile programs that use certain obsolete functions against it (bigcrypt, fcrypt, encrypt, setkey -- nobody should be using these anymore, since they all inherently involve the use of single DES, but it's still a consideration). Fedora seems to have switched to this library in version 28.

C. libcrypt.so.2 built from besser82/libxcrypt (configure option --disable-obsolete-api). This drops binary backward compatibility with both A and B. Fedora seems to have switched to this in version 30.

I expected Linux distributions to converge on B, not C, but I guess "any program using these functions is necessarily using a block cipher with an unacceptably short key" was enough of a reason for Fedora to break them.

I agree with @njsmith that, as a practical matter, it's no longer appropriate for libcrypt.so.1 to be on the manylinux whitelist. I'm not sure how safe it is to vendor A (glibc's libcrypt); the copy on my computer does refer to some internal-use-only symbols from libc.so.6. Vendoring B or C, however, should be safe; libxcrypt uses only public interfaces, and it shouldn't matter to any program if they get
crypt@GLIBC_2.2.5 from B instead of A.

What I'd recommend, therefore, is that the manylinux1 and manylinux2010 base images be modified to supply option C (libcrypt.so.2) as the libcrypt that will be used to build extension modules and vendored into wheels. Specifically, build libxcrypt with --disable-obsolete-api --enable-hashes=all, install it in /usr/local, and then delete the crypt.h, libcrypt.a, and libcrypt.so (but not libcrypt.so.1) installed by glibc in /usr. This will break any extension modules using the obsolete functions, but if that's okay for Fedora I think it should be okay for PyPA as well. Using --enable-hashes=all guarantees backward compatibility with all historical hashed passwords.

If there's a problem building libxcrypt in a CentOS 5 environment, please file a bug report on besser82/libxcrypt and I'll see it gets fixed.

[takluyver]

So if I'm understanding this correctly, the steps to be taken are:

• Remove libcrypt.so.1 from the libraries whitelist for manylinux1 (PEP 513) and manylinux2010 (PEP 571) - PEP 513 & 571: Remove libcrypt.so.1 from whitelist python/peps#1124 .
• Remove it from the profiles in auditwheel - Remove libcrypt.so.1 from library whitelist auditwheel#182
• Add libcrypt.so.2 to the three existing manylinux docker images for building packages, so wheels can bundle that.
  • manylinux2010_x86_64 - manylinux2010: Install libxcrypt #325
  • manylinux1_x86_64 - manylinux1: Install libxcrypt #324
  • manylinux1_i686 - manylinux1: Install libxcrypt #324

[lkollar]

Shouldn't we update the PEPs as well? auditwheel will start reporting previously compliant wheels as non-compliant and given that libcrypt.so.1 is still on the list of allowed libraries in the PEPs, this can create confusion.

[takluyver]

Yup, updating the PEPs is done now: python/peps#1124.

[zackw]

@takluyver When adding libcrypt.so.2to the manylinux docker images, make sure to verify that it gets used for wheels that call the C function crypt, rather than the libcrypt.so.1 that shipped with CentOS. I am 95% sure that the changes I described above

build libxcrypt with --disable-obsolete-api --enable-hashes=all, install it in /usr/local, and then delete the crypt.h, libcrypt.a, and libcrypt.so (but not libcrypt.so.1) installed by glibc in /usr

will accomplish this, but it still needs testing.

[takluyver]

To check that, I assume one can run ldd on a relevant binary and check which libcrypt it's linked against?

It would also be good if someone could provide a sample extension module which calls crypt, because it would probably take me a long time to piece together an example.

Finally: anyone reading this, please feel free to work on the manylinux images. I'm about to be offline for a week, and even when I am here, I'm not that hot on anything that involves writing or compiling C code. I wrote the checklist above as what needs doing, not saying that I'm going to do it. 🙂

[encukou]

I learned about this issue in EuroPython. After some discussion, we're updating Fedora's pip to Recommend libcrypt.so.1. (“Recommend” means a soft dependency that's installed by default.)
This should hide the issue for regular users.
Of course, removing it from the manylinux standards is still the way to go.

When testing on Fedora, please make sure you don't have libcrypt.so.1 (use sudo dnf remove libxcrypt-compat).

[takluyver]

(While I'm not doing systematic testing, I did note somewhere that I haven't come across any wheels which fail to load. So just to confirm: I don't have libcrypt.so.1; removing libxcrypt-compat said "Nothing to do.")