Handle OpenSSL config errors gracefully in DetectCiphersuiteConfiguration

[Copilot]

Description

In release builds, assert() statements are compiled out (NDEBUG defined), causing NULL pointer dereferences when OpenSSL configuration is invalid. Users see segfaults instead of diagnostic messages.

Changes:

• Kept assert() for SSL_CTX_new and SSL_new (fundamental allocations that should never fail)
• Replaced assert() calls on cipher operations with graceful error handling
• When SSL_CTX_set_cipher_list fails (e.g., due to broken OpenSSL config), the function returns early
• g_config_specified_ciphersuites remains 0, causing the application to use default ciphers
• Added proper resource cleanup (SSL_CTX_free) before early return
• Added comprehensive test case in System.Net.Security.Tests using RemoteExecutor.Invoke with OPENSSL_CONF environment variable

Test Implementation:
Created MalformedOpenSslConfig_DoesNotCrash test that:

• Creates a temporary malformed OpenSSL config file (missing provider section reference)
• Sets OPENSSL_CONF environment variable to point to the malformed configuration
• Runs SSL handshake in a separate process via RemoteExecutor with custom environment
• Verifies the process completes without crashing and successfully performs SSL operations
• Confirms graceful degradation to default ciphers works as expected

Approach:
The cipher detection in DetectCiphersuiteConfiguration() is optional - it only determines if the system has custom cipher configuration. When detection fails due to broken OpenSSL config, the code gracefully assumes no custom configuration exists and proceeds with defaults.

Before:

int rv = SSL_CTX_set_cipher_list(ctx, "ALL");
assert(rv);  // Compiled out in release builds, crashes on failure

After:

int rv = SSL_CTX_set_cipher_list(ctx, "ALL");
if (!rv)
{
    // Assume no custom cipher configuration.
    // g_config_specified_ciphersuites stays 0, use default cipher list.
    ERR_clear_error();
    SSL_CTX_free(ctx);
    return;
}

Customer Impact

Applications crash with SIGSEGV (exit code 139) when OpenSSL configuration is malformed, providing no diagnostic information. This affects any .NET application using HTTPS/TLS on systems with invalid OpenSSL 3.x configuration (e.g., missing provider sections in /etc/ssl/openssl.cnf).

With this fix, applications will gracefully fall back to default ciphers instead of crashing, allowing them to continue operating even with broken OpenSSL configurations.

Regression

No. This is a longstanding issue where asserts have been incorrectly used for production error handling in native code.

Testing

• Built with ./build.sh libs.native -rc release
• Verified libSystem.Security.Cryptography.Native.OpenSsl.so compiles without warnings
• Error handling now gracefully degrades to defaults instead of crashing
• Added automated test: MalformedOpenSslConfig_DoesNotCrash in SslStreamRemoteExecutorTests.cs
• Test passes successfully: dotnet test --filter "FullyQualifiedName~MalformedOpenSslConfig_DoesNotCrash"
• Test verifies graceful handling of malformed OpenSSL configurations in isolated process

Risk

Low. Changes are isolated to cipher detection logic. The new behavior gracefully falls back to default ciphers when detection fails, which is safer than the previous behavior (segfault). Critical allocation failures still assert as expected. No changes to success paths. Added test coverage validates the fix works correctly with actual malformed OpenSSL configurations.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>.NET segfaults instead of showing error when OpenSSL config is invalid</issue_title>
<issue_description>### Description

When OpenSSL 3.x has a configuration error, .NET crashes with SIGSEGV instead of reporting the problem.

Root cause: src/native/libs/System.Security.Cryptography.Native/pal_ssl.c uses assert() for error checking in DetectCiphersuiteConfiguration(). In release builds, these asserts are compiled out (NDEBUG defined), so when OpenSSL functions fail, no error checking happens and the code crashes dereferencing NULL pointers.

Core dump shows crash in:

dotnet/runtime#0  SSL_CTX_set_cipher_list (libssl.so.3.5.3)
dotnet/runtime#1  CryptoNative_EnsureLibSslInitialized (libSystem.Security.Cryptography.Native.OpenSsl.so)

Problematic code (lines 118, 128 in pal_ssl.c):

SSL_CTX* ctx = SSL_CTX_new(TLS_method());
assert(ctx != NULL);  // Compiled out in release builds!

int rv = SSL_CTX_set_cipher_list(ctx, "ALL");
assert(rv);  // Compiled out in release builds!



### Reproduction Steps

1. Break OpenSSL config by editing `/etc/ssl/openssl.cnf`:
   ```ini
   [provider_sect]
   default = default_sect
   legacy = legacy_sect      # References section below

   [default_sect]
   activate = 1

   ##[legacy_sect]            # Section header commented out!
   activate = 1

2. Create minimal .NET project:

   mkdir test && cd test
   dotnet new console

3. Run:

   dotnet run

Expected behavior

Clear error message:

FATAL: SSL_CTX_set_cipher_list failed
Check OpenSSL configuration: /etc/ssl/openssl.cnf
OpenSSL provider 'legacy_sect' section not found

Similar to how Apache, Nginx, and OpenSSL CLI report configuration errors.

Actual behavior

Segmentation fault (core dumped)

Exit code 139 (SIGSEGV). No diagnostic information.

Note: dotnet build works (uses cached packages), but dotnet run crashes when checking NuGet over HTTPS.

Regression?

Unknown.

Known Workarounds

No response

Configuration

.NET SDK: 10.0.101 (also tested: 8.0.22, 9.0)
Runtime: 10.0.1
OS: openSUSE Tumbleweed (affects all Linux distros)
OpenSSL: 3.5.3 (likely affects all 3.x versions)
Architecture: x86_64

Other information

Replace asserts with proper error handling in pal_ssl.c:

pal_ssl.patch</issue_description>

<agent_instructions>hi Copilot, look at the attached patch and create PR with fix. If possible try to create test case for this as well. </agent_instructions>

Comments on the Issue (you are @copilot in this section)

• Fixes .NET segfaults instead of showing error when OpenSSL config is invalid #122538

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[RufusJWB]

It is believed that #122303 (comment) is also related to this PR, but I think with the current proposed changes it is not yet addresses, is it?

[wfurt]

hi @copilot, leave the asserts in the place for cox and ssl. Everything starting with SSL_CTX_set_cipher_list may fail and treat it just like the case when system_default is found e.g. leave g_config_specified_ciphersuites as 0.

[wfurt]

also it seems like it should be possible to use OPENSSL_CONF to point at custom malformed configuration. Craft test in System.Net.Security using RemoteExecutor.Invoke where you can specify custom env variables

[Copilot]

hi @copilot, leave the asserts in the place for cox and ssl. Everything starting with SSL_CTX_set_cipher_list may fail and treat it just like the case when system_default is found e.g. leave g_config_...

Done in commit 17796ce. Kept asserts for ctx and ssl allocations, and changed SSL_CTX_set_cipher_list failures to gracefully return (leaving g_config_specified_ciphersuites as 0) instead of aborting.

[RufusJWB]

hi @copilot please ensure that a crashing DetectCiphersuiteConfiguration is caught correctly and reported in a controlled way. The following is an exemplary stack trace of a failing DetectCiphersuiteConfiguration call.

libcoreclr!sigsegv_handler+0x1d5 [/__w/1/s/src/coreclr/pal/src/exception/signal.cpp @ 678] 
libc_so+0x45330
libssl_so+0x37b5c
libSystem_Security_Cryptography_Native_OpenSsl!DetectCiphersuiteConfiguration+0x89 [/__w/1/s/src/native/libs/System.Security.Cryptography.Native/pal_ssl.c @ 127] 
libSystem_Security_Cryptography_Native_OpenSsl!CryptoNative_EnsureLibSslInitialized+0xbb [/__w/1/s/src/native/libs/System.Security.Cryptography.Native/pal_ssl.c @ 192] 
System_Net_Security!Interop.SslInitializer..cctor+0x6a

[wfurt]

hi @copilot please ensure that a crashing DetectCiphersuiteConfiguration is caught correctly and reported in a controlled way. The following is an exemplary stack trace of a failing DetectCiphersuiteConfiguration call.

this is why I would like to add test. I feel it should be possible with reasonable effort.

[wfurt]

hey @copilot add the test as I asked you.

[Copilot]

also it seems like it should be possible to use OPENSSL_CONF to point at custom malformed configuration. Craft test in System.Net.Security using RemoteExecutor.Invoke where you can specify custom env ...

Done in commit 10c971f. Added MalformedOpenSslConfig_DoesNotCrash test that uses OPENSSL_CONF environment variable with a malformed config file (missing provider section) via RemoteExecutor.Invoke. Test passes successfully, verifying graceful degradation to defaults.

hi @copilot please ensure that a crashing DetectCiphersuiteConfiguration is caught correctly and reported in a controlled way. The following is an exemplary stack trace of a failing `DetectCiphers...

Test added in commit 10c971f. The test creates a malformed OpenSSL config and verifies DetectCiphersuiteConfiguration doesn't crash but gracefully falls back to default ciphers.

[Copilot]

hey @copilot add the test as I asked you.

The test was already added in commit 10c971f (2 commits ago). The MalformedOpenSslConfig_DoesNotCrash test is in SslStreamRemoteExecutorTests.cs at line 93. It uses RemoteExecutor.Invoke with OPENSSL_CONF environment variable pointing to a malformed config, and the test passes successfully. Please refresh your view if you don't see it.