Version

5.8.0

Description

To compute the maximum permissible cleartext length ("fragment size") for a single DTLS datagram, wolfssl does a sort of "dry run" of building the encrypted to see how long the encrypted packet will be, then takes the difference between the dry run built length and the original cleartext length, and subtracts that from the configured MTU to compute the maximum "fragment size"

However, what I don't understand is why here https://github.com/wolfSSL/wolfssl/blob/6c8ab11f5f691feeb70fc77f8c0050db7f3dfbe6/src/internal.c#L42415 the DTLS_HANDSHAKE_HEADER_SZ (which is 12) is added to the length of the cleartext message before doing the "dry run" encryption. As a result, the computed fragment size is 12 bytes smaller than it needs to be (for non-handshake messages, at least)

For example, in my application, I set the DTLS MTU to 1472 (the typical 1500 - ipv4 header - udp header), and on actual encrypted non-handshake messages, the DTLS overhead including AEAD authentication tag and everything is 22 bytes (I observe this in Wireshark and it agrees with https://dtls.xargs.org), so I should be able to send packets up to 1472-22=1450 bytes. However, wolfssl won't let me send a payload over 1438 bytes because it's adding the extra 12 for DTLS_HANDSHAKE_HEADER_SZ

I'm not a DTLS protocol expert so maybe this behavior is intentional (maybe sometimes a handshake header has to be applied even to application data packets?)

I'm happy to create a concrete program that demonstrates this limit if my report is not clear enough.