Fix compatibility with UnifiedPush/tink

[p1gp1g]

Bring back openssl3 support and fix encryption

[ClearlyClaire]

Hi! Thank you for your contribution.

First, can you rebase this PR? main should have the OpenSSL 3 fixes already.

I'm not very knowledgeable about RFC8291 and aes128gcm, but my understanding is that the current code is already correct, and your changes amount to:

• checking message size before encryption for earlier error handling
• removing an unnecessary byte of padding after the padding delimiter
• choosing a fixed value of 4096 for the record size

[p1gp1g]

Hi! Thank you for your contribution.

First, can you rebase this PR? main should have the OpenSSL 3 fixes already.

I've rebased the branch

I'm not very knowledgeable about RFC8291 and aes128gcm, but my understanding is that the current code is already correct, and your changes amount to:

* checking message size before encryption for earlier error handling

* removing an unnecessary byte of padding after the padding delimiter

* choosing a fixed value of 4096 for the record size

It is currently not possible to decrypt push messages with other libs, the error may come from that extra padding or from a wrong RS.

Removing that extra padding and using the fixed RS fixes decryption with other libs (tested with tink used by UnifiedPush lib), and also allow testing against the RFC example (that is what I've implemented in mastodon PR)

[ClearlyClaire]

Right, so this PR changes various parameters to use the particular ones used in the RFC example to be able to use it as a verifiable example.

As for tink failing to process our messages without changes, I was not aware of that and will look into it, as it sounds like an issue with tink.

[p1gp1g]

I haven't check if there were an issue on tink side

[ClearlyClaire]

Looking at tink's code, I believe the extra padding should be fine:
https://github.com/tink-crypto/tink-java-apps/blob/4295187cb1c8b05d01146c1a6b5fb45d47ed7fd5/webpush/src/main/java/com/google/crypto/tink/apps/webpush/WebPushHybridDecrypt.java#L271-L278

However, if my understanding is correct, the record size may be the issue, as it is set to 4096 unless explicitly defined otherwise:
https://github.com/tink-crypto/tink-java-apps/blob/4295187cb1c8b05d01146c1a6b5fb45d47ed7fd5/webpush/src/main/java/com/google/crypto/tink/apps/webpush/WebPushHybridDecrypt.java#L135
https://github.com/tink-crypto/tink-java-apps/blob/4295187cb1c8b05d01146c1a6b5fb45d47ed7fd5/webpush/src/main/java/com/google/crypto/tink/apps/webpush/WebPushHybridDecrypt.java#L148-L151

And checked against what is read from the record:
https://github.com/tink-crypto/tink-java-apps/blob/4295187cb1c8b05d01146c1a6b5fb45d47ed7fd5/webpush/src/main/java/com/google/crypto/tink/apps/webpush/WebPushHybridDecrypt.java#L230-L235

https://codeberg.org/UnifiedPush/android-connector does not set the recordSize, so it would default to 4096 and fail the validation.

So my belief is that our current code is correct and UnifiedPush/tink fails to decode it because of an superfluous check.

[p1gp1g]

Maybe the decryption was correct and the error come from that RS indeed 👍

The extra padding could let message with exact length of 3993 to fail even if it could succeed

[p1gp1g]

@ClearlyClaire : There was indeed a bug in tink: tink-crypto/tink-java-apps#5