[connor-cihula]

Description

Give Linux builds the option to choose OpenSSL as the crypto backend
with -DCRYPTO_BACKEND=openssl. MbedTLS remains the default, but can
be explicitly set with -DCRYPTO_BACKEND=mbedtls.

Upstream librtmp added OpenSSL RTMPS support in commit 8251927. This is
a backport from upstream, along with modifications to use non-deprecated
OpenSSL APIs, per-RTMP TLS state, certificate verification, and frontend
signature verification.

Motivation and Context

MbedTLS updates have introduced breaking changes in minor releases, leading to issues like #11020. OpenSSL tends not to have this problem.

How Has This Been Tested?

Tested on Fedora 44 ARM64.
Each following test was executed building against both MbedTLS and OpenSSL. Each one functions identically, regardless of which backend was used.

• Build and install, verify the correct library is used with realdelf.
• Stream RTMP with ffmpeg. Verifies RTMPS changes left RTMP functional.
• Stream RTMPS with ffmpeg and socat successfully.
• Fail RTMPS streaming with invalid SSL certificate when certificate is untrusted.
• Fail RTMPS streaming with invalid SSL certificate when trusted certificate’s hostname mismatches.
• VerifySignature() accepts a valid RSA/SHA256 signature.
• VerifySignature() rejects altered content, signature, and public key.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)

Checklist:

• I have read the contributing document.
• My code has been run through clang-format.
• My code follows the project's style guidelines
• My code is not on the master branch.
• My code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[connor-cihula]

Thanks for reviewing @loqs, good catches.

[Conan-Kudo]

Note that downstream in Fedora, I explicitly indicated OpenSSL is a bad fit due to the high chance of collisions between OpenSSL, BoringSSL, and other OpenSSL forks embedded in libraries (such as with CEF). I have explicitly attempted to avoid OpenSSL as a dependency in the OBS Studio direct dependency chain for this reason.

If things turn out to be fine, that would be nice, but I'm very cautious about that. I have recommended previously to use GnuTLS or something else that is equally validated but doesn't have these problems.

[billatarm]

Note that downstream in Fedora, I explicitly indicated OpenSSL is a bad fit due to the high chance of collisions between OpenSSL, BoringSSL, and other OpenSSL forks embedded in libraries (such as with CEF). I have explicitly attempted to avoid OpenSSL as a dependency in the OBS Studio direct dependency chain for this reason.

If things turn out to be fine, that would be nice, but I'm very cautious about that. I have recommended previously to use GnuTLS or something else that is equally validated but doesn't have these problems.

So libcrypto is already pulled into the process via libcurl, we specifically chose this to avoid a dependency, so the concerns around it breaking seem incorrect unless im missing something.

lddtree `which obs`
obs => /usr/bin/obs (interpreter => /lib/ld-linux-aarch64.so.1)
libcurl.so.4 => /lib64/libcurl.so.4
<snip>
libcrypto.so.3 => /lib64/libcrypto.so.3
...

[niklaskorz]

This PR is becoming more relevant as mbedtls will possibly stop maintenance in April 2027: https://lists.trustedfirmware.org/archives/list/mbed-tls@lists.trustedfirmware.org/thread/YPIGCB7WN6GCHOGEGKOJXUYAP7OK527W/#BSRM3JY6JFNWWEHKMY4JQQON5QB3LP5U

[billatarm]

And we did not stop working on it. Folks asked us to switch it to GnuTLS, which I have working locally. I am finishing the patch up and we should have it up for review shortly. Thanks.

[billatarm]

gnutls/nettle PR here: #14059

[tytan652]

Firstly thinking that adding multiple TLS backend would benefit is a not a good thinking:

• Maintenance cost is ignored
• Other platform are clearly been ignored, MbedTLS was chosen due to being the simplest to deal with on all platforms

Also opening another PR for other TLS stacks is neither convincing nor helpful.

[tytan652]

This PR is becoming more relevant as mbedtls will possibly stop maintenance in April 2027: https://lists.trustedfirmware.org/archives/list/mbed-tls@lists.trustedfirmware.org/thread/YPIGCB7WN6GCHOGEGKOJXUYAP7OK527W/#BSRM3JY6JFNWWEHKMY4JQQON5QB3LP5U

For the "relevant" part, no.
If the EOL is realized, then yes OBS Project may have pick a new "correct" multi-platform TLS backend.
But a PRs for adding backends is not the correct first step due to the maintenance cost.

[billatarm]

For the "relevant" part, no. If the EOL is realized, then yes OBS Project may have pick a new "correct" multi-platform TLS backend. But a PRs for adding backends is not the correct first step due to the maintenance cost.

Really? As someone who maintains tpm2-software project, supporting different crypto backends has been pretty simple, and lets the project scale easily to different environments. The crypto heavy lifting is in rtmp, which already has support for 3 different backends, this is merely utilizing that. The only obs-specific crypto code is signature verification which is pretty trivial to do with almost any crypto stack. So I don't really see the maintenance cost, and if the project wants, I can help support it.

[tytan652]

supporting different crypto backends has been pretty simple, and lets the project scale easily to different environments

Being simple from your point of view does not always mean that it should be considered, OBS Studio is a complex app and I don't think expanding CI to test if the project builds for every supported TLS lib is an idea to consider.

There is also all of our bundled dependencies using MbedTLS on Windows and macOS that needs to be considered too.

So trying to support and bundle only one TLS library is simpler.

The crypto heavy lifting is in rtmp, which already has support for 3 different backends, this is merely utilizing that.

This part of the fork has never been maintained, so I consider that this librtmp has only proper support for MbedTLS.

[billatarm]

supporting different crypto backends has been pretty simple, and lets the project scale easily to different environments

Being simple from your point of view does not always mean that it should be considered, OBS Studio is a complex app and I don't think expanding CI to test if the project builds for every supported TLS lib is an idea to consider.

It doesn't need too support every TLS stack[1], but having two options for different environments would be nice. Especially when one of those futures is uncertain.

1. I can see the confusion here. I am not proposing both OpenSSL and GnuTLS PRs get merged, merely one or the other.

There is also all of our bundled dependencies using MbedTLS on Windows and macOS that needs to be considered too.

So trying to support and bundle only one TLS library is simpler.

The crypto heavy lifting is in rtmp, which already has support for 3 different backends, this is merely utilizing that.

This part of the fork has never been maintained, so I consider that this librtmp has only proper support for MbedTLS.

See our various patches, that's the maintenance. I have also offered to help support the gnutls backend to relieve any perceived maintenance burden. This would only affect users that opt into the different backend, which is mostly going to be all the linux distributions that package this.

So if we/you do decide to have another backend supported, If it's openssl, let me re-spin the CMake files, after doing the gnutls CMake files, I prefer what I did over there.

I will be away for about 10 days, so please don't perceive my silence as ignoring you. I would love to continue this discussion on my return and see what other comments folks leave. Thanks.

[Fenrirthviti]

Appreciate the insight and context. We're considering opening a discussion post for this topic, but as it's not critical right now that we make a choice, and not even clear what the future of mbedtls is going to be, we can leave this PR (and the other, though it being Linux-only is probably a non-starter for something we'd want to accept support for) open for now.

I'll ask for patience as we work through our initial thoughts.