Skip to content

cmake: Add GnuTLS and Nettle as an alternative to MbedTLS - #14059

Open
billatarm wants to merge 1 commit into
obsproject:masterfrom
billatarm:add-gnutls-support
Open

billatarm wants to merge 1 commit into
obsproject:masterfrom
billatarm:add-gnutls-support

Conversation

@billatarm

Copy link
Copy Markdown

Description

Give Linux builds the option to choose GnuTLS and nettle as the crypto backend with -DCRYPTO_BACKEND=gnutls. MbedTLS remains the default, but can be explicitly set with -DCRYPTO_BACKEND=mbedtls.

MbedTLS updates have introduced breaking changes in minor releases, leading to issues like #11020. GnuTLS tends not to have this problem.

Upstream librtmp has GnuTLS support, but some minor tweaking was required to get it to build.

Additionally, mbedtls has announced an EOL if no one steps up:
https://lists.trustedfirmware.org/archives/list/mbed-tls@lists.trustedfirmware.org/thread/YPIGCB7WN6GCHOGEGKOJXUYAP7OK527W/#BSRM3JY6JFNWWEHKMY4JQQON5QB3LP5U

Motivation and Context

mbedtls is slated to be removed from Fedora, so having a different crypto backend means we can still package it without bundling in mbedtls.

How Has This Been Tested?

Set up an rtmp server and tested via rtmp, srtmp, and ran the crypto-helpers unit test.

Fedora 44 aarch64 m4 mac pro

Tests mentioned above. This also had a benefit of removing a dependency as gnutls and nettle were already being dragged along by others (along with openssl).

Types of changes

New feature (non-breaking change which adds functionality)

Checklist:

  • I have read the contributing document.
  • My code has been run through clang-format.
  • My code follows the project's style guidelines
  • My code is not on the master branch.
  • My code has been tested.
  • All commit messages are properly formatted and commits squashed where appropriate.
  • I have included updates to all appropriate documentation.

Give Linux builds the option to choose GnuTLS and nettle as the crypto backend
with `-DCRYPTO_BACKEND=gnutls`. MbedTLS remains the default, but can
be explicitly set with `-DCRYPTO_BACKEND=mbedtls`.

MbedTLS updates have introduced breaking changes in minor releases,
leading to issues like obsproject#11020. GnuTLS tends not to have this problem.

Upstream librtmp has GnuTLS support, but some minor tweaking was
required to get it to build.

Signed-off-by: Bill Roberts <bill.roberts@arm.com>
@tytan652

tytan652 commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

#13676 (comment)

Firstly thinking that adding multiple TLS backend would benefit is a not a good thinking:

  • Maintenance cost is ignored
  • Other platform are clearly been ignored, MbedTLS was chosen due to being the simplest to deal with on all platforms

Also opening another PR for other TLS stacks is neither convincing nor helpful.

@billatarm

Copy link
Copy Markdown
Author

#13676 (comment)

Firstly thinking that adding multiple TLS backend would benefit is a not a good thinking:

  • Maintenance cost is ignored
  • Other platform are clearly been ignored, MbedTLS was chosen due to being the simplest to deal with on all platforms

Also opening another PR for other TLS stacks is neither convincing nor helpful.

I discussed this on Discord eons ago, the general census was to float some patches.

  1. mbedtls is likely going away, see the link in my commit message
  2. mbedtls is going away in Fedora, so for the Fedora package, it will need something else.

@tytan652

tytan652 commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

I discussed this on Discord eons ago, the general census was to float some patches.

If it was eons, then it does not matter and we have now PRs trying to had various TLS backend without thinking that "various" is maybe too much.

  1. mbedtls is likely going away, see the link in my commit message

And I saw the OpenSSL one too, and I don't think it is correct to open a second PR when the first didn't get any feedback from OBS maintainers.

  1. mbedtls is going away in Fedora, so for the Fedora package, it will need something else.

This is Fedora's own issue, please do not consider it as a valid argument since OBS Studio only officially support Windows, macOS, Flatpak and Ubuntu.

@Conan-Kudo

Conan-Kudo commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor
  1. mbedtls is going away in Fedora, so for the Fedora package, it will need something else.

This is Fedora's own issue, please do not consider it as a valid argument since OBS Studio only officially support Windows, macOS, Flatpak and Ubuntu.

It should be valid since Fedora is ahead of what you will eventually experience in Ubuntu. And it'll go away everywhere eventually as Arm is ending support for the library they develop.

Comment on lines +19 to +23
elseif(CRYPTO_BACKEND_U STREQUAL "gnutls")
if(NOT OS_LINUX)
message(FATAL_ERROR
"GnuTLS is only supported as a crypto backend on Linux.")
endif()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unless mbedtls is only used on Linux, I don't think this is really an acceptable outcome. The end state would be that mbedtls will get dropped globally, so GnuTLS should be usable globally too.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have ~6 months to move it to another system, I just don't really work on the other supported platforms. AFAIK, gnutls does build and work on those two systems.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh and the other reason, when I started this, mbedtls wasn't announcing EOL, so the initial plan was mbedtls would stay in place for users that want that, and for linux distros they could switch to a different stack.

@Lain-B

Lain-B commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator

This is Fedora's own issue, please do not consider it as a valid argument since OBS Studio only officially support Windows, macOS, Flatpak and Ubuntu.

It is frustrating that we have people out there who are doing the work to try to help get our software working on popular systems we don't officially support, yet we have responses to their submissions saying "we don't officially support your system, so good luck, you're on your own" any time they try to get something merged into upstream.

I get it, maintaining large software is not easy. But surely it's possible to be patient, understanding, supportive, and kind even in cases where it might be difficult for us, or in cases where there are multiple different submissions that all do the essentially same thing.

@billatarm

Copy link
Copy Markdown
Author

I discussed this on Discord eons ago, the general census was to float some patches.

If it was eons, then it does not matter and we have now PRs trying to had various TLS backend without thinking that "various" is maybe too much.

Sorry it took us 6 months to a year to get the PRs out, I was busy working on some kernel stuff at the time, but I don't see how the delay from Discord to now would make it any less relevant. If anything, it is more relevant.

  1. mbedtls is likely going away, see the link in my commit message

And I saw the OpenSSL one too, and I don't think it is correct to open a second PR when the first didn't get any feedback from OBS maintainers.

It got feedback from others that seemed relevant around licensing concerns. So we did a respin.

  1. mbedtls is going away in Fedora, so for the Fedora package, it will need something else.

This is Fedora's own issue, please do not consider it as a valid argument since OBS Studio only officially support Windows, macOS, Flatpak and Ubuntu.

It's an all system problem, whats the long term support plan for obs-studio when mbedtls goes away? Is obs-studio going to maintain a fork like they are doing for librtmp?

We can always just say, vendor it, in the linux distros, but the proper way would be to decompose the dependencies, especially the crypto library.

@tytan652

Copy link
Copy Markdown
Collaborator

whats the long term support plan for obs-studio when mbedtls goes away? Is obs-studio going to maintain a fork like they are doing for librtmp?

I don't know, the discussion around that was not properly started, but in my opinion the notion to support multiple TLS lib only in the Linux path is not the right path at all.

As far as I understood, MbedTLS was picked because both OpenSSL and GnuTLS were not easy to build/handle on Windows and having 3 code path (Windows TLS lib, macOS TLS lib and one Linux TLS lib) was not preferred at the time.

@billatarm

Copy link
Copy Markdown
Author

whats the long term support plan for obs-studio when mbedtls goes away? Is obs-studio going to maintain a fork like they are doing for librtmp?

I don't know, the discussion around that was not properly started, but in my opinion the notion to support multiple TLS lib only in the Linux path is not the right path at all.

This thread is kind of split between the PRs, but my comment to this is here: #13676. Essentially, don't want to support them all, want one or the other.

As far as I understood, MbedTLS was picked because both OpenSSL and GnuTLS were not easy to build/handle on Windows and having 3 code path (Windows TLS lib, macOS TLS lib and one Linux TLS lib) was not preferred at the time.

Yeah and that's fine. I don't think anyone here is saying that was a poor choice, very reasonable. It does leave a lot of feature sets in the crypto stacks off the table. Eg, OpenSSL engines/providers for example.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants