rustc: Support --reproducible option for requesting reproducible output

[petrochenkov]

Follow up to #159675.
Part of rust-lang/compiler-team#1005.
r? @bjorn3

[petrochenkov]

enum OptionKind doesn't currently support optional arguments for top level options (i.e. both --reproducible and --reproducible=yes are supported).
It can be added later if it becomes necessary, but for now I just kept things as is.

View changes since the review

[bjorn3]

Making diagnostics non-reproducible by default is probably fine., but I don't think we should ever make the compiled executable non-reproducible by default.

[Kobzol]

To extend bjorn's remark, I think that a flag named --reproducible is misleading, because it sounds like it enables some mode that will actually enable full determinism, while we attempt (but do not always succeed) to have reproducible output even by default without the parallel frontend, and this flag does not fundamentally change that idea.

Maybe the flag should be called --reproducible-diagnostics, or something like that, but then it would only have to affect diagnostics.. :)

The MCP shared a list of issues, some of which were said to produce irreproducible binary output. Have you looked more in detail into what causes these reproducibility issues, and whether there is really no way around that? There might be things that are not that important, e.g. alloc IDs (#154278), that perhaps could stay irreproducible, as long as the final binary output stays the same.

[petrochenkov]

The flag is definitely not just about diagnostics, but about binaries too.
I don't think we'll be able to make the binaries reproducible in parallel mode in reasonable time, based on the previous triage of the issues. But I can check again.
As for the default, it will be most likely be reproducible-by-default, and people will have to explicitly add reproducible = false to their cargo configs to benefit from the parallelism.
I want to make it non-reproducible by default during the 3 month testing period on nightly though, otherwise we won't be able to really test anything.

[Kobzol]

I don't think we'll be able to make the binaries reproducible in parallel mode in reasonable time, based on the previous triage of the issues. But I can check again.

I know that the query design of rustc is rather unique, but if the only way to make frontend parallelization for rustc work is to make the compiler output non-deterministic/irreproducible kind of by design (of course there can always be edge cases, some paths being included in the binary by accident, bugs, etc.), then that is IMO a reason to take a step back and re-evaluate the used approach. That is why I was asking if the irreproducibility is really fundamental to the current design, or if it's "just" a matter of stomping out a set of bugs that are resolvable.

As for the default, it will be most likely be reproducible-by-default, and people will have to explicitly add reproducible = false to their cargo configs to benefit from the parallelism.

While I'm very excited for the parallel frontend to land, if it is not enabled by default, then IMO it would just be stabilization for stabilization's sake, and it won't have a large effect across the ecosystem. The default part is what's important, people can already opt in today via the unstable flag. If people have to explicitly opt in to get the build time wins, and choose reproducibility vs compiler speed-up, then that's not really a great outcome. Not to mention that stabilizing a Cargo config option might take some time.. x)

[petrochenkov]

I don't think we'll be able to make the binaries reproducible in parallel mode in reasonable time, based on the previous triage of the issues. But I can check again.

I know that the query design of rustc is rather unique, but if the only way to make frontend parallelization for rustc work is to make the compiler output non-deterministic/irreproducible kind of by design (of course there can always be edge cases, some paths being included in the binary by accident, bugs, etc.), then that is IMO a reason to take a step back and re-evaluate the used approach. That is why I was asking if the irreproducibility is really fundamental to the current design, or if it's "just" a matter of stomping out a set of bugs that are resolvable.

The MCP rust-lang/compiler-team#1005 is written and I'm acting in assumption that the irreproducibility of the binaries is indeed fundamental, but also that the reproducibility doesn't matter in many common cases, and it would be a disservice to users to block the stabilization on it, or delay it for an indeterminate number of years.

Once the parallel frontend is enabled on nightly I'll have time to debug the binary reproducibility issues and figure out how exactly fundamental they are, maybe some of them are actually fixable. I certainly agree that it would be good to fix them, just don't think that it should be a condition for stabilization.

As for the default, it will be most likely be reproducible-by-default, and people will have to explicitly add reproducible = false to their cargo configs to benefit from the parallelism.

While I'm very excited for the parallel frontend to land, if it is not enabled by default, then IMO it would just be stabilization for stabilization's sake, and it won't have a large effect across the ecosystem. The default part is what's important

This is a strange argument, considering that most of stabilized compiler and language feature do not do anything unless explicitly used.
As for the parallel builds specifically, cc-rs, for example, has a feature enabling parallel builds. The feature is opt-in, but it is much better than if the feature didn't exist at all.
Similarly, make or cmake build support -j, but you need to pass it explicitly to enable parallel builds. Now imagine if they couldn't run parallel builds at all.
So I disagree with this position pretty strongly.

people can already opt in today via the unstable flag.

Yes, and my goal is for them to opt in via a stable cargo config, or a stable rustc option in the interim.
If that happens, then maybe we'll get funding to continue working on reproducibility and switch some defaults in a couple of years.

and choose reproducibility vs compiler speed-up

I'm not actually sure that nonreproducible diagnostics are acceptable by default.
That's something I hope to figure out during the 3 month testing period, during which we'd collect various cases in which people had to add --reproducible to their builds.

That's why I think maybe it makes sense to implement the option as --reproducible="binaries"/--reproducible="diagnostics" right away, even if they do the same thing at the moment. But at least the intent will be more clear from the uses.

[Kobzol]

As for the parallel builds specifically, cc-rs, for example, has a feature enabling parallel builds. The feature is opt-in, but it is much better than if the feature didn't exist at all.
Similarly, make or cmake build support -j, but you need to pass it explicitly to enable parallel builds. Now imagine if they couldn't run parallel builds at all.

Well that situation isn't great, right? It should be enabled by default, unless everyone has to remember to set the flag correctly. Not to mention that make -j doesn't do what one would expect and is a footgun. If cargo build was serial by default, then that would be quite bad for Rust compile times :) And I also don't think that make/cmake -j usually makes the output be not reproducible (?)

This PR doesn't stabilize anything, so I won't block it or something, I just wanted to understand how bad are the reproducibility issues. Seems that it is pretty bad, based on what you wrote.

Purely for practical reasons, since the goal here is to test stuff in the wild on nightly, I'm wondering whether having a --reproducible flag is the best way to achieve that:

• People could already use --frontend-jobs=1 or -Zthreads=1 (or sync, I forgot which has the same effect), right? So why add another flag? The chance that this flag survives onto stabilization in exactly the same shape is IMO rather low, and even if it did, people would still need to at least remove -Zunstable-options to switch from the nightly to stable form anyway.
• If we put out a blog post that says that opting out of the frontend is done through a flag called --reproducible, then I worry that some people will be more scared of the current implementation than they should be (at least according to what you said; that you consider the issues to be somewhat mild). Essentially, I'm thinking about people saying "Oh, for sure I want my code to be reproducible, yes please", and enabling the flag because they think that the parallel frontend will break their programs (which we hope it wouldn't). That being said, since they would have to opt out, I don't expect that too many people would do it, but it is something to consider, IMO.

[petrochenkov]

So why add another flag?

Ideally, I'd want to understand better why people have to disable parallel frontend.
If the announcement says ...

• Use --reproducible="diagnostics" if you need reproducible diagnostics for some kind of testing, at cost of some reduced parallelism.
• Use --reproducible="binaries" if you need reproducible binaries for some external caching or publishing, at cost of some reduced parallelism.
• Use --jobs-frontend=<small-number> if you literally have resource consumption issues, like OOMs or youtube lagging when you watch it in parallel with the builds.

... then the people will follow the recommendations, more or less, and we'll be able to collect better results by text search rather than relying on people bothering to report the reasons to us.

Various rust-lang/rust test suites is the first place where I was going to use this distinction.

The chance that this flag survives onto stabilization in exactly the same shape is IMO rather low

I hope that it survives, because --reproducible=false should be useful for optimizing the compiler even in single-threaded scenarios.

[bjorn3]

Use --reproducible="binaries" if you need reproducible binaries for some external caching or publishing, at cost of some reduced parallelism.

If this wouldn't be the default, we did break basically every distro that tries to do reproducible builds instantly as well as Bazel, Buck2, ... And we did make verifying binaries for releases for everyone who hasn't added --reproducible=binaries impossible.

Also somewhat selfishly, it would break the release process for sudo-rs. (the release tag message contains hashes for locally built binaries and CI checks that it reproduces those exact hashes before creating the release)

[petrochenkov]

Use --reproducible="binaries" if you need reproducible binaries for some external caching or publishing, at cost of some reduced parallelism.

If this wouldn't be the default, we did break basically every distro that tries to do reproducible builds instantly as well as Bazel, Buck2, ... And we did make verifying binaries for releases for everyone who hasn't added --reproducible=binaries impossible.

Also somewhat selfishly, it would break the release process for sudo-rs. (the release tag message contains hashes for locally built binaries and CI checks that it reproduces those exact hashes before creating the release)

Do you mean we won't be able to start the 3 month nightly testing as described in the MCP?
How many of those users are using nightly and are not willing to deal with the nightly breakage?

---

A bit of archeology - back when the parallel compiler was done in separate builds (#[cfg(parallel_compiler)]), it enabled thread::available_parallelism threads by default since #57948 (Jan 28, 2019) until #64722 (Sep 24, 2019).

[rustbot]

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

[petrochenkov]

Updated the option to accept none, binaries, diagnostics or binaries,diagnostics (or diagnostics,binaries) as values, with the motivation in #160448 (comment).
I'd still like to merge this to continue the experiments as described in the MCP.