Let a reservoir coupling master spawn its slaves through a wrapper

[hakonhagland]

A reservoir coupling master starts its slaves itself, at runtime, with MPI_Comm_spawn(). That makes a slave hard to debug: a debugger cannot be attached to a process before it exists, and nothing outside the simulator (not the MPI launcher, not gdb's follow-fork-mode) can wrap a process that the MPI runtime spawns on the master's behalf. This PR adds one small hook so that the master can spawn a wrapper executable instead of flow, leaving it to the wrapper to start the simulator plainly or under a debugger.

Add the hidden parameter --rescoup-spawn-wrapper (commit 1)

• New hidden parameter --rescoup-spawn-wrapper=<path>, declared in EclTimeSteppingParams.hpp next to the other reservoir coupling developer flag (--rescoup-sync-at-report-steps), registered and hidden in EclTimeSteppingParams.cpp like --slave.
• When set, ReservoirCouplingSpawnSlaves::spawnSlaveProcesses_() passes that executable to MPI_Comm_spawn() instead of the simulator binary, with the binary path prepended to the slave's usual arguments: <wrapper> <flow> --slave-log-file=<name> <deck> ... --slave=true. The master logs Spawning slave <name> through wrapper <path> at info level, once per slave.
• The wrapper's contract is stated in the code comment: it decides how to start the simulator (typically exec flow "$@" for every rank except the one to debug, and gdb --args flow "$@" for that one — the slave name is in --slave-log-file=…, the rank in OMPI_COMM_WORLD_RANK), and it must keep running as the spawned process until the simulator exits, because the MPI runtime watches the process it started.
• The slaves receive the parameter too, as they receive every other command-line parameter from the master, and ignore it: they never spawn.
• With the parameter unset the code path is unchanged.

Replace a stale TODO about slave output (commit 2)

• The comment above MPI_Comm_spawn() said the handling of the slaves' output was undecided and might need a custom solution. It has been decided for a long time: every slave rank redirects its own stdout and stderr to <slave name>.<rank>.log in Main::initMPI(), from the --slave-log-file argument the master puts first on its command line. The comment now says so and points at Main::maybeRedirectReservoirCouplingSlaveOutput_().

Testing

Tested on a two-slave case (4 + 1 ranks) with a Debug build. With the parameter, rank 2 of the first slave came up under gdb in its own terminal and stopped at a breakpoint in BlackoilWellModel::beginTimeStep() while the coupled run continued and completed; the master log shows both slaves spawned through the wrapper. Without the parameter the same case runs exactly as before. There is nothing to unit-test here without MPI spawning, so the change is covered by the coupled runs only.

[hakonhagland]

jenkins build this please

[hakonhagland]

jenkins build this serial please

[Copilot]

🟢 Approval recommended

No unresolved issues were identified, and the existing spawn path remains unchanged when the option is unset.

Pull request overview

Adds an optional hidden wrapper executable for reservoir-coupling slave processes, enabling debugger integration while preserving existing behavior.

Changes:

• Adds and hides --rescoup-spawn-wrapper.
• Prepends the simulator path to wrapper arguments and logs wrapper usage.
• Updates documentation for slave output redirection.

File summaries

File	Summary
opm/simulators/timestepping/EclTimeSteppingParams.hpp	Declares the wrapper parameter.
opm/simulators/timestepping/EclTimeSteppingParams.cpp	Registers and hides the parameter.
opm/simulators/flow/rescoup/ReservoirCouplingSpawnSlaves.cpp	Implements wrapper-based spawning and updates output documentation.

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[blattms]

Looks good. Maybe be more detailed about how such a wrapper would look in help and comment

[blattms]

I think it would help if you would put some more information about how the wrapper should look here. E.g. from the PR descritpion

[hakonhagland]

Good idea, in b3aaadb the comment now shows the exact command line the wrapper receives, where it finds the slave name and the rank, and a minimal wrapper script that starts one rank of one slave under gdb and every other rank as usual.

[hakonhagland]

@blattms Thanks for the review. I rebased onto master and added b3aaadb which describes the wrapper in both places you mentioned: the help text (shown by --help-all) now states the wrapper's command line, what it must do, and where the slave name and rank come from; the code comment says the same and includes a small example script. The help text is prose only, because the help output reflows it.

[hakonhagland]

jenkins build this please