From c2cc99dcc8c36c770452015f60e6d8eb547cac92 Mon Sep 17 00:00:00 2001 From: Panasun Date: Mon, 21 Sep 2026 15:04:54 +0200 Subject: [PATCH 1/6] Add guide for running Flow in parallel from Python --- python/sphinx_docs/docs/index.rst | 4 + .../sphinx_docs/docs/parallel-in-python.rst | 344 ++++++++++++++++++ 2 files changed, 348 insertions(+) create mode 100644 python/sphinx_docs/docs/parallel-in-python.rst diff --git a/python/sphinx_docs/docs/index.rst b/python/sphinx_docs/docs/index.rst index 87d598f..e5a1e85 100644 --- a/python/sphinx_docs/docs/index.rst +++ b/python/sphinx_docs/docs/index.rst @@ -6,6 +6,9 @@ There are two Python APIs within OPM Flow: - running Flow from Python code using the Python bindings (see :doc:`flow-in-python`) - running a Python script embedded in a simulation (see :doc:`embedded-python`) +The first of these can also be run in parallel across several MPI ranks +(see :doc:`parallel-in-python`). + This site further contains the documentation of: @@ -18,6 +21,7 @@ Contents :maxdepth: 1 flow-in-python + parallel-in-python embedded-python common simulators diff --git a/python/sphinx_docs/docs/parallel-in-python.rst b/python/sphinx_docs/docs/parallel-in-python.rst new file mode 100644 index 0000000..a77261c --- /dev/null +++ b/python/sphinx_docs/docs/parallel-in-python.rst @@ -0,0 +1,344 @@ +Run OPM Flow in parallel from Python +==================================== + +:doc:`flow-in-python` shows how to drive a simulation from a single Python +process. This page covers running the same simulation across several MPI ranks. + +Prerequisites +------------- + +- Flow compiled with Python support **and** MPI enabled, i.e. the cmake flags + ``-DOPM_ENABLE_PYTHON=ON``, ``-DOPM_INSTALL_PYTHON=ON`` and ``-DUSE_MPI=ON``. + See :doc:`flow-in-python` for a sample build script. + +- A graph partitioner available at configure time — Zoltan or ParMETIS — + otherwise the grid cannot be distributed. + +- `mpi4py `_ installed in the same Python + environment as the ``opm`` module. + +- ``PYTHONPATH`` pointing at both build trees, since ``opm-common`` provides + ``opm.io.*`` and ``opm-simulators`` provides ``opm.simulators.*``: + + .. code-block:: bash + + export PYTHONPATH="/python:/python" + + The two merge into one ``opm`` package because neither defines an + ``__init__.py`` at the ``opm`` level — they are PEP 420 namespace packages. + + +What ``mpirun`` actually does +----------------------------- + +.. code-block:: bash + + mpirun -np 4 python3 my_script.py + +This starts **four independent Python interpreters**, not one parallel Python. +Each executes the whole script and is assigned its own MPI rank. Any statement +that is not guarded by a rank check runs four times, and every collective call +must be reached by all ranks or the run blocks forever. + +The simulation is parallel because OPM distributes the grid internally: rank 0 +parses the deck, broadcasts it to the other ranks, the partitioner splits the +grid, and each rank then owns a subset of the cells. + + +A minimal parallel run +---------------------- + +.. code-block:: python + + from mpi4py import MPI # must be imported first: this calls MPI_Init + + from opm.simulators import BlackOilSimulator + + comm = MPI.COMM_WORLD + + sim = BlackOilSimulator(filename="SPE1CASE1.DATA") + + # mpi4py has already initialised MPI, so OPM must not do it again. + sim.setup_mpi(init=False, finalize=False) + + rc = sim.step_init() + if rc != 0: + raise RuntimeError(f"step_init() failed with code {rc}") + + for _ in range(2): + if sim.step() == 0: # 0 means the schedule hit an EXIT keyword + break + + sim.step_cleanup() + comm.barrier() + +Run it with: + +.. code-block:: bash + + mpirun -np 4 python3 my_script.py + +and confirm the rank count in the print file: + +.. code-block:: bash + + grep "Number of MPI processes" SPE1CASE1.PRT + + +Constructing the simulator +-------------------------- + +.. warning:: + + In parallel, only the **filename constructor** works: + + .. code-block:: python + + sim = BlackOilSimulator(filename="SPE1CASE1.DATA") + + The four-argument form documented for serial runs — + ``BlackOilSimulator(deck, state, schedule, summary_config)`` — cannot run on + more than one rank. An ``EclipseState`` built in Python is the *serial* + class, while a parallel run requires ``ParallelEclipseState``, which has no + Python binding. OPM reports:: + + Error: Parallel simulator setup is incorrect as it does not use ParallelEclipseState + + but this is **not** raised as a Python exception; see + :ref:`checking-return-values` below. + +With the filename form, OPM owns the whole pipeline — parsing, broadcast, +partitioning and distribution — so no serial state object ever crosses into the +parallel code path. + + +Initialising MPI +---------------- + +``setup_mpi()`` takes two flags, and both matter when mpi4py is in use: + +.. code-block:: python + + sim.setup_mpi(init=False, finalize=False) + +``init=False`` + ``from mpi4py import MPI`` already called ``MPI_Init``. Letting OPM + initialise MPI a second time is an error. + +``finalize=False`` + Leaves MPI running after the simulator shuts down. With ``finalize=True`` + OPM tears MPI down, and any collective call afterwards — including an + ``allgather`` used for checking results — aborts. + + +.. _checking-return-values: + +Checking return values +---------------------- + +.. warning:: + + ``step_init()`` and ``step()`` use **opposite** success conventions, and + neither raises on failure. Do not reuse one check for the other. + +.. list-table:: + :header-rows: 1 + :widths: 20 20 60 + + * - Method + - Success + - Notes + * - ``step_init()`` + - ``0`` + - Non-zero means setup failed. Nothing is raised. + * - ``step()`` + - ``1`` + - ``0`` means the schedule hit an ``EXIT`` keyword — a normal early stop. + +A failed ``step_init()`` returns a non-zero code and leaves a null simulator +behind. The failure only becomes visible at the *next* call, as a segmentation +fault rather than a Python traceback. Calling ``step_init()`` a second time +reports success, because the internal "already initialised" flag was set +despite the failure. + +Always check the return value: + +.. code-block:: python + + rc = sim.step_init() + if rc != 0: + raise RuntimeError(f"step_init() failed with code {rc}") + +.. note:: + + ``get_dt()`` must not be called before the first ``step()``. It maps to + ``SimulatorTimer::stepLengthTaken()``, which is guarded by an assertion on + the step counter. In a build without ``-DNDEBUG`` this aborts the + interpreter with ``SIGABRT``, which Python cannot catch. + + +Working with distributed arrays +------------------------------- + +After the grid is distributed, each rank holds **only its own cells**, plus an +overlap layer. Array getters return that local array, not the global field. + +For SPE1CASE1 (300 cells) the local lengths are: + +.. list-table:: + :header-rows: 1 + :widths: 20 80 + + * - Ranks + - ``len(sim.get_porosity())`` per rank + * - 1 + - ``[300]`` + * - 2 + - ``[177, 189]`` + * - 4 + - ``[135, 123, 117, 132]`` + +The local lengths sum to more than 300 because overlap cells are counted on +more than one rank. Owned cells sum to exactly 300. + +Setters expect an array of **this rank's** length. The safe pattern is to +derive the new array from the old one, which stays correct at any rank count: + +.. code-block:: python + + poro = sim.get_porosity() + sim.set_porosity(poro * 0.95) # correct on 1, 2 or 4 ranks + +A script that builds a 300-element array from ``DIMENS`` is wrong on every run +with more than one rank. + + +Verifying that the run is really parallel +----------------------------------------- + +A run that does not crash is not evidence of anything — four *independent +serial* runs do not crash either. Two checks distinguish the cases. + +First, the print file should report the rank count: + +.. code-block:: bash + + grep "Number of MPI processes" SPE1CASE1.PRT + +Second, compare values across ranks. Quantities such as the step counter must +agree, while local array lengths must differ: + +.. code-block:: python + + def agree(label, value): + gathered = comm.allgather(value) + same = len({repr(v) for v in gathered}) == 1 + if comm.Get_rank() == 0: + print(f"{'same ' if same else 'DIFFER'} {label:<20} {gathered}") + return gathered + + agree("current_step", sim.current_step()) # expect agreement + agree("len(porosity)", len(sim.get_porosity())) # expect differences + +If the porosity lengths agree at four ranks, the grid was never distributed and +you are running four serial simulations. + + +Complete example +---------------- + +.. code-block:: python + + #!/usr/bin/env python3 + """SPE1CASE1 driven from Python across several MPI ranks. + + mpirun -np 4 python3 spe1case1_parallel.py + """ + + from mpi4py import MPI # must come first: this calls MPI_Init + + from opm.simulators import BlackOilSimulator + + COMM = MPI.COMM_WORLD + RANK = COMM.Get_rank() + SIZE = COMM.Get_size() + + CASE = "SPE1CASE1.DATA" + NSTEPS = 2 + + + def root(msg): + """Print once, from rank 0.""" + if RANK == 0: + print(msg, flush=True) + + + def agree(label, value): + """Report whether every rank produced the same value.""" + gathered = COMM.allgather(value) + same = len({repr(v) for v in gathered}) == 1 + root(f" {'same ' if same else 'DIFFER'} {label:<22} {gathered}") + return gathered + + + def main(): + root(f"SPE1CASE1 on {SIZE} rank(s)") + + sim = BlackOilSimulator(filename=CASE) + sim.setup_mpi(init=False, finalize=False) + + rc = sim.step_init() + if rc != 0: + raise RuntimeError(f"step_init() failed with code {rc} on rank {RANK}") + + agree("current_step", sim.current_step()) + + poro = sim.get_porosity() + lengths = agree("len(porosity)", len(poro)) + if RANK == 0: + print(f" -> {sum(lengths)} local cells across {SIZE} ranks " + f"(300 owned + overlap)", flush=True) + + sim.set_porosity(poro * 0.95) + + for i in range(NSTEPS): + if sim.step() == 0: + root(" EXIT encountered in the schedule -- stopping early") + break + root(f"after step {i}:") + agree("get_dt", sim.get_dt()) + agree("current_step", sim.current_step()) + + sim.step_cleanup() + + # Every rank must reach this, or the others block here forever. + COMM.barrier() + root(f"done -- check 'Number of MPI processes: {SIZE}' in the PRT file") + + + if __name__ == "__main__": + main() + + +Known limitations +----------------- + +.. list-table:: + :header-rows: 1 + :widths: 40 60 + + * - Limitation + - Consequence + * - No Python binding for ``ParallelEclipseState`` + - Decks cannot be built or modified in Python before a parallel run. Use + the filename constructor and edit the deck on disk. + * - ``step_init()`` failures are not raised + - A setup error surfaces as a segmentation fault on the next call. Always + check the return value. + * - Return conventions are inconsistent + - ``step_init()`` returns ``0`` on success, ``step()`` returns ``1``. + * - ``get_dt()`` asserts before the first step + - Aborts the interpreter in builds without ``-DNDEBUG``. + +These apply to ``GasWaterSimulator`` and ``OnePhaseSimulator`` as well, since +the behaviour comes from their shared base class. From 14966022c3fe82f15619d5cd22b93b2e942ca1c9 Mon Sep 17 00:00:00 2001 From: Panasun Date: Mon, 21 Sep 2026 15:54:13 +0200 Subject: [PATCH 2/6] Finalized parallel page --- .../sphinx_docs/docs/parallel-in-python.rst | 305 ++++-------------- 1 file changed, 63 insertions(+), 242 deletions(-) diff --git a/python/sphinx_docs/docs/parallel-in-python.rst b/python/sphinx_docs/docs/parallel-in-python.rst index a77261c..1024b7e 100644 --- a/python/sphinx_docs/docs/parallel-in-python.rst +++ b/python/sphinx_docs/docs/parallel-in-python.rst @@ -7,70 +7,69 @@ process. This page covers running the same simulation across several MPI ranks. Prerequisites ------------- -- Flow compiled with Python support **and** MPI enabled, i.e. the cmake flags - ``-DOPM_ENABLE_PYTHON=ON``, ``-DOPM_INSTALL_PYTHON=ON`` and ``-DUSE_MPI=ON``. - See :doc:`flow-in-python` for a sample build script. +This page assumes you can already run a serial simulation from Python. See +:doc:`flow-in-python` for compiling Flow with Python support and for setting +``PYTHONPATH``. -- A graph partitioner available at configure time — Zoltan or ParMETIS — - otherwise the grid cannot be distributed. +Running in parallel needs three things in addition: -- `mpi4py `_ installed in the same Python - environment as the ``opm`` module. +- **MPI enabled in the build.** Add ``-DUSE_MPI=ON`` to the cmake flags + alongside ``-DOPM_ENABLE_PYTHON=ON`` and ``-DOPM_INSTALL_PYTHON=ON``. -- ``PYTHONPATH`` pointing at both build trees, since ``opm-common`` provides - ``opm.io.*`` and ``opm-simulators`` provides ``opm.simulators.*``: +- **A graph partitioner** present at configure time, either Zoltan or + ParMETIS. Without one, the grid cannot be distributed across ranks. - .. code-block:: bash +- **mpi4py** installed in the same Python environment as the ``opm`` module. + See the `mpi4py documentation `_. - export PYTHONPATH="/python:/python" - The two merge into one ``opm`` package because neither defines an - ``__init__.py`` at the ``opm`` level — they are PEP 420 namespace packages. +A script for parallel run example +---------------------- +.. code-block:: python -What ``mpirun`` actually does ------------------------------ + from opm.simulators import BlackOilSimulator -.. code-block:: bash + # mpi4py owns MPI_Init/MPI_Finalize; importing it initialises MPI for the + # whole process, including the simulator underneath. + from mpi4py import MPI - mpirun -np 4 python3 my_script.py + COMM = MPI.COMM_WORLD + RANK = COMM.Get_rank() -This starts **four independent Python interpreters**, not one parallel Python. -Each executes the whole script and is assigned its own MPI rank. Any statement -that is not guarded by a rank check runs four times, and every collective call -must be reached by all ranks or the run blocks forever. + CASE = "SPE1CASE1.DATA" -The simulation is parallel because OPM distributes the grid internally: rank 0 -parses the deck, broadcasts it to the other ranks, the partitioner splits the -grid, and each rank then owns a subset of the cells. + def main(): + sim = BlackOilSimulator(filename=CASE) -A minimal parallel run ----------------------- + # init=False: MPI is already initialised by mpi4py. + # finalize=False: keep MPI alive until the script exits. + sim.setup_mpi(init=False, finalize=False) -.. code-block:: python + # sim_step_init() return 1 is fail. So we have to check abit + rc = sim.step_init() + if rc != 0: + raise RuntimeError(f"step_init() failed with code {rc} on rank {RANK}") - from mpi4py import MPI # must be imported first: this calls MPI_Init + sim.step() - from opm.simulators import BlackOilSimulator + # The grid is distributed, so each rank sees only its own cells + # (owned + overlap). + poro = sim.get_porosity() + sim.set_porosity(poro * 0.95) - comm = MPI.COMM_WORLD + sim.step() - sim = BlackOilSimulator(filename="SPE1CASE1.DATA") + sim.step_cleanup() - # mpi4py has already initialised MPI, so OPM must not do it again. - sim.setup_mpi(init=False, finalize=False) + if RANK == 0: + print("done -- results written to SPE1CASE1.PRT", flush=True) - rc = sim.step_init() - if rc != 0: - raise RuntimeError(f"step_init() failed with code {rc}") - for _ in range(2): - if sim.step() == 0: # 0 means the schedule hit an EXIT keyword - break + if __name__ == "__main__": + main() - sim.step_cleanup() - comm.barrier() Run it with: @@ -85,35 +84,14 @@ and confirm the rank count in the print file: grep "Number of MPI processes" SPE1CASE1.PRT -Constructing the simulator --------------------------- - -.. warning:: - - In parallel, only the **filename constructor** works: - - .. code-block:: python - - sim = BlackOilSimulator(filename="SPE1CASE1.DATA") - - The four-argument form documented for serial runs — - ``BlackOilSimulator(deck, state, schedule, summary_config)`` — cannot run on - more than one rank. An ``EclipseState`` built in Python is the *serial* - class, while a parallel run requires ``ParallelEclipseState``, which has no - Python binding. OPM reports:: - - Error: Parallel simulator setup is incorrect as it does not use ParallelEclipseState - - but this is **not** raised as a Python exception; see - :ref:`checking-return-values` below. - -With the filename form, OPM owns the whole pipeline — parsing, broadcast, -partitioning and distribution — so no serial state object ever crosses into the -parallel code path. +Good to know +------------ +The rest of this page covers behaviour that is easy to get wrong, +and the reasons behind the recommendations above. Initialising MPI ----------------- +~~~~~~~~~~~~~~~~ ``setup_mpi()`` takes two flags, and both matter when mpi4py is in use: @@ -131,10 +109,27 @@ Initialising MPI ``allgather`` used for checking results — aborts. + +Constructing the simulator +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. warning:: + + In parallel, only the **filename constructor** works: + + .. code-block:: python + + sim = BlackOilSimulator(filename="SPE1CASE1.DATA") + + The four-argument form documented for serial runs — + ``BlackOilSimulator(deck, state, schedule, summary_config)`` — cannot run on + more than one rank. + + .. _checking-return-values: Checking return values ----------------------- +~~~~~~~~~~~~~~~~~~~~~~ .. warning:: @@ -168,177 +163,3 @@ Always check the return value: rc = sim.step_init() if rc != 0: raise RuntimeError(f"step_init() failed with code {rc}") - -.. note:: - - ``get_dt()`` must not be called before the first ``step()``. It maps to - ``SimulatorTimer::stepLengthTaken()``, which is guarded by an assertion on - the step counter. In a build without ``-DNDEBUG`` this aborts the - interpreter with ``SIGABRT``, which Python cannot catch. - - -Working with distributed arrays -------------------------------- - -After the grid is distributed, each rank holds **only its own cells**, plus an -overlap layer. Array getters return that local array, not the global field. - -For SPE1CASE1 (300 cells) the local lengths are: - -.. list-table:: - :header-rows: 1 - :widths: 20 80 - - * - Ranks - - ``len(sim.get_porosity())`` per rank - * - 1 - - ``[300]`` - * - 2 - - ``[177, 189]`` - * - 4 - - ``[135, 123, 117, 132]`` - -The local lengths sum to more than 300 because overlap cells are counted on -more than one rank. Owned cells sum to exactly 300. - -Setters expect an array of **this rank's** length. The safe pattern is to -derive the new array from the old one, which stays correct at any rank count: - -.. code-block:: python - - poro = sim.get_porosity() - sim.set_porosity(poro * 0.95) # correct on 1, 2 or 4 ranks - -A script that builds a 300-element array from ``DIMENS`` is wrong on every run -with more than one rank. - - -Verifying that the run is really parallel ------------------------------------------ - -A run that does not crash is not evidence of anything — four *independent -serial* runs do not crash either. Two checks distinguish the cases. - -First, the print file should report the rank count: - -.. code-block:: bash - - grep "Number of MPI processes" SPE1CASE1.PRT - -Second, compare values across ranks. Quantities such as the step counter must -agree, while local array lengths must differ: - -.. code-block:: python - - def agree(label, value): - gathered = comm.allgather(value) - same = len({repr(v) for v in gathered}) == 1 - if comm.Get_rank() == 0: - print(f"{'same ' if same else 'DIFFER'} {label:<20} {gathered}") - return gathered - - agree("current_step", sim.current_step()) # expect agreement - agree("len(porosity)", len(sim.get_porosity())) # expect differences - -If the porosity lengths agree at four ranks, the grid was never distributed and -you are running four serial simulations. - - -Complete example ----------------- - -.. code-block:: python - - #!/usr/bin/env python3 - """SPE1CASE1 driven from Python across several MPI ranks. - - mpirun -np 4 python3 spe1case1_parallel.py - """ - - from mpi4py import MPI # must come first: this calls MPI_Init - - from opm.simulators import BlackOilSimulator - - COMM = MPI.COMM_WORLD - RANK = COMM.Get_rank() - SIZE = COMM.Get_size() - - CASE = "SPE1CASE1.DATA" - NSTEPS = 2 - - - def root(msg): - """Print once, from rank 0.""" - if RANK == 0: - print(msg, flush=True) - - - def agree(label, value): - """Report whether every rank produced the same value.""" - gathered = COMM.allgather(value) - same = len({repr(v) for v in gathered}) == 1 - root(f" {'same ' if same else 'DIFFER'} {label:<22} {gathered}") - return gathered - - - def main(): - root(f"SPE1CASE1 on {SIZE} rank(s)") - - sim = BlackOilSimulator(filename=CASE) - sim.setup_mpi(init=False, finalize=False) - - rc = sim.step_init() - if rc != 0: - raise RuntimeError(f"step_init() failed with code {rc} on rank {RANK}") - - agree("current_step", sim.current_step()) - - poro = sim.get_porosity() - lengths = agree("len(porosity)", len(poro)) - if RANK == 0: - print(f" -> {sum(lengths)} local cells across {SIZE} ranks " - f"(300 owned + overlap)", flush=True) - - sim.set_porosity(poro * 0.95) - - for i in range(NSTEPS): - if sim.step() == 0: - root(" EXIT encountered in the schedule -- stopping early") - break - root(f"after step {i}:") - agree("get_dt", sim.get_dt()) - agree("current_step", sim.current_step()) - - sim.step_cleanup() - - # Every rank must reach this, or the others block here forever. - COMM.barrier() - root(f"done -- check 'Number of MPI processes: {SIZE}' in the PRT file") - - - if __name__ == "__main__": - main() - - -Known limitations ------------------ - -.. list-table:: - :header-rows: 1 - :widths: 40 60 - - * - Limitation - - Consequence - * - No Python binding for ``ParallelEclipseState`` - - Decks cannot be built or modified in Python before a parallel run. Use - the filename constructor and edit the deck on disk. - * - ``step_init()`` failures are not raised - - A setup error surfaces as a segmentation fault on the next call. Always - check the return value. - * - Return conventions are inconsistent - - ``step_init()`` returns ``0`` on success, ``step()`` returns ``1``. - * - ``get_dt()`` asserts before the first step - - Aborts the interpreter in builds without ``-DNDEBUG``. - -These apply to ``GasWaterSimulator`` and ``OnePhaseSimulator`` as well, since -the behaviour comes from their shared base class. From ae168d933d7217584368548823fb59bc35928dc0 Mon Sep 17 00:00:00 2001 From: Panasun Date: Tue, 22 Sep 2026 10:34:11 +0200 Subject: [PATCH 3/6] Remove the wrong warning massage. --- .../sphinx_docs/docs/parallel-in-python.rst | 38 ------------------- 1 file changed, 38 deletions(-) diff --git a/python/sphinx_docs/docs/parallel-in-python.rst b/python/sphinx_docs/docs/parallel-in-python.rst index 1024b7e..c22d483 100644 --- a/python/sphinx_docs/docs/parallel-in-python.rst +++ b/python/sphinx_docs/docs/parallel-in-python.rst @@ -125,41 +125,3 @@ Constructing the simulator ``BlackOilSimulator(deck, state, schedule, summary_config)`` — cannot run on more than one rank. - -.. _checking-return-values: - -Checking return values -~~~~~~~~~~~~~~~~~~~~~~ - -.. warning:: - - ``step_init()`` and ``step()`` use **opposite** success conventions, and - neither raises on failure. Do not reuse one check for the other. - -.. list-table:: - :header-rows: 1 - :widths: 20 20 60 - - * - Method - - Success - - Notes - * - ``step_init()`` - - ``0`` - - Non-zero means setup failed. Nothing is raised. - * - ``step()`` - - ``1`` - - ``0`` means the schedule hit an ``EXIT`` keyword — a normal early stop. - -A failed ``step_init()`` returns a non-zero code and leaves a null simulator -behind. The failure only becomes visible at the *next* call, as a segmentation -fault rather than a Python traceback. Calling ``step_init()`` a second time -reports success, because the internal "already initialised" flag was set -despite the failure. - -Always check the return value: - -.. code-block:: python - - rc = sim.step_init() - if rc != 0: - raise RuntimeError(f"step_init() failed with code {rc}") From 2bd2a30499f94fad0646c69c7378ac3fa83a47bd Mon Sep 17 00:00:00 2001 From: Panasun Date: Wed, 23 Sep 2026 00:05:07 +0200 Subject: [PATCH 4/6] Fix follow the reviewer comments --- .../sphinx_docs/docs/parallel-in-python.rst | 53 +++++++++++-------- 1 file changed, 31 insertions(+), 22 deletions(-) diff --git a/python/sphinx_docs/docs/parallel-in-python.rst b/python/sphinx_docs/docs/parallel-in-python.rst index c22d483..fff0a2e 100644 --- a/python/sphinx_docs/docs/parallel-in-python.rst +++ b/python/sphinx_docs/docs/parallel-in-python.rst @@ -11,26 +11,35 @@ This page assumes you can already run a serial simulation from Python. See :doc:`flow-in-python` for compiling Flow with Python support and for setting ``PYTHONPATH``. -Running in parallel needs three things in addition: +Running in parallel needs the following in addition: -- **MPI enabled in the build.** Add ``-DUSE_MPI=ON`` to the cmake flags - alongside ``-DOPM_ENABLE_PYTHON=ON`` and ``-DOPM_INSTALL_PYTHON=ON``. +- **MPI available at configure time.** ``USE_MPI`` is ``ON`` by default, so in + practice this just means having an MPI implementation installed where CMake + can find it. -- **A graph partitioner** present at configure time, either Zoltan or - ParMETIS. Without one, the grid cannot be distributed across ranks. +- **A graph partitioner** — Zoltan or METIS/ParMETIS — present at configure + time. The default partitioning method is ``zoltanwell``, so without one of + these libraries you will need ``--partition-method=simple``, which uses + OPM's built-in rectangular partitioning of the Cartesian grid instead. -- **mpi4py** installed in the same Python environment as the ``opm`` module. - See the `mpi4py documentation `_. +- **mpi4py**, *if the script itself needs MPI* — to print from one rank only, + or to combine the per-rank results of ``get_porosity()`` and friends. It is + not needed simply to run in parallel: with the defaults + ``init=True, finalize=True`` OPM initializes MPI itself, and + ``mpirun -np 4 python3 my_script.py`` works with no mpi4py at all. But once + mpi4py is imported, the flag values described under "Initializing MPI" below + become required rather than optional. See the + `mpi4py documentation `_. -A script for parallel run example ----------------------- +An example script for a parallel run +------------------------------------ .. code-block:: python from opm.simulators import BlackOilSimulator - # mpi4py owns MPI_Init/MPI_Finalize; importing it initialises MPI for the + # mpi4py owns MPI_Init/MPI_Finalize; importing it initializes MPI for the # whole process, including the simulator underneath. from mpi4py import MPI @@ -43,14 +52,12 @@ A script for parallel run example def main(): sim = BlackOilSimulator(filename=CASE) - # init=False: MPI is already initialised by mpi4py. + # init=False: MPI is already initialized by mpi4py. # finalize=False: keep MPI alive until the script exits. sim.setup_mpi(init=False, finalize=False) - # sim_step_init() return 1 is fail. So we have to check abit - rc = sim.step_init() - if rc != 0: - raise RuntimeError(f"step_init() failed with code {rc} on rank {RANK}") + # step_init() returns 0 on success and 1 on failure. + sim.step_init() sim.step() @@ -87,10 +94,10 @@ and confirm the rank count in the print file: Good to know ------------ -The rest of this page covers behaviour that is easy to get wrong, +The rest of this page covers behavior that is easy to get wrong, and the reasons behind the recommendations above. -Initialising MPI +Initializing MPI ~~~~~~~~~~~~~~~~ ``setup_mpi()`` takes two flags, and both matter when mpi4py is in use: @@ -101,13 +108,15 @@ Initialising MPI ``init=False`` ``from mpi4py import MPI`` already called ``MPI_Init``. Letting OPM - initialise MPI a second time is an error. + initialize MPI a second time is an error. ``finalize=False`` Leaves MPI running after the simulator shuts down. With ``finalize=True`` OPM tears MPI down, and any collective call afterwards — including an - ``allgather`` used for checking results — aborts. - + ``allgather`` used for checking results — aborts. The teardown happens in + the simulator's destructor, not in ``step_cleanup()``: in the example above + it fires when ``main()`` returns and ``sim`` goes out of scope, so + collectives still work immediately after ``step_cleanup()``. Constructing the simulator @@ -123,5 +132,5 @@ Constructing the simulator The four-argument form documented for serial runs — ``BlackOilSimulator(deck, state, schedule, summary_config)`` — cannot run on - more than one rank. - + more than one rank. It aborts with + ``Parallel simulator setup is incorrect as it does not use ParallelEclipseState``. From 446b84691dcf789f25197a1134eb955bbff76259 Mon Sep 17 00:00:00 2001 From: Panasun Date: Wed, 23 Sep 2026 07:50:47 +0200 Subject: [PATCH 5/6] Address the minor comments --- python/sphinx_docs/docs/parallel-in-python.rst | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/python/sphinx_docs/docs/parallel-in-python.rst b/python/sphinx_docs/docs/parallel-in-python.rst index fff0a2e..4ce4ddc 100644 --- a/python/sphinx_docs/docs/parallel-in-python.rst +++ b/python/sphinx_docs/docs/parallel-in-python.rst @@ -56,7 +56,6 @@ An example script for a parallel run # finalize=False: keep MPI alive until the script exits. sim.setup_mpi(init=False, finalize=False) - # step_init() returns 0 on success and 1 on failure. sim.step_init() sim.step() @@ -78,6 +77,10 @@ An example script for a parallel run main() +``step_init()``, ``step()`` and ``step_cleanup()`` each return a status code: +0 on success, 1 on failure. The example ignores them for brevity; check them +in real scripts. + Run it with: .. code-block:: bash @@ -114,9 +117,10 @@ Initializing MPI Leaves MPI running after the simulator shuts down. With ``finalize=True`` OPM tears MPI down, and any collective call afterwards — including an ``allgather`` used for checking results — aborts. The teardown happens in - the simulator's destructor, not in ``step_cleanup()``: in the example above - it fires when ``main()`` returns and ``sim`` goes out of scope, so - collectives still work immediately after ``step_cleanup()``. + the simulator's destructor, not in ``step_cleanup()``: had the example above + used ``finalize=True``, it would fire when ``main()`` returns and ``sim`` + goes out of scope, so collectives still work immediately after + ``step_cleanup()``. Constructing the simulator @@ -132,5 +136,5 @@ Constructing the simulator The four-argument form documented for serial runs — ``BlackOilSimulator(deck, state, schedule, summary_config)`` — cannot run on - more than one rank. It aborts with + more than one rank. It aborts with ``Parallel simulator setup is incorrect as it does not use ParallelEclipseState``. From 2231c3507c6a8703cfc9104df01c4255d2b8b5aa Mon Sep 17 00:00:00 2001 From: Panasun Date: Wed, 23 Sep 2026 10:35:12 +0200 Subject: [PATCH 6/6] Remove the information about the return value of step_init(), step() and clean_up() --- python/sphinx_docs/docs/parallel-in-python.rst | 4 ---- 1 file changed, 4 deletions(-) diff --git a/python/sphinx_docs/docs/parallel-in-python.rst b/python/sphinx_docs/docs/parallel-in-python.rst index 4ce4ddc..b2a6d73 100644 --- a/python/sphinx_docs/docs/parallel-in-python.rst +++ b/python/sphinx_docs/docs/parallel-in-python.rst @@ -77,10 +77,6 @@ An example script for a parallel run main() -``step_init()``, ``step()`` and ``step_cleanup()`` each return a status code: -0 on success, 1 on failure. The example ignores them for brevity; check them -in real scripts. - Run it with: .. code-block:: bash