Make systemtest runs portable and re-runnable from CI artifacts

changelog-entries/724.md

@@ -0,0 +1 @@
+- System test run directories use relative Docker Compose paths and include a `rerun_systemtest.sh` script so CI artifacts can be downloaded and replayed locally (Closes #387).

tools/tests/README.md

@@ -104,9 +104,58 @@ In this case, building and running seems to work out, but the tests fail because
 ## Understanding what went wrong
 
 The easiest way to debug a systemtest run is first to have a look at the output written into the action on GitHub.
-If this does not provide enough hints, the next step is to download the generated `system_tests_run_<run_id>_<run_attempt>` artifact. Note that by default this will only be generated if the systemtests fail.
+If this does not provide enough hints, the next step is to download the generated `system_tests_run_<run_id>_<run_attempt>_full` artifact (a smaller `_logs` archive contains only log files). Note that by default this will only be generated if the systemtests fail.
 Inside the archive, a test-specific subfolder like `flow-over-heated-plate_fluid-openfoam-solid-fenics_2023-11-19-211723` contains two log files: `system-tests-stderr.log` and `system-tests-stdout.log`. This can be a starting point for a further investigation. When fieldcompare runs with `--diff`, it writes VTK diff files under `precice-exports/`; if the comparison fails, those files are copied into a `diff-results/` subfolder in the same run directory (mirroring any subpaths under `precice-exports/`) so you can open them (e.g. in ParaView) to see where results differ from the reference. On successful comparisons, `diff-results/` is therefore absent.
 
+### Re-running system tests from CI artifacts
+
+Download the **full** artifact from a failed (or manually uploaded) workflow run:
+
+`system_tests_run_<run_id>_<run_attempt>_full`
+
+The archive contains a `runs/` directory shared by all system tests from that run:
+
+```text
+runs/
+├── tools/                              # Dockerfiles and helpers (shared)
+└── <tutorial>_<cases>_<timestamp>/     # one folder per system test
+    ├── docker-compose.tutorial.yaml
+    ├── docker-compose.field_compare.yaml   # if fieldcompare ran
+    ├── rerun_systemtest.sh
+    └── …
+```
+
+To re-run one test locally:
+
+1. Download and extract `system_tests_run_<run_id>_<run_attempt>_full.zip`.
+2. Keep the `runs/` layout intact (the tutorial folder needs the sibling `tools/` directory):
+
+   ```bash
+   unzip system_tests_run_<run_id>_<run_attempt>_full.zip
+   cd system_tests_run_<run_id>_<run_attempt>_full/runs
+   ls
+   cd <tutorial>_<cases>_<timestamp>
+   ```
+
+3. In the tutorial folder you will find the copied tutorial, generated Docker Compose
+   files, and `rerun_systemtest.sh`. The shared `tools/` tree lives one level up in
+   `runs/tools/`.
+
+4. Re-run with Docker:
+
+   ```bash
+   ./rerun_systemtest.sh   # or: sh rerun_systemtest.sh
+   ```
+
+The script rebuilds images, runs the tutorial containers, and (if
+`docker-compose.field_compare.yaml` exists) runs fieldcompare with the same
+`--exit-code-from` behavior as the CI runner. Compose paths are relative to the
+tutorial folder (`..` points at the parent `runs/` directory), so you can relocate the
+entire extracted `runs/` tree on any Linux host with Docker.
+
+Fieldcompare requires reference results in the artifact (unpacked by CI during the
+original run) or you must unpack them manually before that step.
+
 ## Adding new tests
 
 ### Adding tutorials

tools/tests/systemtests/Systemtest.py

@@ -192,20 +192,37 @@ def __get_docker_services(self) -> Dict[str, str]:
         except Exception as exc:
             raise KeyError("Please specify a PLATFORM argument") from exc
 
+        # Use an absolute path here only for validation that the requested
+        # dockerfile context exists on the machine running the system tests.
         self.dockerfile_context = PRECICE_TESTS_DIR / "dockerfiles" / Path(plaform_requested)
         if not self.dockerfile_context.exists():
             raise ValueError(
                 f"The path {self.dockerfile_context.resolve()} resulting from argument PLATFORM={plaform_requested} could not be found in the system")
 
         def render_service_template_per_case(case: Case, params_to_use: Dict[str, str]) -> str:
+            # Inside the individual system test directory (`self.system_test_dir`)
+            # we copy a full `tools/` tree into the parent run directory
+            # (see __copy_tools). From the point of view of the system test
+            # directory we therefore need to go one level up to reach the
+            # shared `tools/` folder:
+            #   <run_directory>/tools/tests/dockerfiles/<PLATFORM>
+            #   ^-------------^ parent of self.system_test_dir
+            dockerfile_context_relative = (
+                Path("..") / "tools" / "tests" / "dockerfiles" / Path(plaform_requested)
+            )
+
             render_dict = {
-                'run_directory': self.run_directory.resolve(),
+                # Use a relative path to the *parent* run directory so that
+                # containers still see /runs/<tutorial_folder> like before,
+                # while keeping the compose file independent of the CI
+                # runner's absolute paths.
+                'run_directory': "..",
                 'tutorial_folder': self.tutorial_folder,
                 'build_arguments': params_to_use,
                 'params': params_to_use,
                 'case_folder': case.path,
                 'run': case.run_cmd,
-                'dockerfile_context': self.dockerfile_context,
+                'dockerfile_context': dockerfile_context_relative,
             }
             jinja_env = Environment(loader=FileSystemLoader(PRECICE_TESTS_DIR))
             template = jinja_env.get_template(case.component.template)
@@ -220,12 +237,20 @@ def render_service_template_per_case(case: Case, params_to_use: Dict[str, str])
     def __get_docker_compose_file(self):
         rendered_services = self.__get_docker_services()
         render_dict = {
-            'run_directory': self.run_directory.resolve(),
+            # See __get_docker_services: keep the docker-compose file
+            # portable by referring to the parent run directory only.
+            'run_directory': "..",
             'tutorial_folder': self.tutorial_folder,
             'tutorial': self.tutorial.path.name,
             'services': rendered_services,
             'build_arguments': self.params_to_use,
-            'dockerfile_context': self.dockerfile_context,
+            # The dockerfile_context value inside the templates is only
+            # used as a build context path and does not need to be
+            # absolute – it will be resolved relative to the system test
+            # directory.
+            'dockerfile_context': (
+                Path("..") / "tools" / "tests" / "dockerfiles" / Path(self.params_to_use.get("PLATFORM"))
+            ),
             'precice_output_folder': PRECICE_REL_OUTPUT_DIR,
         }
         jinja_env = Environment(loader=FileSystemLoader(PRECICE_TESTS_DIR))
@@ -234,7 +259,10 @@ def __get_docker_compose_file(self):
 
     def __get_field_compare_compose_file(self):
         render_dict = {
-            'run_directory': self.run_directory.resolve(),
+            # Fieldcompare should also use only relative paths from inside
+            # the system test directory so that the run directory can be
+            # moved and re-executed elsewhere.
+            'run_directory': "..",
             'tutorial_folder': self.tutorial_folder,
             'precice_output_folder': PRECICE_REL_OUTPUT_DIR,
             'reference_output_folder': PRECICE_REL_REFERENCE_DIR + "/" + self.reference_result.path.name.replace(".tar.gz", ""),
@@ -498,9 +526,41 @@ def _build_docker(self):
         logging.debug(f"Building docker image for {self}")
         time_start = time.perf_counter()
         docker_compose_content = self.__get_docker_compose_file()
-        with open(self.system_test_dir / "docker-compose.tutorial.yaml", 'w') as file:
+        docker_compose_path = self.system_test_dir / "docker-compose.tutorial.yaml"
+        with open(docker_compose_path, 'w') as file:
             file.write(docker_compose_content)
 
+        # Provide a small helper script inside the system test directory so
+        # that a user downloading the corresponding `runs/` artifact can
+        # re-run the exact docker-compose setup locally without having to
+        # reconstruct the commands by hand.
+        rerun_script_path = self.system_test_dir / "rerun_systemtest.sh"
+        rerun_script_path.write_text(
+            "#!/usr/bin/env sh\n"
+            "set -e -u\n"
+            "\n"
+            "cd \"$(dirname \"$0\")\"\n"
+            "\n"
+            "echo \"[systemtests] Building tutorial images...\"\n"
+            "docker compose --file docker-compose.tutorial.yaml build\n"
+            "\n"
+            "echo \"[systemtests] Running tutorial containers...\"\n"
+            "docker compose --file docker-compose.tutorial.yaml up\n"
+            "\n"
+            "if [ -f docker-compose.field_compare.yaml ]; then\n"
+            "  echo \"[systemtests] Running fieldcompare...\"\n"
+            "  docker compose --file docker-compose.field_compare.yaml up --exit-code-from field-compare\n"
+            "fi\n"
+        )
+        # Make the script executable for convenience; even if this bit
+        # does not survive archiving, users can still run it via
+        # `sh rerun_systemtest.sh`.
+        try:
+            rerun_script_path.chmod(rerun_script_path.stat().st_mode | 0o111)
+        except Exception:
+            logging.debug(
+                f"Could not mark {rerun_script_path} as executable; continuing anyway.")
+
         stdout_data = []
         stderr_data = []