System tests: Archive fieldcompare diff files on failure

changelog-entries/441.md

@@ -0,0 +1 @@
+- Archive fieldcompare diff VTK files into a `diff-results/` folder in each systemtest run directory on failure so they are easy to find in CI artifacts when investigating comparison failures (fixes [#441](https://github.com/precice/tutorials/issues/441)). Nested paths under `precice-exports/` are preserved under `diff-results/`.

tools/tests/README.md

@@ -105,7 +105,7 @@ In this case, building and running seems to work out, but the tests fail because
 
 The easiest way to debug a systemtest run is first to have a look at the output written into the action on GitHub.
 If this does not provide enough hints, the next step is to download the generated `system_tests_run_<run_id>_<run_attempt>` artifact. Note that by default this will only be generated if the systemtests fail.
-Inside the archive, a test-specific subfolder like `flow-over-heated-plate_fluid-openfoam-solid-fenics_2023-11-19-211723` contains two log files: a `stderr.log` and `stdout.log`. This can be a starting point for a further investigation.
+Inside the archive, a test-specific subfolder like `flow-over-heated-plate_fluid-openfoam-solid-fenics_2023-11-19-211723` contains two log files: `system-tests-stderr.log` and `system-tests-stdout.log`. This can be a starting point for a further investigation. When fieldcompare runs with `--diff`, it writes VTK diff files under `precice-exports/`; if the comparison fails, those files are copied into a `diff-results/` subfolder in the same run directory (mirroring any subpaths under `precice-exports/`) so you can open them (e.g. in ParaView) to see where results differ from the reference. On successful comparisons, `diff-results/` is therefore absent.
 
 ## Adding new tests
 

tools/tests/systemtests/Systemtest.py

@@ -22,6 +22,8 @@
 GLOBAL_TIMEOUT = int(os.environ.get("PRECICE_SYSTEMTESTS_TIMEOUT", 900))
 SHORT_TIMEOUT = 10
 
+DIFF_RESULTS_DIR = "diff-results"
+
 
 def slugify(value, allow_unicode=False):
     """
@@ -449,6 +451,46 @@ def _run_field_compare(self):
             elapsed_time = time.perf_counter() - time_start
             return FieldCompareResult(1, stdout_data, stderr_data, self, elapsed_time)
 
+    def __archive_fieldcompare_diffs(self) -> None:
+        """
+        Copy fieldcompare diff VTK files from precice-exports/ into diff-results/,
+        preserving paths under precice-exports/ so nested outputs are not skipped
+        and identical basenames in different folders do not overwrite each other.
+        """
+        exports_dir = self.system_test_dir / PRECICE_REL_OUTPUT_DIR
+        if not exports_dir.is_dir():
+            return
+        suffixes = (".vtu", ".vtk", ".vtp")
+        dest_root = self.system_test_dir / DIFF_RESULTS_DIR
+        seen_resolved: set[Path] = set()
+        archived_count = 0
+        for path in exports_dir.rglob("*"):
+            if not path.is_file():
+                continue
+            if path.suffix.lower() not in suffixes:
+                continue
+            if "diff" not in path.name.lower():
+                continue
+            resolved = path.resolve()
+            if resolved in seen_resolved:
+                continue
+            try:
+                rel = path.relative_to(exports_dir)
+            except ValueError:
+                continue
+            seen_resolved.add(resolved)
+            dest_path = dest_root / rel
+            dest_path.parent.mkdir(parents=True, exist_ok=True)
+            shutil.copy2(path, dest_path)
+            archived_count += 1
+        if archived_count:
+            logging.debug(
+                "Archived %d fieldcompare diff file(s) to %s for %s",
+                archived_count,
+                dest_root,
+                self,
+            )
+
     def _build_docker(self):
         """
         Builds the docker image
@@ -626,6 +668,7 @@ def run(self, run_directory: Path):
         std_out.extend(fieldcompare_result.stdout_data)
         std_err.extend(fieldcompare_result.stderr_data)
         if fieldcompare_result.exit_code != 0:
+            self.__archive_fieldcompare_diffs()
             self.__write_logs(std_out, std_err)
             logging.critical(f"Fieldcompare returned non zero exit code, therefore {self} failed")
             return SystemtestResult(