Fix the failure of the test_cleanse.py

[pwalczysko]

What this PR does

The failing test is https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-test-integration/11/testReport/junit/OmeroPy.test.integration.clitest.test_cleanse/TestCleanseFullAdmin/testCleanseNonsenseName/
This PR is removing a line in the test_cleanse.py.

The assertion in that line which checks the existence of a path to a logfile which should have been deleted is not possible on NFS, because:

As the Blitz process is still holding the logfile open, on NFS this file cannot be deleted by NFS although the cleanse script deletes it - it leaves a hidden file behind which the NFS renames to ...nfs.

Testing this PR

Check that the Jenkins build testintegration job is green.

Related reading

See #6461

cc @joshmoore @jburel @sbesson

[joshmoore]

Thanks, @pwalczysko. No immediate objections from my side.

[sbesson]

Do you have any idea on what keeps a reference to the file? In a standard import workflow, I would expect all file handles to be released on completion. Is this another example of latency due to the migration of the infrastructure (i.e. would a delay also allow the test to pass.

[pwalczysko]

Do you have any idea on what keeps a reference to the file?

Blitz, I know that. More precisely dont know, did not go deeper.

(i.e. would a delay also allow the test to pass.

Delay did not work. I did not try polling though, just a plain wait. It was longer than 30 sec afai can remember and the process still did not release the file.

[pwalczysko]

Do you have any idea on what keeps a reference to the file?

new investigation. Definitely a java Blitz process. It is not actively doing something. The leak is not worsening over time. Sometimes the .nfs file gets deleted after a minutes of wait. Sometimes not.

The cmds below shows the file descriptors currently open by process 1114099.

ls -l /proc/1114099/fd | grep .nfs0000008384567c850000160d
l-wx------. 1 omero omero 64 Aug  7 19:22 244 -> /home/omero/workspace/OMERO-test-integration/data/ManagedRepository/c1d4022e-2bcb-47af-a99f-7bb37c1826cf_4813/2026-08/07/.nfs0000008384567c850000160d


...
ps -fp 1114099
UID          PID    PPID  C STIME TTY          TIME CMD
omero    1114099 1114059 20 14:03 ?        01:11:19 java -Xmx9600m --add-opens java.base/java.lang=ALL-UNNAMED --add-exports java.naming/com.sun.jndi.ldap=ALL-UNNAMED -Djava.awt.headless=tru
(.venv3) bash-5.1$ 


jcmd 1114099 VM.command_line | grep blitz.  # has a very long output below, which shows it is blitz
java_command: ome.services.blitz.Entry --Ice.Config=/home/omero/workspace/OMERO-test-integration/src/dist/var/master/servers/Blitz-0/config/config
java_class_path (initial): lib/server/proj4j.jar:lib/server/calcite-linq4j.jar:lib/server/minio.jar:lib/server/commons-lang3.jar:lib/server/ini4j.jar:lib/server/logback-core.jar:lib/server/spring-expression.jar:lib/server/httpcl.....


ls /proc/1114099/fd | wc -l
272
(.venv3) bash-5.1$ lsof -p 1114099 | grep -c nfs.  # at the time of investigation, Blitz had 19 open files (not supposed to do anything though
19

lsof -p 1114099 | grep nfs | awk '{print $9}' | sort | uniq -c
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs000000838452795f000015d0
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs0000008384527962000015ca
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs000000838452796f000015d9
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs0000008384527975000015c8
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs0000008384527977000015ce
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs0000008384527993000015d1
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs000000838452799b000015cc
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs000000838452799d000015d3
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279a0000015d7
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279a2000015d6
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279c0000015d2
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279c2000015cd
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279c5000015c9
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279c6000015cf
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279c7000015d4
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279c8000015cb
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279ca000015d8
      1 /home/omero/workspace/OMERO-test-integration/data/FullText/.nfs00000083845279cb000015d5
      1 /home/omero/workspace/OMERO-test-integration/data/ManagedRepository/c1d4022e-2bcb-47af-a99f-7bb37c1826cf_4813/2026-08/07/.nfs0000008384567c850000160d

I am afraid I am not getting forward. I am told that we need to enable the enable Java file descriptor tracing. I should restart the java process with -Djava.security.debug=access but we are talking here about Blitz in OMERO server... no idea how to do that. Will leave it there, this is, I can continue after AL if please told how to configure the server with such option.

[sbesson]

Thanks for the debugging. It sounds like the file handle is held by the Indexer service. This is consistent with the observation that integration tests have been sporadically failing due to the Indexer being slow - see https://merge-ci.openmicroscopy.org/jenkins/job/OMERO-test-integration/56/testngreports/integration/LightAdminRolesTest/testIndexObjectbyRestrictedSystemUser/

Maybe let's update the comment to mention this finding and then we should get this merged. I don't think this points at a core issue but another piece of evidence that there are caveats with running the binary repository on NFS

[pwalczysko]

As discussed with @sbesson , the jury is out on this one. It stably passes with the fix in this PR, but not optimal and possibly underresearched. Will leave open for now.