Memory Leak: Multiple OpenTap.Python code paths create owning PyObject wrappers without disposing them

[YNjanjo]

Several OpenTap.Python code paths create owning pythonnet PyObject wrappers and then drop them without deterministic disposal. In pythonnet, those wrappers keep Python references alive until Dispose() or later finalization, so repeated plan-load, discovery, annotation, and serialization activity can accumulate refcount retention, finalizer backlog, and observable memory growth. ([GitHub]6)

Affected locations in the current dev branch:

• OpenTap.Python/PythonTypeDataWrapper.cs
CreateInstance() does mem.ToPython().AsManagedObject(innerType.Type) without disposing the temporary PyObject. ([GitHub]2)

• OpenTap.Python/PythonPluginProvider.cs
Search() creates multiple wrappers with no Dispose(), including Py.Import("opentap"), Py.Import(moduleName), module.GetAttr("__dict__"), new PyDict(files), new PyType(_item), and pyType.GetAttr("__module__"). This is especially relevant if plugin discovery is re-run in-process. ([GitHub]3)

• OpenTap.Python/Annotations/PyObjectAnnotator.cs and OpenTap.Python/Annotations/PythonEnumValueProperty.cs
  The annotator leaks temporaries (Py.Import, GetAttr, etc.) and also stores enum-member PyObjects inside PythonEnumValueProperty. That retains Python references for at least the annotation lifetime. ([GitHub]4)

• OpenTap.Python/PyObjectSerializer.cs
Serialize() and Deserialize() leak more than just Py.Import("pickle") / Py.Import("codecs"); they also allocate undisposed temporaries from ToPython() and InvokeMethod(...) chains. ([GitHub]5)

Suggested fix direction:

• Use using var for every temporary PyObject / PyType / PyDict that does not intentionally escape the current scope.
• In PythonTypeDataWrapper.CreateInstance(), wrap the temporary conversion object:
  using var pyObj = mem.ToPython(); return pyObj.AsManagedObject(innerType.Type); ([GitHub]2)
• In PythonPluginProvider.Search(), avoid storing disposable wrappers in HashSet<PyType> if a managed identity (Type, module/name tuple, or handle snapshot) is sufficient. ([GitHub]3)
• In PyObjectAnnotator, either keep the retained enum values bounded and explicitly intentional, or replace per-annotation List<PyObject> creation with a safer cached/managed representation. ([GitHub]4)
• In PyObjectSerializer, dispose all non-escaping intermediates; caching pickle / codecs alone is not enough. ([GitHub]5)

Expected behavior:

All temporary pythonnet wrappers should be deterministically released inside the same lexical scope in which they are created. Only deliberately long-lived Python references should escape, and those should be bounded by design rather than recreated on every search/annotation/serialization pass. ([GitHub]6)

[rmadsen-ks]

Hi @YNjanjo, thanks for raising the issue.

While I agree that its best practice to dispose objects as soon as possible, according to pythonnet docs, those objects gets automatically disposed upon GC.

So this should not cause a leak.

There might be observable growth, but are you sure that this is the problem? According to https://forum.opentap.io/t/python-ram-leaks/2667 there is a leak inside pythonnet itself.

[YNjanjo]

Hi @rmadsen-ks,

Thanks for the feedback. It’s definitely possible the root cause lies within Python.NET itself, as you suggested.

Do you have an idea of when this might be looked into?

[rmadsen-ks]

Hi @YNjanjo, we have fixed a memory leak in OpenTAP 9.33.0 related to leaking dynamic members. its possible that this could fix this issue as well. Could you try reproducing the issue with that version installed?