Manual/deterministic releasing of interface pointers

[gexgd0419]

I wonder, is it possible to release a COM interface pointer manually by calling Release(), instead of waiting for its __del__ to be called?

The problem of this method is that Release() is still called when __del__ is called.

comtypes/comtypes/_post_coinit/unknwn.py

Lines 272 to 284 in 5c564ec

	def __del__(self, _debug=logger.debug) -> None:
	"Release the COM refcount we own."
	if self:
	# comtypes calls CoUninitialize() when the atexit handlers
	# runs. CoUninitialize() cleans up the COM objects that
	# are still alive. Python COM pointers may still be
	# present but we can no longer call Release() on them -
	# this may give a protection fault. So we need the
	# _com_shutting_down flag.
	#
	if not type(self)._com_shutting_down:
	_debug("Release %s", self)
	self.Release() # type: ignore

__del__() in Python cannot be used to achieve deterministic destruction, like destructors in C++ do. In the docs:

del() can be invoked when arbitrary code is being executed, including from any arbitrary thread.

This would cause some problems if I want a certain COM object to be released at or before a certain time.

Although CPython uses reference counting, so usually the object will be released when the last reference is removed, it isn't necessarily the case in other Python implementations.

A common solution is to use the context manager. Implement __enter__ and __exit__, so that it can be used in a with block, and be automatically released when leaving the with block. Many resource objects, such as files, already support the context manager protocol.

[junkmd]

Hi,

Have you read about the automatic invocation of Release() here?
https://comtypes.readthedocs.io/en/stable/com_interfaces.html#comtypes.IUnknown.Release

I believe it would be easier to move the discussion forward if you could provide a reproducer that actually causes a problem.

[gexgd0419]

I think that a way to suppress the automatic releasing would be helpful.

I found this problem when I was trying to fix a problem in NVDA's SAPI 4 driver, where a certain SAPI 4 voice didn't work. SAPI 4 is an old and obsolete speech technology no longer maintained by Microsoft, and even finding its documentation on the Internet becomes difficult nowadays. SAPI 4 is built on COM, where each voice engine implements some SAPI 4 standard interfaces, and SAPI 4 framework provides COM objects to find the voice engines installed on your system and to use the voice engine through the standard interfaces.

The problem is that SAPI 4 voices are allowed to be single-instance. If a voice mark itself as single-instance, then creating another instance of this voice engine object before releasing the previous one can cause problems. So I fixed that by carefully removing all references to the COM pointers to release the object, before creating the next instance.

But this is still relying on CPython implementation details, where removing the last reference to the pointer will usually call __del__ on it. What if we switch to another Python implementation? Will __del__ still be called in time, or calling gc.collect() to force garbage-collection is needed? Because this is not guaranteed in the language standard.

Also, what Python implementations support comtypes? I tried PyPy, but it fails to import comtypes with cannot import name 'pythonapi' from 'ctypes'.

[junkmd]

This package is designed to be used with CPython.

As you pointed out, the timing of the __del__ call depends on CPython's ctypes.

To get a specific release timing, using a context manager is probably a good idea, as you said.
However, I don't have a clear idea of the code, the use case, or how to test it.

It would be helpful for our discussion if you could share your ideal use case.

[gexgd0419]

I am thinking of introducing some kind of release/close/dispose method that can be used to release the COM object, and more importantly, prevent automatic releasing after that.

For example, IUnknown.Release() could be made to set the pointer to null, or set a flag for no automatic releasing. But this change might break the compatibility with some programs that uses AddRef and Release in their own ways.

Maybe we can add another method to IUnknown, but then the method name, such as "Close" or "Dispose", might collide with some other interfaces, since other interfaces are not forbidden to use method names like that.

Maybe we can add a parameter for IUnknown.Release(), so specifying an additional argument can achieve manual releasing.

Or maybe we can introduce a utility function inside another module/class, like C#'s Marshal.ReleaseComObject(). COM object references in C# cannot be used in using blocks (similar to Python's with blocks for automatic releasing when leaving the block), but you can release the COM object manually by calling Marshal.ReleaseComObject() on it.

Honestly I don't have an ideal use case, as my current code is already working with CPython. I'm just expressing my concern about the lack of a deterministic way to safely release a COM object - surely you can call obj.Release(), but it may cause problems later, as automatic releasing is not disabled, and we cannot disable it without some hacks.

---

I wrote some code for showcasing the COM object life time, not a real use case though:

import os
import tempfile
from comtypes import CoInitialize
from comtypes.stream import ISequentialStream
from ctypes import POINTER, WINFUNCTYPE, HRESULT, windll, byref
from ctypes.wintypes import LPCWSTR, DWORD, BOOL

IStream = ISequentialStream  # doesn't matter for this demo
STGM_READ = 0x00000000
STGM_SHARE_EXCLUSIVE = 0x00000010

SHCreateStreamOnFileEx = WINFUNCTYPE(
    HRESULT,
    LPCWSTR, DWORD, DWORD, BOOL, POINTER(IStream), POINTER(POINTER(IStream))
)(("SHCreateStreamOnFileEx", windll.shlwapi))

def able_to_open(path: str) -> bool:
    try:
        with open(path, "rb"):
            return True
    except Exception:
        return False

def test_stream(path: str, raise_error: bool):
    pStream = POINTER(IStream)()
    SHCreateStreamOnFileEx(path, STGM_READ | STGM_SHARE_EXCLUSIVE, 0, True, None, byref(pStream))
    assert not able_to_open(path)
    if raise_error:
        raise RuntimeError("Oops!")

def main():
    CoInitialize()

    path = tempfile.mktemp()  # get a unique temp file path

    try:
        test_stream(path, raise_error=True)
        assert able_to_open(path)
    except Exception:
        # file still locked during exception handling
        assert not able_to_open(path)
    finally:
        assert able_to_open(path)
        os.remove(path)

if __name__ == "__main__":
    main()

This example uses SHCreateStreamOnFileEx to create an IStream that locks the file exclusively. The file will not be closed until the IStream is released, so it's easier to demonstrate the life time of IStream.

With CPython, everything works as usual. A confusing part is that if an exception is thrown inside test_stream, its stack frame (including the local variable pStream) will be included in the exception object, so the stream will not be released during exception handling.

But with an implementation where garbage collection is not deterministic, all the assert able_to_open(path) may fail, as you cannot ensure that the IStream has been released.

[junkmd]

As you said, the automatic release can be handled by carefully implementing code, but it's certainly a concern in corner cases.

Let's keep this issue open.

We can resume the discussion when someone comes up with a more critical problem or an ideal use case.