fix: serialize writes of files shared by wheels installed in parallel

[SulimanAbdulrazzaq]

Pull Request Check List

Resolves: #9158

• Added tests for changed code.
• Updated documentation for changed code.

Description

Poetry installs wheels in parallel (installer.parallel = true by default). When two wheels contain the same file, for example the __init__.py of a pkgutil-style namespace package like dbt-core and its adapters in #9158, two threads can write that file at the same time in WheelDestination.write_to_fs(). Both threads open the file with "wb", so their writes interleave and the installed file ends up corrupted.

This change takes a lock for each target path while the file is written. Different files are still written in parallel. Only writes to the same file wait for each other, so the file ends up with the complete content of the wheel that was written last, the same as with a sequential installation.

The issue suggests writing to a temporary file and moving it into place with os.replace(). I tried that first, but on Windows two concurrent os.replace() calls onto the same target fail with PermissionError: [WinError 5] Access is denied. A lock avoids that.

Tests

The new test test_parallel_installation_of_file_contained_in_several_wheels installs two wheels that ship the same file from two threads. The first write pauses after a few bytes so the second installation can write the same file. Without the fix the test fails on main, because the file contains both contents:

E       AssertionError: assert b'# second\n\...st\n# first\n' == b'# second\n'

With the fix it passes, and pytest tests/installation, the full pytest suite, mypy, and ruff pass on Ubuntu (3.10 and 3.12), macOS (3.12) and Windows (3.12).

Without any mocking, a script that installs two wheels sharing a large ns/__init__.py in parallel 300 times gave these results:

Runner	main	This PR
Ubuntu, Python 3.10	294/300 corrupted	0/300
Ubuntu, Python 3.12	297/300 corrupted	0/300
macOS, Python 3.12	276/300 corrupted	0/300
Windows, Python 3.12	53/300 corrupted	0/300

Reproducer script

"""Reproducer for python-poetry/poetry#9158.

Two wheels ship the same file (ns/__init__.py) with different content.
Poetry's installer installs wheels in parallel threads by default
(installer.parallel = true). This installs both wheels concurrently with
WheelInstaller, like the executor does, and checks the resulting file.
"""

from __future__ import annotations

import base64
import hashlib
import sys
import tempfile
import threading
import zipfile

from pathlib import Path

from poetry.installation.wheel_installer import WheelInstaller
from poetry.utils.env import MockEnv


def make_wheel(directory: Path, name: str, content: bytes) -> Path:
    files = {
        "ns/__init__.py": content,
        f"{name}/__init__.py": b"",
        f"{name}-1.0.dist-info/METADATA": (
            f"Metadata-Version: 2.1\nName: {name}\nVersion: 1.0\n".encode()
        ),
        f"{name}-1.0.dist-info/WHEEL": (
            b"Wheel-Version: 1.0\nGenerator: repro\nRoot-Is-Purelib: true\n"
            b"Tag: py3-none-any\n"
        ),
    }
    record = []
    for path, data in files.items():
        digest = base64.urlsafe_b64encode(hashlib.sha256(data).digest()).rstrip(b"=")
        record.append(f"{path},sha256={digest.decode()},{len(data)}")
    record.append(f"{name}-1.0.dist-info/RECORD,,")
    files[f"{name}-1.0.dist-info/RECORD"] = ("\n".join(record) + "\n").encode()

    wheel = directory / f"{name}-1.0-py3-none-any.whl"
    with zipfile.ZipFile(wheel, "w") as zf:
        for path, data in files.items():
            zf.writestr(path, data)
    return wheel


def main(rounds: int) -> int:
    # Both files are valid Python and only differ in a comment, like
    # real-world duplicates. They are big enough that writing them takes a while.
    tail = b"\n__path__ = __import__('pkgutil').extend_path(__path__, __name__)\n"
    content_a = b"# " + b"a" * 3_000_000 + tail
    content_b = b"# " + b"b" * 2_000_000 + tail

    broken = 0
    with tempfile.TemporaryDirectory() as tmp:
        tmp_path = Path(tmp)
        wheel_a = make_wheel(tmp_path, "pkga", content_a)
        wheel_b = make_wheel(tmp_path, "pkgb", content_b)
        for i in range(rounds):
            env = MockEnv(path=tmp_path / f"env{i}")
            barrier = threading.Barrier(2)

            def install(wheel: Path, env: MockEnv = env, barrier: threading.Barrier = barrier) -> None:
                barrier.wait()
                WheelInstaller(env).install(wheel)

            threads = [
                threading.Thread(target=install, args=(w,)) for w in (wheel_a, wheel_b)
            ]
            for t in threads:
                t.start()
            for t in threads:
                t.join()

            result = (Path(env.paths["purelib"]) / "ns" / "__init__.py").read_bytes()
            if result not in (content_a, content_b):
                broken += 1

    print(f"python {sys.version.split()[0]}: {broken}/{rounds} installs left a corrupted ns/__init__.py")
    return 1 if broken else 0


if __name__ == "__main__":
    sys.exit(main(int(sys.argv[1]) if len(sys.argv) > 1 else 200))

[sourcery-ai]

Hey - I've found 1 issue

Prompt for AI Agents

Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="src/poetry/installation/wheel_installer.py" line_range="41" />
<code_context>
+    with _file_locks_lock:
+        lock = _file_locks.get(key)
+        if lock is None:
+            lock = _file_locks[key] = threading.Lock()
+
+    return lock
+
</code_context>
<issue_to_address>
**issue (bug_risk):** Assigning `threading.Lock()` to the `WeakValueDictionary` raises `TypeError` because `_thread.lock` objects cannot be weak-referenced, so every wheel file installation fails when `_file_lock()` creates its first lock.

**Triggers:** When any file is installed in a process where that path does not already have a lock.

**Suggested fix:** Store the locks in a regular dictionary, or wrap each `threading.Lock` in a weak-referenceable holder object.

```suggestion
_file_locks: dict[str, threading.Lock] = {}
```
</issue_to_address>

Sourcery assessment

Needs a human reviewer. 1 finding to address first, and if the per-path locking is wrong, parallel wheel installation could still leave a shared file with interleaved or otherwise corrupted contents. Reverting prevents future races, but an already corrupted file would need to be repaired by reinstalling or rerunning the installation.

Blocking findings: src/poetry/installation/wheel_installer.py:41

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

[kokokoXUY]

The Sourcery inline finding that storing threading.Lock() in WeakValueDictionary necessarily raises TypeError appears incorrect for supported CPython versions. I ran this minimal check on CPython 3.10.19, 3.11.14, 3.12.12, and 3.13.12 (Windows); every version printed True without an exception:

import threading, weakref
locks = weakref.WeakValueDictionary()
lock = threading.Lock()
locks["path"] = lock
print(locks["path"] is lock)

In this patch, the returned lock is also held strongly by the with _file_lock(target_path_str): statement for the whole write, so a weak-value entry cannot disappear while that critical section uses it. This addresses the bot's specific blocking claim; it is not a claim that I have run the full Poetry suite or verified every concurrency case. I would not switch to a permanent dictionary solely to fix that reported TypeError.

Review and cross-version reproduction assisted by Codex.

[radoering]

How does this affect performance? (Fixing an edge case may not be worth it if installing will be much slower in general.)

[SulimanAbdulrazzaq]

The only added per-file cost is one dict lookup and acquiring an uncontended lock around the write. Different paths never wait on each other; only threads writing the very same file are serialized.

I measured it on Windows with Python 3.11:

• _file_lock() plus acquire/release on its own: about 10 µs per file, i.e. about 60 ms for 6,000 files.
• End to end: WheelInstaller installing numpy, pandas, sympy and botocore (6,102 files) in parallel, 4 threads, into a fresh venv, 8 alternating runs each:

	median	min	max
with the lock	16.05 s	15.05 s	18.11 s
without the lock	16.01 s	15.30 s	17.03 s

The difference is within the run-to-run noise.

[SulimanAbdulrazzaq]

@radoering @Secrus when you have a moment, could one of you take a look at this one? The checks are green on the current head.