Skip to content

fix: keep the root container off app.conf so spawn children start - #42

Merged
lesnik512 merged 1 commit into
mainfrom
fix/container-off-app-conf
Oct 17, 2026
Merged

lesnik512 merged 1 commit into
mainfrom
fix/container-off-app-conf

Conversation

@lesnik512

Copy link
Copy Markdown
Member

Closes #41

Problem

setup_di stored the root container in app.conf. Under the spawn start method (Windows, or forced), billiard pickles the app, conf.changes included, for every pool child. Container is not picklable, so the prefork pool failed to start with cannot pickle '_thread._local' object. Fork, the default on macOS and Linux, never pickles the app, so it was unaffected.

Change

  • setup_di registers the container in a module-level weakref.WeakKeyDictionary keyed by the app object. Nothing DI-related goes into app.conf, so pickle.dumps(app) works.
  • fetch_di_container and @inject resolve the current_app proxy with _get_current_object() before the lookup, since the proxy can't be weakly referenced.
  • The worker_process_init handler opens its container as before, then binds it to the process's current app when that app's main equals the main of the app it was set up for. A spawn child makes the unpickled app current and re-imports the task modules, which re-runs setup_di on a new app object. The binding connects the two.
  • The bind uses setdefault, so an app that already has a container keeps it. Under fork the current app is the module app, which already has its entry, so the bind does nothing there. It also stops a stale handler from an earlier setup_di on another app with the same main from replacing a live container.
  • The other three signal connections are unchanged, so solo, threads, gevent and eventlet behave as before.

Behavior change: app.conf no longer has a modern_di_container key. Code that read it directly instead of calling fetch_di_container will break.

Tests

New tests/test_spawn.py covers a pickle round trip of a configured app, main surviving the round trip, a simulated spawn child (unpickled app as current, setup_di on a re-created app, then worker_process_init), two apps with different main names, and the no-overwrite rule. The existing tests pass unchanged.

I didn't add a real-worker test. I ran one by hand on macOS with celery 5.6.3 and billiard 4.3.1, a filesystem broker, a file result backend and FORKED_BY_MULTIPROCESSING=1:

  • spawn forced, this branch: an @inject task ran in SpawnPoolWorker-1 and returned.
  • fork, this branch: the same task ran in ForkPoolWorker-1.
  • spawn forced, main: the pool failed to start with TypeError: cannot pickle '_thread._local' object.

Docs

The README has a new "Prefork with spawn" note covering fork as the default, the main matching, and FORKED_BY_MULTIPROCESSING=1. CONTEXT.md no longer says the root container lives on conf. The matching note for the Celery page in the modern-di docs will come in a separate follow-up PR in that repo.

setup_di now registers the container in a weak registry keyed by app, so
pickling the app no longer pickles the container. In a spawn child the
worker_process_init handler binds its container to the unpickled current
app when the main names match.

Closes #41
@lesnik512
lesnik512 merged commit 9314065 into main Oct 17, 2026
12 checks passed
@lesnik512
lesnik512 deleted the fix/container-off-app-conf branch October 17, 2026 08:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Prefork pool fails under a forced spawn start method because the container on app.conf cannot be pickled

1 participant