Skip to content

docs: rework the task, bot, rpc and cli integrations pages - #667

Merged
lesnik512 merged 1 commit into
mainfrom
docs/rework-workers
Oct 10, 2026
Merged

lesnik512 merged 1 commit into
mainfrom
docs/rework-workers

Conversation

@lesnik512

Copy link
Copy Markdown
Member

Reworks the arq, Celery, FastStream, taskiq, aiogram, FastMCP, gRPC and Typer pages the same way the Quickstart, Introduction and Providers pages were done. Every behavioural claim was checked against the integration sources (local main of each modern-di-<name> repo, byte-identical to the 4.0.0 releases) and, where possible, by running a script: real arq burst workers against Redis, real celery worker processes per pool, gRPC sync and aio servers on 127.0.0.1:0, FastStream test brokers, aiogram feed_update, the FastMCP in-memory client and Typer's CliRunner.

What was wrong

  • Celery, pool table. The page said prefork and solo both send worker_process_init/worker_process_shutdown. Solo sends only worker_process_init and closes the root on worker_shutdown (celery/concurrency/solo.py:20, confirmed with real worker runs). The table now lists prefork, solo and threads/gevent/eventlet separately.
  • Celery, eager mode. The page implied tasks fail unless you send the worker signals by hand. A constructed container is already open, so eager tasks resolve; only the close needs driving. The Testing example now wraps the eager call in with container:.
  • Celery, async finalizers. "Must be sync (or close_sync-compatible)" became the actual behaviour: an async finalizer makes the task (or the worker-shutdown close) raise FinalizerError.
  • arq. The section titled "Async resolution" described sync resolution. ctx must be the first parameter apart from the FromDI ones, not the first parameter outright (a task with a FromDI parameter before ctx ran through a real worker).
  • taskiq. The warning told readers to pass run_startup=True to run_receiver_task so the root closes. Neither value closes it: the receiver never calls broker.shutdown(), so WORKER_SHUTDOWN never fires. Calling await broker.shutdown() after the receiver stops does close it (verified; the receiver sets is_worker_process).
  • FastStream, Testing. The warning described the wrong mechanism (the AST check sets connect_only, it doesn't decide whether startup hooks run). The rule that holds: enter the test broker first and TestApp after it. A broker alone raises the RuntimeError naming setup_di; TestApp first tries a real NATS connection.
  • FastStream, hook example. It called NatsBroker(settings.nats_url) with settings undefined (Sybil passed only because the hook never ran).
  • aiogram. The page said Message, CallbackQuery and so on resolve by type. Only Update and TelegramObject do; a factory declaring message: Message fails validate(). The intro said aiogram has no DI; it injects middleware data by parameter name. A handler added after startup under auto_inject raises TypeError, and update observers are never wrapped.
  • FastMCP, mounting. setup_di on a mounted server does work for that server's components; it misses only the parent's own components.
  • FastMCP, background tasks. Now says task=True needs fastmcp[tasks] and a TasksExtension, and that the client sees ToolError ("Failed to resolve dependencies for parameter(s): ...") while the server logs the RuntimeError.
  • gRPC. The aio example injected a Greeter that existed only in a hidden block and was never registered (ProviderNotRegisteredError for real). It now reuses RpcReport. The *args section contrasted with Typer, which has no such check; arq and Celery are the ones that raise.
  • Typer. "This adapter is also the answer for Click" was wrong: setup_di on a Click group raises AttributeError: 'Group' object has no attribute 'info'.

What was added

  • A Root container lifecycle section on every page: who opens and closes the root, and with which close method.
  • Testing sections for arq (burst worker, skipped in Sybil since there is no Redis; run by hand against Redis), Celery (eager plus with container:), FastStream (runnable request(...).decode(), replacing the skipped placeholder), taskiq (async with broker:), aiogram (emit_startup/feed_update/emit_shutdown) and Typer (CliRunner, which does not close the root).
  • Celery: a warning that prefork fails on macOS with TypeError: cannot pickle '_thread._local' object, and to use --pool=solo or --pool=threads there (see the library bug below).
  • gRPC: factories must annotate grpc.ServicerContext on the aio server too (grpc.aio.ServicerContext is not a subclass and does not resolve); the sync server closes RPC children with close_sync(), so an async REQUEST finalizer fails the RPC with StatusCode.UNKNOWN.
  • Typer: Scopes (one REQUEST child per command, closed with close_sync()), Action scope rewritten around action_scope(ctx) (the API the table lists), and full-sentence API rows.
  • FastStream: setup_di also takes AsgiFastStream.

Structure changes

Pages now follow one order: intro, How to use, Scopes, Root container lifecycle, Framework context objects, framework-specific sections, Testing, See also, API. The repeated "Sync resolution, ..." sections are folded into Scopes as one or two sentences. Celery's DITask section is down to one example plus a one-line base=DITask variant, without the head_from_fun/__header__/weak=False internals. aiogram's dialog section moved above the API table. On FastStream, "Several brokers" now sits after Testing because its blocks clear the Sybil namespace that the Testing example needs.

Anchors kept

grpc.md#injecting-the-servicercontext, faststream.md#framework-context-objects, faststream.md#testing, celery.md#the-ditask-base-class. celery.md#worker-process-lifecycle was renamed to #root-container-lifecycle; its only link was in-page and now points at Testing. All page paths are unchanged.

Checks

  • just test-docs -q -p no:cacheprovider --no-cov: 152 passed, 21 skipped
  • just test-docs-integrations: 257 passed, 1 skipped (the arq Redis block)
  • just docs-build: no warnings
  • just lint-ci: passed
  • lychee --offline with the GitHub remap: 0 errors
  • By hand: the arq page against a local Redis (DB 7), the FastStream test as a real file (where TestBroker's AST check works), and the Celery DITask, Typer and aiogram claims in scratch scripts.

Library bug found

modern-di-celery: the prefork pool cannot start under the spawn start method (the macOS default). setup_di stores the Container on app.conf, and billiard pickles the app for spawned children.

import pickle
from celery import Celery
from modern_di import Container
from modern_di_celery import setup_di

app = Celery("x")
pickle.dumps(app)               # ok
setup_di(app, Container())
pickle.dumps(app)               # TypeError: cannot pickle '_thread._local' object

A real celery -A wmod worker --pool=prefork on macOS (celery 5.6.3, billiard 4.3.1, Python 3.14) logs Unrecoverable error: TypeError("cannot pickle '_thread._local' object") and exits. --pool=solo, --pool=threads and forcing billiard's fork method work. A fix would keep the container off app.conf (for example a module-level WeakKeyDictionary keyed by app, as modern-di-fastmcp does).

Needed outside this batch

  • docs/integrations/writing-integrations.md (page template, around lines 484-497) could name the Root container lifecycle and Testing sections, and mention FastMCP's default-value FromDI as the exception to "always use Annotated".
  • docs/migration/from-dependency-injector.md:289 and docs/migration/from-that-depends.md:312 say the setup call of FastAPI, Litestar, FastStream and Typer "calls container.close_async() at shutdown". Typer's never closes the root.
  • modern-di-taskiq could document or handle run_receiver_task never firing WORKER_SHUTDOWN.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Benchmark

Details
Benchmark suite Current: bc70f45 Previous: 395e8fb Ratio
benchmarks/test_guard_by_type.py::test_g16_resolve_by_type 5269822.278399695 iter/sec (stddev: 2.227180889282562e-8) 6139935.138982263 iter/sec (stddev: 1.8605923146541302e-8) 1.17
benchmarks/test_guard_by_type.py::test_g17_resolve_by_type_large_registry 5375642.651349847 iter/sec (stddev: 1.791256431136679e-8) 6205832.315667191 iter/sec (stddev: 1.3805592208773412e-8) 1.15
benchmarks/test_guard_cold.py::test_g8_cold_first_resolve 25807.148394489886 iter/sec (stddev: 0.00003882481793776367) 32024.886577255867 iter/sec (stddev: 0.000030238944431934) 1.24
benchmarks/test_guard_cold.py::test_g8b_cold_first_resolve_cached 20792.830389596085 iter/sec (stddev: 0.00015248265389928512) 24960.133332489408 iter/sec (stddev: 0.00011886986138363566) 1.20
benchmarks/test_guard_concurrency.py::test_g14_concurrent_cached_hit[1] 685.9738390910055 iter/sec (stddev: 0.00011019839283477558) 775.0388204206938 iter/sec (stddev: 0.00004473754687217169) 1.13
benchmarks/test_guard_concurrency.py::test_g14_concurrent_cached_hit[2] 634.0802887200944 iter/sec (stddev: 0.00010827955702154118) 764.2436132277375 iter/sec (stddev: 0.0001144571715558005) 1.21
benchmarks/test_guard_concurrency.py::test_g14_concurrent_cached_hit[4] 602.9152150356276 iter/sec (stddev: 0.0000980323629285478) 680.0345534095648 iter/sec (stddev: 0.0000803400475442327) 1.13
benchmarks/test_guard_concurrency.py::test_g15_concurrent_first_resolve[1] 15314.291373273636 iter/sec (stddev: 0.000016530995313624612) 10419.935559673875 iter/sec (stddev: 0.00001230746171334532) 0.68
benchmarks/test_guard_concurrency.py::test_g15_concurrent_first_resolve[2] 5419.645646403735 iter/sec (stddev: 0.0000301704109692264) 5634.100392630293 iter/sec (stddev: 0.00003435893765767916) 1.04
benchmarks/test_guard_concurrency.py::test_g15_concurrent_first_resolve[4] 3176.0883740233294 iter/sec (stddev: 0.00004545063337832835) 2968.9836953557647 iter/sec (stddev: 0.000061919224501018) 0.93
benchmarks/test_guard_concurrency.py::test_g15b_concurrent_first_resolve_sibling_children[1] 12246.228799328605 iter/sec (stddev: 0.000007886308592929258) 12389.480670083722 iter/sec (stddev: 0.000019500905757513246) 1.01
benchmarks/test_guard_concurrency.py::test_g15b_concurrent_first_resolve_sibling_children[2] 3765.667648637206 iter/sec (stddev: 0.000040368378221854074) 4843.535831204409 iter/sec (stddev: 0.00003981695650002605) 1.29
benchmarks/test_guard_concurrency.py::test_g15b_concurrent_first_resolve_sibling_children[4] 1879.113644016476 iter/sec (stddev: 0.00006681225016127219) 1868.0290700812566 iter/sec (stddev: 0.00007765838735564648) 0.99
benchmarks/test_guard_concurrency.py::test_g15c_worker_pool_floor_control[1] 17286.455730663034 iter/sec (stddev: 0.000008097580366193873) 24896.9783783031 iter/sec (stddev: 0.000017881784016521047) 1.44
benchmarks/test_guard_concurrency.py::test_g15c_worker_pool_floor_control[2] 11824.94237064147 iter/sec (stddev: 0.00002078025032926536) 11645.85197075752 iter/sec (stddev: 0.000015482573556939863) 0.98
benchmarks/test_guard_concurrency.py::test_g15c_worker_pool_floor_control[4] 5672.9646501995 iter/sec (stddev: 0.00004723000635303754) 5052.235696391264 iter/sec (stddev: 0.000048475453192008314) 0.89
benchmarks/test_guard_lifecycle.py::test_g6_build_child_container 2101735.671118386 iter/sec (stddev: 4.2066069926754305e-8) 2335541.2567186337 iter/sec (stddev: 3.860763352243666e-8) 1.11
benchmarks/test_guard_lifecycle.py::test_g6b_build_child_container_auto_scope 2292034.012871781 iter/sec (stddev: 3.558335371604026e-8) 2505907.6773532387 iter/sec (stddev: 3.905850495221895e-8) 1.09
benchmarks/test_guard_lifecycle.py::test_g7_request_lifecycle_batch 3356.2693131512488 iter/sec (stddev: 0.000017978061776230364) 3317.2697581338502 iter/sec (stddev: 0.000052891745120707064) 0.99
benchmarks/test_guard_lifecycle.py::test_g7c_event_loop_floor_control 78112.4851830058 iter/sec (stddev: 0.0000027086744758087456) 83100.58991551855 iter/sec (stddev: 0.0000014527703096612033) 1.06
benchmarks/test_guard_lifecycle.py::test_g7b_request_cycle_sync 475165.44250959443 iter/sec (stddev: 1.2394827535113792e-7) 468518.9908558271 iter/sec (stddev: 8.515899938405845e-8) 0.99
benchmarks/test_guard_lifecycle.py::test_g13_teardown_at_scale 70909.2434303564 iter/sec (stddev: 0.0000027782747092240103) 78785.45603292529 iter/sec (stddev: 0.0000017671089125152717) 1.11
benchmarks/test_guard_lifecycle.py::test_g13b_teardown_at_scale_async_no_finalizers 621.3493342413783 iter/sec (stddev: 0.0013603197027562005) 586.9668417253587 iter/sec (stddev: 0.0013708543619692236) 0.94
benchmarks/test_guard_resolve.py::test_g1_transient_resolve 2991696.3224817435 iter/sec (stddev: 4.157090149691814e-8) 3091773.339921797 iter/sec (stddev: 3.1905368666696455e-8) 1.03
benchmarks/test_guard_resolve.py::test_g2_cached_resolve 5054368.323745746 iter/sec (stddev: 1.712731860059663e-8) 5585002.481125255 iter/sec (stddev: 1.7326975696337195e-8) 1.10
benchmarks/test_guard_resolve.py::test_g3_deep_chain 1019718.1626471496 iter/sec (stddev: 5.779632453520447e-8) 1053634.7223852123 iter/sec (stddev: 4.376268176758002e-8) 1.03
benchmarks/test_guard_resolve.py::test_g4_wide_resolve 580397.4869603864 iter/sec (stddev: 4.1063887829558055e-7) 602475.4027370019 iter/sec (stddev: 3.669391702377282e-7) 1.04
benchmarks/test_guard_resolve.py::test_g5_cross_scope 2584365.313476234 iter/sec (stddev: 3.715010938792363e-8) 2722743.1964451196 iter/sec (stddev: 2.2608711961763022e-8) 1.05
benchmarks/test_guard_resolve.py::test_g9_context_resolve 1881938.0291938286 iter/sec (stddev: 1.6413346832874623e-7) 2095884.4060996436 iter/sec (stddev: 1.4539795764318154e-7) 1.11
benchmarks/test_guard_resolve.py::test_g12_override_active_resolve 1040992.5155770762 iter/sec (stddev: 1.4124621669601707e-7) 1056924.0255533508 iter/sec (stddev: 4.539064934432262e-8) 1.02
benchmarks/test_guard_resolve.py::test_g18_alias_hop 5332125.855645337 iter/sec (stddev: 1.766459180387573e-8) 5564689.374851157 iter/sec (stddev: 1.595292015837515e-8) 1.04
benchmarks/test_guard_validate.py::test_g10_validate_deep_chain 40396.43118530917 iter/sec (stddev: 0.000015876340377316337) 23951.926949647954 iter/sec (stddev: 0.000031960404140896095) 0.59
benchmarks/test_guard_validate.py::test_g11_validate_wide 23910.734914167984 iter/sec (stddev: 0.00002488399705853336) 21724.253571453744 iter/sec (stddev: 0.00003130808206721572) 0.91

This comment was automatically generated by workflow using github-action-benchmark.

@lesnik512
lesnik512 merged commit e83abd0 into main Oct 10, 2026
12 checks passed
@lesnik512
lesnik512 deleted the docs/rework-workers branch October 10, 2026 19:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant