Docs harness: run inline blocks with a prelude, run docstring examples, and check lambda names and error checks #704

Test run

uv run --no-sync pytest tests/test_docs_examples.py tests/test_docs_migrations.py tests/test_documentation_features.py -q -p no:cacheprovider --no-cov

426 passed, 5 skipped, 0 failed (95.8 s). Skips:

• tests/test_docs_migrations.py:739: FERRO_POSTGRES_URL is not set: this output needs Postgres (one Postgres-only output block).
• tests/test_documentation_features.py:687/709/733/756: four many-to-many tests, all @pytest.mark.skip(reason="Many-to-many join tables not automatically created - see coming-soon.md"). The reason is stale: docs/pages/coming-soon.md does not exist, and docs/examples/relationships.py (m2m-usage) and existence_tests.py (m2m) create M2M join tables through auto_migrate=True and pass. These four tests should be unskipped or deleted.

Which tests execute which examples

Executed by no test in a meaningful way:

• testing_conftest.py: it runs as a script, but main() is a no-op ("import smoke check only"). The db and db_transaction fixtures the Testing how-to tells readers to copy are never run under pytest.
• _migrations_project.py: a helper, imported by the migration examples (excluded by the _ prefix, by design).
• Postgres paths: every example runs on SQLite only. row_level_security*.py exercises the SQLite "PostgreSQL-only" warning, not the policies the page shows (see that section).
• No twin equivalence check exists for the 11 non-migration pairs (quickstart, aggregations, existence_tests, partial_selects, populated_relations, predicates, relationships, row_level_security, soft_deletes, table_checks, timestamps, traversal). Each twin only has to run. Its main() body is a smaller smoke test, not the parent's assertions. I diffed every embedded twin section by hand. All are equivalent apart from the declaration style (findings below).

Inline (non-snippet) code blocks in docs/pages: the rule 9 gap

pytest-examples finds 364 Python fences in docs/pages: 180 are snippet-backed (--8<--, excluded from the inline test) and 184 are inline. The inline test compiles each of the 184 with ast.PyCF_ALLOW_TOP_LEVEL_AWAIT and nothing more. The opt-ins {test=run} and {lint=check} exist in tests/test_docs_examples.py, but 0 of 184 blocks use either one. So 184 inline Python examples are syntax-checked only: undefined names, wrong APIs, stale signatures and I-8 violations all pass. Non-Python fences (text, sql, bash, toml, yaml) are untested except the <!-- output: id --> blocks test_docs_migrations.py compares on five schema pages.

Evidence that compile-only lets defects through: 57 lines in page inline blocks use one-letter lambda parameters (I-8), e.g. howto/pagination.md lines 59–60 and its Annotated twin (lambda t: t.id > cursor for Article), guide/queries.md ×10, howto/upgrade-guide.md ×8, api/queries.md ×5. Also, the inline Assignment/Annotated tab pairs on pages such as guide/mutations.md:39–67 and howto/pagination.md:40–95 satisfy I-7's "compile-checked" clause, but no test checks that the two tabs declare the same model.

Cross-cutting patterns

1. I-8 one-letter lambda parameters in four embedded examples: 66 lambdas (aggregations 24, partial_selects 16, populated_relations 15, existence_tests 11), plus 6 in their twins' bodies and 57 lines in inline page blocks. predicates.py, traversal.py, relationships.py and quickstart.py show the correct form. Fix the four files and add a lint test (an AST check of docs/examples/*.py and inline blocks: a lambda passed to where/select/order_by/include/exists takes a parameter of at least two characters, ideally snake_case(Model)).
2. Error checks that cannot fail: 12 try/except blocks assert only inside except, with no else (aggregations ×5, populated_relations ×3, partial_selects ×1, mutations ×3). The "loud limits" the pages advertise are untested. table_checks.py:38 shows the fix.
3. Results hidden below the section marker in about 50 sections across 9 files. The reader sees a query, never its answer (rule 8). Only three snippets print anything, and no page shows printed output from an example.
4. Placeholder data (rule 10) in 5 files: "a1"/"b1"/"o1"/"ledger-a"/"Pair left/right", with tied totals that make a top-N example nondeterministic. relationships.py, predicates.py and migrations_backfill.py show the real-looking alternative.
5. No equivalent SQL beside any example (rule 13). Comments describe the SQL ("renders NOT EXISTS", "COUNT(column)") in 6 files, and the query pages show 5 SQL blocks for 101 Python blocks.
6. 184 inline page blocks are compile-only (rule 9). The {test=run}/{lint=check} opt-ins exist and are used 0 times. Five pages hold 83 of them (models-and-fields 24, migrate-from-sqlalchemy 18, upgrade-guide 17, row-level-security 13, connections 11). tests/test_documentation_features.py tests hand copies, not the docs, and 4 of its tests are skipped with a stale reason.
7. Running-example models the page never shows (rule 3) in 4 files: Customer (mutations), Account (transactions), Event (raw_sql), Metric (multiple_databases). The models sit outside any section, so the I-7 twin is also skipped.
8. Two Annotated idioms across twins: Field(default=None, …) inside Annotated in 11 pairs, against FerroField(…)] = None in 7 pairs. Plus 18 # noqa: E711/E712 comments rendered on pages, avoidable with a ruff per-file ignore for docs/examples/*.