Repository navigation
docs: panel round 1 on #777 (harness and pages) — output blocks regenerated, one fence reader and runner, twin equivalence, CLI reference complete - #780
Conversation
… marked - tests/_docs_harness.py: fenced_blocks() reads every fence (language, info, dedented body, opening line, the <!-- kind: value --> note above it) and refuses an unclosed one; run_python() is the one subprocess runner; check_fence() holds a fence to its contract; page_examples() also collects fences from Markdown a page pulls in with --8<-- (CONTRIBUTING.md). - The README, TOML, step-file and output-block readers and the example, migrations, fixtures and autogenerate runners use them. - A docstring fence without a test mode is refused, as a page fence is. - The nine entry points of story 72 must each keep a test=run example. - CONTRIBUTING.md's test example says why it only compiles. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hb9U9A7Zu9ypdHTsDvdScP
…rinted output captured(example, dialect, url) runs an example once and reports, per snippet region, the statements it sent and the lines it printed, plus its stdout and the models' SchemaIR; check_block and check_same_on_postgres compare a page's block with it. test_docs_sql.py is now the gate over PAGES, and its capture tests assert through by_region() and captured() on a scratch example. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hb9U9A7Zu9ypdHTsDvdScP
… is regenerated tests/test_docs_output.py refuses a ```text block with no note above it and compares each <!-- prints: <example>.py:<snippet> --> block with what that region prints (captured once per example), and each <!-- prints: above --> block with what the run fence above it prints, byte for byte. Text no script prints says so with <!-- shows: <what it is> -->. The aggregation error regions print the error they catch, so the page shows each message whole instead of an elided copy; the backends page runs the unsupported-scheme connect it quotes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hb9U9A7Zu9ypdHTsDvdScP
… pinned equivalent tests/_docs_twins.py rewrites an Assignment example into the one Annotated idiom (options inside Annotated, the default or default_factory assigned, a quoted declaration in a comment quoted the same way) and leaves every other line alone. All 29 twins are regenerated with it, so both tabs run the same data, queries and prints; before, most twins were shorter, different programs. tests/test_docs_twins.py fails when a twin is not its script rewritten, or when the pair declares a different schema (SchemaIR at each connect and at the end) or prints different output. A default_factory stays an assigned Field in the Annotated tab, because type checkers see no default inside Annotated (ty: missing-argument). The conventions record that rule, the generated twins and the printed-output notes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hb9U9A7Zu9ypdHTsDvdScP
…atch it Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hb9U9A7Zu9ypdHTsDvdScP
…s against --help tests/test_docs_cli_reference.py fails when a ferro migrate verb's --help prints a flag (bar the global options and positional keyword spellings) that the verb's section of reference/cli.md does not list. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hb9U9A7Zu9ypdHTsDvdScP
…s, not four; drop the unused prelude The global region now starts at the imports and connect(), as the Queries page's first program does. The aggregations prelude was cited by no fence; it is removed, and a test fails on any prelude no fence asks for. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hb9U9A7Zu9ypdHTsDvdScP
…table half changed Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hb9U9A7Zu9ypdHTsDvdScP
…d anchors and narrow MOVED_ANCHORS's claim The page keeps its path, which two API docstrings link. MOVED_ANCHORS now says what it covers: the split pages, the deleted Next steps page and six renamed sections outside links use (get-or-create, nesting-behavior, sessions-recommended, what-is-ferro, writing-your-first-test, save-no-longer-upserts), each redirected. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hb9U9A7Zu9ypdHTsDvdScP
…pture runs under the interpreter's hash seed Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hb9U9A7Zu9ypdHTsDvdScP
0x054
left a comment
There was a problem hiding this comment.
Spec: FAIL
Spec = the panel findings this slice owns (PR #777, comment 6319609050). No Closes #N: accepted, this is a fix slice under epic #698 with no sub-issue.
- Important · docs/pages/guide/schema/auto-migrate.md:67,91,190,236,243; deploying.md:68,129,144; overview.md:45; renames-and-retirements.md:168; reference/configuration.md:90; reference/row-level-security.md:206; docs/solutions/patterns/docs-page-conventions.md:95 at 0860323 · Missing · F7 is narrowed, not fixed. These 12
textblocks quote Ferro's own errors and warnings (the NOT NULL refusal, the SQLite check warning, the kept-default warning, both run-lock texts,PendingMigrationsError, bothDatabaseAheadErrortexts, the tracked-database refusal, the retired-labelValidationError, the forced-RLS warning). They are marked<!-- shows: ... -->, whichcheck_printed(tests/_docs_sql.py:583) never compares. The conventions now defineshows:to include "an error from a scenario no example runs", so the exemption is part of the rule. Every text matches its source in src/ and crates/ today, so nothing is wrong yet. But a reworded message leaves a stale page with every gate green, which is the defect F7 names. These are also the texts most likely to change, and most of them can be produced cheaply: the retired-label error and the tracked-database refusal need only a model and a connect, and the deploying.md errors fit test_docs_migrations' existingrefusal:scenario mechanism. There is no**Narrowed:**entry, and the PR body calls them "the next candidates forprints:" (I-6: a documented residual is not a resolution). · F7: "mark each output block with its example and snippet …, run the script once, record stdout per snippet region, and compare." - Minor · docs/pages/guide/raw-sql.md:69,98,130; guide/connections.md:57,206,298; concepts/backends.md:88; guide/schema/auto-migrate.md:147; src/ferro/session.py:295,422 at 0860323 · Missing · F25 was not done, and the stated reason holds for only some of the blocks. The Litestar/FastAPI/ASGI reason covers the five row-level-security.md blocks. For these ten plain-Postgres blocks the other obstacle ("the runner would have to patch
connect()") is one this PR already solves:_run_captureredirectsferro.connectto a given URL (tests/_docs_sql.py:374), and test_docs_sql already runs regions on Postgres. · F25: "these could run under a postgres-only fence setting."
Checked and addressed: Spec/story 17 (--allow-ahead both forms, cli.md:202-203; test_docs_cli_reference.py fails on c80e3eb's cli.md); Spec/RLS tab (zensical.toml:49, test_the_row_level_security_split_sits_in_its_three_tabs; the page keeps guide/row-level-security.md, and the links at src/ferro/models.py:278 and rowsecurity.py:283 resolve under --strict); F8 (test_docstring_examples.py RUNNING_ENTRY_POINTS, plus a scratch-package negative); F9 (fenced_blocks, _docs_harness.py:149; no other fence regex is left in the docs tests); F10 (run_python, _docs_harness.py:65; the remaining subprocess.run calls assert on non-zero exits and are outside F10's five); F11 (tests/_docs_twins.py: an AST rewrite plus ruff format, deterministic against the locked ruff 0.15.1, which pytest-examples pulls into ci-test; test_docs_twins.py compares source after conversion, SchemaIR and stdout for all 29 pairs); F13 (aggregations.py global region now mirrors predicates.py first-query; aggregations.md:9,15 name five aggregates and five numbers); F14 (auto-migrate.md:230); F15 (conventions lines added; no twin or page puts default_factory inside Annotated, e.g. timestamps_annotated.py:40); F17 (CONTRIBUTING.md:317 now collected, id CONTRIBUTING.md:317-334); F18 (six redirects in zensical.toml:195-200, each target id present in site/, docstring narrowed at test_docs_site.py:572); F19 (ci.yml:108 glob, plus the misnamed-module pin); F21 (check_fence); F22 (tests/_docs_sql.py); F23 (capture tests go through captured()/by_region()); F24; Coherence (unmarked docstring fence refused; prelude deleted, test_every_prelude_is_cited_by_a_fence).
Harness refactor, collected ids at c80e3eb vs 0860323: every existing parametrized gate collects the same blocks (only line numbers shift where notes were added), apart from one extra test_docs_examples id, the CONTRIBUTING fence (F17). test_docs_output collects 107 page blocks.
Cannot verify: whether the six redirected anchors are the ones outside links actually use (the docstring claims that).
Quality: PASS
- Minor · tests/_docs_sql.py:583 at 0860323 ·
check_printedpasses any<!-- output: ... -->/<!-- refusal: ... -->block on any page. test_docs_migrations readsoutput:only onOUTPUT_PAGESandrefusal:only on cli.md, so either note on another page escapes both gates. There is none today. · Fix: refuse those kinds outside the pages test_docs_migrations reads. - Minor · tests/test_docs_site.py:316 at 0860323 · The new "docstring" sits after the first statement of
test_ci_docs_job_runs_every_docs_test_and_the_lint, so it is a no-op string expression, not a docstring. · Fix: move it abovejobs = .... - Minor · PR body · The counts do not match the tree: there are 107
textblocks, not 116 (116 is the module's test count). 20prints:blocks come from regions, not 21. 12shows:blocks are errors or warnings, not 11, and 6 are other kinds, not 7. - Depth check (
_docs_sql,_docs_twins,fenced_blocks/run_python/check_fence): no check fired.
Verify: PR head 0860323 passed (zensical build --clean --strict "No issues found"; pytest tests/test_doc*.py tests/test_docstring_examples.py -n 8 932 passed, 109 skipped; the same with FERRO_POSTGRES_URL=… --db-backends=sqlite,postgres 1043 passed, Postgres accepted connections; python -m tests._docs_checks docs/pages docs/examples exit 0; ruff check tests docs/examples clean; ruff format --check . 348 files formatted). The capture of mutations.py is byte-identical under PYTHONHASHSEED 1, 7, 42 and 1234.
Trial merge: not run (base unchanged; merge-base = origin/feat/docs-rewrite = c80e3eb)
…nario that raises it Twelve text blocks quoted Ferro's own errors and warnings under shows:, which nothing compared. Each is now a prints: block backed by an example region: docs/examples/auto_migrate_errors.py (the required column, the SQLite check, both run-lock texts, the foreign default on Postgres), migration_errors.py (behind, ahead, ahead by a contract step, the tracked database), a region of migrations_retire_label.py and one of row_level_security.py (the forced-table warning, on Postgres). prints: takes (postgres): the region runs on PostgreSQL as <example>.py <database-url> <role> under postgres_only. shows: is narrowed to text Ferro does not print, and a shows: block that reads like Ferro's output fails. Regenerating found two stale blocks: the forced-table warning names 'document' too, and the warnings start 'ferro auto-migrate:'. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hb9U9A7Zu9ypdHTsDvdScP
…ostgres blocks run A test=run fence marked backend=postgres runs under postgres_only with every connect() in its prelude and block pointed at the test database, and only compiles without Postgres. The Raw SQL, Connections, Backends and Auto-migrate blocks that needed a server now run on it; the five web-framework blocks on the RLS how-to stay compile-only (Litestar and FastAPI are not installed). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hb9U9A7Zu9ypdHTsDvdScP
…SQL too Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hb9U9A7Zu9ypdHTsDvdScP
0x054
left a comment
There was a problem hiding this comment.
Scoped re-review: 0860323..21ed328
Spec: PASS
- F7 (Important, narrowed): ADDRESSED. The 12 Ferro error and warning blocks are now
prints:blocks, each regenerated from a region: docs/examples/auto_migrate_errors.py (required-column, sqlite-check, lock-wait, lock-timeout, foreign-default), docs/examples/migration_errors.py (behind, ahead, ahead-by-contract, tracked), migrations_retire_label.py:77 (retired-write) and row_level_security.py:202 (forced_warning). The two Postgres ones carry(postgres)and run intest_a_postgres_text_block_is_what_its_example_prints_there(tests/test_docs_output.py:55), which CI's backend-matrix job reaches through-m "backend_matrix or postgres_only"(ci.yml:296). Regeneration corrected three stale texts: theferro auto-migrate:prefix,'document', 'invoice', and schemamainwith1s.shows:is narrowed in docs-page-conventions.md:96 and guarded bytest_a_shows_block_holds_nothing_ferro_prints(tests/test_docs_output.py:83). The renumbered migrations on deploying.md:124-152 and api/migrations.md:65-71 (0002_author_slug, 0003_author_slug_required, 0003_drop_bio) agree with each other and with what migration_errors.py prints. - F25 (Minor): ADDRESSED. 8 page blocks are now
{test=run prelude=… backend=postgres}: raw-sql.md:69,98,130, connections.md:57,206,298, backends.md:88, auto-migrate.md:147. All 8 are collected astest_docs_examples_on_postgresunderpostgres_only. Without Postgres they are reported as skipped, not passed, and the SQLite gate only compiles them (check_fencereturns beforerun_fence, tests/_docs_harness.py:426).run_fenceraises if asked to run one with no URL (pinned bytest_a_postgres_fence_without_a_database_is_refused_by_run_fence). The 5 RLS blocks that stay compile-only (row-level-security.md:128,163,214,237,313) need Litestar, FastAPI, an ASGI server or theauth_uid()setup, which no test group installs, so that reason holds. session.py:295,422 are out of this slice.
Cannot verify: none
Quality: PASS
- Minor · tests/test_docs_output.py:75-80 at 21ed328 · The one exception (
VARIES) is honest about why: the statement-log lines end in a duration (src/ferro/_sql_log.py:54,{duration_ms:.1f}ms), and the block elides the column list. But it is not the I-6 answer, and the repo already has the primitive that is: test_docs_migrations'_normalisemasks(\d+ ms)on both sides. A region that turns on theferro.sqllogger, plus a mask for the trailing duration (and a printed statement short enough not to need(...)), would make the block aprints:block. The exemption is also keyed by page, so any latershows:block on connections.md skips the check too. · Fix: regenerate with the duration masked, or at least keyVARIESby page and line. - Minor · tests/test_docs_output.py:68-71 at 21ed328 ·
FERRO_OUTPUTkeys on a prefix (ferro.,ferro auto-migrate:,…Error:,N validation error,connect(). Three of the twelve texts this round converted carried none of those prefixes when they wereshows:(Cannot add NOT NULL column…,Check constraint 'ck_…,author.slug has default…), so the guard would not have caught them. The conventions sentence ("Ashows:block that reads like Ferro's output … fails the test") claims more than the test enforces. - Minor · docs/pages/concepts/backends.md:88, guide/connections.md:57,298 at 21ed328 ·
point_connect_at(docs/examples/_preludes/init.py:87) replaces the whole URL, so?ferro_search_path=test_42,?ferro_search_path=tenant_aand?sslmode=requirenever reachconnect(). Those blocks run, but the option each one demonstrates goes untested. auto-migrate.md:147 likewise connects to an empty database, so no label is added. · Fix (follow-up): carry the block's query parameters onto the test URL (a per-test schema name forferro_search_path; drop onlysslmode). - Minor · docs/examples/migration_errors.py:63,77,90 at 21ed328 · The regions hand-write the exception's dotted name (
f"ferro.migrations.DatabaseAheadError: {error}"), so a renamed or moved class would leave the page's first word stale with the gate green. · Fix:f"{type(error).__module__}.{type(error).__qualname__}: {error}", or the convention'stype(exc).__name__. - No new Critical or Important breakage. Both new scripts run under
test_docs_example_scriptson SQLite. migration_errors.py works inproject()'s temporary directory and leaves nothing behind. auto_migrate_errors.py's Postgres half runs only when given a URL.
Out-of-scope observations: none.
Verify: PR head 21ed328 passed (zensical build --clean --strict "No issues found"; pytest tests/test_doc*.py tests/test_docstring_examples.py -n 8 948 passed, 121 skipped, each Postgres test skipped with "No Postgres provider is configured"; with FERRO_POSTGRES_URL=… --db-backends=sqlite,postgres 1070 passed, 1 skipped (the empty docstring backend=postgres set), Postgres accepted connections; python -m tests._docs_checks docs/pages docs/examples exit 0; ruff check tests docs/examples clean; ruff format --check . 350 files formatted)
Trial merge: not run (base unchanged at c80e3eb)
Panel round 1 on #777, the harness and page findings (the
src/findings landed in #779; this branch is rebased on it).Part of #698 (panel round 1 on PR #777)
What a reader gets
Before, the output under "It prints:" was typed by hand. The aggregations page showed
and nothing checked it. Now every ```text block on a page carries a note naming where it comes from, and the test regenerates it:
Verify block corrected: no
Fix report: review round 1 (Spec FAIL on F7 and F25)
F7: the 12 error and warning blocks are now regenerated
Each block that quoted Ferro's own error or warning text is now a
prints:block. An example region triggers the error and prints it:auto_migrate_errors.py:required-columnauto_migrate_errors.py:sqlite-checkauto_migrate_errors.py:foreign-default (postgres)auto_migrate_errors.py:lock-wait. The example holds the SQLite lock file itself.auto_migrate_errors.py:lock-timeout, withlock_timeout = "1s"migration_errors.py:behindmigration_errors.py:aheadmigration_errors.py:ahead-by-contractmigration_errors.py:trackedmigrations_retire_label.py:retired-writerow_level_security.py:forced_warning (postgres)Regenerating found three blocks that no longer matched Ferro:
'document'.ferro auto-migrate:.main.The page prose now says the example sets
"1s"on SQLite. deploying.md's migration names follow the scenario's numbering (0002_author_slug,0003_drop_bio), as does the API page'scontract_stepsexample.New form:
<!-- prints: <example>.py:<snippet> (postgres) -->. The region runs on PostgreSQL as<example>.py <database-url> <role>underpostgres_only.captured()gainedargsfor this.shows:is narrowed to text Ferro does not print: a sketch, a file tree, another tool's shell session, a prompt to paste.test_a_shows_block_holds_nothing_ferro_printsfails anyshows:block that reads like Ferro output (ferro.,ferro auto-migrate:,SomethingError:,N validation error for,connect().VARIESwith its reason: connections.md's two sample statement-log lines. Each ends with how long the statement took, which differs on every run, so no capture reproduces it byte for byte.TDD
shows:blocks.tests/test_docs_output.py: 127 passed, 2 skipped on SQLite; 129 passed with--db-backends=sqlite,postgres.tests/test_docs_twins.pystays green with the regeneratedmigrations_retire_labelandrow_level_securitytwins.F25:
backend=postgres{test=run prelude=<name> backend=postgres}:postgres_only. The prelude runner sends everyconnect()to the test database, whatever URL the prelude or block names.backend=on a mode other thanrun, or a value other thanpostgres, is refused.test_docstring_example_on_postgresruns docstring fences the same way.session.py:295and:422can droptest=compilefortest=run prelude=<name> backend=postgres; the test that runs them is already in place.auth_uid()function set up earlier on that page;test_docs_examples_on_postgres(8 blocks, all passing on Postgres),test_a_postgres_fence_runs_on_postgres,test_backend_needs_test_run,test_only_postgres_is_a_backend,test_a_postgres_fence_without_a_database_is_refused_by_run_fence.Verify on 21ed328 (base c80e3eb)
Interfaces changed in this round:
prints:takes(postgres).captured(..., args=()).check_printed(..., dialect="sqlite").run_fence(example, *, database_url=None)andcheck_fence(..., database_url=None).backend=postgres.FERRO_DOCS_DATABASE_URL.🤖 Generated with Claude Code
https://claude.ai/code/session_01Hb9U9A7Zu9ypdHTsDvdScP