Skip to content

Add generated compile-time UI contracts - #83

Closed
lodyai[bot] wants to merge 1 commit into
mainfrom
add-compile-time-ui-tests
Closed

Add generated compile-time UI contracts#83
lodyai[bot] wants to merge 1 commit into
mainfrom
add-compile-time-ui-tests

Conversation

@lodyai

@lodyai lodyai Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Summary

This branch adds a generated Rust compile-time UI harness to resolve the
remaining Testing (Compile-Time / UI) warning from PR #79. It retains the
Python-generated project rejection tests for Makefile integration while
placing reviewed compiler, Clippy, and Rustdoc diagnostic contracts inside
each generated Rust project.

Review walkthrough

Validation

  • make check-fmt: passed; 31 files already formatted.
  • make lint: passed with no warnings.
  • make typecheck: passed; no issues in 31 source files.
  • uvx --with pytest-copier --with pyyaml --with syrupy --with make-parser --with hypothesis pytest tests/test_template/test_compilation.py::test_generated_compile_time_ui_contracts -vv: passed; 1 test.
  • make test: passed; 111 passed, 2 skipped, and 1 snapshot passed.
  • make spelling: passed with typos 1.48.0.

Validation did not use TRYBUILD=overwrite, warning suppressions, or reduced
lint severities.

Notes

trybuild owns the compiler pass case and reviewed unsafe_code and
missing_docs .stderr files. The Rust integration harness invokes focused
Cargo commands for the Clippy and Rustdoc policies and compares their output
with narrow reviewed diagnostic fragments.

References

Summary by Sourcery

Add a generated Rust compile-time UI harness and fixtures to enforce lint and documentation diagnostic contracts in rendered projects.

New Features:

  • Introduce a generated compile_ui.rs Rust test harness that runs trybuild, Clippy, and Rustdoc UI contracts.
  • Add Rust UI case projects and fixtures, including compile-pass, compile-fail, and Cargo-based lint/doc examples with reviewed expected diagnostics.

Enhancements:

  • Extend template developer and contributor documentation to describe the compile-time UI workflow, diagnostic review process, and TRYBUILD usage constraints.
  • Update repository layout documentation to surface the new UI harness and fixture directories as part of the generated project structure.
  • Strengthen policy-guidance tests to cover the new UI contracts and documentation expectations around compile-time diagnostics and TRYBUILD usage.

Build:

  • Add trybuild as a dev-dependency in the Rust template Cargo manifest to support compile-time UI tests.

Tests:

  • Add a Python template test that renders a project, validates the generated Rust UI harness and fixtures, and runs the compile-time UI test suite.
  • Expand policy-guidance tests to assert inclusion of the new UI guidance in AGENTS and developer documentation.

Render a `trybuild` pass/fail suite with reviewed compiler diagnostics.
Cover Clippy and Rustdoc through a Rust Cargo-command UI harness because
`trybuild` invokes only `rustc`, while retaining the Python Makefile
integration tests.
@coderabbitai

coderabbitai Bot commented Aug 2, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 2503c1d0-9844-45b2-92e8-a74743a64690

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@sourcery-ai

sourcery-ai Bot commented Aug 2, 2026

Copy link
Copy Markdown

Reviewer's Guide

This PR adds a generated Rust compile-time UI harness and fixtures, wires them into the template Cargo config, and documents the lint/diagnostic contract and validation workflow while extending parent and policy tests to cover the new behaviour.

Sequence diagram for test_generated_compile_time_ui_contracts and Rust UI harness

sequenceDiagram
    actor Pytest
    participant PythonParentTests
    participant GeneratedProject
    participant RustUIHarness as tests_compile_ui_rs
    participant Trybuild
    participant CargoClippy
    participant CargoRustdoc

    Pytest->>PythonParentTests: test_generated_compile_time_ui_contracts
    PythonParentTests->>GeneratedProject: render_template_project
    PythonParentTests->>RustUIHarness: run_tests

    RustUIHarness->>Trybuild: run_supported_pass_fixture
    Trybuild-->>RustUIHarness: compiler_pass_ok

    RustUIHarness->>Trybuild: run_unsafe_code_compile_fail
    Trybuild-->>RustUIHarness: compare_unsafe_code_stderr

    RustUIHarness->>Trybuild: run_missing_docs_compile_fail
    Trybuild-->>RustUIHarness: compare_missing_docs_stderr

    RustUIHarness->>CargoClippy: cargo_clippy_ui_cases
    CargoClippy-->>RustUIHarness: output
    RustUIHarness->>RustUIHarness: compare_clippy_output_with_tests_ui_expected

    RustUIHarness->>CargoRustdoc: cargo_doc_ui_cases
    CargoRustdoc-->>RustUIHarness: output
    RustUIHarness->>RustUIHarness: compare_rustdoc_output_with_tests_ui_expected

    RustUIHarness-->>PythonParentTests: all_ui_contracts_pass
    PythonParentTests-->>Pytest: assert_success
Loading

File-Level Changes

Change Details Files
Add a Rust UI harness that uses trybuild plus focused Cargo commands for Clippy and Rustdoc diagnostics.
  • Introduce tests/compile_ui.rs in the template that runs trybuild pass/fail cases and invokes cargo clippy/doc for curated UI fixtures.
  • Define UiCase structs and CLIPPY_CASES/RUSTDOC_CASES arrays with embedded expected .stderr fragments via include_str!.
  • Implement run_cargo_ui_case and assert_expected_failure helpers to execute Cargo commands, enforce failure, and assert that all reviewed diagnostic fragments are present.
template/tests/compile_ui.rs
Introduce generated UI fixtures and expected diagnostic files for compiler, Clippy, and Rustdoc lint contracts.
  • Add a dedicated UI fixtures Cargo package with lints configured to deny targeted Clippy and Rustdoc behaviours.
  • Create pass-case and compile-fail Rust sources covering supported code, unsafe code, and missing docs, with adjacent .stderr files for trybuild.
  • Add Clippy and Rustdoc UI binaries plus a feature-gated lib fixture that intentionally trigger specific diagnostics.
  • Check in narrow expected diagnostic fragment files under tests/ui/expected/ for each Clippy/Rustdoc case.
template/tests/ui/cases/Cargo.toml
template/tests/ui/pass/supported.rs
template/tests/ui/cases/src/lib.rs
template/tests/ui/cases/src/bin/rustdoc_invalid_codeblock_attributes.rs
template/tests/ui/compile_fail/unsafe_code.rs
template/tests/ui/cases/src/bin/clippy_missing_assert_message.rs
template/tests/ui/cases/src/bin/rustdoc_bare_urls.rs
template/tests/ui/cases/src/bin/rustdoc_invalid_html_tags.rs
template/tests/ui/cases/src/bin/rustdoc_unescaped_backticks.rs
template/tests/ui/compile_fail/missing_docs.rs
template/tests/ui/cases/src/bin/clippy_disallowed_methods.rs
template/tests/ui/cases/src/bin/rustdoc_broken_intra_doc_links.rs
template/tests/ui/cases/src/bin/rustdoc_missing_crate_level_docs.rs
template/tests/ui/compile_fail/missing_docs.stderr
template/tests/ui/compile_fail/unsafe_code.stderr
template/tests/ui/expected/clippy_disallowed_methods.stderr
template/tests/ui/expected/clippy_missing_assert_message.stderr
template/tests/ui/expected/rustdoc_bare_urls.stderr
template/tests/ui/expected/rustdoc_broken_intra_doc_links.stderr
template/tests/ui/expected/rustdoc_invalid_codeblock_attributes.stderr
template/tests/ui/expected/rustdoc_invalid_html_tags.stderr
template/tests/ui/expected/rustdoc_missing_crate_level_docs.stderr
template/tests/ui/expected/rustdoc_private_intra_doc_links.stderr
template/tests/ui/expected/rustdoc_unescaped_backticks.stderr
Wire trybuild as a dev-dependency into generated Cargo manifests.
  • Add a [dev-dependencies] section to the template Cargo.toml with a pinned trybuild version.
  • Ensure generated projects have trybuild available for the compile_ui harness without extra manual setup.
template/Cargo.toml.jinja
Add a focused parent test that validates the generated compile-time UI contracts end-to-end.
  • Extend test_compilation.py with test_generated_compile_time_ui_contracts that renders a project, inspects the manifest and harness, and asserts presence of key UI fixtures and expected diagnostics.
  • Use parse_toml_file in addition to read_generated_text to parse Cargo.toml and verify trybuild dev-dependency and harness content.
  • Run the generated compile_ui test via cargo with RUSTFLAGS/RUSTDOCFLAGS that deny warnings.
tests/test_template/test_compilation.py
Update documentation and policy tests to describe and assert the new UI harness workflow and constraints.
  • Add a Compile-time UI tests section to the template developers guide explaining trybuild usage, Clippy/Rustdoc coverage limits, and TRYBUILD=overwrite rules.
  • Update the concrete developers-guide.md to describe the fourth parent test and the non-overwrite policy for diagnostics.
  • Extend repository-layout docs to list tests/compile_ui.rs and tests/ui/* directories and their roles.
  • Update AGENTS.md guidance to cover trybuild .stderr review, non-use of TRYBUILD=overwrite in validation, and the division of responsibility between Python rejection tests and Rust UI harness.
  • Expand policy-guidance tests to assert presence of the new documentation strings and references to test_generated_compile_time_ui_contracts and TRYBUILD policy.
template/docs/developers-guide.md.jinja
docs/developers-guide.md
template/docs/repository-layout.md.jinja
template/AGENTS.md.jinja
tests/test_template/test_policy_guidance.py

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@leynos

leynos commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Nah, that's just silly.

@leynos leynos closed this Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant