feat: 7 - living doc copilot - #9
Open
miroslavpojer wants to merge 88 commits into
Open
Conversation
…DME and documentation for clarity
…nes and skill overview
…lete glossary file
…mpact tracing, and entity validation - Implemented `next_id.py` for auto-assigning IDs to user stories, features, functionalities, and acceptance criteria. - Created `compute_gaps.py` to analyze a catalog snapshot and identify gaps in documentation and testing coverage. - Developed `trace_impact.py` to trace the impact of code changes on features, functionalities, and user stories based on a catalog. - Added `validate_entity.py` to validate living doc entities against a canonical schema, ensuring required fields and referential integrity.
- Implemented `scan_ac_links.py` to check for missing or malformed AC link headers in Gherkin scenarios. - Validates AC ID format and checks for duplicates within the same feature file. docs: create gherkin-scenario skill documentation - Added `SKILL.md` for `gherkin-scenario` detailing standards for writing BDD scenarios in Gherkin. - Covers traceability requirements, language use, and anti-pattern avoidance. docs: create gherkin-step skill documentation - Added `SKILL.md` for `gherkin-step` outlining best practices for implementing Gherkin step definitions. - Emphasizes keeping steps thin, encapsulating selectors, and sharing state correctly. docs: create living-doc-pageobject-scan skill documentation - Added `SKILL.md` for `living-doc-pageobject-scan` detailing how to generate and maintain PageObject classes. - Describes modes for creating and maintaining PageObjects, including selector preferences and output artifacts. feat: add manifest diff script for PageObject validation - Implemented `manifest_diff.py` to compare the manifest against PageObject files on disk. - Identifies stale manifest entries and undocumented PageObjects. docs: create living-doc-scenario-creator skill documentation - Added `SKILL.md` for `living-doc-scenario-creator` detailing the process of generating BDD scenarios from User Stories. - Includes workflow steps for mapping acceptance criteria to scenarios and identifying missing steps. feat: add coverage report script for AC tracking - Implemented `coverage_report.py` to generate a report on AC coverage by Gherkin scenarios. - Scans feature files for AC links and compares them against User Stories to identify gaps.
- Updated Gherkin step definitions to remove Java and Scala references, focusing on Python behave and TypeScript Cucumber. - Enhanced Living Doc PageObject scan to include TypeScript examples and clarified output artifact locations. - Revised Living Doc Scenario Creator to improve missing step handling and stub generation, ensuring better integration with PageObjects. - Created a comprehensive implementation roadmap for the Agentic Engineering Toolkit, detailing progress, file layout, and validation checklists.
…n skills with new trigger phrases and improved descriptions
- Introduced trigger evaluations for Gherkin living doc sync to identify sync-related queries. - Added evaluation scenarios for Gherkin scenario creation, focusing on writing and reviewing BDD scenarios. - Implemented evaluations for Gherkin step definitions, emphasizing the distinction between Gherkin text and step binding code. - Created evaluations for living doc page object scanning, including bootstrap and maintain modes for PageObject generation. - Established evaluations for living doc scenario creation, generating BDD scenarios from user stories and handling coverage reports.
…s and new trigger phrases
…nality - Updated SKILL.md for living-doc-create-functionality to clarify Functionality naming and acceptance criteria elicitation. - Enhanced living-doc-create-user-story to streamline narrative elicitation and improve AC generation process. - Revised living-doc-gap-finder to normalize script output and report gaps more effectively. - Improved living-doc-impact-analysis to flag missing coverage and provide a re-test checklist. - Enhanced living-doc-pageobject-scan to better handle fragile selectors and update PageObjects. - Updated living-doc-scenario-creator to ensure accurate scenario generation and coverage reporting. - Refined living-doc-update to maintain AC ID stability and improve documentation practices. - Added gap-report.json to track documentation coverage and identify gaps in User Stories and Functionalities.
…-bdd-copilot agents
…y and completeness
…ntext and function naming conventions
…for living documentation analysis
miroslavpojer
marked this pull request as ready for review
July 15, 2026 08:50
miroslavpojer
requested review from
lsulak,
oto-macenauer-absa and
tmikula-dev
as code owners
July 15, 2026 08:50
…cross User Stories and Acceptance Criteria; clarify inheritance and extension rules.
…ld to cross-reference headers; clarify its usage for step-specific behaviors.
…efinition, user story authoring, impact analysis, and test management - Introduced `living-doc-create-feature` skill for documenting system surfaces as Feature entities. - Added `living-doc-create-functionality` skill to define atomic behaviors with acceptance criteria. - Created `living-doc-create-user-story` skill for authoring user stories with business-level acceptance criteria. - Implemented `living-doc-gap-finder` skill to audit documentation for coverage gaps. - Developed `living-doc-impact-analysis` skill to trace code changes and identify affected documentation entities. - Added `living-doc-pageobject-scan` skill for discovering UI surfaces and generating Playwright PageObject classes. - Introduced `living-doc-scenario-creator` skill for generating Gherkin scenarios from user stories and functionalities. - Created `living-doc-update` skill for amending living documentation entities. - Added `pr-review` skill for structured code reviews on pull requests. - Introduced `tdd-workflow` skill to guide test-driven development processes. - Added `test-data-management` skill for consistent test data setup across tests. - Implemented `test-mocking-patterns` skill to guide test double selection and implementation. - Created `test-unit-review` skill for auditing unit tests against standards. - Added `test-unit-standards` skill to define comprehensive unit testing standards. - Introduced `test-unit-write` skill for generating unit tests from scratch. - Added `token-saving` skill to enforce concise AI responses across all interactions.
oto-macenauer
left a comment
oto-macenauer
left a comment
There was a problem hiding this comment.
Big, well-structured effort — but not mergeable as-is. Inline comments carry the specifics; items with no single code anchor:
PR description vs diff
- Claims new
docs/responsible-agent-use.mdand enhanceddocs/troubleshooting.md— neither is in the diff. - Claims AC states normalized to Title-case
[Planned, In Review, Active, Deprecated]— the branch canonicalizes to lowercaseplanned/in_review/active/deprecated(glossary, schemas, coverage_report.py). Branch is consistent; description is stale. - Assertion counts inflated ~3× (e.g. create-user-story "247 assertions" → actual 18 cases / 78 expectations).
Structure
skills/grows to 23 flat sibling dirs; 13 tightly-coupled living-doc/BDD skills land next to unrelated ones. Worth deciding grouping/prefix convention before merge — renames after adoption break references.skills/shared/is not a skill (no SKILL.md) — breaks the one-dir-one-skill convention and standalone installs (see next_id.py comment).- The three create-* skills share identical structure, a byte-identical
next_id.py, the same validator, and colliding triggers ("write acceptance criteria for X" routes ambiguously) — consider one entity-parameterized create skill. - Three AC-ID formats coexist (
AC:US-001-01shared lib,US-042-AC-1living-doc-update SKILL/evals,-AC-split in trace_impact.py) — unify before anything depends on them.
Testing
- None of the 12 operational Python scripts has tests; only the shared lib does (happy-path), and nothing in CI runs even that.
Must-fix before merge: remove the 3 committed .DS_Store files, fix the 3 unparseable evals.json, fix trace_impact.py AC matching, fix the broken doc links from the docs/ moves, correct the PR description.
| @@ -0,0 +1,5 @@ | |||
| **/__pycache__/* | |||
| **/.DS_Store | |||
There was a problem hiding this comment.
This rule doesn't untrack already-committed files — this PR commits three .DS_Store files (repo root, skills/living-doc-gap-finder/scripts/, skills/living-doc-impact-analysis/scripts/). Please git rm --cached them.
| "id": 2, | ||
| "category": "happy-path", | ||
| "prompt": "I want to document the Orders API as a Feature.", | ||
| "expected_output": "Agent identifies surface_type as API. Asks for endpoint group description, linked User Stories, owned Functionalities, team owners, and external dependencies (e.g. order-db, notification-service). Runs next_id.py to get the next Feature ID from the catalog. Outputs Feature JSON with id=FEAT-002 (or next numeric ID), surface_type=API.",, |
There was a problem hiding this comment.
Stray ,, after the string makes this file invalid JSON (also lines 97 and 112). Any eval loader dies on parse.
| "id": 10, | ||
| "category": "happy-path", | ||
| "prompt": "I need to document this atomic behavior for the checkout domain: 'Reject order when all items in the cart are out of stock'. The Feature hasn't been created yet.", | ||
| "expected_output": "Agent forms verb-phrase name: 'Reject order when all items are out of stock'. Asks which Feature (Checkout Page / FEAT-001 or equivalent) owns this behavior. Runs completeness checklist: boundary conditions (all items out of stock vs. some items out of stock), partial cart availability, zero-quantity items. Notes that the parent Feature must already exist in the catalog with a numeric ID. Outputs canonical Functionality JSON with feature_id set to the correct numeric ID.",, |
| }, | ||
| { | ||
| "id": 14, | ||
| "prompt": "A Feature (e.g., FEAT-001 Checkout Page) is linked to no User Stories at all. What gap type is this?",, |
|
|
||
| ## Does NOT | ||
|
|
||
| - **Write unit or integration tests** — decline and direct the user to `@sdet-copilot` (not yet deployed). Do not write or modify any test code. |
There was a problem hiding this comment.
@sdet-copilot doesn't exist anywhere on this branch (only this agent is in .github/agents/), yet two agent eval cases (ids 6, 25) assert this handoff. Either ship a stub, or reword to a plain decline so the evals don't test a reference to a nonexistent agent.
| return methods | ||
|
|
||
|
|
||
| def collect_called_methods(steps_dir: Path) -> set[str]: |
There was a problem hiding this comment.
Calls are collected only from --steps-dir: a PO method called from another PageObject (composition, base-class helpers) or from fixtures.ts is reported as a false-positive "unused". The docstring claims fixtures are included, but nothing outside steps_dir is scanned — note find_unused_po_components.py does include fixtures.ts.
|
|
||
| # Inline credential literals are a security violation in seed.yaml. | ||
| # Negative lookaheads allow safe patterns (env:, ${, <, ~, null) both quoted and unquoted. | ||
| CREDENTIAL_LEAK_RE = re.compile( |
There was a problem hiding this comment.
Keyword must be the key's prefix, so db_password:, aws_secret_access_key:, admin_token: are not caught; YAML list items (- password: hunter2) also slip through the ^\s* anchor. Consider [\w-]*(password|…)[\w-]* and allowing a leading - .
| HEALING (fix selector drift in failing tests only). | ||
| Triggers on: "scan this webapp", "generate pageobjects", "crawl the UI", "explore the app", | ||
| "discover routes", "seed.yaml", "manifest.json", "first scan", "create page objects", | ||
| "pageobject drift", "re-scan", "refresh manifest", "heal pageobjects", "fix failing tests", |
There was a problem hiding this comment.
"fix failing tests" / "tests are failing" (next line) are far too broad for this skill's description — they'll hijack ordinary unit-test failures that test-unit-write/test-unit-review own. Scope to selector/PageObject failures explicitly.
| missing data-cy (use data-cy-instrument); deleting deprecated BDD files (use bdd-maintain). | ||
| Pairs with data-cy-instrument, living-doc-create-feature, and living-doc-scenario-creator. | ||
| license: Apache-2.0 | ||
| compatibility: GitHub Copilot |
There was a problem hiding this comment.
All 12 new skills declare compatibility: GitHub Copilot, several hardcode .copilot/bdd/ paths, and the agent is Copilot-CLI-only format. The skills themselves are tool-agnostic. Repo-level decision worth making explicit in this PR: is the toolkit Copilot-only, or should compatibility be dropped and the artifact dir parameterized everywhere (some spots use <bdd_artifacts_dir>, others hardcode)?
| When asked for the expected row format, emit this concrete row shape with a specific element description (for example `Checkout confirm button`), not a generic placeholder. | ||
| Keep the issue link as a plain-text URL — no rich-link or terminal-link formatting. | ||
|
|
||
| **User-facing phase labels:** some teams call template instrumentation “Phase 1” and PageObject sync “Phase 3”. Mirror that shorthand. For `⚠️ PROPOSED` or remaining `coverage_gaps`, say: “run Phase 1 first to add the attribute, then Phase 3 to update the PageObject.” |
There was a problem hiding this comment.
The dual numbering (internal Phases 1–7 vs user-facing shorthand where "Phase 1" = internal Phase 4, "Phase 3" = internal Phase 5) is an internal-contradiction machine — the body itself mixes both conventions under internally-numbered headings, and the description's "phases 1, 3, and 5 are framework-agnostic" is ambiguous under this scheme. Pick one numbering and rename the shorthand (e.g. "instrument step" / "sync step").
… corresponding tests
…tag.py; update related scripts to use new library
…ests for validate_entity.py
…d tests for fixture and composition calls feat: update main function in scan_ac_links.py to support --allow-empty option; improve error handling test: add tests for zero feature files handling in test_scan_ac_links.py fix: ensure compute_gaps.py handles orphan features correctly; update gap detection logic fix: update file reading in multiple scripts to use UTF-8 encoding for compatibility test: add tests for manifest_diff.py to ensure forward-slash paths on Windows test: implement tests for validate_artifacts.py's credential leak detection regex
… for missing dependencies
…tation for clarity and consistency
…sition imports; add corresponding tests feat: add test cases for find_unused_steps.py to cover core matching logic feat: implement tests for next_id.py CLI wrapper to verify subprocess functionality
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Living documentation & BDD automation skills
Summary
Adds a 12-skill family for living documentation (User Story / Feature / Functionality
catalog entities, Acceptance Criteria, impact analysis, gap finding) and BDD automation
(Playwright exploration, PageObject generation and healing, Gherkin scenario/step
generation, traceability sync, dead-code maintenance), plus the
@living-doc-bdd-copilotagent that orchestrates them and the
sharedlibrary skill they depend on.What's new
12 skills (
skills/):living-doc-create-user-storyliving-doc-pageobject-scanliving-doc-create-featuredata-cy-instrumentliving-doc-create-functionalityliving-doc-scenario-creatorliving-doc-updatebdd-maintainliving-doc-impact-analysisgherkin-stepliving-doc-gap-findergherkin-living-doc-syncPlus
skills/shared/— the AC-ID grammar, entity-ID assignment, and glossary/schemareferences these 12 skills share, installable as its own skill so a standalone
single-skill install can pull it in as a sibling (see
skills/shared/lib/README.md).1 agent (
.github/agents/living-doc-bdd-copilot.agent.md) — dispatches to the12 skills above by intent (catalog vs. automation, mode-specific routing), with an
Installation section listing the exact install command for each dependency.
13 operational Python scripts across the 12 skills (ID auto-assignment, AC-tag
scanning, gap computation, impact tracing, coverage reporting, artifact validation,
dead-code detection) — see Testing for coverage.
16 new doc files under
docs/guides/anddocs/testing/(one guide per skill,the agent guide, and an agent-testing methodology doc), plus
docs/skill-testing.mdmoved to
docs/testing/skill-testing.md.Conventions this PR establishes
AC:<PARENT>-<nn>, where<PARENT>isUS-<n>orFUNC-<nnn>(numeric only — no slugs, no
FEAT; Features own Functionalities, not ACs directly).Enforced by
skills/shared/lib/ac_tag.py, shared by every script that parses orvalidates an AC ID so the grammar can't fork across scripts again.
planned,in_review,active,deprecated— consistent across the glossary, JSON schemas, and everyscript that reads or writes state.
compatibilityrequirement;they're plain Markdown instructions plus Python scripts, usable by any tool that can
load a
SKILL.md. Only the agent definition itself (.github/agents/, Copilot CLItool namespace) is Copilot-specific by design.
Testing
evals/evals.json(happy-path, regression, negative, edge-case, and output-formatcategories).
12 skills'
evals/trigger-eval.json, verifying each skill'sdescriptionfires onthe right prompts and stays out of the way of unrelated ones.
(
test_*.py, plain-assert style, runnable directly — seeskills/shared/lib/README.mdfor the convention):
scan_ac_links.py,coverage_report.py,validate_entity.py,trace_impact.py,manifest_diff.py,compute_gaps.py,find_unused_po_methods.py,validate_artifacts.py. The shared library (living_doc_id.py,ac_tag.py) istested independently (
skills/shared/lib/test_living_doc_id.py).review-fix-roadmap.md.Review history
This branch went through a full review pass (
AbsaOSS/agentic-toolkit#9, review4961870872) — 22 inline comments plus a summary verdict, 30 action items across 9problem groups. All fixes are tracked with per-item verification notes in
review-fix-roadmap.md; 26/30 items are resolved as of thisrevision, with the remaining 4 (test-coverage completion + this description's accuracy)
addressed in this same update.