Skip to content

toolchain.md

Latest commit

 

History

History
218 lines (165 loc) · 15.4 KB

toolchain.md

File metadata and controls

218 lines (165 loc) · 15.4 KB
Raw

CodeGraph — Python Toolchain Specification

Related docs: Methodology · Graph Schema · Parser Extension Guide

Overview

The public design names 36 Python tools organised by pipeline phase. Their intended roles, inputs, and outputs are specified below, but their implementations, dependency files, command-line parsers, and interchange examples are not included in this repository. Treat the rows as recovery/build targets, not as executable files.

The described tools are intended to be independently runnable, to avoid modifying their own inputs, and to return clear failures. Those properties require tests in an implementation; prose alone does not establish them.

flowchart LR
    P1["Phase 1\nSource Ingestion\n8 scripts"]
    P2["Phase 2\nDocumentation Validation\n4 scripts"]
    P3["Phase 3\nReference Generators\n7 scripts"]
    P4["Phase 4\nProcess Discovery\n12 scripts"]
    P5["Phase 5\nVault Assembly\n5 scripts"]

    P1 -->|"graph.json\nextraction.json\nNeo4j DB"| P2
    P1 -->|"graph.json\nextraction.json"| P3
    P2 -->|"Validated VPs"| P3
    P3 -->|"Reference layer"| P4
    P2 -->|"VPs"| P4
    P4 -->|"E2E docs"| P5
    P3 -->|"Reference docs"| P5
    P2 -->|"VPs"| P5
Loading

Phase 1 — Source Ingestion (8 scripts)

Core Graph Build

Tool Role Input Output
Graph Builder Scans the declared source scope. Parses source files and schema definitions using registered parsers. Builds graph JSON containing detected nodes, edges, and provenance; completeness depends on parser and source coverage. Source dir, schema files, parser config graph.json (~14 MB in the origin case), import.cypher
Document Extractor Queries the graph using 8 extraction methods to produce structured records: CRUD matrices, call chains, screen inventory, status fields, integration endpoints, include tree, coverage gap analysis, entry point detection. graph.json + Neo4j extraction.json (11K+ records)

Graph Upload and Sync

Tool Role Input Output
Graph Uploader Uploads the graph to Neo4j. --wipe mode for initial setup (full wipe + upload). --incremental mode for updates (merge + tag-and-prune). --prune flag to delete stale nodes after confirmation. Unresolved: this table says import.cypher; the illustrative command below says graph.json. Recover the PoC contract before implementation. Neo4j database
Graph CSV Exporter Alternative to Graph Uploader for bulk-import scenarios. Exports nodes and edges as CSVs compatible with neo4j-admin import. Faster for very large graphs. graph.json neo4j_csv/ directory

Graph Validation

Tool Role Input Output
Graph Validator Parses VP markdown documents and runs Cypher-equivalent checks to detect broken references, programs mentioned in VPs that don't exist in the graph, and coverage gaps. VP docs + Neo4j graph_validation_report.md
Graph Verifier Replays every provenance proof for every graph edge against the original source files. Confirms that the cited source file exists, the cited line number is in range, and the line contains an expected pattern for the edge type. This verifies provenance recording, not extraction precision or recall. Graph edges + source files verification-report.json
Screen Graph Auditor Compares screens found on disk against screen nodes in the graph. Produces two lists: screens parsed but not indexed, and screens indexed but no longer on disk. Source file tree + Neo4j screen_audit.json

Parser Registry

Tool Role Input Output
Parser Registry Central registry for all language parsers. Maps file extensions to parser implementations. Used by the Graph Builder to select the correct parser for each file. See Parser Extension Guide to add a language. parser_config.json Parser instances

Phase 2 — Documentation Validation (4 scripts)

Tool Role Input Output
Documentation Validator Cross-references table and field claims in VP documents against parsed database schema files. Any unresolvable schema reference is a failure; this does not evaluate all claim types. VP markdown files + schema files Schema-reference report (failures + line numbers)
Documentation Validator (Final) Full-coverage audit variant of the Documentation Validator. Runs a comprehensive check including partial-match detection, aliased table names, and field-level verification. Run before marking a phase-2 sprint complete. VP files + schema files Final audit report with full detail
VP Marker Injector Inserts structured <!-- BEGIN AUTO --> / <!-- END AUTO --> markers around auto-generated sections in VP documents. Enables targeted regeneration of auto-generated content without overwriting human-reviewed sections. VP markdown files Updated VP files with markers
Documentation Generator Generates or augments VP markdown from graph data. REPLACE mode regenerates auto-generated sections (between markers). AUGMENT mode adds content to empty sections without touching existing content. graph.json + markers Updated VP sections

Phase 3 — Reference Generators (7 scripts)

Some Phase 3 rows depend on later information: the Screen Catalog can consume Phase 4 mappings and the Glossary Enricher consumes E2E terminology. These are documented iteration points, not evidence that the pipeline ran once in strict phase order. The exact PoC re-run and revalidation rules are not recovered.

Tool Role Input Output
Schema Parser Parses database schema definition files into structured Python objects (tables, fields, indexes, sequences). Shared dependency for all reference generators that need schema data. Schema definition files Parsed schema objects (in-memory + cached JSON)
Undocumented Table Generator Produces a markdown inventory of all tables present in the schema but not referenced in any VP, cross-referenced with graph metadata (usage count, related programs, field count). extraction.json + schema parser undocumented-tables.md
Undocumented Program Generator Produces a markdown inventory of all programs present in the source but not covered by any VP, with module, file type, and call-chain depth from the graph. extraction.json + source headers undocumented-programs.md
Screen Catalog Generator Generates one markdown file per screen from extraction records. Each file covers: screen identifier, type, module, primary table, user role signals, navigation context, related E2E processes (if Phase 4 has run), and user stories. extraction.json + screen classification ~500 SCR-*.md files
Status Index Generator Generates a master status field index grouped by database and table. For each status field: all values found in source, all programs that write to it, and a stub Mermaid state diagram. extraction.json Status index markdown + state diagram stubs
Screen Classifier Classifies all screens into functional roles (inquiry, report, maintenance, processing, monitor, menu) based on their program type, widget composition, and CRUD patterns. Assigns stable SCR-NNN identifiers used across all documentation. extraction.json + audit data screen_classification.json
Glossary Enricher Expands the project glossary with terms discovered during E2E process analysis. Adds Obsidian alias entries for all identified acronyms and project-specific terminology. Vault content + E2E terminology Updated glossary.md

Phase 4 — Process Discovery (12 scripts)

Discovery

Tool Role Input Output
E2E Discovery Engine Implements the four-layer hybrid discovery framework. Layer 1: candidate entity lifecycle from status evidence. Layer 2: candidate screen entry points (no recorded parent). Layer 3: candidate cross-module handoffs (shared-table access). Layer 4: merge, score, gap-detect. extraction.json + Neo4j + schema candidates.json (scored candidate list)
Screen Process Discoverer Clusters all screens by shared table access patterns, common call chain membership, and user role signals. Produces screen cluster data that supplements the signal-based discovery with usage-pattern candidates. Screen catalog + Neo4j screen_clusters.json
E2E Candidate Triage Auto-classifies all candidates as new_e2e_process, sub_workflow, or infrastructure. Applies configurable confidence thresholds. Filters out candidates with insufficient evidence. screen_clusters.json + classification + mapping e2e_candidate_triage.json

Documentation

Tool Role Input Output
E2E Document Generator Generates initial E2E process documents from triaged candidates following the canonical 11-section template. Produces one markdown file per process with all evidence citations included. Triage + mapping + classification + Neo4j E2E-NN-*.md files
E2E Enricher Enriches all E2E documents to the full canonical template. Adds schema references, status machine cross-links, and CRUD matrices sourced from graph queries. Neo4j + screen docs + status machines Updated E2E docs
E2E Documentation Enricher Adds Screen Catalog sections to Tier 1 E2E documents (manually/discovery-authored). Links individual screens to their process steps with SCR-NNN identifiers. E2E screen mapping + screen classification Updated Tier 1 E2E docs

Mapping and Validation

Tool Role Input Output
E2E Screen Mapper Maps all E2E processes to their screens bidirectionally. Updates both screen documentation headers (which E2E processes use this screen) and E2E documents (which screens appear in each step). Screen clusters + classification e2e_screen_mapping.json
E2E Subworkflow Updater Updates screen document E2E headers for triaged sub-workflows and newly identified processes. Propagates triage results back to individual screen files. e2e_candidate_triage.json + mapping Updated screen docs
E2E Citation Auditor Verifies every evidence citation in all E2E documents ([CODE:], [SCHEMA:], [GRAPH:], etc.). Blocking on any failure. Produces audit report with pass/fail per citation. E2E docs + source files + Neo4j Audit report
E2E Cross Auditor Verifies consistency between E2E documents and screen documents. Checks that bidirectional screen-to-E2E mappings are symmetric — if an E2E doc references a screen, that screen's doc must reference the E2E process. E2E docs + 500+ screen docs e2e_cross_audit_results.json
E2E Enrichment Verifier Verifies that all E2E documents meet the 11-section canonical template specification. Checks section presence, minimum content length, and required fields per section. All E2E docs e2e_verify_results.json
Program-to-Screen Mapper Scans the screen catalog filesystem and builds a bidirectional mapping between program names and their SCR-NNN identifiers. Used as a lookup table by multiple other tools. Screen catalog files program_to_scr.json

Phase 5 — Vault Assembly (5 scripts)

Tool Role Input Output
VP Enricher Adds E2E participation sections and SCR-NNN cross-references to all 19+ VP documents. Connects the domain-level view to the process-level view. VP docs + E2E registry + screen mapping Updated VP files
Vault Wikilinker Converts ordinary markdown cross-references to Obsidian [[wikilinks]]. Adds YAML frontmatter to every file: tags, aliases, and sprint provenance. Idempotent — running twice produces the same result. All vault .md files Updated vault with wikilinks + YAML
Vault Optimizer Generates Maps of Content (MOC) pages for each major section. Builds the HOME.md audience navigation entry point. Merges Obsidian workspace configuration. Entire vault MOC pages + HOME.md + .obsidian/ config
Vault Verifier Final integrity pass. Verifies: all wikilinks resolve to existing files, all SCR-NNN identifiers are consistent, all required content sections are present in their expected locations, all YAML tags are valid. Produces a pass/fail report. Entire vault Pass/fail verification report
Index Refresher Regenerates all _index.md files from the current filesystem state. Validates wikilink targets. Idempotent — safe to run at any point during or after assembly. Entire vault Updated index files + wikilink_validation.json

Data Files

Key JSON data files produced and consumed by the pipeline:

File Typical Size Contents
graph.json 10–20 MB Recorded knowledge graph (detected nodes, edges, provenance)
extraction.json 1–3 MB Structured extraction records (CRUD, call chains, screens, etc.)
screen_classification.json 100–200 KB Screen functional role and SCR-NNN assignments
candidates.json 200–500 KB E2E discovery candidate list with scores
screen_clusters.json 50–100 KB Screen clustering analysis results
e2e_candidate_triage.json 50–100 KB Triaged E2E candidates with classifications
e2e_screen_mapping.json 100–250 KB Bidirectional screen-to-process mapping
wikilink_validation.json 1–5 KB Vault wikilink integrity results

Illustrative Command Shapes

Not executable from this repository. The commands below preserve the public interface sketch for implementers. The files they invoke are not shipped. In particular, the Graph Uploader input conflicts with its table entry above and must be recovered rather than guessed.

# Phase 1 — Build graph and extract records
python tools/phase1/graph_builder.py \
  --source /path/to/codebase \
  --schemas /path/to/schemas \
  --output graph.json

python tools/phase1/graph_uploader.py \
  --input graph.json \
  --neo4j-uri $NEO4J_URI \
  --wipe  # First run only; use --incremental for updates

python tools/phase1/document_extractor.py \
  --graph graph.json \
  --neo4j-uri $NEO4J_URI \
  --output extraction.json

# Phase 2 — Validate documentation (after AI agent authoring)
python tools/phase2/documentation_validator.py \
  --vps docs/validation-packages/ \
  --schemas /path/to/schemas

# Phase 3 — Generate reference layer
python tools/phase3/screen_catalog_generator.py \
  --extraction extraction.json \
  --output docs/reference/03-screen-catalog/

python tools/phase3/status_index_generator.py \
  --extraction extraction.json \
  --output docs/reference/09-status-state-machines/

# Phase 4 — Discover E2E processes
python tools/phase4/e2e_discovery.py \
  --extraction extraction.json \
  --neo4j-uri $NEO4J_URI \
  --schemas /path/to/schemas \
  --output candidates.json

python tools/phase4/e2e_candidate_triage.py \
  --candidates candidates.json \
  --threshold 0.7 \
  --output e2e_candidate_triage.json

# Phase 5 — Assemble vault
python tools/phase5/vault_wikilinker.py \
  --vault docs/

python tools/phase5/vault_verifier.py \
  --vault docs/

See Methodology for the full guided workflow including the AI agent phases.