Related docs: Architecture · Agent Architecture · Graph Schema · Toolchain
The CodeGraph pipeline is organised into five named phases. The graph is the shared foundation, but the public documents also describe later enrichment and gap checks that feed back into earlier artifacts. Treat the diagram as a dependency view, not a strict one-pass waterfall. The exact PoC re-run points and revalidation rules have not yet been recovered.
flowchart LR
P1["Phase 1\nSource Ingestion"]
P2["Phase 2\nDomain Documentation"]
P3["Phase 3\nReference Layer"]
P4["Phase 4\nProcess Discovery"]
P5["Phase 5\nVault Assembly"]
P1 -->|"Graph + Records"| P2
P1 -->|"Graph + Records"| P3
P2 -->|"VPs"| P3
P2 -->|"VPs"| P4
P3 -->|"Reference Layer"| P4
P2 -->|"VPs"| P5
P3 -->|"Reference docs"| P5
P4 -->|"E2E docs"| P5
Important: Phases 2–5 are human-in-the-loop processes. The tools automate extraction, generation, and enrichment. The documentation quality comes from the guided agent workflow — running the scripts alone does not produce the complete output.
Origin-case results (reported): Over 15 sprints, the engagement produced 19 Validation Packages, 851 documented programs, 572 business rules, 248 user stories, 506 catalogued screens, and 18 E2E processes. The PoC team reported no detected hallucinations on its first complete pass and 191 explicit SME-review items. The public repository does not include the claim-level audit protocol or accepted artifacts needed to reproduce that result. See Baseline Status.
Goal: Build a provenance-tagged graph of the relationships detected within the declared source and parser scope. Completeness must be measured separately.
Inputs: Source files, database schema definitions, compiler cross-reference output (if available)
Outputs: graph.json (recorded graph with provenance), extraction.json (structured records), Neo4j database
Legacy vs. modern languages: The origin case used regex-based detection for a proprietary 4GL. Modern parsers can add AST, type, bytecode, or compiler evidence, but syntax recognition does not guarantee semantic target resolution. Higher-tier evidence is available only for relationships a tool actually resolves and tests; no modern-language implementation is published here.
The Graph Builder performs static analysis of the entire codebase in a single pass. It processes source files, schema definitions, and — where available — compiler-generated cross-reference files. Compiler cross-reference output is the highest-confidence evidence source because it is produced by the language compiler itself, not inferred from patterns.
Every relationship edge written to the graph carries four provenance properties at write time:
source: "xref" | "regex" | "schema" | "derived"
source_file: path/to/the/file/that/provided/evidence
source_line: integer line number (0 if not applicable)
confidence: "compiler" | "regex" | "schema" | "derived"
This origin-case source enum cannot represent AST detection honestly. The public
parser guide documents ast as a PROPOSED-EXPERIMENT, not as a silently adopted
change to the validated baseline.
The Document Extractor then queries the graph using 8 extraction methods to produce a flat structured record set used by all subsequent phases:
| Extraction Method | What It Produces |
|---|---|
| CRUD matrix | Every program's read/write/create/delete access per table |
| Call chains | Full program-to-program call graphs by module |
| Screen inventory | All UI screens with type, module, and parent callers |
| Status fields | All fields containing status/state values, with programs that update them |
| Integration endpoints | Cross-module and external system interaction points |
| Include tree | All include file dependencies and parameterisation |
| Coverage gap analysis | Programs, tables, and screens not referenced in any VP |
| Entry point detection | Candidate roots — programs with no recorded inbound CALLS edges |
Coverage against VPs cannot be completed before VPs exist. Initial Phase 1 output can provide the program/table/screen inventory; the VP coverage comparison is a later or repeated check. The precise placement used in the PoC is not recovered and must not be inferred from the current phase numbering.
The Graph Uploader writes the graph to Neo4j. It supports two modes:
--wipe: Full reset — wipes existing graph and uploads from scratch. Use for initial setup.--incremental: Tag-and-prune — merges new data, marks stale nodes, optionally prunes with--prune. Use for updates.
Not all graph edges are equally reliable. The confidence tier system makes this explicit:
| Tier | Source | Trust Level |
|---|---|---|
compiler |
Compiler/type output or an AST relationship resolved against known symbols | Highest for that resolved relationship |
schema |
Schema definition files (.df, .sql, etc.) | High — schema-verified |
regex |
Pattern-matching on source code | Medium — inferred |
derived |
Documentation files, filesystem structure | Lower — document-derived |
Documentation agents receive the confidence tier for every piece of data they use, and are instructed to prefer compiler-tier evidence when making structural claims.
Goal: Produce a Validation Package (VP) for each business domain — a complete, evidence-grounded document a domain expert can validate.
Inputs: Extraction records, knowledge graph, raw source files Outputs: One VP markdown file per domain
Each VP is a self-contained document covering one business domain:
- Overview — domain scope, program count, module coverage
- Program inventory — all programs with type, purpose, entry points, and tables
- Screen descriptions — character-mode and GUI screens with fields and user-facing functions
- CRUD matrix — which programs read, write, create, and delete which tables
- Process flows — Mermaid flowcharts showing step-by-step business logic
- Business rules — numbered, cited rules extracted from source (BR-01, BR-02, ...)
- User stories — functional requirements in "As a [role], I want [action], so that [benefit]" format
- Validation checklist — questions for domain expert review
sequenceDiagram
participant G as Neo4j Graph
participant E as Extraction Records
participant A as VP Author Agent
participant V as Documentation Validator
participant T as Validation Triad
participant C as Citation Auditor
A->>G: Query domain programs, tables, screens
A->>E: Retrieve CRUD matrix, call chains, status fields
A->>A: Author VP with citation tags on every claim
A->>V: Submit VP for schema cross-reference
V->>V: Verify all table/field references against .df files
V-->>A: Schema-reference report (must be zero unresolved references)
A->>T: Submit corrected VP to Defender
T->>T: Defender argues correctness (cites evidence)
T->>T: Challenger argues against (identifies gaps)
T->>T: Judge issues binding verdict
T-->>A: Required corrections
A->>A: Apply corrections
A->>C: Submit to Citation Auditor
C->>C: Verify every [TAG] against source
C-->>A: Verification report (must be all pass)
A VP is not complete until it passes all three quality gates:
- Documentation Validator — zero unresolvable table/field references
- Validation Triad verdict — Judge's required corrections are applied
- Citation Auditor — 100% of citation tags verified against source
Goal: Build the complete structural knowledge base that supports process discovery, compliance analysis, and architectural understanding.
Inputs: Knowledge graph, extraction records, validated VPs Outputs: 12 reference sections — data dictionary, program inventory, screen catalog, dependency maps, process flows, integration maps, modernisation assessment, new insights, include files, status state machines, cross-VP orchestration, compliance gap designs
| Section | What It Contains |
|---|---|
| Data Dictionary | All tables, fields, indexes, and sequences across all databases |
| Program Inventory | All programs by module with type, purpose, and coverage status |
| Screen Catalog | Per-screen documentation: layout, widgets, user role, navigation |
| Dependency Map | Program→table CRUD, screen→program, include tree, cross-module calls |
| Process Flows | Domain-level Mermaid flowcharts for all major workflows |
| Integration Maps | External system channels — protocols, data formats, staging tables |
| Modernisation Assessment | Complexity scores, dead code candidates, migration risk register |
| New Insights | Shared state coupling, API surface map, hardcoded decision trees |
| Include Files | Shared macro and utility include inventory, parameter documentation |
| Status State Machines | All status fields with Mermaid state diagrams per domain |
| Cross-VP Orchestration | End-to-end processes spanning multiple VPs (foundations for Phase 4) |
| Compliance Gap Designs | Regulatory gap identification and remediation design specifications |
Status fields (fields named *_status, *_stat, or *_state) are indexed from the extraction records. For each status field, the Graph Builder has captured which programs write to it and what values they assign. The Status Index Generator uses this data to produce Mermaid state diagrams:
stateDiagram-v2
[*] --> NEW : creation program writes NEW
NEW --> RELEASED : planner program writes RELEASED
RELEASED --> IN_PROGRESS : execution program writes IN_PROGRESS
IN_PROGRESS --> COMPLETE : completion program writes COMPLETE
COMPLETE --> CLOSED : archive program writes CLOSED
COMPLETE --> REOPENED : exception handler writes REOPENED
REOPENED --> IN_PROGRESS : execution program writes IN_PROGRESS
Write evidence establishes that a program assigns a value. It does not establish the
ordering between two values unless the code also reads or guards on the prior state,
or another accepted evidence source establishes that order. Generated transitions
without ordered evidence must be marked [INFERRED:reason].
Where Phase 3 identifies code that does not meet regulatory requirements, a gap design specification is produced. Each specification includes: the regulatory requirement, the current code behaviour, the gap description, evidence citations, and a design specification for the remediation — not just a flag, but a design that a developer could implement.
Goal: Discover and document all end-to-end business processes that span multiple modules or VPs.
Inputs: Discovery uses the knowledge graph, extraction records, and schema. Documentation and cross-linking additionally use validated VPs and the reference layer. Outputs: End-to-end process flow documents
Business processes rarely live inside a single module. A goods receipt that begins in warehouse management triggers inventory updates in the core system, which may generate integration messages through a messaging layer. No single Validation Package captures this. Phase 4 discovers these cross-domain processes systematically.
flowchart TD
subgraph L1["Layer 1 — Entity Lifecycle"]
SIG1["Status fields transitioning\nacross modules"]
end
subgraph L2["Layer 2 — Screen Entry Points"]
SIG2["Screens with no parent caller\nin the graph"]
end
subgraph L3["Layer 3 — Cross-Module Handoffs"]
SIG3["Tables written by one module\nread by another"]
end
subgraph L4["Layer 4 — Merge & Score"]
MERGE["Merge · Score · Gap-detect"]
end
subgraph CLUSTER["Screen Clustering"]
SC["Cluster by shared tables\nand call chains"]
end
L1 --> L4
L2 --> L4
L3 --> L4
CLUSTER --> L4
L4 --> CAND["Scored candidates\nHigh / Medium / Low"]
CAND --> TRIAGE["Auto-triage\nnew process / sub-workflow / infra"]
TRIAGE --> DOC["E2E Process Documents"]
Each discovered process is documented in a canonical template:
- Discovery metadata (layers that triggered it, confidence score, VP coverage)
- Entry point and primary entity
- Step table (action, program, tables read/written, evidence per step)
- Mermaid process flow diagram
- Data handoff map (cross-module staging tables)
- Screen catalog section (screens involved by step)
- Business rules referenced
- Schema references (tables and key fields)
- Status machine cross-links
- CRUD matrix for the full process
- SME review items (gaps requiring human confirmation)
In the proof-of-concept, the team reported that Layer 2 produced no cross-module candidates and Layer 3 identified six staging tables associated with inter-module communication. The engagement concluded that the system used a pure data-handoff architecture rather than direct inter-module calls.
Shared read/write access alone does not establish temporal handoff direction, and an empty call set is bounded by parser recall. The conclusion is therefore a reported PoC finding supported by converging analysis, not something this public repository can independently reproduce without the original graph and review records. The team reported that it materially changed the modernisation strategy.
Goal: Unify all outputs into a navigable, wikilinked Obsidian vault with audience-based navigation.
Inputs: All Phase 1–4 outputs Outputs: Complete documentation vault ready for use and living documentation deployment
The VP Enricher adds E2E participation sections and screen identifier cross-references to all Validation Packages, connecting the domain-level view to the process-level view.
Because this modifies previously validated documents, affected sections must be revalidated before release. The public repository does not record which gate was re-run in the PoC; implementations must keep this unresolved rather than assume that the Citation Auditor alone was sufficient.
The Vault Wikilinker converts all markdown cross-references in the full vault to Obsidian [[wikilinks]] format, and adds YAML frontmatter to every file — tags (content type, domain, sprint, review status), aliases (for fuzzy search), and cross-reference metadata.
The Vault Optimizer generates Maps of Content (MOC) pages — one per major section — that serve as navigable indexes. It also builds a HOME.md entry point that routes four audience types (domain expert, IT architect, manager, new reviewer) to the content most relevant to them.
The Vault Verifier performs a final integrity pass: all wikilinks resolve, all cited programs and tables exist in the graph, all required content sections are present.
flowchart TD
HOME["HOME.md — Entry point"]
DE["Domain Expert"]
ITA["IT Architect"]
MGR["Manager / Stakeholder"]
NEW["New Team Member"]
VP["Validation Packages\nBusiness rules · Process flows"]
E2E["E2E Processes\nCross-module flows"]
COMP["Compliance Gaps\nRisk register"]
PATH["Review Path\nSequenced reading"]
HOME --> DE --> VP
HOME --> ITA --> E2E
HOME --> MGR --> COMP
HOME --> NEW --> PATH
Re-running phases. The described structural scripts are intended to be
repeatable on identical logical input. Agent output is not deterministic and must
pass the same gates whenever regenerated. Phase 1's documented --incremental mode
merges new data, but the public specification does not yet define citation
re-anchoring or affected-document propagation across source revisions.
Partial runs. Individual reference sections in Phase 3 can be regenerated independently. Individual VPs in Phase 2 can be re-authored without affecting other VPs.
Human review points. Phase 2 VP authoring and Phase 3 reference expert work involve significant human review between script runs. The scripts prepare the data and verify the output — the agents and the reviewers produce the quality.
SME review tracking. All [SME-REVIEW-REQUIRED:ID] items across the vault are aggregated into an SME Review Dashboard — a single file listing all open items with their IDs, locations, and questions. This is the handoff document for domain expert review.