Repository navigation
fix: address grounding, Wiki, and graph benchmark gaps - #274
Merged
Merged
Conversation
… changes Regenerate the managed instruction block in the root and tool-config CLAUDE.md/AGENTS.md copies so they carry the new wiki query line, pin src/agent-prompt.ts as the private prompt-file writer that replaced the setup population writer, and expect resolver version compiler-first-v3.
Allowing every language's call to resolve to its own caller turned super().save(), other.save() and imported same-named calls inside a method into false self-loops (61 in flask, 2,324 in django; Python call precision 0.828 -> 0.745 against the SCIP oracle). Outside C#, a self edge now needs an unimported bare name inside a function, or a self/this/cls receiver inside a method. Python call F1 returns to 0.731 (main 0.732).
crate:: paths were rooted at the first path segment, so every use in a workspace member (crates/<name>/src/...) missed its module file, and a use of a sibling member's library never resolved. Root crate:: at the nearest ancestor holding lib.rs or main.rs, and bind a leading segment that names a unique workspace member library to that member's src root. ripgrep against the SCIP oracle: file dependency F1 0.006 -> 0.411 (precision 1.000), call F1 0.329 -> 0.372 bare, 0.304 -> 0.345 qualified.
C# references resolved only within one file, so no cross-file call, heritage or construction edge existed and impact on a C# file was always empty. Resolve a type reference, or a call whose receiver names a type, by C# lookup order: enclosing types, each enclosing namespace from the inside out, then the file's using directives and the global namespace, binding only when exactly one type answers at the first level that has any. Qualified and global:: names bind to the type in that namespace, never to a same-named lexical type. serilog against the SCIP oracle: call F1 0.295 -> 0.549 (precision 0.966), file dependency F1 0.000 -> 0.293 (precision 1.000).
…evidence The benchmark-gap change stopped fresh reads from caching the body the committed hash names. With no old side, the comment, layout, rename and nearby relaxations could not run, and every edit read changed: on the Hono knowledge benchmark (132 facts, C0-C5) false alarms rose from 168 to 240 and fresh retention fell from 0.675 to 0.579, while stale facts served stayed at 0.271 (62 caught either way). Restore the content-addressed write on a fresh read. Keep the stronger rebind rule only where it has evidence: a moved symbol is judged by its old body when this checkout has it (rename-only, layout, comment or nearby), and otherwise by structure as #229 settled. A rename hash that cannot be computed is no evidence either way. A legacy grounding without a body hash is compared by structure again instead of turning every older scaffold unverified.
The keyword guard (only, all, no, not, default, ...) and the control-flow guard (any edit containing if, return, await, ...) forced "touched" for most facts and edits. Measured on the Hono knowledge benchmark with the body cache restored: they added 36 false alarms (none-trap facts 135 -> 171) and caught no stale fact (62 caught with and without them). Restore the anchor-and-expression rule; the h018 CORS case stays touched under it.
A grounding with no committed body hash can only be compared by structure, which cannot see an edited literal. Keep the benchmark-gap rule that reports a structural match as unverified, with the GROUNDING_NO_BASELINE notice and the reviewed reground that records the baseline. Fixtures written before that rule now commit a body hash where they test something else, and the neighbourhood lease test states the legacy verdicts explicitly.
An arrow-function or other declaration whose rename cannot be hashed skipped the changed-nearby check and fell straight to the structural fallback, so moved Hono helpers whose edit was far from the fact read changed. Run the nearby check first; only a rebind with no verdict falls back to structure.
theyashasvipandey
force-pushed
the
codex/benchmark-gap-fixes
branch
from
October 14, 2026 06:20
f3ecdc2 to
14ef1e2
Compare
Add the Unreleased changelog entries with their upgrade effects (graph rebuild, Wiki index v5 rebuild, renamed assignment nodes, legacy groundings reading unverified), correct the compatibility notes, router state and the durable-change-signal pattern to the behaviour that shipped, and remove the implementation session report, which described the reverted cache and nearby rules.
theyashasvipandey
marked this pull request as ready for review
October 14, 2026 06:45
The quiet window after MEX_PTY_CHILD_READY began as soon as the marker was read. When the PTY delivered the child's own line ending in a later read, that "\r\n" landed in the window and failed the no-MEX-output check intermittently. Match the marker with its line ending first.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Repair grounding, Wiki, graph-impact, and evaluator behaviors identified in the supplied benchmark reports. A moved symbol with a changed body now retains its warning, missing baselines stay unverified, root metadata contributes bounded Wiki search hints, and file impact exposes callers before seed definitions exhaust the existing output cap.
The implementation report and owner verification checklist records every gap, local evidence, remaining decisions, pinned subjects, and required major runs. Compatibility guidance documents explicit Graph refresh and Wiki-index rebuild requirements.
Why
Benchmark findings exposed correctness and visibility gaps that can hide changed knowledge or omit useful graph/search evidence. This PR implements feasible fixes and keeps unresolved corpus-dependent or architectural decisions explicit. It makes no claim of improved benchmark-wide scores.
This is a draft stacked on the open release preparation PR #271, targeting
codex/release-0.8.4ata86f9b6. All changes were made in an isolated worktree; the original checkout's unrelated skills, local configuration, and artwork are excluded.Type of change
How to test
Local checks were kept lightweight and sequential with one worker. Passed results include 81 graph/protocol/recursion fixtures, 91 grounding/Wiki/synthesis fixtures, 20 transport/setup fixtures, graph CLI exit regressions, and all 46 graph evaluator self-tests. The report lists the additional focused selections and their exact scope. Staged diff whitespace checks passed.
Full
npm run typecheckdid not complete: root TypeScript exhausted the local 768 MiB old-space cap; Hub typechecking was not reached. Syntax/transpile checks and two isolated helper typechecks passed, but do not replace whole-tree typechecking. Full build, full suite, platform, resource, and benchmark gates remain pending.Owner verification from this PR's commit:
npm ci npm run typecheck npm test -- --maxWorkers=1 --no-file-parallelism npm run eval:test npm run build npm run test:hub:packageThen follow the report's pinned knowledge/graph subjects, immutable-read and recovery cases, native Windows prompt checks, and existing resource/platform CI. Preserve labels, scoring rules, and budgets. The original custom
bench/harness and raw manifests are absent from this checkout; the owner must supply their entry commands. Actual AI-assisted h018 safety and benchmark-wide results are unmeasured here.Checklist
npm test) — full suite pending; focused results recorded above.Code-graph changes
main— intentionally stacked oncodex/release-0.8.4/ Release 0.8.4: terminal setup by default and clearer failures #271 to keep its diff scoped.LanguageExtractororFrameworkResolverinterface.core / discuss-firstissue is linked above — extractor/resolver identities and grounding semantics change as documented; maintainer review required before merge. Graph schema remains v4.