Skip to content

conformance(loader,report): reject non-standard JSON constants in loads_strict and validate record_digest/build inputs (#648) - #653

Open
drk-ram wants to merge 1 commit into
agentrust-io:mainfrom
drk-ram:drk-ram-fix-conformance-loader-report-648
Open

drk-ram wants to merge 1 commit into
agentrust-io:mainfrom
drk-ram:drk-ram-fix-conformance-loader-report-648

Conversation

@drk-ram

@drk-ram drk-ram commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

What this changes

Closes #648.

Hardens conformance/src/trace_tests/loader.py and conformance/src/trace_tests/report.py:

  • Rejects non-standard JSON constants (NaN, Infinity, -Infinity) via parse_constant=_refuse_constants and non-bytes inputs (not isinstance(raw, bytes)) in loads_strict with LoadError, preserving loads_strict and parse_record's documented LoadError-only boundary while keeping invalid_canonical_non_finite_float.json (1e400) reachable for TR-SIG.
  • Rejects non-dict record inputs in record_digest and build, and validates results_by_level (requiring exact int keys in LEVEL_NAMES and dict values) in build.

Validation commands and results:

  • PYTHONPATH=src;conformance/src python -m pytest tests/test_conformance_report_parity.py -q -> passed (26 passed)
  • PYTHONPATH=src;conformance/src python -m pytest conformance/tests/test_report.py conformance/tests/test_canonicalization_refusals.py conformance/tests/unit/test_loader.py -q -> passed (51 passed)
  • python -m ruff check tests/test_conformance_report_parity.py -> passed (All checks passed!)
  • python tools/check_dashes.py -> passed (no dashes outside examples/, spec/trace-v0.1.md)
  • PYTHONPATH=src python tools/review_bar.py --base origin/main --head HEAD -> passed (14 changed test cases fail on behaviour at origin/main)

Type of change

  • Editorial (typo, link fix, clarification: no normative effect)
  • Non-breaking spec change (new optional field, new platform profile, informative addition)
  • Breaking spec change (requires 14-day comment period and Project Lead sign-off)
  • Schema change
  • Example addition

Spec section

conformance/src/trace_tests/loader.py and conformance/src/trace_tests/report.py (conformance test suite loader and report builder; no normative change to spec/trace-v0.2.md)

Tests for a fix

  • parse_constant=_refuse_constants and isinstance(raw, bytes) in loads_strict: tests/test_conformance_report_parity.py::test_loads_strict_and_parse_record_reject_non_standard_constants_and_non_bytes
  • isinstance(record, dict) in record_digest and build: tests/test_conformance_report_parity.py::test_record_digest_and_report_build_reject_non_dict_record
  • results_by_level level and dict validation in build: tests/test_conformance_report_parity.py::test_report_build_rejects_invalid_results_by_level

Changelog entry

Checklist

  • DCO sign-off on all commits (git commit -s)
  • Breaking changes marked with <!-- CHANGED: #NNN: description --> in spec text
  • Backward compatibility statement included (for breaking changes)

@drk-ram
drk-ram requested review from a team, lywinged and rajnisht7 as code owners October 30, 2026 15:18
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

🟡 Contributor Check: MEDIUM

Check Result
Profile MEDIUM
Credential LOW
Overall MEDIUM

Automated check by AgenTrust Contributor Check.

@github-actions github-actions Bot added the needs-review:MEDIUM Contributor check flagged MEDIUM risk label Oct 30, 2026
@drk-ram
drk-ram force-pushed the drk-ram-fix-conformance-loader-report-648 branch from ce4cb3e to 43fb214 Compare October 30, 2026 15:19
…ds_strict and validate record_digest/build inputs (agentrust-io#648)

- Reject non-standard JSON constants (NaN, Infinity, -Infinity) via parse_constant and non-bytes inputs in trace_tests.loader.loads_strict with LoadError.
- Reject non-dict record inputs in trace_tests.report.record_digest and trace_tests.report.build, and validate results_by_level level keys and module dicts in build.

Closes agentrust-io#648

Signed-off-by: Raghuvar Karthik Durga <177662415+drk-ram@users.noreply.github.com>
@drk-ram
drk-ram force-pushed the drk-ram-fix-conformance-loader-report-648 branch from 43fb214 to cd9bf9e Compare October 30, 2026 15:20
@rajnisht7

rajnisht7 commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Hello @drk-ram, thank you for the continued interest in contributing to agentrust-io, before we go any further, we need to sort out what's already open.

Quick recap of where things stand across agentrust-io: 120 PRs opened in the last 17 days, averaging over 7 a day. 69 are currently open, 46 merged, 5 closed without merging, and almost 98 PRs landed in earlier 7 days.

You also mentioned in your maintainer interest issue #640 that you review all code diffs and test output yourself before submitting, with AI agents working under your supervision. Given what we're actually seeing in the open PRs, that review step doesn't seem to be happening consistently. Some examples:

  • Several PRs have gone through 2-3 rounds of review comments and still aren't merge ready
  • A few have failing test coverage that's not addressed yet
  • Several have merge conflicts that just sitting there unresolved
  • Several fail CI because the branch was not rebased or updated against main yet
  • Some PR descriptions and comments have broken characters in them, like �zure_vtpm._b64url_uint &, which is the kind of thing a human proofreading their own PR description would normally catch
  • Some already have changes requested that are not addressed yet
  • And most of the PR batches landed almost at the same time, which isn't really how one person writing, testing, and reviewing each PR individually would show up

If each of these was reviewed before opening, these issues would have been caught before submission, not after. At this volume and with these gaps, it's hard for us to review each, and also whether it's you behind each PR or something more automated opening them faster than they can reasonably be checked.

So kindly avoid opening new PRs until the current backlog is cleared,kindly go back through your own open PRs and actually review them yourself first, fix the ones with pending changes requested, resolve the conflicts, fix the failing tests and CI, check if there are still bugs or issues which still exists in the diff and fix them and get them into merge ready state

And separately, it would help if you could just clarify directly: are you personally testing, and reviewing each of these before they go up, or is there is an automated process submitting PRs on this account with less oversight than described? We'd rather have a straight answer than keep guessing from the pattern.

@drk-ram

drk-ram commented Oct 31, 2026

Copy link
Copy Markdown
Contributor Author

Thank you @rajnisht7 for the direct feedback — point taken, and I am pausing all new PR submissions across agentrust-io until the existing open PRs are reviewed and cleared.

To answer your question directly: I work on Windows 11 and had been batching my local fixes and submitting them together using a PowerShell gh pr create script. In PowerShell double-quoted strings, a backtick before a, b, e, r, or t (such as `azure_vtpm or `request_sha256) is interpreted as an escape sequence (\a BEL / ^G, \b, \e, \r, \t), which caused the broken characters in weight-custody-manifest#224 and integrations#364. Pushing those batches at once also meant that when upstream main moved (e.g., CHANGELOG.md merges or the pip-audit dependency bumps), multiple open branches conflicted or failed CI at the same time.

I have gone back through every open PR across all 8 repositories to resolve all merge conflicts, fix the PR descriptions, and address every pending review comment:

  1. agentrust-io/trace-spec:
  2. agentrust-io/ca2a:
    • Formatted #275 (4af1465), #276 (f231b13), #277 (e7008a1), #278 (8eb696d), #287 (e2fd3a4), and #288 (d9e1939) with ruff format src/ tests/ per @carloshvp's review (#270, #289, #290, #291 are already APPROVED).
  3. agentrust-io/cmcp:
    • #805 (5d057eb): Added TPMS_ATTEST type == 0x8018 (TPM_ST_ATTEST_QUOTE) check, required ak.get("kty") == "RSA", and added unit tests for 100% patch coverage.
    • #806 (d897245): Enforced len(report_data_hex) <= 128 in both direct and quote TDX helpers and added 130-char negative regressions.
    • #807 (39fea26): Rebased onto origin/main (1fbc1b2) to resolve the tests/unit/test_policy_bundle.py conflict and added strict UTF-8 / duplicate-key / non-finite validation to _approval_schema().
    • #808 (4ab488a), #816 (f06f3ff), #817 (51c0d56), #818 (417e073): Rebased onto origin/main (1fbc1b2) so they pick up the pip-audit fix.
    • #819 (e4cc625): Added finite parse_float validation on EAR payloads and top-level/nested overflow tests (#800 and #815 are already APPROVED).
  4. agentrust-io/agent-manifest:
  5. agentrust-io/weight-custody-manifest, trace-registry, agentrust-telemetry, integrations:
    • Rebased all 9 open weight-custody-manifest PRs onto 8ae4da6 and fixed the escaped characters in the PR descriptions of weight-custody-manifest#224 and integrations#364 (all 5 open integrations PRs are now APPROVED).

I will hold off on opening any new PRs until the current queue is worked through.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-review:MEDIUM Contributor check flagged MEDIUM risk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

conformance(loader,report): reject non-standard JSON constants in loads_strict and validate record_digest/build inputs

2 participants