feat(manifest): faster + more correct manifest path resolution (cache, index precedence, id validation)

[vilenarios]

Draft — PR 1 of 2 in the manifest-resolution work (see follow-up for the persistent resolveFromIndex index). No schema changes here.

What & why

Every ArNS site is an Arweave manifest (a map of URL paths → data ids), and each asset on a page is a separate path resolution against the same manifest. Today the gateway streams and re-parses the entire manifest body on every resolution. This PR removes that redundant work and fixes two correctness issues, all within the streaming resolver — no DB/migration.

Changes

perf: cache resolved manifest paths (1c).
A manifest tx is immutable, so a resolved (manifest id, path) → data id mapping is valid forever. StreamingManifestPathResolver now keeps a bounded LRU: resolveFromData populates it; resolveFromIndex serves hits with complete: true, so the caller (sendManifestResponse) skips fetching and re-parsing the body. Positive and negative resolutions are both cached. Size via MANIFEST_RESOLUTION_CACHE_SIZE (default 5000).

fix: deterministic v0.2.0 index precedence (1a).
Index resolution is deferred until the index object closes, so index.id deterministically wins over index.path regardless of JSON key order. Previously a manifest whose paths preceded index (with path before id) resolved to the id index.path mapped to — meaning the served index could differ from other gateways (a cross-gateway consistency bug). Depth guard excludes a paths entry keyed "index".

fix: reject malformed data ids (1b).
Manifest entries whose id isn't 43-char base64url are rejected before a data-retrieval attempt.

Before/after (real manifests, 200 repeat resolutions, body in memory → isolates parse cost)

manifest	before	after	speedup	parse cost/call today
small (100 paths, 9 KB)	129 ms	5.8 ms	22×	0.65 ms
medium (269 paths, 28 KB)	285 ms	3.8 ms	76×	1.4 ms
large (7,784 paths, 500 KB)	11,363 ms	59 ms	192×	57 ms

The large real manifest costs ~57 ms of CPU parse per resolution today — a 30-asset homepage burns ~1.7 s of pure parse CPU per view, eliminated on cache hits. When the body is not locally cached, the current path additionally pays a full upstream fetch per resolution (not measured here) — the follow-up persistent index removes that too.

Tests

• src/lib/encoding.test.ts: +4 (deterministic index-id precedence regression; malformed-id rejection). 29/29 pass.
• src/resolution/streaming-manifest-path-resolver.test.ts (new): cache hit/miss, negative caching, trailing-slash key normalization, per-manifest keying. 5/5 pass.
• lint:check clean.

Docs

MANIFEST_RESOLUTION_CACHE_SIZE added to docs/envs.md and docker-compose.yaml.

Not in this PR

• 1e observability metrics (small follow-up).
• 1d persistent manifest_index_id/manifest_fallback_id index implementing resolveFromIndex from a dedicated table (separate PR — carries the migration).

🤖 Generated with Claude Code

https://claude.ai/code/session_01LPbcF2M3P1XcM8mU8FWEf9

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 4098f144-1382-4a40-b1a5-cd9910d6895c

📥 Commits

Reviewing files that changed from the base of the PR and between e0e4e64 and cea59e4.

📒 Files selected for processing (2)

• src/lib/encoding.ts
• src/resolution/streaming-manifest-path-resolver.ts

🚧 Files skipped from review as they are similar to previous changes (2)

• src/resolution/streaming-manifest-path-resolver.ts
• src/lib/encoding.ts

---

📝 Walkthrough

Walkthrough

Manifest parsing now validates IDs and applies order-independent index precedence. Streaming manifest path resolution adds bounded caching for successful and unsuccessful results. Cache capacity is configurable through environment settings and system wiring.

Changes

Manifest resolution

Layer / File(s)	Summary
Manifest parsing and ID validation src/lib/encoding.ts, src/lib/encoding.test.ts, test/mock_files/manifests/*, test/stubs.ts	Parsing validates Arweave IDs and defers index resolution until the complete index object is available. Tests and fixtures cover key order and malformed IDs.
Streaming resolution cache src/resolution/streaming-manifest-path-resolver.ts, src/resolution/streaming-manifest-path-resolver.test.ts	The resolver caches positive and negative results by manifest ID and normalized path. Tests cover misses, hits, normalization, and manifest isolation.
Cache configuration and system wiring src/config.ts, src/system.ts, docker-compose.yaml, docs/envs.md	The cache size is configurable through MANIFEST_RESOLUTION_CACHE_SIZE, defaults to 5,000, and is passed to the resolver.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant StreamingManifestPathResolver
  participant LRUCache
  participant ManifestBody
  Client->>StreamingManifestPathResolver: resolve manifest path
  StreamingManifestPathResolver->>LRUCache: look up manifest ID and normalized path
  alt cache hit
    LRUCache-->>StreamingManifestPathResolver: return cached result
  else cache miss
    StreamingManifestPathResolver->>ManifestBody: resolve path from manifest data
    ManifestBody-->>StreamingManifestPathResolver: return positive or negative result
    StreamingManifestPathResolver->>LRUCache: store result
  end
  StreamingManifestPathResolver-->>Client: return resolution

Loading

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the cache, index precedence, and ID validation changes in the manifest resolver.
Description check	✅ Passed	The description directly explains the manifest resolver changes, performance goals, correctness fixes, tests, and configuration updates.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/manifest-resolution-improvements

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

src/lib/encoding.ts (1)

227-232: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Add TSDoc for the new helper and method.

The touched TypeScript APIs use line comments instead of TSDoc.

• src/lib/encoding.ts#L227-L232: add TSDoc for isValidManifestId.
• src/resolution/streaming-manifest-path-resolver.ts#L35-L39: add TSDoc for cacheKey.

As per coding guidelines, “Add or improve TSDoc comments on code you touch.”

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/encoding.ts` around lines 227 - 232, Add concise TSDoc describing the
purpose and validation behavior of isValidManifestId in src/lib/encoding.ts at
lines 227-232, and add TSDoc describing the cacheKey API in
src/resolution/streaming-manifest-path-resolver.ts at lines 35-39; update both
documented symbols without changing their implementation.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/lib/encoding.ts`:
- Around line 227-232: Add concise TSDoc describing the purpose and validation
behavior of isValidManifestId in src/lib/encoding.ts at lines 227-232, and add
TSDoc describing the cacheKey API in
src/resolution/streaming-manifest-path-resolver.ts at lines 35-39; update both
documented symbols without changing their implementation.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: fe3a286b-c70a-48c4-a00f-a621ea265f31

📥 Commits

Reviewing files that changed from the base of the PR and between 3e81d66 and e0e4e64.

📒 Files selected for processing (11)

• docker-compose.yaml
• docs/envs.md
• src/config.ts
• src/lib/encoding.test.ts
• src/lib/encoding.ts
• src/resolution/streaming-manifest-path-resolver.test.ts
• src/resolution/streaming-manifest-path-resolver.ts
• src/system.ts
• test/mock_files/manifests/example_manifest_invalid_ids.json
• test/mock_files/manifests/example_manifest_v0.2.0_paths_before_index_path_first.json
• test/stubs.ts

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.85%. Comparing base (d2c2663) to head (cea59e4).
⚠️ Report is 49 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #832      +/-   ##
===========================================
+ Coverage    78.77%   78.85%   +0.07%     
===========================================
  Files          133      134       +1     
  Lines        50969    51113     +144     
  Branches      3847     3858      +11     
===========================================
+ Hits         40152    40303     +151     
+ Misses       10765    10758       -7     
  Partials        52       52              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[vilenarios]

Superseded by #833. That PR now contains these commits (1a deterministic index precedence, 1b id validation, 1c resolution cache) plus the persistent index (1d), as a single self-contained PR against develop so CodeRabbit reviews the full change. No code is lost — #833 includes everything here.