Found while fixing #4643 (PR #4647). The fix only stops the server stall; the calculation itself is still slow.
Problem
tokenizer.calculateStats on a 1.24M-row active epoch takes ~45–57 s: one worker.postMessage and one promise per message, a single tokenizer worker. The renderer's CALCULATION_TIMEOUT_MS (60 s, WorkspaceConsumerManager.ts) fired in local browser runs under load on both base and fix (Calculation failed … Calculation timeout), leaving the Stats/Cost tab empty even though the backend finishes and persists session-usage.json shortly after. Peak live heap during the calculation is ~1.9 GB (message array + 1.24M pending requests).
Options
- Batch worker requests (one message per N texts) to cut clone/promise overhead.
- Approximate counts for epochs far beyond any model's context window.
- Let the renderer pick up the persisted cache when its request times out.
Repro recipe and probe: see the root-cause comment on #4643.
Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high • Cost: $9.90
Found while fixing #4643 (PR #4647). The fix only stops the server stall; the calculation itself is still slow.
Problem
tokenizer.calculateStatson a 1.24M-row active epoch takes ~45–57 s: oneworker.postMessageand one promise per message, a single tokenizer worker. The renderer'sCALCULATION_TIMEOUT_MS(60 s,WorkspaceConsumerManager.ts) fired in local browser runs under load on both base and fix (Calculation failed … Calculation timeout), leaving the Stats/Cost tab empty even though the backend finishes and persistssession-usage.jsonshortly after. Peak live heap during the calculation is ~1.9 GB (message array + 1.24M pending requests).Options
Repro recipe and probe: see the root-cause comment on #4643.
Generated with
xum• Model:anthropic:claude-opus-5-5• Thinking:high• Cost:$9.90