[Roadmap] Post-hit CLIP quality evaluation against cold-run baselines

[cyyecao-lappland]

I would like to take ownership of the roadmap item “Post-hit CLIP-quality evaluation doc” from the README.

The goal would be to make the quality/latency trade-off of approximate cache reuse reproducible against a matched cold-run baseline, so users can choose rerank_score_threshold using quality evidence rather than hit rate alone.

Proposed first PR scope

• Add an offline paired-evaluation script for cold and cache-reused outputs.
• Use fixed seeds and matched generation parameters for each pair.
• Record the model/profile, prompt pair, seed, resolution, sampler settings, reranker score, configured threshold, selected skip depth K, latency, and output paths.
• Compute frame-level CLIP-based similarity and aggregate it per clip, then summarize results by skip depth and reranker-score bucket.
• Emit machine-readable per-sample results plus a compact human-readable summary.
• Add documentation for constructing a small evaluation manifest and reproducing the comparison.
• Add CPU-only unit tests for manifest validation and metric aggregation, plus a small opt-in GPU smoke test.

This first PR would remain an offline evaluation tool: it would not change the current lookup policy, default threshold, or online behavior.

For initial validation I can use up to 4×RTX 3090 GPUs, with reduced resolution or clip length where needed. I will report this as a functional/quality validation rather than as a replacement for the README's Hopper performance setting.

Once paired quality data are available, they could support a separate follow-up discussion about workload-specific threshold calibration. I would keep that policy work outside this roadmap PR.

Before implementation, could the maintainers confirm:

1. the preferred CLIP backbone / aggregation convention, if one is already used internally;
2. the Wan2.2 entry point and minimal model profile that should be the first documented path;
3. whether JSONL per-sample records plus a Markdown summary fit the expected output format?

If this scope matches the roadmap intent, I am happy to implement and test it.