Skip to content

Generic Remotion leaderboard video for benchmark composite scores - #352

Open
HeyGarrison wants to merge 19 commits into
masterfrom
devin/1787255794-remotion-leaderboard
Open

Generic Remotion leaderboard video for benchmark composite scores#352
HeyGarrison wants to merge 19 commits into
masterfrom
devin/1787255794-remotion-leaderboard

Conversation

@HeyGarrison

@HeyGarrison HeyGarrison commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

Genericizes packages/remotion-video so the same leaderboard composition can be rendered for any benchmark in results/<benchmark>/latest.json.

  • prepare.ts now accepts a benchmark via pnpm generate <benchmark> or BENCHMARK=.... It derives the title from a small mapping (with a BENCHMARK_TITLE env override), reads the result JSON, and uses each provider's compositeScore when present. For sandbox-dax (which has no compositeScore), it falls back to the median phasesCompleted / 7 * 100.
  • Logo resolution moved to a dotcom-aligned src/logo.ts using providerBrandId, logomarkSlug, and MISSING_LOGO_BRAND_IDS so cloud-rungoogle-cloud-run, browserusebrowser-use, and lightning/sail fall back to the local SVGs served by dotcom. Display names use the same capitalize helper as dotcom (CodeSandbox, Cloud Run, Lightning AI, etc.).
  • Composition.tsx keeps the 30-second, dark-mode, full-width race layout, but the header now places the dark ComputeSDK wordmark and title in a single flex container so they stay vertically aligned and the logo is sized larger than the title.

Try it

pnpm install
pnpm --filter @computesdk/remotion-video generate              # defaults to burst_tti
BENCHMARK=browser pnpm --filter @computesdk/remotion-video generate
pnpm --filter @computesdk/remotion-video render                # out.mp4
pnpm --filter @computesdk/remotion-video render:png            # out.png at frame 840

Sample output

Opening frame (frame 0):

Opening frame

Final leaderboard at frame 840:

Final leaderboard

Watch the 30s race video

Only composite scores are displayed; no median/p95/p99 metrics appear in the video.

Link to Devin session: https://app.devin.ai/sessions/e276f3c69f6a4e4783b1d3f09f60d930
Requested by: @HeyGarrison


Open in Devin Review

…e scores

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@open-cla

open-cla Bot commented Aug 20, 2026

Copy link
Copy Markdown

Contributor License Agreement

All contributors are covered by a CLA.

devin-ai-integration[bot]

This comment was marked as resolved.

…st logo fallback

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
devin-ai-integration[bot]

This comment was marked as resolved.

HeyGarrison and others added 2 commits August 20, 2026 20:07
…ed leaderboard

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
devin-ai-integration[bot]

This comment was marked as resolved.

HeyGarrison and others added 5 commits August 20, 2026 20:41
…ion range

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…top 5

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…e video

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
devin-ai-integration[bot]

This comment was marked as resolved.

HeyGarrison and others added 9 commits August 20, 2026 21:05
… gap

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ks site; relabel 'even more'

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…reveal timings

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…s and race

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…-one finishes

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…sors label

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
- Read any benchmark result JSON via BENCHMARK arg or env

- Derive title from benchmark path mapping, score from compositeScore (or dax phases)

- Use dotcom-aligned logo manager (providerBrandId, logomarkSlug, MISSING_LOGO_BRAND_IDS)

- Fix ComputeSDK logo/title alignment in Composition

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration devin-ai-integration Bot changed the title Add Remotion leaderboard video for sandbox composite scores Generic Remotion leaderboard video for benchmark composite scores Aug 20, 2026

@superagent-security superagent-security Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superagent found 1 security concern(s).

return BENCHMARK_TITLES[benchmark] ?? humanizeBenchmark(benchmark);
}

function getResultPath(benchmark: string): string {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Path traversal vulnerability in benchmark result resolution

Unsanitized BENCHMARK env/arg used in path.resolve enables directory traversal to read arbitrary files.

Validate benchmark names against an allowlist and ensure resolved paths remain within the intended results directory.

AI prompt
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name="packages/remotion-video/src/prepare.ts">
<violation number="1" location="packages/remotion-video/src/prepare.ts:94">
<priority>P2</priority>
<title>Path traversal vulnerability in benchmark result resolution</title>
<evidence>The `getResultPath` function constructs a filesystem path using an unsanitized `benchmark` parameter (`process.argv[2]` or `process.env.BENCHMARK`) passed directly into `path.resolve`: `return path.resolve(PKG_ROOT, '../../results', benchmark, 'latest.json');`. An attacker controlling the `BENCHMARK` environment variable or CLI argument can traverse outside the intended `results` directory to read arbitrary files on the filesystem (e.g., `BENCHMARK=../../../etc/passwd`).</evidence>
<recommendation>Sanitize the `benchmark` input before using it in a path. Validate it against an allowlist of known benchmark names, or reject values containing path traversal sequences (e.g., `..`, absolute paths). Use `path.join` combined with a check that the resolved path remains within the intended base directory.</recommendation>
</violation>
</file>

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant