Skip to content

Track Performance #593

New issue
New issue
Open
1 of 5 issues completed
Open
Track Performance#593
1 of 5 issues completed
Assignees
ludfjig
Labels
A-perfAddresses performanceS-blockedFor issues blocked by another issue or external factor
Milestone
Upcoming

Description

@ludfjig
ludfjig
opened on Jun 9, 2025

Hyperlight Performance Plan

Core Benchmarks

Microbenchmarks (Time per Task)

Measure round-trip time (RTT) and lifecycle operations:

  • Guest call RTT (empty, no workload)
  • Guest call RTT (large param/return, no workload)
  • Sandbox create, evolve, and drop

Test across:

  • Hypervisors: Windows Server 2025/2022 Hyper-V, Linux KVM (5.x/6.x), MSHV (5.x/6.x)
  • CPU vendors: Intel, AMD

Native vs Hyperlight

Compare native host code vs Hyperlight (x86_64-unknown-none) on compute-heavy tasks to evaluate performance overhead due to limited CPU feature use (e.g. SSE/AVX).

HyperlightJS & HyperlightWASM

Load Testing

Benchmark lightweight compute workloads (per-request tasks):

  • Metrics: HTTP throughput, latency, memory use
  • Workloads: header rewrite, JWT validation, TBD CPU-intensive task

Comparisons

Evaluate:

  • HyperlightJS
  • HyperlightWASM (with JS component)
  • Wasmtime (with JS component, baseline)

Test across:

  • Hypervisors: Windows Server 2025/2022 Hyper-V, Linux KVM (5.x/6.x), MSHV (5.x/6.x)
  • CPU vendors: Intel, AMD

Performance Monitoring

Track metrics over time to catch regressions and improvements:

  • Run benchmarks on every PR merge and release
  • Allow local runs during dev
  • Visualize data on public dashboard

Notes

  • Benchmarks run in nested virtualization on CI (Standard_D4(a)ds_v5 VMs, premium SSD)
  • Expect some noise, but useful for trends and comparisons

Activity

  1. github-actions
    added
    S-needs-triageThe issue has not yet been reviewed.
    on Jun 9, 2025

  2. dblnz commented on Jun 11, 2025

    @dblnz
    dblnz
    on Jun 11, 2025
    Contributor

    Two important points that have been brought up in the last community meeting:

    • When we compare measurements between commits we should run the benchmarks on the same machine (maybe keep a separate machine only to run benchmarks). Ideally we want to see the most accurate measurements without outside factors' interference.
    • We should measure performance with different sandbox size(1MB, 10MB, 100MB ..), parameters size and other similar variations

  3. simongdavies commented on Jun 12, 2025

    @simongdavies
    simongdavies
    on Jun 12, 2025
    Member

    I also think we need to measure creating, using and dropping sandboxes in parallel with various degrees of parallelism

  4. ludfjig
    added this to the v0.7.0 milestone on Jun 18, 2025
  5. ludfjig
    self-assigned this
    on Jun 18, 2025
  6. danbugs
    added
    A-perfAddresses performance
    S-confirmedBug is verified or proposal seems reasonable
    and removed
    S-needs-triageThe issue has not yet been reviewed.
    on Jun 18, 2025
  7. danbugs
    added this to Hyperlight Milestoneson Jun 18, 2025

  8. jsturtevant commented on Jun 26, 2025

    @jsturtevant
    jsturtevant
    on Jun 26, 2025
    Contributor

    Add and few notes from #152:

    At present there are a small number of performance benchmarks run as a part of PR validation but there are no scalability benchmarks.

    We need to have some benchmarks that measure the scalability of hyperlight when a host process creates varying numbers of sandboxes and invokes functions on them concurrently.

    These benchmarks should attempt to measure how hyperlight behaves in a number of dimensions, for example:

    Different usage patterns e.g.
    Create a Sandbox, Use, Drop
    Create a Sandbox, initialize it, cache, use, return to cache
    Different parameter types and sizes

  9. jsturtevant
    moved this to In Progress in Hyperlight Milestoneson Jun 26, 2025

  10. syntactically commented on Jun 30, 2025

    @syntactically
    syntactically
    on Jun 30, 2025
    Member

    Especially for the microbenchmarks, I think the numbers will be a lot more useful on a dedicated host without NV. We have seen before that there is a ton of variance in the GH CI benchmarks, because on the time scales we are looking at with the microbenchmarks, even a single extraneous entry into the host kernel scheduler shows up quite badly.

  11. jsturtevant
    modified the milestones: v0.7.0, v0.8.0 on Jun 30, 2025
  12. syntactically
    added
    S-blockedFor issues blocked by another issue or external factor
    and removed
    S-confirmedBug is verified or proposal seems reasonable
    on Jul 9, 2025

  13. syntactically commented on Jul 9, 2025

    @syntactically
    syntactically
    on Jul 9, 2025
    Member

    Converting this to a tracking issue for having the visibility into performance/regular benchmarks that we would like to see overall.

  14. syntactically
    modified the milestones: v0.8.0, Upcoming on Jul 9, 2025
  15. syntactically
    added sub-issues on Jul 9, 2025
  16. syntactically
    added a sub-issue on May 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

A-perfAddresses performanceS-blockedFor issues blocked by another issue or external factor

Type

No type

Projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions