Design and Implement an Extra vs. Hermes Agent Benchmark

[Asaf-prog]

Benchmark Extra vs. Hermes Agent

Goal

Design and implement a reproducible benchmark comparing Extra with Hermes Agent.

Since the projects have different architectures and goals, the first part of the task is to decide what should be compared and how to measure it fairly.

Scope

1. Define the benchmark

Research both systems and document:

• Shared and unique capabilities.
• Which characteristics should be compared.
• Which characteristics are not directly comparable.
• Metrics and success criteria for each area.
• Fairness constraints and known limitations.

Potential areas:

• Routing and orchestration.
• Tool and MCP execution.
• Authentication and authorization.
• Human approval.
• Memory and context handling.
• Error recovery.
• Observability.
• Developer experience.
• Performance and cost.

2. Implement the benchmark

Create a shared benchmark harness that:

• Runs equivalent scenarios on both systems.
• Uses the same model, prompts, tools, and data where possible.
• Captures correctness, latency, token usage, cost, and failures.
• Exports machine-readable results and a Markdown report.
• Pins the tested versions or commit SHAs.

Deliverables

• Capability comparison matrix.
• Final benchmark criteria and metrics.
• Shared test scenarios.
• Extra and Hermes adapters.
• Reproducible benchmark runner.
• Results and comparison report.
• Documentation for running the benchmark.

Acceptance criteria

• The comparison criteria are documented and justified.
• Every selected characteristic has a clear metric.
• At least five scenarios are implemented.
• Both systems are tested under equivalent conditions.
• Unsupported or non-comparable capabilities are documented separately.
• All conclusions are supported by benchmark results.