A command-line harness for regression-testing LLM agents, locally and in CI.
Live demo: https://agenteval-six.vercel.app
When you change a prompt, a model, or a tool, an agent usually still returns something, and ordinary unit tests still pass. AgentEval checks behaviour instead: you write the expected behaviour as YAML test cases, run the agent through an adapter, and compare the scores against a baseline stored in git. If quality drops, the command exits non-zero and the CI job fails.
It also keeps a local record of failures you have seen in real use, so a failure that was reviewed and approved can be turned into a permanent test case.
A project page with a static demo is at nishanttyagi28.github.io/agenteval.
Python 3.10 or newer.
pip install nishanttyagi-agenteval
agenteval --versionFrom source:
git clone https://github.com/nishanttyagi28/agenteval.git
cd agenteval
pip install -e ".[dev]"Optional framework extras: crewai, autogen, openai-agents, karmasakshi, e.g. pip install "nishanttyagi-agenteval[crewai]".
From the repository root. This uses a scripted mock agent, so no API key or network is needed:
python -m agenteval run --agent mock_agent --registry examples/mock_agent/agents.yaml
python -m agenteval compare --agent mock_agent --registry examples/mock_agent/agents.yamlrun scores three cases and writes a run JSON under examples/mock_agent/runs/. compare checks that run against examples/mock_agent/baseline.json and exits non-zero if any gate fails. (Use python -m agenteval here so the repo's examples package is importable from the current directory.)
To start a suite for your own agent:
agenteval init # scaffolds agents.yaml, a sample suite and a CI workflow
agenteval templates list # bundled starter suitesinit writes an agents.yaml entry for my_agent and a sample suite in tests/golden/my_agent.yaml. Point adapter at a class that subclasses agenteval.adapters.base.AgentAdapter (or use one of the bundled framework adapters), set enabled: true, replace the sample cases with real ones, and run agenteval run --agent my_agent.
- Correctness: exact, contains, numeric and table matching, or an optional LLM judge.
- Hallucination: claims in the answer that the ground truth doesn't support.
- Tool calls: precision, recall and F1 against the tools the case expects.
- Trajectory: the sequence of steps against the expected one (
agenteval diffshows the difference). - Latency and cost: p50/p95 and total cost, with optional budget gates.
- Flakiness: repeat a case and label it stable, flaky or unstable.
- RAG and SQL: context relevance, faithfulness and citation checks; a structural safety scanner for generated SQL (
agenteval sql).
Adapters are included for CrewAI, AutoGen, the OpenAI Agents SDK, LangGraph, and custom agents. Results are available as JSON, a self-contained HTML report (agenteval report), a Streamlit dashboard, or a read-only local API (agenteval serve). Run agenteval --help for the full command list.
agenteval memory stores failing traces in a local SQLite database (.agenteval/failure-memory.db). Secrets are redacted before anything is written. Similar failures are grouped, can be replayed and reduced to a minimal case, and only become a test after a person approves them. Approved cases are written to .agenteval/production-regressions.yaml and run with agenteval run --production-cases <file>.
agenteval gate --local serves the same review flow as a small browser page on 127.0.0.1:8741. It has no login, so it refuses non-loopback addresses unless you pass --allow-remote.
An offline end-to-end demo:
python examples/failure_memory_demo_v21/run_demo.pySee docs/failure-memory.md and docs/release-desk.md.
- uses: nishanttyagi28/agenteval@v0.5.0
with:
agent: my_agent
config-file: agents.yaml
cases-file: tests/golden/cases.yaml
baseline-file: baselines/my_agent.jsonPin a release tag; main moves. A full workflow example is in examples/github-actions/agenteval.yml.
- Alpha. CLI flags, YAML schema and report formats can still change; see docs/v1-readiness.md.
- The failure database is local and single-user. It is not a hosted service or an OpenTelemetry collector.
- Redaction is pattern-based and will miss secrets that don't look like secrets.
- With no API key,
llm_judgecases are scored by a deterministic offline judge, which is much cruder than a real model. SetAGENTEVAL_JUDGE_PROVIDERtoopenai,groqoranthropicto use one. - Evaluating a live agent still needs that agent's runtime and whatever provider keys it uses.
- Cost figures fall back to estimates when a provider doesn't report token usage.
pip install -e ".[dev]"
python -m pytest -qTests that need Docker, FastAPI or the karmasakshi extra are skipped when those aren't installed. See CONTRIBUTING.md and CHANGELOG.md.
More docs: templates, plugins, multi-turn evaluation, tool efficiency, red-team generation, SQL scanner, KarmaSakshi bridge, compatibility.
MIT. See LICENSE.
Parts of this code were written with AI coding assistants (Cursor and Claude). I set the design and the tests the code has to pass, and agent-written commits carry a Co-authored-by trailer.