A benchmark for evaluating how well AI systems understand and document legacy COBOL code.
Ground truth-based evaluation with execution testing.
📄 Read the Paper: LegacyCodeBench Paper
220+ billion lines of COBOL still run critical infrastructure—banking, insurance, government. Modernization projects consistently fail, and the failure mode is almost always the same: the business logic was never properly understood before conversion began.
LegacyCodeBench tests whether AI can actually understand legacy code well enough to document it accurately.
| Component | Weight | Description |
|---|---|---|
| Structural Completeness (SC) | 30% | All business rules, data structures, control flow, and external calls must be documented |
| Documentation Quality (DQ) | 20% | Structure, traceability, readability, and abstraction level |
| Behavioral Fidelity (BF) | 50% | Execution-based verification (claims, BSM pattern matching, IUE classification) |
LCB_Score = (0.30 × SC) + (0.20 × DQ) + (0.50 × BF)
Critical failures (hallucinated functions, incorrect business rules) result in a score of 0 for that task.
# Install
pip install -e .
# Set API keys
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
# Run the benchmark
legacycodebench run-full-benchmark --enable-execution- 200 tasks across banking, insurance, retail, and government COBOL systems
- T1: 50 basic tasks (Single programs, simple batch processing)
- T2: 41 moderate tasks (File handling, sub-routine calls)
- T3: 50 complex tasks (Control flow complexity, multi-file dependencies)
- T4: 59 enterprise tasks (CICS, DB2, extensive copybooks, high Cyclomatic complexity)
- Fully deterministic evaluation (removes LLM-as-judge)
- Execution-based testing for behavioral fidelity (requires Docker)
Behavioral Fidelity testing requires Docker to compile and run COBOL:
# Build COBOL execution environment
cd docker/cobol-sandbox
docker build -t legacycodebench-cobol:latest .
cd ../..
# Run with execution testing
legacycodebench run-full-benchmark --enable-execution --task-limit 200Note: Without Docker, BF evaluation falls back to heuristic verification (claim quality analysis). For full accuracy, Docker is recommended.
| Model | Provider | Environment Variables |
|---|---|---|
claude-sonnet-4 |
Anthropic | ANTHROPIC_API_KEY |
gpt-4o |
OpenAI | OPENAI_API_KEY |
gemini-2.5-flash |
GOOGLE_API_KEY |
legacycodebench run-full-benchmark --models "claude-sonnet-4,gpt-4o"legacycodebench interactive # Guided setup
legacycodebench run-full-benchmark # Full evaluation
legacycodebench leaderboard # Generate rankings
legacycodebench evaluate # Score single submission
legacycodebench load-datasets # Clone COBOL repositories
legacycodebench create-tasks # Generate task definitions# Generate leaderboard JSON
legacycodebench leaderboard
# Or with detailed output
legacycodebench leaderboard --detailed
# Serve web UI
cd web
python -m http.server 8080
# Open http://localhost:8080legacycodebench/
├── evaluators_v213/ # Evaluator (deterministic, default)
├── static_analysis/ # COBOL parsing, ground truth extraction
├── execution/
│ ├── bsm/ # Behavioral Similarity Matching
└── cli.py
datasets/ # COBOL source files (auto-cloned from GitHub)
tasks/ # 200 task definitions (JSON)
results/ # Evaluation outputs
web/ # Leaderboard UI
docker/ # GnuCOBOL 3.2 execution environment
Run the benchmark and open a PR with your results/ directory. Results will be added to the public leaderboard.
MIT License (code) · CC-BY 4.0 (data)
Built by Kalmantic AI Labs