Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

LegacyCodeBench

A benchmark for evaluating how well AI systems understand and document legacy COBOL code.

Ground truth-based evaluation with execution testing.

📄 Read the Paper: LegacyCodeBench Paper


Why This Exists

220+ billion lines of COBOL still run critical infrastructure—banking, insurance, government. Modernization projects consistently fail, and the failure mode is almost always the same: the business logic was never properly understood before conversion began.

LegacyCodeBench tests whether AI can actually understand legacy code well enough to document it accurately.


What It Measures

Component Weight Description
Structural Completeness (SC) 30% All business rules, data structures, control flow, and external calls must be documented
Documentation Quality (DQ) 20% Structure, traceability, readability, and abstraction level
Behavioral Fidelity (BF) 50% Execution-based verification (claims, BSM pattern matching, IUE classification)
LCB_Score = (0.30 × SC) + (0.20 × DQ) + (0.50 × BF)

Critical failures (hallucinated functions, incorrect business rules) result in a score of 0 for that task.


Quick Start

# Install
pip install -e .

# Set API keys
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."

# Run the benchmark
legacycodebench run-full-benchmark --enable-execution

Benchmark Details

  • 200 tasks across banking, insurance, retail, and government COBOL systems
    • T1: 50 basic tasks (Single programs, simple batch processing)
    • T2: 41 moderate tasks (File handling, sub-routine calls)
    • T3: 50 complex tasks (Control flow complexity, multi-file dependencies)
    • T4: 59 enterprise tasks (CICS, DB2, extensive copybooks, high Cyclomatic complexity)
  • Fully deterministic evaluation (removes LLM-as-judge)
  • Execution-based testing for behavioral fidelity (requires Docker)

Full Setup (With Execution Testing)

Behavioral Fidelity testing requires Docker to compile and run COBOL:

# Build COBOL execution environment
cd docker/cobol-sandbox
docker build -t legacycodebench-cobol:latest .
cd ../..

# Run with execution testing
legacycodebench run-full-benchmark --enable-execution --task-limit 200

Note: Without Docker, BF evaluation falls back to heuristic verification (claim quality analysis). For full accuracy, Docker is recommended.


Supported Models

Model Provider Environment Variables
claude-sonnet-4 Anthropic ANTHROPIC_API_KEY
gpt-4o OpenAI OPENAI_API_KEY
gemini-2.5-flash Google GOOGLE_API_KEY
legacycodebench run-full-benchmark --models "claude-sonnet-4,gpt-4o"

Commands

legacycodebench interactive         # Guided setup
legacycodebench run-full-benchmark  # Full evaluation
legacycodebench leaderboard         # Generate rankings
legacycodebench evaluate            # Score single submission
legacycodebench load-datasets       # Clone COBOL repositories
legacycodebench create-tasks        # Generate task definitions

View Results

# Generate leaderboard JSON
legacycodebench leaderboard

# Or with detailed output
legacycodebench leaderboard --detailed

# Serve web UI
cd web
python -m http.server 8080
# Open http://localhost:8080

Project Structure

legacycodebench/
├── evaluators_v213/     # Evaluator (deterministic, default)

├── static_analysis/     # COBOL parsing, ground truth extraction
├── execution/           
│   ├── bsm/            # Behavioral Similarity Matching

└── cli.py

datasets/                # COBOL source files (auto-cloned from GitHub)
tasks/                   # 200 task definitions (JSON)
results/                 # Evaluation outputs
web/                     # Leaderboard UI
docker/                  # GnuCOBOL 3.2 execution environment

Submit Results

Run the benchmark and open a PR with your results/ directory. Results will be added to the public leaderboard.


License

MIT License (code) · CC-BY 4.0 (data)


Built by Kalmantic AI Labs

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages