AgentExecutor

AgentExecutor is a small, readable framework for training coding agents. It supports the two mainstream training schemes with real sandbox execution:

Scheme	Rollout	Training	Output
Teacher distillation	A teacher agent executes tasks in E2B sandboxes.	Off-policy LoRA SFT on successful tool trajectories.	output/*-agent-lora-sft
On-policy RL	The policy executes tasks in E2B through ms-swift GRPO.	On-policy LoRA GRPO with verifier rewards.	output/*-agent-grpo

The complete flow is: environment setup -> model download -> teacher rollout -> SFT LoRA -> GRPO LoRA -> Swift inference.

Quickstart

# 1. Install Python dependencies (add --with-vllm for GRPO)
python scripts/setup_env.py --with-vllm

# 2. Download the 27B base model to models/Qwen3.8-27B
python scripts/download_model.py

# 3. Run the offline smoke test
python tests/test_offline_pipeline.py

For a real run, export your keys and execute the full workflow:

export E2B_API_KEY=...
export LLM_BASE_URL=https://api.deepseek.com/v1
export LLM_API_KEY=...

bash scripts/run.sh all

run.sh all performs teacher collection, SFT LoRA, GRPO LoRA, real inference, and refreshes data/demo/. It defaults to EPOCHS=2; adjust SAMPLES_PER_TASK, CONCURRENCY, and MAX_TURNS to control cost.

Scripts

Command	Purpose
python scripts/setup_env.py --with-vllm	Install base, training, vLLM, and model-download dependencies.
python scripts/download_model.py	Download Qwen3.8-27B from ModelScope.
bash scripts/run.sh sft	Teacher rollout -> exported SFT data -> LoRA SFT.
bash scripts/run.sh rl	Continue GRPO from the latest SFT LoRA.
bash scripts/run.sh all	Full SFT -> GRPO -> inference -> demo workflow.
bash scripts/run.sh infer	Inference with the latest RL LoRA.
bash scripts/run.sh demo	Refresh the minimal real-run demo bundle.

Output Layout

output/ contains runtime artifacts only and is safe to delete:

• *-agent-lora-sft/: teacher-distillation SFT runs.
• *-agent-grpo/: on-policy GRPO runs. In this project, the rl stage specifically uses GRPO.
• vN-<timestamp>/: ms-swift creates a new version directory for each training invocation so old runs are not overwritten.
• checkpoint-N/: LoRA checkpoints saved by step or epoch.
• round-NNN/: created only by the optional rl-loop command for repeated rollout-and-training cycles; run.sh rl does not create them.

Project Layout

agentexecutor/
  core/         # shared types, tools, verifier, LLM clients
  sandbox/      # E2B and mock sandbox backends
  agents/       # builtin tool loop and codex teacher
  data/         # trajectory recording, SFT export, demo creation
  training/     # ms-swift SFT, GRPO, inference, patch, closed loop
  cli.py        # one command-line interface
tasks/          # one minimal verified task
data/demo/      # one real end-to-end sample bundle
scripts/        # setup, model download, workflow runner
tests/          # offline end-to-end tests

CLI

The scripts are thin wrappers around the stable CLI:

python -m agentexecutor --help
python -m agentexecutor run --prompt "create /workspace/hi.py" --backend mock --llm mock
python -m agentexecutor collect --tasks tasks/sample_tasks.jsonl --backend e2b --agent codex
python -m agentexecutor export --data-dir data --out data/agent_lora_sft/sft/train.jsonl
python -m agentexecutor sft-train --dry-run
python -m agentexecutor rl-prepare --tasks tasks/sample_tasks.jsonl
python -m agentexecutor rl-train --dry-run
python -m agentexecutor infer --query "What is the capital of France?" --dry-run
python -m agentexecutor create-demo

Demo Data

data/demo/ is intentionally minimal and versioned:

• teacher_trajectory.jsonl: one successful teacher rollout.
• sft_train.jsonl: the same rollout exported for ms-swift SFT.
• inference.jsonl: one real base + LoRA inference result.
• validation_report.json: checkpoint and inference summary.

The task source lives only in tasks/sample_tasks.jsonl. GRPO datasets are generated by agentexecutor rl-prepare and are not committed. Runtime data, model weights, checkpoints, and logs are excluded by .gitignore.

Real Validation Requirements

• GPU count: the validated full run uses 4 GPUs. Fewer GPUs are not validated with the default 27B configuration.
• GPU memory: the validated cards provide 80–97GB each. The observed peak was about 88GiB per GPU, or about 352GiB aggregate.
• Default configuration: Qwen3.8-27B, LoRA rank 8, ZeRO-3, vLLM colocate rollout, VLLM_GPU_MEMORY_UTILIZATION=0.25, and VLLM_MAX_MODEL_LEN=2048.
• Lower-memory guidance: use a smaller base model, shorter context, fewer generations, or lower precision before assuming that fewer than four 80GB GPUs will work.

Training Notes

1. Hardware: see Real Validation Requirements for the GPU count and memory footprint of the validated run.
2. First GRPO initialization can take several minutes: model loading, JIT, and CUDA graph capture occur before step 1.
3. Memory: defaults use VLLM_GPU_MEMORY_UTILIZATION=0.25, VLLM_MAX_MODEL_LEN=2048, VLLM_MAX_NUM_SEQS=2, and vLLM weight offload. The real run peaked around 88GiB/GPU.
4. Tensor parallelism: VLLM_TP defaults to the visible GPU count and must divide NPROC_PER_NODE.
5. ms-swift workaround: run.sh rl automatically calls agentexecutor patch-swift, an idempotent fix for a redundant first-rollout base-weight sync in ms-swift 4.5.2 LoRA GRPO.
6. Thread settings: vLLM requires a positive OMP_NUM_THREADS; run.sh replaces empty or zero values with 1.
7. Secrets: keys are read only from environment variables and are not written to tasks, trajectories, or demo data.
8. Cost: E2B creates a real cloud sandbox per episode. Start with one task and one sample.

Validated Run

The repository includes a real demo generated from:

• 4/4 successful teacher episodes.
• 2-epoch SFT LoRA training.
• 2-epoch GRPO LoRA training with reward 1.0.
• Real Swift inference that answered “Paris” for the capital of France.

Roadmap

• Real E2B teacher collection.
• Teacher-distillation LoRA SFT.
• On-policy GRPO LoRA.
• Swift inference and minimal real demo.
• Held-out evaluation harness.
• Process reward models.