Platform Evolution Engine — Distill the best from the combinatorial mess.

Retort applies statistical Design of Experiments (DoE) to systematically evaluate AI-assisted development tooling stacks. It generates fractional factorial designs across languages, coding agents, and frameworks, executes experiments in isolated playpens, scores the results, and promotes or retires stacks based on measured confidence.

Retort is pre-1.0 and under active development. The CLI surface, scoring metrics, and storage schema may change between commits. The code is published so others can read it, fork it, and reproduce experiments — not yet as a stable tool to depend on.

What works today: LocalRunner (executes the claude CLI on the host), 3 built-in scorers (code_quality, build_time, token/cost capture), fractional-factorial design generation, ANOVA + effects reporting, SQLite storage, and resumable runs (retort run --resume) with per-run workspace archival.

What does not yet work end-to-end: DockerRunner (skeleton only — LocalRunner is the supported path), agents other than claude-code, the intake/scheduler paths, and the bundled:// task sources beyond rest-api-crud. See Implementation Status for details.

Currently in flight: Experiment 1 is being re-run with replicates: 3 to get past the n=1 results below. Rust was added as a fourth language. Results will replace the table when the run completes.

📊 Browse the live web report → (sortable leaderboard with per-stack drill-downs, token/cost data, and links to per-run code reviews)

Full data is also in experiment-1/reports/ — ANOVA, per-stack maturity, full CSV, and the same static-HTML web report. Below is the headline.

Setup: 6 languages (python, typescript, go, rust, java, clojure) × 2 models (opus, sonnet) × 2 tooling (none, beads) × 3 replicates = 72 runs against the bundled rest-api-crud task. Java + clojure added in a follow-up extension run as 4 parallel --shard 0/4..3/4 shards. Final tally: 67 of 73 runs completed, 6 failed. Total cost ≈ $25, ≈ 25.8M tokens.

Multiplicative ANOVA (default: log10 transform, since cost/tokens/duration scale by ratios not constants):

Switching from additive to multiplicative ANOVA surfaced model + tooling effects on the cost-like metrics that the additive model treated as noise. See reports/anova.txt.

Top stacks by maturity: Java sweeps quality 1.000 in all four model/tooling combinations. Go's sonnet/beads ties at quality 1.000 too. Generate the full maturity report with retort maturity --db experiment-1/retort.db.

Sortable + drill-downable in the web report. Bold quality means perfect 1.000.

Headline:

• Java is the surprise winner — perfect 1.000 quality across every (model, tooling) combination, and java/sonnet/none does it for $0.33/run.
• Best value: go/sonnet/none at $0.303 quality 0.956, or java/sonnet/none at $0.326 quality 1.000.
• Worst value: clojure/opus/beads at $0.762 quality 0.556 (paying the most for the worst output).
• Beads helps only for Go. Python, typescript, rust, java, clojure all score lower (or no different) with beads enabled — beads' overhead isn't recouped on this small task.

📊 Web report →

A second experiment run against brazil-bench/benchmark-template — a much harder task: MCP server, CSV ingest of Kaggle data, BDD tests with 16 canonical requirements. 24 cells, 1 replicate each, screening pass. Results combine with experiment-1 to give cross-task ANOVA insights.

Single-task ANOVA on code_quality: only language significant (consistent with experiment-1).

Cross-task ANOVA (89 rows = experiment-1's 67 + experiment-2's 22, task as a factor):

The model:task interaction is the headline finding. Opus vs sonnet behaves differently on hard (brazil-bench) vs easy (rest-api-crud) tasks. The simple "best stack on rest-api-crud is best everywhere" assumption from experiment-1 doesn't fully generalize for the resource-cost dimensions.

Pareto frontier across both tasks (retort report pareto --data combined.csv --metric code_quality --metric -_cost_usd):

Every other stack is dominated. go / sonnet / beads is the only stack that no other stack beats on both quality AND cost simultaneously.

For prediction of unmeasured cells (when running fewer cells of a fractional design), use retort analyze --predict — emits 95% CIs for cells you didn't run, fitted from cells you did.

pip install only fetches the Python deps. To actually run experiments you also need:

git clone https://github.com/adrianco/retort.git
cd retort
pip install -e ".[dev,test]"

.devcontainer/ provisions Python 3.12 + the C++ toolchain + Node + Go + Rust + the claude and bd CLIs via post-create.sh. Open the repo in GitHub Codespaces or VS Code "Dev Containers: Reopen in Container" and the prereqs are handled automatically. You'll still need to authenticate claude interactively the first time.

retort --help                # CLI loads → Python deps OK
claude --version             # Claude CLI present
bd --version                 # Beads present (only needed for tooling=beads)

# Initialize a workspace
retort init my-eval
cd my-eval

# Edit workspace.yaml to define your factors, responses, and tasks
# Then generate a screening design matrix
retort design generate --phase screening --config workspace.yaml -o design.csv

# Execute experiment runs
retort run --phase screening --config workspace.yaml

# Compute main effects and interactions
retort report effects --db retort.db --matrix-id 1 --metric code_quality

# Evaluate a promotion gate
retort promote my-stack --from screening --to trial \
    --evidence '{"p_value": 0.05}' --config workspace.yaml

Retort runs standalone — pip install + claude CLI is enough to drive every command above. There is no Gas Town dependency.

That said, real experiments are long-running, parallelizable, and benefit from an orchestrator. Gas Town is the orchestrator we use during development; it adds:

• Parallel execution. gt sling dispatches a slice of the design to a polecat (a worker agent in its own git worktree). Multiple polecats share one retort.db via retort run --shard N/M --resume. The --shard partition is a deterministic hash, so two polecats never both pick the same cell, and per-run sqlite commits keep concurrent writers safe.
• Patrol + escalation. witness watches the merge queue; refinery patrols it; mail/escalations route to the mayor agent if anything sticks.
• Auto-evaluation. With gt + bd (beads) installed, the evaluate-run and file-run-issues skills file findings as tracked beads in your project — survives session resets and shows up in queries.

Pattern:

gt sling re-ucc retort --crew alpha   --args "retort run --phase screening --config experiment-2/workspace.yaml --resume --shard 0/4"
gt sling re-ucc retort --crew bravo   --args "retort run --phase screening --config experiment-2/workspace.yaml --resume --shard 1/4"
gt sling re-ucc retort --crew charlie --args "retort run --phase screening --config experiment-2/workspace.yaml --resume --shard 2/4"
gt sling re-ucc retort --crew delta   --args "retort run --phase screening --config experiment-2/workspace.yaml --resume --shard 3/4"

Without Gas Town, the same parallelism works in plain bash:

for s in 0 1 2 3; do
    nohup retort run --phase screening --config experiment-2/workspace.yaml \
        --resume --shard $s/4 > shard-$s.log 2>&1 &
done
wait

Caveats if you choose the gt path:

• Some current gt 0.12.0 rough edges (project-id mismatches, missing bd agent subcommand, gt dolt fix-metadata vs rig-config-sync disagreement) are documented in re-o5b in the beads tracker. None block work; they produce noisy telemetry warnings.
• Concurrent runs multiply per-second token usage. Keep the shard count to a value the Anthropic API tier comfortably supports — start at 2× and monitor rate-limit headers before going higher.

Retort workspaces are configured via workspace.yaml:

factors:
  language:
    levels: [python, typescript, go]
  agent:
    levels: [claude-code, cursor, copilot]
  framework:
    levels: [fastapi, nextjs, stdlib]

responses:
  - code_quality
  - token_efficiency
  - build_time
  - test_coverage

tasks:
  - source: bundled://rest-api-crud

playpen:
  runner: local            # 'local' is the supported path; 'docker' is a skeleton
  replicates: 3
  timeout_minutes: 30

design:
  screening_resolution: 3
  significance_threshold: 0.10

promotion:
  screening_to_trial: { p_value: 0.10 }
  trial_to_production: { posterior_confidence: 0.80 }

src/retort/
├── cli.py              # Click-based CLI entry point
├── analysis/           # ANOVA, Bayesian updating (conjugate NIG), Pareto frontier, residual diagnostics
├── config/             # Pydantic config schema and YAML loader
├── design/             # Factor registry and fractional factorial generator (pyDOE3)
├── playpen/            # Isolated execution: DockerRunner, task loading, prompt building
├── plugins.py          # Pluggy-based plugin system for custom scorers and runners
├── scoring/            # Score collection with pluggable scorers (code_quality, token_efficiency, build_time)
├── promotion/          # Lifecycle state machine, configurable gates, immutable changelog
├── reporting/          # Effects computation, export (text, JSON, CSV), and status dashboard
├── scheduler/          # Candidate intake, D-optimal augmentation, budget tracking, run queue
└── storage/            # SQLAlchemy models and Alembic migrations (SQLite)

• Factors: Variables under test (language, agent, framework) with discrete levels
• Design Matrix: Fractional factorial design that efficiently covers the factor space
• Playpen: Isolated Docker environment where each experiment run executes
• Scoring: Pluggable metrics collected from run artifacts
• Promotion: Evidence-based lifecycle transitions (candidate → screening → trial → production → retired)

Honest accounting. "Code exists" ≠ "tested end-to-end against real data."

• DockerRunner not validated — use runner: local in workspace.yaml for now
• Only claude-code is wired up as an agent
• New scorers (test_coverage, defect_rate, maintainability, idiomatic) are wired but not yet exercised against an experiment with replicates
• idiomatic is opt-in via the responses: list in workspace.yaml (each invocation makes a Claude haiku call, ~$0.001/run, cached per workspace)
• retort intake / D-optimal augmentation untested against a real candidate
• Bundled tasks cli-data-pipeline and react-dashboard exist but haven't been run end-to-end; brazil-bench integration available now as a github://brazil-bench/benchmark-template/<spec>.md task source — running it end-to-end is tracked as re-cn5

# Install dev dependencies
pip install -e ".[dev,test]"

# Run tests
pytest

# Lint
ruff check src/ tests/

# Type check
mypy src/retort/

Apache-2.0 — see LICENSE.