You write the checks; Claude, Grok, Copilot, or a local model works until they pass; and the run ends in a report you can hand to a reviewer — not the agent's word that it's finished.
# offline demo — no API key needed
$ npm run loopgen -- run examples/building-blocks/mock-demo.loop.yaml
[loopgen] running baseline evaluation (no agent) — disable with limits.baseline: false
[loopgen:iter0] starting iteration 1/5
iter 1: retry — agent ok — ✗ answer-check · 1 file(s)
[loopgen:iter0] result: not yet — failing: answer-check
[loopgen:iter1] starting iteration 2/5
iter 2: PASS — agent ok — ✓ answer-check · 1 file(s)
[loopgen:iter1] result: PASS — all checks passed
✓ SUCCESS — mock-demo
outcome: success — all checks passed
baseline: checks fail as expected — failing: answer-check
iterations: 2, time: 0.0s
changed: 1 file(s)
Abridged. The real run also warns that its workspace isn't a git repo, so
change detection fell back to content hashes and there's no unified diff.
That caveat is part of the receipt — a thinner receipt says so. Run it a
second time and it fails baseline-vacuous after zero agent
iterations: the answer is already there, so there's nothing to earn.
Overnight, in CI, across a fleet. Every run ends in a report with one of nine outcomes, each iteration's results, and the diff of the real work — and streams the agent's trajectory as JSONL or standard OTLP spans to whatever you already use.
Cut a product into slices a check can observe, give each its own red-then-green checks, and run the whole graph as one batch. A ten-slice coaching portal built this way came back 10/10 for about $1.77 — and the report showed the work, not just the "done".
The loop is a versioned file your team code-reviews, not a script on someone's laptop. Spend is capped per run. The report is an artifact you can keep — the difference between a party trick and a process.
One .loop.yaml describes the task, the workspace the agent
may edit, and the checks that are red today and go green only when the
work is really finished. The engine drives the agent, feeds every
failure back, and repeats — until the checks pass or the budget runs out.
If you can't write a check that's red before the work and green only
when the requirement is met, the work isn't ready for an agent yet.
loopgen lint and generate --verify tell you
that before a token is spent.
The agent is a plug-in — claude-agent-sdk,
grok, github-copilot, opencode
with a local model, or a scripted mock — held to the same
contract whichever you pick. Each round it sees what still fails and
what it changed last time.
What the agent changed, which checks it passed, what it cost, and whether anything it wasn't allowed to touch was touched. If the green wasn't earned, the report says so in one word — and says why.
Reward hacking isn't hypothetical — frontier coding models have been caught special-casing tests, hard-coding expected values, and editing the very test files that grade them. Most loop runners take the agent's word for it. This one treats every green as a claim to be checked — and the report tells you what it checked. All of it is on by default; the one thing you add yourself is a spend ceiling, because only you know the number.
Every iteration is diffed with build and runtime artifacts excluded. A green run that changed no source files is flagged as likely vacuous, not celebrated.
Always on → warning on a green runDrivers report an honest stopReason. An agent that
errored or ran out of turns on an otherwise-green run puts a caveat in
report.warnings — a warning on a green run is a finding.
The test files a check runs are the contract. They're
hash-watched across iterations — detected from the command, plus any
explicit guard: paths.
evaluatorGuard: error) → run fails evaluator-tampered
The .loop.yaml is excluded from the work diff and
hash-watched when it lives in the workspace. A mid-run edit isn't a
clever fix; it's a failed run.
specGuard: error) → run fails spec-tampered
Baseline evaluation runs every check before the agent's
first turn. Already passing? Then the checks don't measure the
requirement, and no agent turn is spent finding out. Set
baseline: false only for checks whose side effects must not
run twice.
baseline: strict) → run fails baseline-vacuous
maxCostUsd and maxTokens cap cumulative
spend. A satisfied iteration always reports success first — only
further spend is refused.
budget-exceeded
success — checks pass, guards quiet
spec-tampered · evaluator-tampered
baseline-vacuous — checks never went red
budget-exceeded · max-iterations
preflight-failed · aborted · error
No outcome is inferred from the agent's self-report. "Done" is a rule over your check results — never the model's opinion.
loopgen lint catches the misconfigurations that fake
results upstream — self-fulfilling smoke tests, wrong workspace,
missing binaries, destructive checks — in milliseconds. When you don't
trust the verdict, the trace shows the agent's every turn.
One loop delivers one checkable outcome. An application is a graph of
them. Cut it into the smallest slices a check can observe, order them by
what each needs built first, give every slice its own red checks, and run
the graph as a .batch.yaml — dependency-ordered, concurrency
capped, never two loops in one workspace at once.
frame-app turns a product spec into a dependency graph
of slices. frame-checks turns one slice into acceptance
checks that are red now, for the right reason, and hard to fake.
author-loop writes the spec and proves it lints clean and
starts red. debug-loop reads a failed or suspicious run
and finds the fix without spending agent budget. They load when you
open the repo in Claude Code.
A ten-slice leadership-coaching portal — sign-in, clients, surveys, notes, a client portal with role separation — built from a plan, ten check files, and one batch manifest. Result: 10/10 trustworthy greens under a strict baseline with the evaluator guard armed, about $1.77 and nine minutes of agent time. The write-up also records what the experiment didn't show.
Version it, review it, reuse it. loopgen generate -i --verify
scaffolds one interactively and proves it's lint-clean and starts RED.
name: add-retry-to-fetchUser
task:
type: function
workspace:
dir: ./target
requirements: |
Add exponential backoff (max 3 retries) to fetchUser(). Keep the signature.
driver:
uses: claude-agent-sdk
evaluators:
- uses: command
as: tests
options: { command: npm test }
- uses: command
as: typecheck
options: { command: npx tsc --noEmit }
success:
type: all-pass
limits:
maxIterations: 6
baseline: strict # fail fast if the checks were already green
specGuard: error # editing the spec fails the run
evaluatorGuard: error # editing the tests fails the run
maxCostUsd: 5.0
observability:
observers:
- uses: otlp # stream the run as spans to any OTLP collector
options: { endpoint: http://localhost:4318/v1/traces }The guards are three lines of YAML. Everything after that is the engine's problem — including proving it to whoever reads the report.
claude-agent-sdk, grok,
github-copilot, opencode — including LM Studio
and Ollama models through it — or a scripted mock for
offline runs. Swap with --driver; the spec doesn't change.
command covers anything with a CLI and an exit code —
tests, linters, type checkers, builds. experiment compares
a numeric metric against thresholds or baselines. Or register your own
in ~15 lines.
New backend? loopgen verify-driver runs it against a
behavioral contract — reports a name, does requested work, applies
feedback, handles aborts — before it's trusted inside a loop.
Overnight and CI runs nobody is watching. A failing test that encodes the new behavior. A metric to move: p95 under X ms, coverage ≥ 90%, bundle under Y KB. Repo-wide remediation against a concrete check. Fleets of agents whose word you'd rather not take.
"Improve the architecture." "Make the UI delightful." No exit-zero definition of done means the loop can only tell you the checks were the wrong contract. A passing check doesn't make subjective work good.
And honesty about the guards themselves: they make mis-specified checks, gamed metrics, and context drift visible — they don't eliminate them. If you can't write a check that's RED before the work and GREEN only when the requirement is met, the task isn't ready for a loop. How to prove that up front →
Drivers, evaluators, task types, and observers are typed registries — register your own and pass them in. The engine, and its guards, never change.
// a custom evaluator, complete
import { type Evaluator } from "loop-generator";
export const coverage: Evaluator = {
type: "coverage",
async evaluate(ctx) {
const pct = await measure(ctx.workdir);
return {
passed: pct >= 0.9,
score: pct,
feedback: `coverage ${(pct*100).toFixed(1)}% (need ≥ 90%)`,
};
},
};// a custom observer: ship the evidence anywhere
import { type Observer } from "loop-generator";
export const slackObserver: Observer = {
name: "slack",
begin({ runId, spec }) {
return {
onRunEnd: (report) =>
postToSlack(
`${spec.name}: ${report.outcome} ` +
`in ${report.iterations.length} iterations`,
),
};
},
};
Embedding it? The engine is a library too —
new LoopEngine(createDefaultRegistries()), with an
onAgentEvent callback for streaming the trajectory yourself.
Seven modules, no API key. Step through scripted runs — the happy path, a vacuous baseline, a spec tamper, a blown budget. Build the exact prompt the agent sees. Pick an attack and watch which guard catches it, and which two don't yet. The demos import the engine's own functions, so what you see is what ships.
Write the checks once. Let any agent grind toward them. Trust the report because it shows the work — not because the agent said so.