Getting started

loop-generator is a definition-of-done engine for coding agents. You write the checks that are red today and go green only when the work is really finished; the runner invokes a coding agent and re-invokes it, feeding back what still fails after each turn, until the checks pass or the budget runs out — and hands you a report you can review.

Install

npm install

The Claude Agent SDK, Grok Build CLI, GitHub Copilot CLI, and opencode are optional backends. For real agent runs, set credentials for the driver you use:

export ANTHROPIC_API_KEY=...  # claude-agent-sdk (or Claude login / Bedrock / Vertex)
export XAI_API_KEY=...        # grok (or run `grok` interactive login)
# github-copilot: install the `copilot` CLI and run it once to authenticate
#                 (or set GH_TOKEN / GITHUB_TOKEN for an unattended run)
# opencode: install the `opencode` CLI; runs against local models, no key needed

Local models (LM Studio / Ollama)

LM Studio and Ollama are inference servers — they return tokens. They do not edit files or run tools. loop-generator does not ship an HTTP coding-agent driver; local inference goes through the existing opencode driver. OpenCode owns the tool loop; you point it at a local provider.

1. Install the opencode CLI and load a tool-calling model in LM Studio (or Ollama). A chat-only GGUF will answer in prose and write nothing.
2. Confirm the ids match: curl http://127.0.0.1:1234/v1/models (what LM Studio has loaded) and opencode models (what OpenCode will accept). The spec's model must be provider/model — without the lmstudio/ (or ollama/) prefix OpenCode mis-resolves the id.
3. Write the driver block:

   driver:
     uses: opencode
     options:
       model: lmstudio/qwen/qwen3-coder-next
       dangerouslySkipPermissions: true

4. Scaffold a RED target and run the building-block example:

   npm run loopgen -- init-target opencode-feature
   npm run loopgen -- run examples/building-blocks/opencode-feature.loop.yaml

Preflight warns (does not fail) if model is omitted, lacks a provider prefix, or an lmstudio/ model can't reach http://127.0.0.1:1234/v1/models. If you need to own the tool loop yourself, that is a separate coding-agent project — loop-generator will not grow an in-tree HTTP agent.

Quick start

Run the offline demo:

npm run loopgen -- run examples/building-blocks/mock-demo.loop.yaml

Then run it again. The second run fails with baseline-vacuous after zero agent iterations: the answer is already in the workspace, so the check is green before any work and there is nothing for the agent to earn. That's the default strict baseline doing its job. Reset with rm -rf examples/building-blocks/.workspace.

Generate a new loop and run it:

npm run loopgen -- generate -i           # interactive
npm run loopgen -- generate -i --verify  # …and prove it's lint-clean + starts RED
npm run loopgen -- run my-loop.loop.yaml

--verify encodes the authoring contract: after writing the spec it lints it and runs the checks once with no agent turns, confirming they start RED — a green check before any work probably doesn't test the requirement. Exit codes match lint: 2 on lint errors, 1 on a vacuous/GREEN check set, else 0.

Add --trace <file> to any run to write a JSONL execution trace — the loop's iterations plus the agent's inner tool-call trajectory (see Observing runs):

npm run loopgen -- run my-loop.loop.yaml --trace trace.jsonl

List what's registered, or verify a driver:

npm run loopgen -- list
npm run loopgen -- verify-driver mock

After npm run build && npm link — or a global install — you can use the loopgen binary directly instead of npm run loopgen -- <args>.

Concepts

The generic command evaluator already covers tests, linters, type checkers, and benchmarks: anything with a CLI and an exit code. The experiment evaluator reads a numeric metric (from a command's JSON output or a file) and compares it against thresholds or baselines, which suits A/B tests and performance work.

The spec

version: 1
name: add-retry-to-fetchUser
task:
  type: function
stack:
  language: typescript
  packageManager: npm
workspace:
  dir: ./target          # the directory the agent edits (relative to this file)
  snapshot: none         # none (default) | git — git writes checkpoint refs under refs/loopgen/<run>/ for inspect/reset
  ignore: []             # extra globs excluded from change detection, on top of the built-in artifact defaults
requirements: |
  Add exponential backoff (max 3 retries) to fetchUser(). Keep the signature.
driver:
  uses: claude-agent-sdk
  options:
    model: claude-opus-4-8
    maxTurns: 30
evaluators:
  - uses: command
    as: tests
    options: { command: npm test }
  - uses: command
    as: typecheck
    options: { command: npx tsc --noEmit }
success:
  type: all-pass         # all evaluators must pass
limits:
  maxIterations: 6
  baseline: strict       # strict (default) | true | false — checks must be RED before the agent
  specGuard: error       # error (default) | warn | off — agent edits this spec mid-run
  evaluatorGuard: error  # error (default) | warn | off — agent edits the test files a check runs
  maxCostUsd: 5.0        # optional — stop with "budget-exceeded" past this cumulative cost
  maxTokens: 2000000     # optional — same, on cumulative input+output tokens
evaluation:
  concurrency: 1         # run evaluators sequentially (default; safe for shared DB/state)
observability:          # optional — stream the run's telemetry
  observers:
    - uses: jsonl        # or otlp; see Observing runs
      options: { file: trace.jsonl }

Field-by-field guidance — including which trust settings to pick and why — lives in Authoring a loop you can trust; the observability block is covered in Observing runs.

Examples

The examples directory is a guided tour, ordered from the simplest offline loop up to full loop patterns:

• Building blocks — start with the offline mock-demo (no API key), then one spec per mechanism: each driver, the function / api / experiment task types, and a batch run.
• Loop patterns — established patterns written as specs with the trust guards on: the Ralph Wiggum loop, the evaluator-optimizer, and a discover → implement → verify harness.
• Self-contained projects — examples that run end-to-end without pointing at your own repo.

Batch runs (punch lists)

To run many units of work across one or more codebases, list them in a .batch.yaml manifest and run them with one command:

# punch-list.batch.yaml
version: 1
concurrency: 2          # items run in parallel up to this many…
continueOnError: true   # …and a failure doesn't stop the others
defaults:
  maxIterations: 6      # merged into every item (item-level values win)
items:
  - name: add-retry
    spec: loops/add-retry.loop.yaml
    base: /repos/service-a          # which repo this item's workspace resolves in
  - name: fix-pagination
    spec: loops/fix-pagination.loop.yaml
    base: /repos/service-a
    needs: [add-retry]              # runs only after add-retry succeeds
  - name: dark-mode
    spec: loops/dark-mode.loop.yaml
    base: /repos/web

npm run loopgen -- batch punch-list.batch.yaml --report batch-report.json
# offline demo:
npm run loopgen -- batch examples/building-blocks/punch-list.batch.yaml

The scheduler honors needs ordering and the concurrency cap, and guarantees that two items resolving to the same workspace never run at once — same-repo items auto-serialize, so parallelism is safe across distinct repos. A failed or skipped dependency cascades: its dependents are skipped. You get a per-item summary (status · iterations · files · cost · warnings), an aggregate JSON report, and a non-zero exit if any item failed. Items can also inline: a full spec instead of referencing a file.

Extending it

The whole system is four plug-in points — drivers, evaluators, task types, and observers. Register your own and pass them to the engine — adding a plug-in never means editing the engine. (Custom observers are covered in Observing runs.)

A new evaluator (feedback tool)

import { type Evaluator } from "loop-generator";

export const coverageEvaluator: Evaluator = {
  type: "coverage",
  async evaluate(ctx) {
    const pct = await measureCoverage(ctx.workdir);   // your logic
    return {
      passed: pct >= 0.9,
      score: pct,
      feedback: `coverage ${(pct * 100).toFixed(1)}% (need ≥ 90%)`,
    };
  },
};

A new driver (agent backend)

Implement AgentDriver, then validate it against the conformance harness:

import { runDriverConformance, formatConformanceReport } from "loop-generator/testing";
import { myDriver } from "./my-driver";

const report = await runDriverConformance({ makeDriver: () => myDriver });
console.log(formatConformanceReport(report));   // ✓/✗ per behavioral contract

The harness drives your agent against temp workspaces and asserts the contract: it reports a name, creates a requested file, applies feedback across iterations, and handles aborts. Prompt-driven drivers (like the Claude SDK) work out of the box; scripted drivers supply an optionsFor mapping. The CLI exposes it too:

npm run loopgen -- verify-driver claude-agent-sdk

Working in Claude Code? The in-repo add-driver skill (.claude/skills/add-driver/) scaffolds a new backend and loops on verify-driver for you, fixing one failing scenario at a time — part of the skill pipeline the repo ships.

Use the engine as a library

import { LoopEngine, createDefaultRegistries, parseSpec } from "loop-generator";

const engine = new LoopEngine(createDefaultRegistries());
const report = await engine.run(parseSpec(spec), { baseDir: process.cwd() });
console.log(report.success, report.reason);

Development

npm run typecheck
npm test
npm run build

Test quality gates:

npm run coverage   # vitest + v8 coverage (gate: 85% lines/functions/statements, 80% branches)
npm run mutation   # Stryker mutation testing (gate: 60% mutation score)