How to Build an Agentic Harness With a Decision Layer Your Code Can Check

Most AI agents make dozens of small choices per task. Which model handles this. Whether to read the code first or start editing. Which tools are relevant right now. Whether to retry or stop and ask.

Almost none of those choices leave a trace. When the agent does well you cannot say which decision earned it, and when it goes wrong you get a summary written by the thing that went wrong.

There is a pattern that fixes this, and it is narrower than it first sounds. You do not hand the model more authority. You hand it one question at a time, from a list your own code wrote, and you check the answer before you act on it. The model gets to suggest the next move. Your code still owns the move.

This is a build guide for that pattern. What to wire, in what order, with the code, and with the parts that are easy to get wrong called out where they bite.

The contract, in three lines

Every decision you hand to a model needs three properties. If a decision is missing any one of them, do not wire it up yet.

1. Named options. You can write down every possible answer before you run it.
2. A result your code can check. Not prose. A value from the set you defined, plus a confidence number.
3. A record of what happened afterwards. Otherwise you can never tell whether the extra call earned its place.

That third one is the one people skip and the one that turns this from a demo into something you can improve. An extra model call in a hot loop has to pay for itself, and without the outcome you have no way to know if it did.

The shape that follows from those three lines is always the same. The host prepares the menu, the model picks from it, the host checks the pick again. "Host" here means the ordinary application code around the model, the part that knows the current state of the world.

Read that right to left and you get the security property. The model cannot widen its own options. An ineligible route is gone before the call, so no confidence score can bring it back. A response containing something you never offered is not a dangerous model. It is a rejected response, and you log it as one.

It is the same discipline as a parameterized query. You are not trusting the caller to behave. You are removing its ability to misbehave.

Decision point one: which route runs this task

The first place this earns its keep is choosing which provider and model handles a new piece of work.

The whole job is in the candidate builder, not in the model call. If you cannot state in code why an option is on the list, it should not be on the list.

type RouteId = string; // "anthropic:sonnet", "openai:gpt-5-mini", ...

interface RouteCandidate {
  id: RouteId;
  label: string;        // shown to the selector, no runnable details
  costTier: "low" | "mid" | "high";
  contextWindow: number;
}

function buildRouteCandidates(task: Task, env: Env): RouteCandidate[] {
  return env.routes
    // The host has to be ready to run every option it offers.
    .filter((r) => env.providers.isInstalled(r.provider))
    .filter((r) => env.providers.isEnabled(r.provider))
    .filter((r) => env.models.exists(r.provider, r.model))
    .filter((r) => r.contextWindow >= task.estimatedTokens)
    .filter((r) => env.supportsReasoningLevel(r, task.reasoningLevel))
    .map(toCandidate);
}

Four things about that function matter more than the model you put behind it.

Candidates carry ids and labels, never runnable details. The selector sees anthropic:sonnet, not an endpoint and not a key. It is choosing a label that refers to something your host already knows how to run. That is what stops a selection from turning into an instruction.

Filter before you ask, not after. A provider that is not installed, a model that does not exist, a reasoning level that is not supported: all of these are removed before the call. "Pick the best model" without a boundary is not a decision, it is a wish.

Pinned routes never enter this path. If a person has already chosen a route, that choice stands. Same for a session that is already running: do not quietly reroute live work. Automatic selection belongs to fresh, unpinned tasks and nowhere else. This is a product rule rather than an optimization, and it is the first thing users notice if you get it wrong.

Passing your filter is not the same as working. The provider still authenticates when the session opens, and that can fail after a route cleared every check you wrote. Your fallback has to cover it.

Decision point two: what the agent does next

The second place is inside the agent loop itself, and it is a smaller question with a tighter blast radius.

The agent is offered four focuses: inspect, implement, verify, answer. The selector returns one focus id. The host turns that id into a set of allowed tools.

type Focus = "inspect" | "implement" | "verify" | "answer";

const BUNDLES: Record<Focus, string[]> = {
  inspect:   ["search", "read_file", "list_dir"],
  implement: ["read_file", "write_file", "move_file", "delete_file"],
  verify:    ["run_tests", "run_build", "run_linter"],
  answer:    [], // empty on purpose
};

function resolveBundle(focus: Focus, env: Env): ToolDef[] {
  // Re-check every named tool against the CURRENT definitions.
  // A bundle written five minutes ago can name a tool that no longer exists.
  return BUNDLES[focus]
    .map((name) => env.tools.lookup(name))
    .filter((t): t is ToolDef => t !== undefined);
}

The answer focus getting no tools is the detail worth pausing on. It is not an oversight to fix later. If answer can still call tools, the loop never terminates cleanly, because there is always one more thing worth checking. An empty bundle is what answer means.

Note that the two decision points are not the same job and should not share a code path. "Which worker" and "which tools next" have different lifecycles and different failure modes. Choosing an expensive model for a rename wastes a little time. Choosing a write-capable bundle when the agent should have been reading wastes a workspace.

Selection is not permission

This is the rule that keeps the whole thing safe, and it is worth writing on a sticky note.

A selector can choose a candidate. It cannot approve an action. If a tool requires user permission, the normal permission path still runs, exactly as it would have without a decision layer. A high confidence number grants nothing. A model that picked the option with 0.98 confidence has still only picked an option.

Keep the two lanes separate in the code, because once they blur in a diagram they blur in the implementation:

async function dispatch(focus: Focus, call: ToolCall, ctx: Ctx) {
  const bundle = resolveBundle(focus, ctx.env);

  // Lane 1: was this choice within the offered set?
  if (!bundle.some((t) => t.name === call.tool)) {
    return reject("out-of-bundle", { focus, tool: call.tool });
  }

  // Lane 2: is the caller ALLOWED to run it? Unchanged by any selector.
  const verdict = await ctx.permissions.check(call, ctx.user);
  if (verdict !== "allow") {
    return reject("permission-denied", { tool: call.tool, verdict });
  }

  return ctx.tools.run(call);
}

The same separation applies to anything that proposes a physical action. A selector picking a low-risk action id out of a host-prepared list is a different responsibility from carrying that action out. Keep the picking and the doing in different functions even when it feels like ceremony.

There is a related trap that shows up the moment you start adding capability. Reading a provider's advertised command list does not give your host a way to run those commands. A slash command listed in an interface does not automatically become an action a selector can call. Those are two separate registries, and conflating them is how a harness quietly hands its model something nobody reviewed. Drawing another arrow on the architecture sheet does not create an API.

Re-check before you act

A decision is made at one moment and acted on at another. In between, the world moves.

Two re-checks cover almost everything:

• Before starting a route, confirm the chosen candidate is still eligible and the task state has not changed. Provider lists change. Sessions get pinned while a call is in flight.
• At tool dispatch, confirm the named tool still exists in the current definitions, not in the definitions you snapshotted when you built the bundle.

Both checks do the same job: stop an old decision from acting on a changed system. They are boring, and boring is the correct texture for a safety check.

The four outcomes, and why you write all four first

A selector call has four possible endings, and a harness is mostly the three that are not the happy path. Decide what each one does before you write the call, not after you hit it in production.

The abstention is the one to design first, because it defines your worst case.

"I cannot choose" is a valid answer, not an error. When it comes back, the host runs whatever it would have run with no decision layer at all. That gives you a property most AI features do not have: if the selector abstains on everything, or goes down entirely, your harness still works. It just stops getting the routing benefit. You have added a component whose failure mode is "no improvement" rather than "wrong action taken confidently."

Then instrument it. The abstain rate is the most useful number you will collect. A rate climbing over time usually means your candidate lists have drifted into being too similar to tell apart, which is a bug in your candidate builder, not in the model.

One rule about bookkeeping: record a fallback as a fallback. If a fallback gets counted as a selector win, every measurement you take afterwards is wrong in the flattering direction.

The decision record

This is the part that separates a demo from something you can put in front of a client, and it is five fields.

interface DecisionRecord {
  id: string;
  at: string;                 // ISO timestamp
  point: "route" | "focus";   // which decision point
  candidates: string[];       // exactly what was offered
  result: { kind: "pick"; id: string; confidence: number }
        | { kind: "abstain" };
  validation: "accepted" | "rejected-out-of-set" | "rejected-stale";
  fallback?: { used: true; to: string; reason: string };
  outcome?: {                 // written later, when the work finishes
    status: "success" | "failure" | "abandoned";
    durationMs: number;
    retries: number;
  };
}

Each field answers a question somebody eventually asks.

candidates is how you prove the model was never offered the thing it stands accused of choosing. result is the pick and the confidence. validation says whether your host accepted or refused it. fallback says whether you ended up somewhere else and why.

outcome is the field people leave out, and it is the only one that makes the system improvable. Without it you have logs. With it you have labelled data.

None of this is the model's reasoning, and it should not be. An explanation generated after the fact does not prove why something happened. What you want to know is what was on offer, what was chosen, whether the host allowed it, and what followed.

Turning records into a confidence threshold

Here is what those records buy you, and it is the step most teams never get to.

Once you have a few hundred records with the outcome field populated, you have a labelled sample. Now you can set the auto-accept threshold from data instead of picking 0.8 because it looks like a reasonable number.

The method is ordinary:

1. Take every record where the host accepted the pick and the outcome is known.
2. Sort by confidence, bucket into tenths.
3. For each bucket, compute the share that ended in success.
4. Find the lowest bucket where that share is acceptable for the cost of being wrong at this decision point.
5. Set the auto-accept threshold at the bottom of that bucket. Route the band below it to a fallback or a person. Re-run the whole thing monthly, because it drifts.

The cost of being wrong is not the same at both decision points, which is why they get different thresholds. A wrong route costs you some latency and some money. A wrong tool bundle can cost you a workspace.

Choosing your selector

Three properties decide this, and the one everyone leads with is the least important of the three.

Calibration first. If you gate on a confidence score, that score has to correspond to reality. The measure is expected calibration error, and lower is better. One independent paper reports 0.246 for hosted Jev against 0.081 for the self-hosted Laya model after temperature scaling, on the same task set.⁵ Laya is explicitly trained against Brier score, and Jev's training method is undisclosed.⁵ If your thresholds are load-bearing, this is the number that decides whether they are real.

Latency second, because it sets how many times per loop you can afford to ask. The same paper puts a single hosted Jev request at 236 to 276 ms at the median, and Laya at 32.8 to 39.5 ms on a T4 GPU, which is 217 to 254 decisions a minute against 1,500 to 1,818.⁵

Accuracy third, and be honest about it. On the independent benchmark a typed decision model does not beat a frontier model. Jevals, covering 31,500 scored decisions across seven models at 300 questions per task asked five times each, puts Jev at a Decision Score of 67.8 on pick-one questions where Gemini 3.8 Flash scores 74.1, and 69.0 against 73.0 on yes-or-no questions, at roughly a twenty-eighth of the price.⁴ That is a Decision Score where 100 is perfect and 0 is guessing the label base rates, so it is not a percentage.

Which leads to the arithmetic that actually settles it. The comparison is not selector cost against nothing. It is total task cost either way: selector time, plus worker time, plus retries, plus the review effort at the end. A routing layer that takes longer than the task it is routing is an elaborate waiting room. At 250 ms per call on a task that runs for four minutes, you can ask twenty times and lose two percent of wall clock. At 250 ms per call on a two second task, one question has already cost you an eighth of the job.

Run that sum for your own loop before you pick a selector. The answer changes which of the three properties you should be optimizing.

The reference implementation worth reading is keel, an MIT-licensed local-first macOS coding workspace built in Rust on GPUI, targeting Apple Silicon and macOS 15 or later.² It ships three mutually exclusive modes: a local Laya model over Core ML as the default, an opt-in hosted Jev call using a credential already on the machine, and a normal mode with no selector at all.² That third mode existing is the useful signal. The architecture is the product and the selector is swappable.

If you want the plain-language version of what this class of model is and is not before you commit to one, we wrote that up separately in Jev explained in plain English.

Where your harness stops

One boundary catches people out as soon as they go from one agent to several.

A harness can reach external coding agents through the Agent Client Protocol. The client launches the agent as a subprocess and the two speak JSON-RPC 2.0 over stdio, negotiating capabilities, sessions, streaming prompt turns, permission requests before sensitive operations, and client-provided filesystem and terminal access.⁶ The protocol has moved quickly: created in June 2025, built into JetBrains IDEs since December 2025, a public agent registry launched in January 2026, and a headline feature of Zed 1.0 in April 2026.⁷

What that gives you is a clean call boundary and a permission prompt. What it does not give you is control of the other agent's loop. It runs its own reasoning, its own sub-steps, and its own tool configuration inside its own process. Your decision layer can choose whether to hand work to it. Your decision layer does not choose what happens once it has the work.

So the ownership line fits on one sentence: you own every loop running in your process, and you own the decision to enter any loop that does not.

Worth being precise about the same thing when describing where your models run. "Local selector" describes the selector. The coding worker it routed to may still be calling a hosted model, and so may a locally running external agent. Trace each hop before you describe a workflow as local or private, especially if a client is going to read that description.

What self-improving honestly means here

The phrase gets used loosely, so it is worth being exact about what these records do and do not give you.

They do not train anything. Recording decisions does not mean the system learns from chat history or quietly rewrites its own policy after a bad run. Nothing above amounts to an agent that improves itself while you sleep.

What you get is better than that for anything you have to maintain: a receipt becomes a replayable scenario.

Take a failure. A selector chose a provider that became unavailable a second later. Instead of logging "the model got confused," you saved the offered candidates, the task state, the chosen id, the validation result and the fallback. That is enough to reconstruct the decision exactly. So you can:

1. Reproduce the same candidate set and task flags.
2. Run the current policy as a baseline.
3. Run your proposed change under identical conditions.
4. Compare valid selection rate, abstention rate, fallback rate, latency and downstream result.
5. Have a person look at the difference and decide whether to keep it.

Hold the task set and the scoring rules fixed while you do it, and keep a separate set of tasks for the final comparison. If you tune against every case you have and then cite the familiar ones as proof, you have measured your own tuning. The awkward runs are the valuable ones, because they expose the missing rule that clean demos hide.

That improvement loop can propose changes to candidate construction, the typed schema, the fallback policy, the model behind the selector, or the tool bundle attached to a focus. Every one of those keeps the old version, replays the scenarios, and gets a human approving the version that becomes the new baseline. No silent retraining, no "it changed because it learned" with no diff and no way back.

Less cinematic. But a changed rule, an old result and a new result is something you can work with.

This pattern is not only for coding agents

Strip out the code editor and what is left is generic: a bounded choice, made cheaply, from a list your own code controls, with a defined answer for "not sure," and a record of every call.

That is the same shape as deciding which of several owners a new inbound lead belongs to, which we took apart in lead routing at the moment of form submit. It is the same shape as scoring a lead against fixed bands, in building an automated lead scoring model without a platform. And it is the same shape as assigning one intent label to each of fifty thousand keywords, in classifying search intent at scale.

None of those need an agent harness. All of them need the same three-step discipline, the same four outcomes, and the same five-field record.

Start with one decision

If you are building this week, write down four things before you write any code.

1. One decision the selector is allowed to make. One. Not a category.
2. The exact set of options it can see, and the code that produces that set.
3. What happens when it abstains, written and tested before the happy path.
4. How you will know whether the outcome helped, which means deciding now what goes in the outcome field.

Then build the narrowest path that enforces those four rules. Start small enough that a bad choice has nowhere to hide, and only widen it once the records tell you the extra call is paying for itself.

If you are weighing whether this belongs in something you already run, that is the work we do under AI agent development. The first conversation is usually about which single decision in your process is bounded enough to be worth automating, and it is a short conversation.

Frequently asked questions

What is an agentic harness?

The application code that owns an AI agent's loop. It holds the session, defines which tools exist and which are reachable right now, prepares the options for any decision, calls the model, and validates what comes back before acting. The model is a component inside the harness. Calling a model in a for loop is not a harness, because nothing in that arrangement decides what the model is allowed to choose from.

What does a decision layer add to an agent?

A bounded choice, made cheaply enough that you can afford to ask on every iteration, returned in a shape you can validate. It does not make the agent smarter, it does not reason, and it cannot explain itself. In practice there are two places it earns its keep: choosing which provider route runs a new task, and choosing which of four focuses the agent moves to next.

How many decision points should a harness have?

Start with one. The reference implementations that work use two. Every additional point is another thing to validate, another record type, and another threshold to maintain, so each one should be justified by a measured problem rather than by symmetry.

What happens when the selector abstains?

The host runs whatever it would have run with no decision layer at all. Abstention is a valid answer rather than a failure, and designing that path first is what gives the whole system a defined worst case of "no improvement." Record it as a fallback and watch the rate: a climbing abstain rate usually means your candidate lists have become too similar to distinguish.

Does a high confidence score grant permission to act?

No, and keeping those separate is the most important safety rule in the pattern. Selection and permission are different lanes. If a tool needs user approval, it still needs user approval regardless of what the selector returned or how confident it was. A selector picks from a list. It does not authorize anything.

How do I choose between a hosted and a self-hosted decision model?

By calibration first, latency second, accuracy third. One independent paper reports expected calibration error of 0.246 for hosted Jev against 0.081 for self-hosted Laya after temperature scaling, with medians of 236 to 276 ms and 32.8 to 39.5 ms on a T4 respectively. If you gate on confidence or you call the selector many times per loop, those numbers favor self-hosting. If you have no GPU and no appetite for running inference, a fifth of a second and no infrastructure is a reasonable trade.

What should a decision record contain?

Five things: the candidates offered, the result and its confidence, whether validation accepted or refused it, whether a fallback ran and why, and the observed outcome afterwards. The outcome field is the one people omit and the only one that lets you set a threshold from data later.

How do I set the confidence threshold?

From a labelled sample, not from instinct. Collect a few hundred records where the outcome is known, bucket them by confidence, compute the success rate per bucket, and set auto-accept at the bottom of the lowest bucket whose success rate is acceptable for the cost of being wrong at that decision point. Different decision points get different thresholds, because a wrong route and a wrong tool bundle do not cost the same. Re-run it monthly.

Does a decision layer control external agents too?

No. Agents reached over the Agent Client Protocol run as separate subprocesses speaking JSON-RPC over stdio, with their own loops and their own tool configuration. Your harness decides whether to hand work to one of them. It does not decide what happens inside. Only loops running in your own process are yours to govern.

Does recording decisions mean the agent improves itself?

Not on its own. Records give you replayable scenarios: the conditions to reconstruct a decision, run a proposed change against the same inputs, and compare the results. A person still reviews the difference and approves the new baseline. That is a real improvement loop, and it is not the same thing as a system that retrains itself without a diff or a rollback path.

Where does this pattern apply outside a coding tool?

Anywhere the answer set is fixed, the volume is high, the call is cheap to reverse, and nobody needs a written justification. Lead routing, lead scoring bands, search intent labelling, moderation gating and bulk triage all have the same shape, and the same three-step discipline transfers even when none of the tooling does.

Facts checked on 26 September 2026. The protocol, the models and the reference implementation are all weeks old and moving. Where a figure here comes from a single exploratory study or from a vendor, it says so, and we will correct this page rather than quietly leave it wrong.