The Orchestration Layer: How to Build Reliable AI Agents

Most AI agent demos work. Most AI agent products struggle. Gartner expects over 40% of agentic AI projects to be cancelled by the end of 2027, mainly because of rising costs and unclear business value (Reuters report).

The gap between demo and product is rarely the model. It is the layer around the model: the part that plans the work, hands it out, remembers what happened, stops for a human and picks up after a crash. That is the orchestration layer. This post explains its eight parts in plain English, with short Python you can copy.

The orchestration layer is the manager, not the worker

Think of a restaurant. The model is the chef: brilliant at one dish at a time. The orchestration layer is the kitchen manager. It reads the order, decides which station cooks what, tracks which dishes are done, checks the plate before it leaves and restarts a dish that got dropped.

In code, the orchestration layer answers five questions on every step:

1. What runs next? A fixed step, or one the model picks.
2. Who runs it? A single model call, a tool, or another agent.
3. What do we remember? The state that survives between steps.
4. Does a human need to say yes? Before anything that can't be undone.
5. What if it fails? Retry, resume or stop safely.

Rule zero: start with the simplest thing that works

Anthropic separates two kinds of systems in Building Effective Agents. Workflows run models and tools through code paths you wrote in advance. Agents let the model decide its own steps and tool use. Workflows are predictable. Agents are flexible but cost more and can stack up errors. Their advice: find the simplest solution possible, and add complexity only when it measurably helps.

OpenAI says the same in A Practical Guide to Building Agents: start with one agent and its tools, and split into several agents only when the prompt gets too complex or the agent keeps picking the wrong tool.

So the eight parts below are a toolbox, not a checklist. Use the ones your problem needs.

1. Pick the shape of the work first

Every workflow is built from a few shapes. Anthropic names five:

2. Plan ahead when you know the steps

If you can write the steps on a sticky note, write them in code. This is ahead-of-time planning: the order is fixed, so every run behaves the same and is easy to test. Google's Agent Development Kit (ADK) has workflow agents (sequential, parallel and loop) that run sub-agents without asking a model what to do next, for exactly this reason.

# A fixed workflow: the plan lives in code, not in the model.
def research_lead(state):
    # One model call with one clear job. The result goes into shared state.
    state["notes"] = call_llm(f"Summarise what {state['company']} sells and who buys it.")
    return state

def draft_email(state):
    # This step reads the previous step's output from state, not from memory.
    state["draft"] = call_llm(f"Write a 4-line email to {state['company']} using: {state['notes']}")
    return state

PIPELINE = [research_lead, draft_email]   # the plan, fixed in code

def run_pipeline(state):
    for step in PIPELINE:                  # same order on every run = easy to test
        state = step(state)
    return state

call_llm stands for your model call (OpenAI, Claude, Gemini or a local model). The pattern doesn't care which.

3. Let the model plan only when the steps can't be known

Some tasks can't be planned in advance. "Research these five competitors" might need two searches or twenty. Here the model builds the plan at runtime. This is dynamic planning, and the orchestrator–workers shape from part 1.

The trick is to treat the model's plan as data you check, not instructions you obey. Ask for a list of steps in JSON, reject anything you didn't build, and cap the length. Google's ADK team put it well: "The model makes the decision; the graph makes it reliable, observable, and resumable."

import json

ALLOWED = {"research_lead", "draft_email", "send_email"}   # the only steps the model may pick

def make_plan(goal):
    raw = call_llm(f"Goal: {goal}. Return a JSON list of steps from {sorted(ALLOWED)}.")
    plan = json.loads(raw)                  # the plan is data, not free text
    unknown = [s for s in plan if s not in ALLOWED]
    if unknown:                             # never run a step you didn't build
        raise ValueError(f"Model asked for unknown steps: {unknown}")
    return plan[:6]                         # hard cap, so a confused plan can't run forever

Plans can also mark which steps are independent so they run in parallel. The LLMCompiler paper did this with a planner that builds a task graph, and reported up to 3.7× faster runs and up to 6.7× lower cost than a step-by-step ReAct agent on its tests.

Size the effort to the task too. Anthropic's research agent uses 1 agent and 3–10 tool calls for a simple fact, 2–4 sub-agents for a comparison, and 10+ only for complex research.

4. Use many agents only when the work truly splits

Multi-agent systems can be powerful. Anthropic's research system, with a lead agent and parallel sub-agents, beat a single agent by 90.2% on its internal research tests. The catch: it used about 15 times the tokens of a normal chat, and token use alone explained 80% of the variation in results on one benchmark. More agents mostly means more spending.

They also break in new ways. Cognition, the team behind Devin, showed it with a Flappy Bird example: one sub-agent built a Mario-style background, another built a bird that didn't match, and the final agent had to glue two mismatched pieces together. Each agent made silent decisions the others couldn't see. A Berkeley-led study of seven multi-agent frameworks, Why Do Multi-Agent LLM Systems Fail?, sorted 14 failure modes into three groups: poor system design, agents misaligned with each other, and weak checking of the final result.

Cognition's April 2026 update gives the rule that works today: many agents can contribute, but writes stay single-threaded. Sub-agents that only read (search, review) work well. Several agents editing the same thing don't.

Two patterns cover most needs, as OpenAI's Agents SDK docs describe. With agents as tools (manager), one agent keeps control and combines results. With handoffs, a triage agent passes the whole conversation to a specialist.

When you delegate, write a proper brief. Anthropic found each sub-agent needs four things: an objective, an output format, guidance on which tools to use and clear task boundaries. Vague briefs led to sub-agents duplicating each other's work.

When agents run in different systems, two open standards help. MCP (Model Context Protocol) connects an agent to tools and data. A2A (Agent2Agent) lets agents from different vendors find each other and hand off tasks, without exposing their internal memory or tools. Both now sit under the Linux Foundation's Agentic AI Foundation: MCP from its launch in December 2025, and A2A since August 2026, after its first stable v1.0 spec in March 2026.

5. Ask a human before anything you can't undo

Human-in-the-loop (HITL) means the workflow pauses and waits for a person to approve or reject. OpenAI suggests two triggers: high-risk actions (refunds, payments, cancellations) and repeated failure (the agent misses its goal after a few tries).

The best moment to pause is between the model choosing an action and your code running it. Dex Horthy's 12-Factor Agents makes this point: many frameworks can pause, but not at that exact spot, which is the one that matters.

And don't keep a process waiting. Save the state, exit, and resume when the answer arrives, even if that's three days later. The code in part 6 does this.

If you use LangGraph, its interrupt function does the same job (adapted from their docs):

from langgraph.types import Command, interrupt

def approval_node(state):
    # Pauses the graph here and shows this payload to your UI.
    decision = interrupt({"question": "Send this email?", "draft": state["draft"]})
    # Runs only after resume. True = go to "send", False = go to "cancel".
    return Command(goto="send" if decision else "cancel")

config = {"configurable": {"thread_id": "lead-acme-001"}}  # the run ID
graph.invoke({"draft": "Hi Acme..."}, config)   # runs until the pause
graph.invoke(Command(resume=True), config)      # later: approve and continue

6. Save after every step so any run can pause and resume

A checkpoint is a saved copy of the workflow's state. Give each run a unique ID, save after every step, and you get three things for free: pausing for approval, resuming after a crash, and a history you can debug.

Here is a small orchestrator that does it all: a run ID, a checkpoint after each step, and an approval gate before the risky step. It builds on the steps from parts 2 and 3, and SQLite stands in for any database.

import json, sqlite3, uuid

db = sqlite3.connect("runs.db")
db.execute("CREATE TABLE IF NOT EXISTS runs (id TEXT PRIMARY KEY, state TEXT)")

def save(run_id, state):
    # Write the whole state after every step. A crash loses at most one step.
    db.execute("REPLACE INTO runs VALUES (?, ?)", (run_id, json.dumps(state)))
    db.commit()

def load(run_id):
    row = db.execute("SELECT state FROM runs WHERE id = ?", (run_id,)).fetchone()
    return json.loads(row[0])

RISKY = {"send_email"}   # steps that can't be undone need a human yes
STEPS = {"research_lead": research_lead, "draft_email": draft_email}
# send_email is added in part 7

def run(run_id):
    state = load(run_id)
    while state["next"] < len(state["plan"]):
        step = state["plan"][state["next"]]
        if step in RISKY and state.get("approved") != step:
            state["status"] = "waiting_for_approval"   # pause: save and exit
            save(run_id, state)
            return state
        state = STEPS[step](state)      # do the work
        state["next"] += 1              # move forward only after the step succeeds
        save(run_id, state)             # checkpoint
    state["status"] = "done"
    save(run_id, state)
    return state

def start(goal, company, to):
    run_id = str(uuid.uuid4())          # the ID you resume with later
    state = {"run_id": run_id, "company": company, "to": to,
             "plan": make_plan(goal), "next": 0, "status": "running"}
    save(run_id, state)
    return run(run_id)

def approve(run_id):
    # Called by your Approve button, Telegram bot or webhook, any time later.
    state = load(run_id)
    state["approved"] = state["plan"][state["next"]]
    save(run_id, state)
    return run(run_id)                  # resumes exactly where it paused

Long-running coding agents use the same idea with plain files. Anthropic's harness for long-running agents keeps a progress log and a JSON feature list, and treats each git commit as a checkpoint the next session can read or roll back to.

7. Recover from failure without doing anything twice

Crashes happen: a timeout, a deploy, a laptop lid. The danger isn't the crash. It's the retry that sends the email twice or books two flights.

Databases solved this decades ago with a write-ahead log (WAL): write down what you're about to do before doing it, and after a crash, read the log to see what finished. For agents, that becomes a journal of steps plus an idempotency key: a unique ID sent with the action so the receiving system ignores repeats. Stripe's API works this way: retry a payment with the same key and you get the first result back, not a second charge.

def send_email(state):
    # Same run + same step = same key on every retry.
    key = f"{state['run_id']}:send_email"
    email_api_send(to=state["to"], body=state["draft"], idempotency_key=key)
    state["sent"] = True
    return state

STEPS["send_email"] = send_email   # register the risky step with the orchestrator

journal = {}   # in production: a database table, not a dict

def run_once(run_id, step, fn, *args):
    key = f"{run_id}:{step}"
    if journal.get(key, {}).get("status") == "done":
        return journal[key]["result"]      # already done: reuse the result
    journal[key] = {"status": "started"}   # log intent BEFORE acting (the WAL idea)
    result = fn(*args)
    journal[key] = {"status": "done", "result": result}
    return result

This journal also saves money on model calls. Wrap the model call itself in run_once, and a crashed run reuses the saved answer instead of asking again. That is what durable execution engines like Temporal do for you: they record every model and tool result, and after a crash replay the record instead of asking the model again. Their example is a travel agent that would otherwise book a second, conflicting trip.

Two more habits from Anthropic's research system: resume from the last good state instead of restarting, and roll out new versions gradually so runs already in progress aren't broken by a deploy.

8. Keep two kinds of memory apart

Agents have two kinds of state, and mixing them causes most memory bugs:

• Workflow state: what's done, what's next, the run ID, approvals. Structured, saved in a database, never lost. Parts 6 and 7.
• Context: what the model sees on this call. Limited and expensive, so it must be curated.

Context needs care because models get worse as it fills. Anthropic calls this context rot and suggests three fixes for long tasks: compaction (summarise the history and start fresh), notes the agent writes to a file and reads back, and sub-agents that explore in their own clean context and return a short summary, often 1,000–2,000 tokens.

The Manus team shared lessons from production that are easy to copy:

• Keep the start of the prompt stable so the provider can cache it. Cached input cost them 10× less.
• Use files as memory. Keep a URL or path in context, not the whole page.
• Rewrite a to-do list as the agent works, so the goal stays at the end of the context where the model pays most attention.
• Leave errors in. A failed attempt in context stops the model repeating it.

Which framework should you use?

All of these can build the eight parts. Pick by where you already work.

Whatever you pick, read what it does underneath. Anthropic's warning is that frameworks can hide the prompts and responses you need to see when debugging.

A short checklist before you ship

• Can a fixed workflow do this? If yes, skip the agent.
• Does every run have an ID, and is state saved after every step?
• Is every irreversible action behind an approval and an idempotency key?
• Is the model's plan checked against an allowed list, with a length cap?
• Do retries have a limit, and does repeated failure go to a human?
• Is workflow state kept separate from what the model sees?
• Can you replay a failed run from its saved state and see every prompt?
• Have you tested on 20 real tasks? Anthropic started its evals with about 20 real queries, not hundreds.

What to do today

Take one agent or automation you already run. Write its steps as a list. Circle each step that can't be undone. Put an approval gate and an idempotency key on those, and save state after every step. That alone fixes most "it worked in the demo" problems.

For the basics underneath (prompts, tools, testing), read the AI Engineering Guide. Taipreneur builds AI automations with approval steps for small businesses; see AI automation.