Skip to content

Durable Execution

Hands-on

Full runnable recipe: durable-execution/ in the companion cookbook — a minimal event-log agent, run as two genuinely separate process invocations, with a real deliberate crash mid-tool-call and a real idempotent resume.

TL;DR — quick revision
  • Durable execution means a crash resumes the conversation instead of restarting it. Temporal's own framing: "When a Worker crashes, the Temporal Service hands the work to another Worker, which replays the Event History and resumes at the line where execution stopped, with local variables and progress intact." For an agent specifically: "The loop is a Workflow, each model call and tool call is an Activity, and a crash resumes the conversation instead of restarting it."
  • Replay and checkpointing are two different mechanisms that solve the same problem differently. Temporal-style engines replay a recorded event history — re-running workflow code but skipping already-completed steps using their recorded results. LangGraph-style checkpointing persists application state directly as a snapshot. Both need a persistent backend to survive a real process crash: LangGraph's own docs note MemorySaver and InMemorySaver "store checkpoints in RAM. When the process restarts, all checkpoints are lost."
  • A real, from-scratch repro of the exact failure mode worked cleanly. A crash was deliberately triggered right after a tool's side effect committed but before that fact was durably logged — the single most dangerous instant for a naive retry. Resuming in a genuinely separate process replayed the completed steps with zero new model calls, then completed the task with no duplicated side effect.
  • Idempotency is the second line of defense, not a redundant one. Replaying the event log tells you what's known to have completed — it can't tell you about the gap between "the side effect happened" and "the log says it happened." An idempotent tool, checking its own persisted state before acting, is what actually prevents a double charge or a duplicate booking in that gap.
  • This is a real, documented interview topic, not a hypothetical: "How do you make sure agents do not double-execute side-effectful operations like charging a card or booking a ticket twice?" and "Suppose your booking agent sometimes reserves the same hotel twice — walk through how you'd debug and fix this" are both real, sourced interview questions.

Why agents need this more than typical request/response services

A normal web request either succeeds or fails within a few hundred milliseconds, and a crash mid-request is usually safe to just retry from scratch. An agent loop is different in a way that makes crash recovery genuinely harder: it can run for minutes, hours, or (per real production harnesses) days, accumulating real side effects — a charge, a reservation, a sent email — at unpredictable points along the way. Restarting "from scratch" after a crash doesn't mean replaying an idempotent GET request; it means re-running a sequence that may have already charged a card once, and might charge it again.

Replay vs. checkpointing

Two different real mechanisms answer "how does the system know what already happened," and it's worth being precise about which is which, since the terms get used loosely. Temporal's model persists an event history — every significant step (starting a workflow, an activity being scheduled, an activity completing) is an event, and recovery means re-executing the workflow's own code from the top while replaying already-recorded events instead of re-running the corresponding activities: "replays the Event History and resumes at the line where execution stopped, with local variables and progress intact." LangGraph's model instead checkpoints application state directly — a snapshot of the graph's state at each step, persisted so a later run can "resume exactly where it left off." The practical catch, stated plainly in LangGraph's own docs: the default in-memory checkpointers ("MemorySaver and InMemorySaver") "store checkpoints in RAM. When the process restarts, all checkpoints are lost" — durability, in either model, requires a real persistent backend, not just the presence of a checkpointing API.

The code below is a minimal version of that event history: a plain log of which steps have completed, written to disk after each one, so a later run can tell "already done" apart from "still needs doing" without asking the model again.

import json
import os

LOG_FILE = "durable_agent_log.jsonl"


def append_event(event: dict) -> None:
    with open(LOG_FILE, "a") as f:
        f.write(json.dumps(event) + "\n")


def read_log() -> list:
    if not os.path.exists(LOG_FILE):
        return []
    with open(LOG_FILE) as f:
        return [json.loads(line) for line in f if line.strip()]

That log is the whole mechanism on the orchestrator's side — nothing fancier than reading and appending records. What it can't do is discussed next: it only knows what it was told, and a crash can happen before it gets told.

Idempotent tools: the second line of defense

An event log answers "what does the orchestrator know completed" — but there's a real gap it can't close on its own: the instant between a tool's side effect actually committing and that fact being durably recorded. A crash in that exact window means the orchestrator will, on resume, believe the step never happened and ask for it again. The fix isn't a smarter log — it's a tool that can tell the difference between "do this" and "you already did this," checked against its own persisted state, independent of whatever the orchestrator believes.

The code below shows that check in practice: before a tool performs its real side effect, it looks itself up by a stable key (an order ID, not a fresh ID generated per attempt) and returns the prior result if it already ran.

LEDGER_DIR = "ledgers"  # each tool's own durable, idempotency-checked state


def _ledger_path(name: str) -> str:
    os.makedirs(LEDGER_DIR, exist_ok=True)
    return os.path.join(LEDGER_DIR, f"{name}.json")


def _read_ledger(name: str) -> dict:
    path = _ledger_path(name)
    if not os.path.exists(path):
        return {}
    with open(path) as f:
        return json.load(f)


def _write_ledger(name: str, ledger: dict) -> None:
    with open(_ledger_path(name), "w") as f:
        json.dump(ledger, f)


def charge_card(order_id: str, amount: float) -> dict:
    ledger = _read_ledger("charges")
    if order_id in ledger:
        return {"status": "already charged (idempotent replay)", "charge_id": ledger[order_id]}
    charge_id = f"ch_{order_id}"
    ledger[order_id] = charge_id
    _write_ledger("charges", ledger)
    return {"status": "charged", "charge_id": charge_id, "amount": amount}


def reserve_item(order_id: str, sku: str) -> dict:
    ledger = _read_ledger("reservations")
    if order_id in ledger:
        return {"status": "already reserved (idempotent replay)", "reservation_id": ledger[order_id]}
    reservation_id = f"rsv_{order_id}"
    ledger[order_id] = reservation_id
    _write_ledger("reservations", ledger)
    return {"status": "reserved", "reservation_id": reservation_id, "sku": sku}


def send_confirmation(order_id: str) -> dict:
    ledger = _read_ledger("confirmations")
    if order_id in ledger:
        return {"status": "already sent (idempotent replay)", "confirmation_id": ledger[order_id]}
    confirmation_id = f"conf_{order_id}"
    ledger[order_id] = confirmation_id
    _write_ledger("confirmations", ledger)
    return {"status": "sent", "confirmation_id": confirmation_id}

That self-check is what makes a tool safe to call twice — it doesn't rely on the orchestrator getting the "already done" bookkeeping right, because it keeps its own record independently.

A real crash, a real resume, no duplicate side effect

The code below is the agent loop itself, wiring the event log and the idempotent tools together: on each turn it checks the log before asking the model for a decision, and appends to the log the moment a tool call returns.

_IMPLS = {"charge_card": charge_card, "reserve_item": reserve_item, "send_confirmation": send_confirmation}

_TASK = (
    "Process order ORD-8842: charge the card $49.99, reserve item SKU-772, then send an order "
    "confirmation. Do exactly one tool call per turn, in that order, then report done once all "
    "three steps are complete."
)

# Deliberately crash right after this tool's side effect commits, but
# before the fact that it happened gets durably logged -- the exact
# instant a real worker crash is most dangerous for a naive retry.
_CRASH_AFTER_TOOL = "send_confirmation"


def run(client, model, tools, max_steps=6) -> None:
    log = read_log()
    messages = [{"role": "user", "content": _TASK}]
    replayed_steps = 0

    # Reconstruct in-memory conversation state purely from the durable log
    # -- no model call, no tool re-execution, for every step already
    # completed in a prior run.
    for entry in log:
        messages.append({"role": "assistant", "content": entry["assistant_content"]})
        messages.append({"role": "user", "content": entry["tool_result_content"]})
        replayed_steps += 1

    if replayed_steps:
        print(f"[resume] replayed {replayed_steps} completed step(s) -- no new model calls for these.")

    for step in range(replayed_steps, max_steps):
        response = client.messages.create(model=model, max_tokens=400, tools=tools, messages=messages)
        messages.append({"role": "assistant", "content": response.content})

        if response.stop_reason != "tool_use":
            final_text = "".join(b.text for b in response.content if b.type == "text")
            print(f"[done] {final_text}")
            return

        block = next(b for b in response.content if b.type == "tool_use")
        result = _IMPLS[block.name](**block.input)

        # Only crash on the FIRST genuine execution of this step, not on a
        # later idempotent replay -- a real crash happens once, and resume
        # should actually reach completion the second time around.
        is_fresh_execution = not result["status"].startswith("already")
        if block.name == _CRASH_AFTER_TOOL and is_fresh_execution:
            os._exit(1)  # hard exit -- nothing below this line ever runs; the log entry is never written

        tool_result_content = [{"type": "tool_result", "tool_use_id": block.id, "content": json.dumps(result)}]
        messages.append({"role": "user", "content": tool_result_content})
        append_event({"assistant_content": _serialize_content(response.content), "tool_result_content": tool_result_content})


def _serialize_content(content_blocks) -> list:
    """Anthropic content blocks aren't directly JSON-serializable -- convert
    to the plain dicts the API itself accepts back in a later `messages` list."""
    serialized = []
    for block in content_blocks:
        if block.type == "text":
            serialized.append({"type": "text", "text": block.text})
        elif block.type == "tool_use":
            serialized.append({"type": "tool_use", "id": block.id, "name": block.name, "input": block.input})
    return serialized

A real run — two genuinely separate process invocations, not two calls in one script

Task, identical across both invocations: "Process order ORD-8842: charge the card $49.99, reserve item SKU-772, then send an order confirmation. Do exactly one tool call per turn, in that order, then report done once all three steps are complete."

Invocation 1 — real model decisions, real tool executions, then a deliberate crash: 

[step 1] real model call decided: charge_card({'order_id': 'ORD-8842', 'amount': 49.99})
[step 1] tool executed: {'status': 'charged', 'charge_id': 'ch_ORD-8842', 'amount': 49.99}
[step 2] real model call decided: reserve_item({'order_id': 'ORD-8842', 'sku': 'SKU-772'})
[step 2] tool executed: {'status': 'reserved', 'reservation_id': 'rsv_ORD-8842', 'sku': 'SKU-772'}
[step 3] real model call decided: send_confirmation({'order_id': 'ORD-8842'})
[step 3] tool executed: {'status': 'sent', 'confirmation_id': 'conf_ORD-8842'}
[crash] simulating a worker crash right after send_confirmation's side effect committed, before this step's completion is durably logged.
At the moment of the crash: the confirmation genuinely was sent — its own ledger has the entry — but the event log holds only 2 completed steps, not 3. The process is dead. Nothing about step 3 exists anywhere the orchestrator can see.

Invocation 2 — a completely separate process, started fresh, reading only what's on disk: 

[resume] replayed 2 completed step(s) -- no new model calls for these.
[step 3] real model call decided: send_confirmation({'order_id': 'ORD-8842'})
[step 3] tool executed: {'status': 'already sent (idempotent replay)', 'confirmation_id': 'conf_ORD-8842'}
[done] Done! All three steps have been completed successfully for order ORD-8842:
1. ✓ Card charged $49.99 (charge_id: ch_ORD-8842)
2. ✓ Item SKU-772 reserved (reservation_id: rsv_ORD-8842)
3. ✓ Order confirmation sent (confirmation_id: conf_ORD-8842)
Steps 1 and 2 cost zero new model calls — reconstructed purely from the durable log, exactly matching Temporal's own framing of what replay buys you. Step 3 needed a real new model call, since it was never durably marked complete, and the model correctly (from its own point of view) asked to send the confirmation again — but the tool's own ledger check caught it: one confirmation_id, sent exactly once, across a crash and a full resume. Total real API calls for the entire crash-and-resume story: 4.

Interview angle

Weak answer to "how do you make sure an agent doesn't charge a customer's card twice if it crashes mid-task": "Just log everything and replay the log on restart." This names half the mechanism and skips the actual failure mode this page's own real run demonstrates: a crash can happen in the gap between a side effect committing and that fact being logged, and no amount of replaying the log helps with a step the log never recorded as attempted.

Strong answer: durability needs two separate mechanisms working together, not one. A durable, replayable event log (or persisted checkpoint) answers "what does the system already know completed," so a resumed run doesn't waste calls or make new decisions for steps that are genuinely finished — this page's own resumed run needed zero new model calls for its first two steps. Idempotent tools, keyed on something stable like an order ID or an explicit idempotency key, answer the harder question the log can't: what happens when the orchestrator asks for a step again that already silently succeeded. The real production discipline is designing every side-effecting tool call to be safe to receive twice, not just designing the orchestrator to avoid asking twice.

Follow-up to expect: "doesn't the crash simulation in your demo feel artificial — how often does a crash really land in that exact instant?" It doesn't need to land there often to matter: a long-running agent making many tool calls over hours has many such instants, and the cost of getting it wrong (a real double charge, a real duplicate booking) is asymmetric — one occurrence in production is a real incident, however rare the timing. That asymmetry is exactly why production durable-execution engines (Temporal, DBOS, Restate) exist as dedicated infrastructure rather than something teams build ad hoc per agent, and why the real interview questions on this topic ("how do you make sure agents do not double-execute side-effectful operations") are framed around debugging exactly this class of rare-but-costly failure.

Build it yourself — 30 minutes

  1. Pick a multi-step task with at least one side-effecting action per step, and design each step's tool to check its own persisted state (a local file, a dict, a real database in production) before acting — keyed by something stable across retries, not a fresh UUID generated each attempt.
  2. Build the durable event log: append a record after every completed step, and write a replay function that reconstructs conversation state from the log before making any new model call.
  3. Deliberately crash the process (a hard os._exit, not a caught exception) right after a tool executes but before you log that it did — the specific instant this page's own repro targets.
  4. Run the exact same command again as a genuinely new process invocation, not a retry loop inside the same script. Verify two things separately: that already-completed steps needed zero new model calls, and that the step that was mid-flight during the crash didn't produce a second real side effect.

Scenario Check

Sources & further reading

  • Temporal, "Why Temporal" — the Event History replay model and the agent-specific framing (loop as Workflow, calls as Activities).
  • LangChain, "Durable execution" — checkpointing, and the explicit caveat that in-memory checkpointers don't survive a process restart.
  • DBOS — "Database-Backed Durable Python Workflows", an alternative durable-execution approach backed directly by a database rather than a separate orchestration service.
  • Restate — restatedev/sdk-python, a third real durable-execution SDK in the same space.