An onboarding agent reads a customer's documents, drafts a case, gets a manager's approval, and creates the case in a CRM. The CRM accepts the request. Before the agent saves the response, its worker crashes.
The worker restarts. The conversation is intact. The approval is visible. There is no saved case ID.
Should it create the case again?
That question is where durable AI agents become an engineering problem. Keeping a transcript helps, but recovery also needs evidence of what the system attempted, what the outside world accepted, and which next action is safe.
This guide develops that design through a hypothetical onboarding workflow. It includes a failure drill and a short Pi Durable case study; the example is architectural guidance, not a report of a deployed customer system.
Temporal's Activity documentation recommends idempotent operations so retries avoid duplicate effects. An agent that changes external records needs to address that same boundary.
Key takeaways
- Save accepted decisions and completed results so recovery can reuse them.
- Give each external action a stable identity and a defined way to resolve an uncertain outcome.
- Test recovery after the receiving system accepts a request, before the agent saves its receipt.
What is a durable AI agent?
A durable AI agent preserves enough execution state outside its worker process to continue useful work after an interruption. That includes completed decisions, pending work, tool outcomes, and any conditions that must be satisfied before the next action.
“Durable” needs a stated failure boundary. Surviving a worker restart, losing a machine, and restoring an entire region are different promises. The runtime, storage configuration, and hosting arrangement determine which promise the application can keep.
For a workflow owner, a useful recovery contract answers four questions:
- What is already established? Accepted inputs, saved decisions, completed results, and approvals.
- What remains pending? A model request, tool attempt, timer, review, or external response.
- What is uncertain? An operation may have succeeded without leaving a local receipt.
- Who can move it forward? The scheduler, an authorised reviewer, or an operator resolving an exception.
That contract gives the agent something concrete to resume. Our production agent workflow guide covers the wider system around an agent; this article focuses on the recovery boundary.
Conversation memory and execution state serve different jobs
A summary saying “the manager approved onboarding” leaves several operational questions unanswered. Which version was approved? Was the CRM request sent? Did the provider return a result? Has anything changed since approval?
For this example, keep these records distinct:
| Record | What it establishes | Example |
|---|---|---|
| Conversation | What the user and agent discussed | The customer needs an onboarding case |
| Working context | Information useful for reasoning | Customer preferences and document summaries |
| Execution state | Where the workflow can continue | Draft version 3 is waiting for publication |
| Operation record | What an external action attempted | Create case using operation op-27 |
| Receipt | Evidence returned by the receiving system | CRM case case-814 was created |
| Approval | Who authorised which proposed action | Manager approved the digest of draft version 3 |
This distinction also appears in LangGraph's persistence model: checkpointers retain state for a thread, while stores retain information across threads. Both can be useful, but neither name establishes that a particular external write is safe to repeat.
Design the recovery view so an operator can inspect these records without asking the model to reconstruct events from prose. The model can explain the history; recorded facts should determine the next permitted transition.
Put recovery boundaries around decisions and actions
Consider the onboarding workflow as six separate steps:
- Read source documents.
- Save extracted facts and source versions.
- Save the proposed case.
- Wait for approval of that proposal.
- Attempt the CRM operation.
- Save its receipt and finish.
Each boundary identifies work that can be accepted independently. If extraction and publication share one large retryable function, a failure during publication can cause the extraction and drafting to run again. The new draft might differ from the approved one.
Save the accepted model output, its relevant inputs, and the version of the prompt or policy that produced it. Recovery should use that decision unless the workflow explicitly enters a revision step. A new model call is new work, even when its prompt looks identical.
Temporal's workflow documentation illustrates one implementation: workflow code reconstructs state from recorded history, while completed Activity results are reused during replay. External interactions, including model calls, belong in Activities rather than directly in replayed orchestration code. An unfinished Activity attempt may still need another attempt.
The practical design question is where to place those units. Separate model reasoning from the external write, and separate each external write that needs its own recovery policy. Our guide to agentic loops explains how reasoning, tool use, and verification fit together within a run.
Treat a lost receipt as an unknown outcome
A successful external action and a missing local receipt can exist at the same time. This illustration shows the conceptual failure window.
In the opening example, a timeout or crash does not prove that the CRM rejected the request. Record that uncertainty explicitly. A useful set of application states is prepared, in_flight, succeeded, failed, and unknown.
Here is an illustrative operation record, not a framework API:
{
"operationId": "op-27",
"workflowId": "onboarding-184",
"action": "create_onboarding_case",
"proposalVersion": 3,
"payloadDigest": "sha256:<digest-of-approved-payload>",
"idempotencyKey": "onboarding-184:create-case:v3",
"status": "unknown",
"providerReceipt": null
}
Create the operation identity and persist its intended payload before making the external request. Reuse that identity when retrying the same logical operation. A newly generated key on every attempt makes each retry look like a new action to the receiver.
There are three useful recovery paths:
- The receiver supports idempotency. Repeat the same operation with the same key and payload within the receiver's supported guarantees.
- The receiver exposes authoritative status. Look up the operation by a unique reference, validate the result, and save its receipt. An empty search result is insufficient if the original request can still finish or the search is eventually consistent.
- Neither guarantee exists. Keep the action unresolved and route it to an operator. Define a manual reconciliation procedure before enabling automatic writes.
Stripe's idempotent request documentation is a concrete example of a receiver contract: API v1 stores a response for a key and compares retry parameters. Its keys can be pruned after at least 24 hours, so deduplication has a lifetime. Check the actual provider's rules before relying on the pattern.
An operation table is evidence of intent. It cannot, by itself, make a remote API call and a local database commit atomic. Temporal's idempotency discussion also describes the race in checking whether a record exists before creating it. Enforce uniqueness at the receiving boundary where possible, and account for overlapping attempts.
Make human approval a durable, bounded decision
Suppose the manager approves draft version 3, then the worker stops overnight. On restart, the agent discovers a new document and rewrites the draft. The earlier approval should not silently authorise version 4.
For this workflow, save the approval against the proposal version and a digest of the action payload. Include the reviewer identity, decision time, expiry policy, and the scope of the permitted action. Authenticate the reviewer and check their authority in application code.
Before executing, verify that the pending action still matches the approved proposal and that current permissions permit it. If the payload changes or approval expires, create a new review request. Duplicate approval callbacks should resolve the same pending decision rather than starting a second workflow.
The waiting state must survive a process restart too. LangGraph interrupts provide a mechanism for pausing and resuming with external input. Its documentation also explains that a resumed node starts again from its beginning, so code before an interrupt needs appropriate handling for repeated execution.
Treat “waiting for a person” as a stored state with an owner and a deadline. A browser tab left open is a poor place to keep that responsibility.
Match the storage promise to the failure you expect
Saving a checkpoint is meaningful only if the storage survives the failure in question. Before choosing a backend, write down the required behaviour:
| Failure or operating change | Design question |
|---|---|
| Worker process exits | Can another invocation read the last committed state? |
| Machine or container disappears | Is the state stored on a retained volume or external service? |
| Two workers overlap | What grants ownership, and what prevents an old worker from writing? |
| Application is redeployed | Can new code interpret pending states created by the previous version? |
| Storage is restored from backup | Could an external action have completed after the restored checkpoint? |
| History grows for months | Which evidence must remain, and what can be archived? |
A database backup restores a point in your history. It does not reverse actions already accepted by other services. After restoration, reconcile relevant operations before enabling automatic writes again.
Keep large source files and generated artefacts in suitable object storage, with versioned references in execution state. Define retention and access controls for prompts, documents, tool responses, and approval records. Recovery history can contain the same sensitive material as the source systems.
Pi Durable is one concrete implementation of the pattern
Pi Durable's README, inspected at this revision, describes an experimental agent harness that persists conversations, model turns, tool calls, and application state. Reopening persistent storage can resume unfinished work.
Its tool contract makes the recovery boundary explicit: intent is committed before execution; an interrupted tool reruns only if declared replay: "safe". Otherwise it produces an interrupted result. That declaration still leaves external idempotency to the tool implementation.
The same revision documents memory, SQLite, JSONL, and Cloudflare Durable Object storage. Memory does not persist. The Node SQLite adapter uses WAL with synchronous = NORMAL: process crashes and host or power failures have different durability guarantees. Storage has one owning process and no cross-process locking.
For a Pi-based application, those are useful constraints to evaluate against the table above. Pin a version and validate its recovery behaviour; the README explicitly marks the API as experimental.
Choose a runtime by the recovery work it must own
The first choice is the unit of work you need to preserve. A conversation, an agent graph, and a business process spanning several services have different shapes.
| Starting point | What to examine | Question for your prototype |
|---|---|---|
| An embedded agent harness, such as Pi Durable | Persisted turns, tool replay policy, host and storage ownership | Can this host recover the session and safely resolve interrupted tools? |
| An agent graph, such as LangGraph | Checkpoints, thread identity, interrupts, and restart boundaries | Which node code runs again, and what happens to its external writes? |
| A workflow engine, such as Temporal | Recorded history, Activities, retries, and deployment compatibility | Which decisions belong in orchestration, and which operations need independent retry contracts? |
These are architectural starting points, not interchangeable capability claims. Use the linked documentation to verify the specific version and hosting arrangement you plan to run.
A short read-only task may need only a saved result and a job queue. A workflow that waits for approval overnight, changes external records, and must survive deployments needs a more explicit recovery design. Choose against those requirements before adding infrastructure.
Run a failure drill before trusting automatic recovery
Use a sandbox receiver and a worker you can terminate. Observe the receiver's records as well as the agent history. A successful final response alone cannot reveal duplicate actions.
The following is a proposed acceptance plan for the onboarding example, not measured test results:
| Stop or disturb the workflow here | Expected recovery behaviour |
|---|---|
| After the draft is saved | Restore the same proposal version without silently generating a replacement |
| While approval is pending | Restore the pending review; accept one authorised decision |
| After approval, before the request | Verify the approved payload and current permissions before execution |
| After the receiver accepts, before saving the receipt | Reconcile or retry under a verified idempotency contract; preserve uncertainty otherwise |
| After the receipt is committed | Reuse the saved result without another create operation |
| While two delivery attempts overlap | Deduplicate at the receiving boundary and retain the evidence from both attempts |
| After deploying changed workflow code | Resume through a supported version path or park the run with an actionable reason |
For every drill, record the operation ID, proposal version, attempts, receiver outcome, and next allowed action. Verify the business invariant directly: for this workflow, one approved logical operation must produce one onboarding case.
Also test a permanently failing step. Set limits on attempts, elapsed time, and model spend. When a limit is reached, retain the evidence, stop further automatic action, and give a named operator a clear recovery task.
Give operators a recovery view they can act on
A useful run page should make the next decision legible: “waiting for approval of draft 3,” “CRM outcome unknown; reconcile operation op-27,” or “completed with case case-814.” Include the evidence needed to establish that state.
Track unresolved outcomes, pending review age, recovery duration, repeated attempts, and runs parked for intervention. Define exactly what each metric counts. An attempt count alone cannot tell you whether any useful work completed.
If a model or retrieval error produced the wrong proposal, durable execution may preserve it perfectly. Keep recovery checks alongside evaluation and monitoring, and evaluate the decisions inside the agent loop as well as the machinery that resumes it.
For your next agent workflow, choose one external action and write its recovery contract before increasing autonomy. Specify the saved decision, operation identity, approval scope, receiver guarantee, and handling of an unknown result. Then terminate the worker at the receipt boundary and inspect what happens.
If you need help turning that contract into an operating workflow, our AI agent development service covers integrations, evaluation, review, and recovery design.
