Wire a checkpointer into a LangGraph agent and a lot of hard problems go away. You pass a thread_id in the config, and after each super-step the graph's state gets written somewhere durable, MemorySaver while you're developing and PostgresSaver once it's real. The thread can be stopped and picked up later. That's what human-in-the-loop is built on, and time-travel debugging, and an agent surviving a pod restart. Resume a thread and the completed nodes are skipped, the pending one runs again, and the agent carries on from where it stopped.
What a checkpointer does is useful, but there's a gap in what it actually protects you from.
The window
The checkpoint is written after the node returns. So for any node that does something irreversible, there's a stretch of time where the thing has happened and nothing durable knows it.
Take a two-node booking agent. Node one reserves a seat in your own inventory, node two confirms that booking with the supplier. The pod gets rescheduled in the middle.
run 1
│
├─ node "reserve" ───► seat held in your inventory, checkpoint written
│
├─ node "confirm"
│ │
│ └─ confirm_booking() ───► supplier API seat is booked
│
╳ process dies here
│
└─ no checkpoint for "confirm" was ever written
run 2 resumes the same thread_id from cold
│
├─ node "reserve" ───► skipped. its checkpoint exists.
│
└─ node "confirm" ───► runs again, because nothing says it ran
│
└─ confirm_booking() ───► supplier API booked a second time
Re-running node two is the only correct thing the checkpointer could do with what it knows. As far as the durable record goes, that node never happened. But it did happen. The confirmation left the process through a socket and the write that would have recorded it never ran. The supplier now holds two bookings for one passenger, and there's no header you can send that would have stopped it.
A checkpointer records what your agent decided. It can't record what your agent did to the outside world, because it never saw it happen. Every checkpointer in every agent framework has this window. None of them sit in the call path of your HTTP client.
Worth saying plainly: your own inventory is fine. Node one committed, so the resumed run skipped it and there's exactly one seat held. Nothing was lost or corrupted. If you read this as a bug report about LangGraph's checkpointer you'll reach for the wrong fix.
It isn't only restarts
The pod restart is the easy one to picture. It isn't the only shape.
Two workers arrive at the same job. A thread_id is a storage key. It doesn't claim anything. Nothing stops two processes invoking the same thread at once, and that happens more than people expect: a retry that fires while the first attempt is still running, a queue that redelivers, two replicas picking up the same message. Five identical invocations of one thread at once and the supplier gets five bookings.
An advisory lock doesn't fix this, which surprises people. Mutual exclusion and idempotency get treated as the same property. A lock means one worker runs the node at a time. It says nothing about how many times the booking happens across all of them, and the measured answer there is also five.
The worker holding the job dies and doesn't come back. Not a crash loop, where the pod restarts and retries. The process is gone: the node was killed, the container was evicted, the host went away. Whatever marked that work as in-progress is now held by something that no longer exists.
Several agents reach the same conclusion independently. Five agents working one support ticket, each deciding a refund is owed. They aren't retrying each other, they reasoned separately, and the customer is owed one refund. Nothing in the thread model connects them, because they're five different threads.
What people already do about it
If the API provider supports an idempotency key, use that. Stripe has one. If it doesn't and it's an internal API you own, add one. The receiver dedupes on its own side, which beats anything the caller can do about it.
It buys less than it sounds, because nobody hands you the key. Stripe dedupes a retry only if the retry presents the same key, and your agent has to produce that key twice across a process that died in between. Generate it inside the node and the resumed run generates a different one. Derive it from the model's arguments and two agents that reasoned differently derive two keys, which is a measured result rather than a worry: in a harness where two agents each decided a refund was owed and disagreed on the amount, putting the amount in the key let both of them through. Stripe's keys also expire, at least 24 hours in v1, which covers a retry loop and not a thread somebody resumes on Tuesday.
The key has to come from somewhere durable, be the same across replicas, and outlive the process. That's the actual work, and you do it whether the receiver dedupes or you do.
What's left is the calls with no such header, which for an agent is most of them. A partner endpoint, a Slack message, an internal service another team owns, a tool someone wrote last quarter.
For those the usual answer is a claims table, the Stripe-style idempotency key written by hand. Insert a row keyed on the operation before you do the work. A second caller's insert conflicts, so it knows someone else owns the operation and waits for their answer, and you store the response against the key when you're done. It's a good answer, it's what teams actually write, and it handles both the restart and the concurrent workers.
It's the dead holder that breaks it. The row says in-progress and it's held by a process that no longer exists, so nobody else can finish the operation without a human clearing it. The usual next move is a TTL on the claim, and the instinct is right, but a TTL whose expiry is the only thing authorising takeover can't tell a dead holder from a slow one. It takes over from both. From the dead one that's correct. From the paused one you get two workers who each believe they own the operation.
Making it safe takes a heartbeat, so the holder can say it's still alive, and a fencing token, so a superseded holder's late write gets refused instead of applied. At which point you've written a lease, and you own it.
This is also where the workflow engines sit, and it's worth being clear that they don't remove the problem. Temporal makes the orchestration durable: the workflow survives, the step re-runs on replay, and you get timers and retries and history for free. Its own documentation says activities execute at least once and that you should make them idempotent, because an activity can run more than once and can partially complete more than once. Run Temporal and you still write something like the claims table inside your activity, since an activity retry is exactly the thing that books the seat twice.
Neither approach solves the case where multiple agents independently decide to perform the same operation. Five agents on five threads derive five different keys, so every guard sees a different operation and lets it through. Catching it means deriving the key from the ticket rather than from the thread, which is a decision you have to make on purpose whatever you're using.
Which is roughly what I've been building
Cellaflow is an engine that holds those leases, and I should say up front that it's mine, so weigh the rest accordingly. A tool call takes a lease before it runs. If a later call derives the same key, it's answered from the record of the first attempt instead of executing. If the holder dies, the lease expires and the work gets reclaimed by whoever takes it, which is the case a claims table can't reach.
For LangGraph the integration is two things: a decorator on the tool you can't take back, and a context manager around invoke.
from cellaflow import durable_tools
from cellaflow import tool as leased
The tool that hands the parcel to the courier gets leased:
@langchain_tool
@leased(tool_name="dispatch_order")
def dispatch_order(order_id: str, courier: str) -> str:
"""Hands `order_id` to `courier` for delivery. Cannot be recalled."""
DISPATCHES.append(order_id)
return f"dispatched {order_id} with {courier}"
Read it bottom-up. Cellaflow leases the call, then LangChain builds the model's tool schema from the signature and the docstring. The decorator wraps with functools.wraps, so the signature, annotations and docstring survive it and the schema LangChain generates is the one you wrote.
The agent's other tool isn't leased, because repeating a read costs nothing and a lease on it would buy nothing:
@langchain_tool
def lookup_order(order_id: str) -> str:
"""Returns the customer, item count, and status of an order."""
Then the run:
config = {"configurable": {"thread_id": f"dispatch-{uuid.uuid4().hex[:8]}"}}
with durable_tools(config, target=ENGINE):
result = app.invoke(
{"messages": [("user", f"Dispatch order {ORDER_ID} to the courier.")]},
config,
)
durable_tools reads configurable.thread_id out of the config you're already passing to invoke and derives a tool session from it, so the same thread always reaches the same leases. That's what survives the restart. The resumed run derives the same key, and the leased call is answered from the record of the first attempt instead of executing again.
LangGraph needs nothing further because it copies the calling context into the executor it runs a node on, so a tool invoked from the tools node reaches the session opened around invoke. Other frameworks vary. AutoGen doesn't propagate it at all, and there the session has to be bound inside the tool.
Does it hold up
There's a runnable version of that two-node agent in the repo. Hold a seat, confirm the booking with the supplier, kill the process between the confirmation and the checkpoint. Two arms, the same graph, the same PostgresSaver, the same crash at the same instant, and the only variable is whether the tool call holds a lease.
| label | seats held | confirmations |
|---|
PostgresSaver alone | 1 | 2 |
| the same graph, leased tool call | 1 | 1 |
The left column is the control, and it's what makes the right one mean anything. One seat held in both arms, because node one committed before the crash and the resumed run skipped it. The lease isn't fixing the checkpointer. It's covering the node that was still mid-call when the process died.
docker run -d --name cellaflow -p 50051:50051 \
-e CELLAFLOW_ALLOW_UNAUTHENTICATED=true ghcr.io/cellaflow/cellaflow:latest
git clone https://github.com/cellaflow/cellaflow-sdks
cd cellaflow-sdks/examples/at_most_once_proof
docker compose up -d
pip install -r requirements.txt
python proof.py
run_arm() takes the two tool functions and a context manager; one arm passes something that does nothing, the other passes durable_tools. The graph, the crash, the resume and the checkpointer are shared, so there's nowhere for a thumb to rest on the scale. Both arms assert their own number and the script exits non-zero if either stops behaving. A proof that cannot fail is not a proof.
The counters live inside the tool bodies, so they count executions and never cache hits. If the lease works the body is never entered and nothing is appended, which means a suppressed call can't be counted as a success.
Some things it doesn't measure. There's no throughput or latency in it and no comparison of storage engines; it's a correctness demonstration, and a deterministic one, in that the confirmation either repeated or it didn't on any hardware. It runs one writer, one thread, one crash, so the concurrent scenarios above come from a separate harness. It kills a process rather than cutting power to the host. The engine is available in RocksDB and Postgresql variants.
One thing I'd like to know
If you've built the claims-table version, and most people with this problem have, I'd like to hear how it went. Specifically whether a dead holder ever cost you anything, or whether it just never came up.
I'll be in the comments answering any questions or thoughts. My DMs are also open.
Live demo: https://cellaflow.com/#live-demo
Github: https://github.com/cellaflow/cellaflow-sdks