r/LangChain • • 46m ago

Projects I got tired of paying $0.04/paper for LlamaParse, so I built a deterministic $0.005/paper extractor for ArXiv with citations & RAG chunks

1 Upvotes

Hey everyone,

Like many of you building RAG pipelines or local agent research monitors, my biggest headache has always been academic PDFs.

Tools like LlamaParse and Unstructured do great work, but they charge ~$0.003/page. When you're indexing hundreds of 15–30 page papers, that quickly runs into $50–$100 bills for simple batch jobs—and vision-based models often hallucinate equation numbers or fail on two-column layouts.

So I wrote a deterministic, sub-3-second extractor in Python combining PyMuPDF (`fitz` in C) with the official ArXiv Atom API, and packaged it as a Pay-Per-Event (PPE) Actor on Apify.

### What it extracts cleanly:

  1. **Authoritative Metadata**: Title, authors, affiliations, date, canonical ArXiv ID, and DOI.
  2. **Decomposed Sections**: Headings (`1. Introduction`, `3.2 Methodology`) with automatic hyphenation de-wrapping.
  3. **Analytics**: Exact count of display equation blocks and table captions with context.
  4. **Structured Citations**: Parses references into `{raw_text, title_guess, year, arxiv_id_if_present}` so you can build paper-citation graphs.
  5. **RAG-Ready Chunks**: Overlapping, token-bounded chunks pre-tagged with section breadcrumbs (`[Section: Methodology]\n...`), ready to drop straight into Qdrant/Pinecone/Chroma.

### Cost & Speed Benchmark (15-page Paper: "Attention Is All You Need"):

- **LlamaParse**: ~$0.045 / paper (12–25 seconds)

- **This Actor**: $0.005 / paper (3.8 seconds)

- **Failure Guarantee**: If an invalid PDF or 404 occurs, Apify charges $0.00.

Code is open on GitHub: https://github.com/ali1556/arxiv-pdf-structured-extractor

Apify Store Actor: https://apify.com/ali1556/arxiv-pdf-structured-extractor

Would love feedback from anyone running paper RAG pipelines—what fields or chunk formats are you missing most?


r/LangChain • • 54m ago

Discussion Langchain vs llamaindex in rag

Thumbnail reddit.com
1 Upvotes

r/LangChain • • 2h ago

Discussion Lang chain/langraph vs llamaindex vs haystack

Thumbnail reddit.com
0 Upvotes

r/LangChain • • 2h ago

Question | Help Lang chain/langraph vs llamaindex vs haystack

Thumbnail reddit.com
1 Upvotes

r/LangChain • • 3h ago

Discussion Tool-calling agent built on LangChain scores rank 8/53 on ExtractBench - architecture breakdown

1 Upvotes

Sharing because this was built with LangChain's tool calling and I think the patterns might be useful to others here.

KYNEX is a document extraction agent. You give it a PDF + JSON schema, it fills the schema by navigating the document with tools. No RAG retrieval step, the agent decides what to read.

LangChain-specific bits:

- Tools defined with u/tool decorator, bound via model.bind_tools()

- Agent loop runs up to 30 rounds with AIMessage/ToolMessage cycling

- SystemMessage includes a Table of Contents built from parsed headings + table page ranges

- Works with ChatOpenAI and AzureChatOpenAI (swappable via config)

The agent has a fallback escalation chain for tables: extract_table → reconstruct_table → Azure DI → vision. Each step is a separate tool. If one fails, the agent tries the next. This was key for handling the 67 different document types in the benchmark.

Results on ExtractBench (370 docs, verified by LlamaIndex):

- 94.6 F1 (rank 8/53)

- GPT-5.4 through this agent scored 92.97, beating GPT-6 Astra standalone (91.9)

Open source under Apache 2.0. Curious if anyone else has built multi-tool agents with this kind of escalation pattern, found it way more effective than single-tool approaches.


r/LangChain • • 23h ago

Discussion Does a second model catch citation errors the first model misses

2 Upvotes

When an assistant cites the wrong policy and the citation looks clean no reader spots the error until the policy gets enforced. For instance one builder in a LangChain thread today said the same model checking its own work found nothing real in his experience. However a second call from the same model may share the same blind spot because it read the same chunks and learned the same shortcut. That is why I want to know whether a different model is the only honest check or whether a cheaper test catches the same errors. If you can tell me what your citation check looks like in practice I would learn more from that than from another eval number.


r/LangChain • • 1d ago

Question | Help Which AI tool gateway should I use so my agent can call multiple external APIs without seperate keys?

2 Upvotes

Building a small research agent that needs domain info, recent social post, and company lookups, and wiring up separate auth wrappers for every provider is getting old fast. I keep seeing people mention tool gateways like monid or running a custom MCP proxy in front of everything. For anyone actually running agents that hit multiple external APIs which gateway or router has held up best for you? Mostly worried about reliability and the model picking weird endpoints.


r/LangChain • • 1d ago

Question | Help langchain check for citations supporting claims in an association assistant

2 Upvotes

In a langchain workflow the retriever can return the right handbook page and the final answer can still stretch beyond what that page says. I'm looking at this for a customgpt.ai project. Say the source provides a deadline and a separate answer guesses the late fee. Both claims might arrive next to the same citation

The check I have in mind would keep the original passages with the response and ask a verifier for the specific span that backs each claim. If it only finds support for the deadline, the assistant can give the deadline and refer the fee question to staff. I'd want to test what the verifier does when two passages look related and neither contains the number

Where would you put that check in a langchain flow? If you have an example that was easier to debug, share what the checker returned to the next step


r/LangChain • • 1d ago

Question | Help How should apps control what AI agents can access?

2 Upvotes

AI agents are starting to interact with more than just APIs. They can work with desktop apps, local files, browser sessions, and even information displayed on screen.

That creates a tricky problem for developers.

A user might be happy to let an agent summarize a document, but definitely not want it reading private messages or accessing unrelated files.

And a simple allow/deny permission doesn't always feel like enough.

I'm curious how other developers are thinking about this:

  • Do you prefer permissions granted per app, per action, or per task?
  • Should an agent have to ask again when a task requires access to something new?
  • How do you handle permissions that should expire after a task is finished?
  • Is screen access fundamentally different from API access?

I'd rather give an agent fewer permissions and have it ask when needed than give it broad access and hope for the best.

How are you handling this in your own projects?


r/LangChain • • 2d ago

Announcement LangSmith Tool Gateway: Secure, governed tool use for agents

5 Upvotes

Hey r/langchain! I’m on the LangChain team, and I’m excited to announce that today, we’re launching LangSmith Tool Gateway in Public Beta, on the LangSmith Agent Platform. 

Tool Gateway is a managed tool layer that handles credentials, identity, and access for every agent tool call, making it simple to govern the actions your agents take.

Tool Gateway makes it easier and faster for engineering teams to grant agents access to new tools in a controlled manner and without hand-rolling auth each time. Security and compliance teams can use LangSmith to trace and audit tool calls as part of agent monitoring.

You can get started in 5 easy steps

  1. Get access. Open Tools in LangSmith.
  2. Add a server. Register an MCP server URL or turn on a built-in.
  3. Connect your account for each tool. Sign in once, or save an API key.
  4. Build a kit. Pick the tools your agent needs. You get one MCP URL.
  5. Point your client at the kit URL. Tool Gateway works with any MCP client, including LangChain, Deep Agents, Managed Deep Agents, Claude Code, and Cursor.

Let us know what you think!


r/LangChain • • 2d ago

Discussion Past ~40 tools, my AI coding agent started picking the wrong one. It didn't fail, it just looked right. Here's what fixed it

0 Upvotes

I kept adding tools and MCP servers to my coding setup and hit a problem I didn't expect. Context wasn't the issue. The agent started choosing the wrong tool.

There was no error. It would call a similar "search" tool, get a plausible answer and move on. That's what makes it easy to miss.

I asked around, and here's what worked for people using Cursor, Claude Code, Codex and custom agents:

  1. The limit is about choosing, not context size

Problems showed up around 40-50 tools, long before the context window was full. When two tools sound alike, the pick is close to a coin flip.

2. Say when to use each tool

Not two tools that both say "search", but "search code in this repo" vs "search the public docs".

3. Flag tools that write

If the model thinks a tool is read-only, it calls it freely. Never let a short description hide a write.

4. Load details only when needed

Keep a short summary in context and load the full schema only when a task needs it.

5. Use a search_tools + call_tool pair

One tool finds the right tools, another runs them. Someone said this keeps idle context under ~1k tokens whether there are 30 tools or 300.

6. Run the same task twice

If it picks different tools each time, your descriptions overlap.

My question:

When you have similar tools, do you merge them into one tool with a parameter, or keep them separate with very specific descriptions? Merging seems cleaner, but I wonder if it just moves the confusion into the arguments.

And how many tools do you keep enabled day to day? Is anyone running 100+ without issues?


r/LangChain • • 2d ago

Question | Help Looking to Validate Chatbot/RAG Agent Automated Test Harness

3 Upvotes

Hi all! I built an automated LLM evaluation harness in Python for customer support bots and RAG pipelines. It uses a two-pass generation workflow (Pass 1 creates test scenarios/rubrics; Pass 2 defines inclusion/exclusion rules) and an LLM Judge to grade responses automatically.

I'm looking to test the harness against real, live customer support chatbot endpoints instead of synthetic sandboxes. If you have an AI bot or RAG endpoint and want a free diagnostic audit on hallucination bounds, prompt injections, and edge-case behavior, drop a comment or DM me your endpoint/API details.


r/LangChain • • 2d ago

Discussion A report on Muse Charm says agentic coding is close — but orchestration is the hard part

2 Upvotes

A report on Muse Charm is making the rounds today, looking at how far agentic coding tools have come and where they still stumble. The pattern it describes will sound familiar to anyone building agents: the model can write good code in one shot, but keeping a long task on track, remembering decisions, and recovering from mistakes is where things fall apart.

That is an orchestration problem more than a model problem. The tools that win will be the ones that manage context, tool calls and state well, not just the ones with the biggest model behind them.

What do you think: is the bottleneck in agentic coding the model, the orchestration layer, or the way we prompt and supervise agents? And what would it take for you to trust an agent with a multi-hour coding task?

Follow more AI news and discussion in r/AINewsRumors.


r/LangChain • • 2d ago

Discussion Crash-Test Guard v1.0.0 — CI/CD reliability gate for stateful AI agents (Go, single binary, protocol-first)

3 Upvotes

Hey everyone,

I've been working on this for a few months and just released v1.0.0.

The problem: Your AI agent crashes mid-workflow (payment processed, email sent, etc.). After restart, does it resume correctly? Charge twice? Send duplicates? Is the checkpoint even readable?

The solution: Crash-Test Guard — a single static Go binary that drops into any CI pipeline.

How it works:

Starts your agent (any language, any framework)

Waits for a specific workflow event (e.g., "step_completed: process_payment")

Crashes it (SIGKILL, docker kill, docker rm -f + fresh run)

Restarts it, sends the checkpoint to /resume

Verifies: checkpoint consistency, monotonic progress, no duplicate side effects (via idempotency keys), workflow completion, output equivalence vs baseline

Key design decisions:

Protocol-first: 6 HTTP endpoints, agent implements them, guard calls them. Zero framework coupling.

Event-based triggers: deterministic, not sleep-based.

Opaque checkpoint envelope: agent uses any serialization (pickle, JSON, msgpack, custom).

Operation ledger: verifies committed business effects, not just intent.

Baseline comparison: runs normal + crash-injected, compares final business state.

Install:

macOS/Linux: brew tap crash-test-guard/tap && brew install crash-test-guard

Windows: scoop bucket add crash-test-guard https://github.com/crash-test-guard/scoop-crash-test-guard && scoop install crash-test-guard

Binary: GitHub Releases (Linux/macOS/Windows, amd64/arm64)

Example config (crash-test.yaml):

version: "1.0"

agent:

command: ["python", "my_agent.py"]

endpoint: "http://localhost:8080"

crash_scenarios:

- type: process_kill

trigger:

event: step_completed

step_id: process_payment

timing: after_event

verification:

checkpoint_consistency: true

duplicate_side_effects: true

workflow_completion: true

baseline_comparison: true

Repo: https://github.com/adirwt852-cloud/crash-test-guard

Protocol spec: https://github.com/adirwt852-cloud/crash-test-guard/blob/main/docs/protocol.md

Reference adapter (LangGraph + FastAPI + PostgreSQL): examples/langgraph-agent/

Looking for:

People to try it against their agents

Missing crash scenarios for your use cases

Adapter contributions (AutoGen, CrewAI, custom frameworks)

CI integration feedback

Happy to answer questions!


r/LangChain • • 3d ago

Discussion Your LangGraph checkpointer is working. It still booked the seat twice.

2 Upvotes

Wire a checkpointer into a LangGraph agent and a lot of hard problems go away. You pass a thread_id in the config, and after each super-step the graph's state gets written somewhere durable, MemorySaver while you're developing and PostgresSaver once it's real. The thread can be stopped and picked up later. That's what human-in-the-loop is built on, and time-travel debugging, and an agent surviving a pod restart. Resume a thread and the completed nodes are skipped, the pending one runs again, and the agent carries on from where it stopped.

What a checkpointer does is useful, but there's a gap in what it actually protects you from.

The window

The checkpoint is written after the node returns. So for any node that does something irreversible, there's a stretch of time where the thing has happened and nothing durable knows it.

Take a two-node booking agent. Node one reserves a seat in your own inventory, node two confirms that booking with the supplier. The pod gets rescheduled in the middle.

run 1
  │
  ├─ node "reserve"  ───►  seat held in your inventory, checkpoint written
  │
  ├─ node "confirm"
  │     │
  │     └─ confirm_booking()  ───►  supplier API     seat is booked
  │
  ╳  process dies here
  │
  └─ no checkpoint for "confirm" was ever written


run 2   resumes the same thread_id from cold
  │
  ├─ node "reserve"  ───►  skipped. its checkpoint exists.
  │
  └─ node "confirm"  ───►  runs again, because nothing says it ran
        │
        └─ confirm_booking()  ───►  supplier API     booked a second time

Re-running node two is the only correct thing the checkpointer could do with what it knows. As far as the durable record goes, that node never happened. But it did happen. The confirmation left the process through a socket and the write that would have recorded it never ran. The supplier now holds two bookings for one passenger, and there's no header you can send that would have stopped it.

A checkpointer records what your agent decided. It can't record what your agent did to the outside world, because it never saw it happen. Every checkpointer in every agent framework has this window. None of them sit in the call path of your HTTP client.

Worth saying plainly: your own inventory is fine. Node one committed, so the resumed run skipped it and there's exactly one seat held. Nothing was lost or corrupted. If you read this as a bug report about LangGraph's checkpointer you'll reach for the wrong fix.

It isn't only restarts

The pod restart is the easy one to picture. It isn't the only shape.

Two workers arrive at the same job. A thread_id is a storage key. It doesn't claim anything. Nothing stops two processes invoking the same thread at once, and that happens more than people expect: a retry that fires while the first attempt is still running, a queue that redelivers, two replicas picking up the same message. Five identical invocations of one thread at once and the supplier gets five bookings.

An advisory lock doesn't fix this, which surprises people. Mutual exclusion and idempotency get treated as the same property. A lock means one worker runs the node at a time. It says nothing about how many times the booking happens across all of them, and the measured answer there is also five.

The worker holding the job dies and doesn't come back. Not a crash loop, where the pod restarts and retries. The process is gone: the node was killed, the container was evicted, the host went away. Whatever marked that work as in-progress is now held by something that no longer exists.

Several agents reach the same conclusion independently. Five agents working one support ticket, each deciding a refund is owed. They aren't retrying each other, they reasoned separately, and the customer is owed one refund. Nothing in the thread model connects them, because they're five different threads.

What people already do about it

If the API provider supports an idempotency key, use that. Stripe has one. If it doesn't and it's an internal API you own, add one. The receiver dedupes on its own side, which beats anything the caller can do about it.

It buys less than it sounds, because nobody hands you the key. Stripe dedupes a retry only if the retry presents the same key, and your agent has to produce that key twice across a process that died in between. Generate it inside the node and the resumed run generates a different one. Derive it from the model's arguments and two agents that reasoned differently derive two keys, which is a measured result rather than a worry: in a harness where two agents each decided a refund was owed and disagreed on the amount, putting the amount in the key let both of them through. Stripe's keys also expire, at least 24 hours in v1, which covers a retry loop and not a thread somebody resumes on Tuesday.

The key has to come from somewhere durable, be the same across replicas, and outlive the process. That's the actual work, and you do it whether the receiver dedupes or you do.

What's left is the calls with no such header, which for an agent is most of them. A partner endpoint, a Slack message, an internal service another team owns, a tool someone wrote last quarter.

For those the usual answer is a claims table, the Stripe-style idempotency key written by hand. Insert a row keyed on the operation before you do the work. A second caller's insert conflicts, so it knows someone else owns the operation and waits for their answer, and you store the response against the key when you're done. It's a good answer, it's what teams actually write, and it handles both the restart and the concurrent workers.

It's the dead holder that breaks it. The row says in-progress and it's held by a process that no longer exists, so nobody else can finish the operation without a human clearing it. The usual next move is a TTL on the claim, and the instinct is right, but a TTL whose expiry is the only thing authorising takeover can't tell a dead holder from a slow one. It takes over from both. From the dead one that's correct. From the paused one you get two workers who each believe they own the operation.

Making it safe takes a heartbeat, so the holder can say it's still alive, and a fencing token, so a superseded holder's late write gets refused instead of applied. At which point you've written a lease, and you own it.

This is also where the workflow engines sit, and it's worth being clear that they don't remove the problem. Temporal makes the orchestration durable: the workflow survives, the step re-runs on replay, and you get timers and retries and history for free. Its own documentation says activities execute at least once and that you should make them idempotent, because an activity can run more than once and can partially complete more than once. Run Temporal and you still write something like the claims table inside your activity, since an activity retry is exactly the thing that books the seat twice.

Neither approach solves the case where multiple agents independently decide to perform the same operation. Five agents on five threads derive five different keys, so every guard sees a different operation and lets it through. Catching it means deriving the key from the ticket rather than from the thread, which is a decision you have to make on purpose whatever you're using.

Which is roughly what I've been building

Cellaflow is an engine that holds those leases, and I should say up front that it's mine, so weigh the rest accordingly. A tool call takes a lease before it runs. If a later call derives the same key, it's answered from the record of the first attempt instead of executing. If the holder dies, the lease expires and the work gets reclaimed by whoever takes it, which is the case a claims table can't reach.

For LangGraph the integration is two things: a decorator on the tool you can't take back, and a context manager around invoke.

from cellaflow import durable_tools
from cellaflow import tool as leased

The tool that hands the parcel to the courier gets leased:

@langchain_tool
@leased(tool_name="dispatch_order")
def dispatch_order(order_id: str, courier: str) -> str:
    """Hands `order_id` to `courier` for delivery. Cannot be recalled."""
    DISPATCHES.append(order_id)
    return f"dispatched {order_id} with {courier}"

Read it bottom-up. Cellaflow leases the call, then LangChain builds the model's tool schema from the signature and the docstring. The decorator wraps with functools.wraps, so the signature, annotations and docstring survive it and the schema LangChain generates is the one you wrote.

The agent's other tool isn't leased, because repeating a read costs nothing and a lease on it would buy nothing:

@langchain_tool
def lookup_order(order_id: str) -> str:
    """Returns the customer, item count, and status of an order."""

Then the run:

config = {"configurable": {"thread_id": f"dispatch-{uuid.uuid4().hex[:8]}"}}

with durable_tools(config, target=ENGINE):
    result = app.invoke(
        {"messages": [("user", f"Dispatch order {ORDER_ID} to the courier.")]},
        config,
    )

durable_tools reads configurable.thread_id out of the config you're already passing to invoke and derives a tool session from it, so the same thread always reaches the same leases. That's what survives the restart. The resumed run derives the same key, and the leased call is answered from the record of the first attempt instead of executing again.

LangGraph needs nothing further because it copies the calling context into the executor it runs a node on, so a tool invoked from the tools node reaches the session opened around invoke. Other frameworks vary. AutoGen doesn't propagate it at all, and there the session has to be bound inside the tool.

Does it hold up

There's a runnable version of that two-node agent in the repo. Hold a seat, confirm the booking with the supplier, kill the process between the confirmation and the checkpoint. Two arms, the same graph, the same PostgresSaver, the same crash at the same instant, and the only variable is whether the tool call holds a lease.

labelseats heldconfirmations
PostgresSaver alone12
the same graph, leased tool call11

The left column is the control, and it's what makes the right one mean anything. One seat held in both arms, because node one committed before the crash and the resumed run skipped it. The lease isn't fixing the checkpointer. It's covering the node that was still mid-call when the process died.

docker run -d --name cellaflow -p 50051:50051 \
  -e CELLAFLOW_ALLOW_UNAUTHENTICATED=true ghcr.io/cellaflow/cellaflow:latest

git clone https://github.com/cellaflow/cellaflow-sdks
cd cellaflow-sdks/examples/at_most_once_proof

docker compose up -d
pip install -r requirements.txt
python proof.py

run_arm() takes the two tool functions and a context manager; one arm passes something that does nothing, the other passes durable_tools. The graph, the crash, the resume and the checkpointer are shared, so there's nowhere for a thumb to rest on the scale. Both arms assert their own number and the script exits non-zero if either stops behaving. A proof that cannot fail is not a proof.

The counters live inside the tool bodies, so they count executions and never cache hits. If the lease works the body is never entered and nothing is appended, which means a suppressed call can't be counted as a success.

Some things it doesn't measure. There's no throughput or latency in it and no comparison of storage engines; it's a correctness demonstration, and a deterministic one, in that the confirmation either repeated or it didn't on any hardware. It runs one writer, one thread, one crash, so the concurrent scenarios above come from a separate harness. It kills a process rather than cutting power to the host. The engine is available in RocksDB and Postgresql variants.

One thing I'd like to know

If you've built the claims-table version, and most people with this problem have, I'd like to hear how it went. Specifically whether a dead holder ever cost you anything, or whether it just never came up.

I'll be in the comments answering any questions or thoughts. My DMs are also open.

Live demo: https://cellaflow.com/#live-demo

Github: https://github.com/cellaflow/cellaflow-sdks


r/LangChain • • 3d ago

Projects How I solved the "terrifying write-access" problem for autonomous AI agents (Semantic Undo & Rollback proxy)

Thumbnail reddit.com
1 Upvotes

r/LangChain • • 3d ago

Projects How I solved the "terrifying write-access" problem for autonomous AI agents (Semantic Undo & Rollback proxy)

Thumbnail reddit.com
1 Upvotes

r/LangChain • • 3d ago

Discussion A meta-analysis of 516 multi-agent LLM configurations: once the single agent gets the same samples or compute, multi-agent systems average 4.2 points worse and tie or lose in 74% of comparisons

Thumbnail
reddit.com
5 Upvotes

r/LangChain • • 3d ago

Discussion I'm building FlowX: an open-source compiler for self-evolving AI agent workflows. Looking for collaborators.

Thumbnail reddit.com
2 Upvotes

r/LangChain • • 3d ago

Discussion Perplexity is launching teams of AI agents that split research tasks

5 Upvotes

Perplexity is rolling out agent teams for research: instead of one agent doing everything, a team splits the work — one gathers sources, another cross-checks, another drafts the synthesis. It is the multi-agent orchestration pattern moving from frameworks into a consumer product.

For people building multi-agent systems: how do you split roles in practice, and what breaks first — handoffs, shared context, or cost? Does a fixed team structure beat dynamic spawning?

We track stories like this daily in r/AINewsRumors, come join the discussion.


r/LangChain • • 4d ago

Resources A Hands-on Guide to Building Agentic RAG with LangGraph

26 Upvotes

An open-source project for learning, building, and experimenting with Agentic RAG systems using LangGraph.

The repository includes interactive notebooks and a modular implementation covering the main components of an agentic retrieval pipeline.

🚀 Key Features

  • 🧠 Agentic RAG: Dynamic retrieval and reasoning workflows powered by LangGraph.
  • 🔍 Hierarchical Retrieval: Multi-level document retrieval for better context selection.
  • 🤖 Multi-Agent Workflows: Specialized agents collaborating to handle complex queries.
  • 💬 Conversation Memory: Context-aware interactions across multiple turns.
  • ❓ Query Clarification: Handling ambiguous or incomplete user queries.
  • 🔄 Self-Correction: Iterative refinement of retrieval and generated responses.
  • 📊 RAGAS Evaluation: Measuring retrieval quality and answer generation performance.
  • 🦙 Local LLM Support: Running models locally through Ollama.
  • ☁️ Cloud LLM Support: Integration with cloud-based language models.
  • 📓 Interactive Notebooks: Step-by-step examples for exploring the architecture and its components.

🔗 Repository

https://github.com/GiovanniPasq/agentic-rag-for-dummies

A practical resource for exploring Agentic RAG architectures, from individual components to complete workflows.


r/LangChain • • 3d ago

Discussion How do you debug a multi-step AI agent workflow when only the final result is wrong?

Thumbnail reddit.com
2 Upvotes

r/LangChain • • 3d ago

Discussion Does the “stale fact” multi-step failure match what you’ve actually seen?

2 Upvotes

A while back I asked this sub how people stop agents from making multi-step mistakes, and the answers shaped my capstone. The failure that kept coming up: every tool call is fine on its own, but the sequence acts on a fact that changed. The agent reads a record, an action gets approved, the record changes underneath, and the agent acts on the old value. One-call-at-a-time checks never see it.

I built a guardrail that re-reads the facts an irreversible action depends on right before it fires, and blocks if a relevant one moved since approval. On a 27-scenario benchmark it reports two numbers separately: how many real failures it catches, and how often it wrongly blocks a safe run. The false-alarm side turned out to be the hard part, not the catching.

Mostly I want to know if the problem is real for you:

  • does the stale-fact failure match something you’ve actually hit, or is it a problem you don’t really have?
  • what’s a multi-step failure you’d want caught that this wouldn’t?

Repo: https://github.com/rayyanmaan/agent-guardrail


r/LangChain • • 4d ago

Resources Stop letting your LangGraph agent write to the database. Make it propose instead.

10 Upvotes

A pattern that saved us in production (an AI assistant used by 25k+ people): the agent never
writes data directly. It proposes the write, the graph pauses, a human approves, and then it resumes.

LangGraph makes this simple with `interrupt` + a checkpointer:

from langgraph.types import interrupt, Command

def apply_change(state):
decision = interrupt({
"action": "update_record",
"proposed": state["proposed_change"],
})
if decision["approved"]:
save(state["proposed_change"])
return {"status": "applied"}
return {"status": "rejected", "reason": decision.get("reason")}
# graph compiled with a checkpointer (Postgres in prod)
# ...later, when the user clicks Approve in the UI:
graph.invoke(Command(resume={"approved": True}), config)

What we learned:
- Checkpoint in Postgres, not memory. Approvals can take hours, and servers restart.
- Show the diff, not the raw tool call. Users approve "Change end date: Mar 1 → Apr 1",
not JSON.
- Cap iterations per mode. A loop that keeps proposing and getting rejected burns tokens fast.
- Trace every interrupt (we use Langfuse) so you can see where users reject the agent most.
That's your roadmap.

How are others handling approval UX? Batch approvals, or one at a time?


r/LangChain • • 3d ago

Resources I built an audit trail for AI agents: 10 data models, 33 pages, 3 visits last month, 0 users. Looking for platform engineers to break the schema.

Thumbnail reddit.com
1 Upvotes