r/LangChain • • 21h ago

Discussion Does a second model catch citation errors the first model misses

2 Upvotes

When an assistant cites the wrong policy and the citation looks clean no reader spots the error until the policy gets enforced. For instance one builder in a LangChain thread today said the same model checking its own work found nothing real in his experience. However a second call from the same model may share the same blind spot because it read the same chunks and learned the same shortcut. That is why I want to know whether a different model is the only honest check or whether a cheaper test catches the same errors. If you can tell me what your citation check looks like in practice I would learn more from that than from another eval number.


r/LangChain • • 1h ago

Discussion Tool-calling agent built on LangChain scores rank 8/53 on ExtractBench - architecture breakdown

1 Upvotes

Sharing because this was built with LangChain's tool calling and I think the patterns might be useful to others here.

KYNEX is a document extraction agent. You give it a PDF + JSON schema, it fills the schema by navigating the document with tools. No RAG retrieval step, the agent decides what to read.

LangChain-specific bits:

- Tools defined with u/tool decorator, bound via model.bind_tools()

- Agent loop runs up to 30 rounds with AIMessage/ToolMessage cycling

- SystemMessage includes a Table of Contents built from parsed headings + table page ranges

- Works with ChatOpenAI and AzureChatOpenAI (swappable via config)

The agent has a fallback escalation chain for tables: extract_table → reconstruct_table → Azure DI → vision. Each step is a separate tool. If one fails, the agent tries the next. This was key for handling the 67 different document types in the benchmark.

Results on ExtractBench (370 docs, verified by LlamaIndex):

- 94.6 F1 (rank 8/53)

- GPT-5.4 through this agent scored 92.97, beating GPT-6 Astra standalone (91.9)

Open source under Apache 2.0. Curious if anyone else has built multi-tool agents with this kind of escalation pattern, found it way more effective than single-tool approaches.