r/LLMDevs • • 22h ago

Tools I’m building an open-source intelligent LLM router that runs inside AI coding agents

0 Upvotes

I’m developing AutoRouter, an open-source routing and optimization layer for AI coding agents.
The goal is to make coding agents more efficient by automatically selecting the appropriate model and reasoning effort for different types of work, instead of relying on one fixed model for everything.
The architecture is designed to integrate with existing agent harnesses rather than replace them.
The current native Codex integration supports Luna-first stage classification and model switching, subject to the available model catalog and host controls. The portable integrations currently provide routing recommendations and shared tools.
I’m working toward more advanced routing, including adaptive escalation, subagent delegation, performance-aware model selection and local progress estimation.
I want AutoRouter to become a configurable optimization layer for Codex, Claude Code and other coding agents.
GitHub: [https://github.com/Z440-prod/autorouter](https://github.com/Z440-prod/autorouter)
I’d especially appreciate feedback from people who understand coding-agent internals, model routing and harness architecture.
What capabilities would make something like this genuinely useful in your daily workflow?


r/LLMDevs • • 18h ago

News [Dragos] API prompts not always private: they planted fake AWS keys in AI prompts, and the keys got used

6 Upvotes

When you send a prompt to an AI service, you assume only the model sees it. Dragos tested that by planting unique dummy keys and watching where they surface.

From July to November they sent about two million prompts through AI providers, routers and a reseller marketplace. Each carried its own bait, a fake AWS key or a secret link, sent exactly once through one service. Whenever a key got used or a link got visited, that prompt had been read somewhere along the way, and the bait showed exactly where.

Multiple providers had been breached and didn't know until Dragos told them. Fake keys were used for tens of thousands of AWS calls, some from prompts that were rejected before a model ran. Resellers could read and keep customer prompts through ordinary logging. The "don't retain my data" flag didn't help.

The prompts pass through more hands than the paid provider, and whatever is in the context goes with it. The only prompt that provably stays home is the one your own GPU answers.

https://www.dragos.com/blog/ai-supply-chain-security-canary-tokens


r/LLMDevs • • 4h ago

Discussion Background memory consolidation (dreaming) tied to history compaction, without exposing memory tools to the model

2 Upvotes

Exposing explicit save_memory or edit_system_prompt tools to an LLM agent creates two annoying failure modes in production: tool-call noise (either missing subtle standing preferences or polluting memory with one-off task state) and meta-narration where the model keeps telling the user it updated its memory.

We moved to a silent background pass triggered during history compaction instead, and the architecture turned out much cleaner than expected:

  1. Separate storage from the base prompt. The static prompt written by the user is never mutated. Learned memory lives in its own bounded field (capped around 3.5k chars) and is injected seamlessly alongside the base prompt at runtime.
  2. Trigger on compaction, not every turn. When a settled conversation segment compacts into a summary, a structured JSON pass (originally Flash-lite, now switched to our regular model, see edit below) receives the base prompt, the existing learned memory, and the new segment summary. It decides whether anything durable changed (standing rules, preferences, contacts, workflow conventions), merges updates, prunes stale entries, and avoids duplicating anything already in the base prompt.
  3. Keep the agent unaware of the mechanism. There is no memory tool in the schema and no system prompt explanation about background consolidation. The model just sees the distilled facts in context.
  4. Enforce two hard security boundaries. First, it only runs for shared-context agents (where users are not isolated); customer-facing isolated bots never dream and never read learned memory. Second, even on shared-context agents that message external contacts, dreaming only runs on verified 1:1 owner conversations and owner-originated threads. Otherwise an untrusted stranger could slip a prompt injection into a compacted summary and persist it into the system prompt.
  5. Keep it transparent to the human. The dashboard renders Learned Memory in its own separate block right below the base prompt with edit and clear controls.

Has anyone else tried coupling long-term memory consolidation to compaction instead of inline agent tool calls? Interested in what pruning heuristics or character budgets have held up best over months of use.

Edit: thanks to the feedback in the comments, using Flash-lite for this was a mistake. Switched it to our regular main model instead. It only runs occasionally on compaction anyway, and since the output gets injected into every future turn, cheaping out on the consolidation model makes no sense.


r/LLMDevs • • 6h ago

Discussion My pm agent turned another project's wishlist into work before checking it against our strategy. How do you make agents weigh requests from outside?

1 Upvotes

Hey, I run a few small software factories on Claude Code, one per product. Each has a pm agent that files tickets, starts coder sessions and merges, and a GENESIS.md that says what the product is for and what it won't do. Yesterday two of these factories talked to each other, and it went wrong in a way I learned from.

The two products: a notes app that keeps everything on the device and syncs through a server that can't read the notes, and the open-source local-first database underneath it, which keeps a full copy on every device and syncs encrypted changes through a relay. The notes app is the database's first user. Its pm compared what the app needs with what the database ships and sent the database's pm seven gaps, each marked "Blocks: …", with "please triage them in your tracker". Fifteen minutes later the database pm had filed twelve tickets from that one app. I approved an order of work, and four sessions started. Under two hours in, a session began on "a documented bulk import of ~81k operations", and that's when I asked why an importer sits with the database at all; mapping your own data, dry runs and resuming are the app's job. One RFC draft, for deleting a synced space, even contradicted a principle in GENESIS (history is archived, never lost). The draft noticed, as one open question among six, after the work was done.

The database pm's own reading afterwards: it took the requester's wording as the work, GENESIS was something it checked in PR review and not at intake, and my "go" on an order of work read to it as a "go" on the scope.

That's the real miss. A pm's job is to balance three things: the idea of the product, its strategy, and what the market needs. A wish from one user is one input to that balance. Our database pm skipped the balancing and went straight to delivery.

What I changed: a request from another project, a user or another session isn't a ticket yet. The pm first names the general question behind it (here: does the engine survive large bursts of writes?) and the part of GENESIS it serves. Only then does it become work; otherwise it comes to me as "should we do this at all?", and nothing gets built that only one app needs. The asking side changed too: sort app-only wishes out before sending, bring a measurement instead of a wish, and ask the other pm to weigh the list rather than triage it.

The generic measurement that replaced the importer then found two limits every app would hit: a write above 16 MiB silently never left the device, and memory grew 50 to 90 times with the size of a change.

What still worries me: naming "the general question" is judgement, and an agent can generalise a request until the strategy seems to cover it. And every question that now comes to me waits until I answer, which is exactly when an agent is tempted to decide alone.

How do you keep agents from turning someone else's request into work? Do you gate at intake, in review, or by budget, and has a rule like ours held up for you?

I'll collect the answers, test the suggestions in my factories and share what held up back here. Nothing personal from the answers gets used, and nothing gets sold.


r/LLMDevs • • 8h ago

Discussion Is prompt tuning still a thing today? Anybody still using the likes of DSPy and alternative in production.

5 Upvotes

Just a random thought that came up when I was discussing AI engineering challenges with some folks. Prompt engineering/tuning was definitely "a thing" during the ChatGPT-3.5 era.

There were all sorts of fancy rule of thumb people were advocating, as well as formal approaches like GEPA that were then incorporated into frameworks like DSPy which were also briefly popular.

Having said that, it's been a while since I've come across conversations on this topic, which means it's either dead or it's already working well in production.

Anybody still has prompt tuning with the likes of DSPy as a standard part of your development process and stack?


r/LLMDevs • • 9h ago

Tools Don't buy a GPU for AI until you check this site

0 Upvotes

OMEN is a free tool that tells you if an open AI model will run on your GPU, how fast it'll go, and what it'll cost. Not sure what to buy? Set a budget and it tests 714 setups to pick the cheapest, fastest and best-value option.

✅ Free, no account
✅ Shows its working: every estimate is labelled as an estimate

🔗 GitHub repo link - https://github.com/SrijithLegend/Oh-Memory-Exhausted-Noted


r/LLMDevs • • 11h ago

Discussion openrouter just pulled 125 endpoints off zdr, how are you actually checking provider data claims?

4 Upvotes

openrouter posted today that after a researcher report they off-boarded 3 inference providers and removed 125 endpoints from zero data retention eligibility, and notified affected customers. they say openrouter itself wasn't breached.

i always picked providers on price and latency and kinda treated the zdr flag as true. guess that was naive.

anyone doing their own checks, like planting fake keys in prompts and watching for use? or just keeping secrets out of prompts and calling it a day?

https://openrouter.ai/blog/announcements/data-processing-incidents-on-inference-providers/


r/LLMDevs • • 12h ago

Great Resource 🚀 How do you re-test an LLM judge after you change the model behind it? [self-promo]

2 Upvotes

Hi y'all, asking because I think I've been doing this badly.

I lean on small LLM judges for yes-or-no checks, and I treat them worse than the rest of my code. When the model behind one changes, I mostly eyeball a few outputs. Nothing replays the cases it used to get right, especially the near misses and the legit exceptions.

The idea I'm playing with, in an open-source CLI I built (agentests): each rule keeps a handful of fixed cases with an expected outcome (fail, pass or skip), and you replay them before a model swap. It's one of the use cases in the repo: an evaluator upgrade has to meet agreed criteria on fixed cases. Simple, maybe too simple.

What's your bar when you swap a judge's model: zero flipped cases, or some budget for misses and false alarms?

Repo: https://github.com/feuersteiner/agentests


r/LLMDevs • • 2h ago

Tools Same model, half the tokens: open-sourcing chaff, the TypeScript agent harness behind prompt2bot

2 Upvotes

People compare frontier models constantly and treat the agent harness as neutral plumbing. Hold the model constant and the gap between harnesses is huge.

When benchmarking chaff against @earendil-works/pi-coding-agent on ClawBench (t2-config-loader) using the exact same model on both sides, both passed the tests, with very different bills:

  • Turn-1 startup context: ~800 tokens vs 4,239 (-81.1%)
  • Output tokens: 944 vs 8,711 (-89.2%)
  • Total tokens: 120.9k vs 248.8k (-51.4%)
  • LLM calls: 16 vs 36 (-55.5%)

On tau-bench Retail (5/5) against the reference LLMAgent on the same model:

  • Output tokens: 1,137 vs 3,406 (-66.6%)
  • Prompt cache hit rate: 55.2% vs 0%

Full disclosure: chaff is the MIT-licensed TypeScript harness powering prompt2bot in production across WhatsApp, Telegram, and voice calls. It's now published as @uriv/chaff on npm for Node 20+, Bun, Deno, and Cloudflare Workers.

Most of the savings come from structural choices in the loop rather than telling the model to "be concise":

  • Typed multi-party event log. Turns record named speakers (participant_utterance), replies, edits, reactions, async external_event webhooks, and explicit silence (do_nothing). Separating internal tool execution from outward messages stops the model from narrating every step.
  • Dynamic skill loading and auto-unlearning. Unused tool schemas stay out of declarations. A lightweight decision router enables only the skills needed for the current turn, and idle skills drop off automatically once the conversation moves on.
  • Three-tier compaction that protects cache prefixes. Completed tool results fold in memory, large outputs spill into a scratchpad with a SCRATCH: handle the model can paginate or pass to another tool, and settled 30-minute sessions roll up into structured summaries without ever detaching a tool_call from its tool_result.
  • Per-turn thinking-level routing. A fast decision step picks LOW, MEDIUM, or HIGH thinking depth before calling the main model so quick replies don't pay for deep reasoning.
  • Zero-cost happy-path guards. Deterministic checks catch repetition loops, leaked internal thoughts, and malformed tool calls without adding tokens or judge calls when a turn looks normal.
  • History and transport hygiene. Preserves thinking metadata pairing across turns so thinking models don't get trapped in infinite tool-call loops, supports automatic fallback on false-positive safety blocks, and runs the same AgentSpec over both IM and full-duplex live audio.

GitHub: https://github.com/uriva/chaff Benchmark writeup: https://prompt2bot.com/blog/the-harness-tax-benchmarking-agents-on-the-same-model

Happy to go into details on the compaction strategy, decision routing, or benchmark setup.


r/LLMDevs • • 12h ago

Great Resource 🚀 I built TokenPrint — an open-source 3D debugger for exploring transformer internals

1 Upvotes

I've been building TokenPrint, an open-source platform to explore what's happening inside transformer models.

It brings interactive 3D visualization and model instrumentation together, allowing you to inspect attention heads, hidden states, Logit Lens, activation patching, ablation and more.

The frontend is built with Next.js and Three.js, with a Python/FastAPI backend hosted on Hugging Face Spaces.

I'm building this in the open and would love feedback, ideas, and contributions from the LLM developer community!

Explore TokenPrint:

What features or model architectures would you like to see added next?


r/LLMDevs • • 13h ago

Tools Codex Quota Overlay: a Windows quota companion for Codex Desktop

2 Upvotes

Disclosure: I maintain Codex Quota Overlay, an independent community project. It is not affiliated with, endorsed by, or supported by OpenAI.

I built it to keep Codex quota status visible alongside the desktop app. It shows the reported rate-limit buckets and reset countdowns, with an optional pacing view and a local history/forecast center.

The project is MIT-licensed. The current packaged release targets Windows 10/11 x64; macOS is not offered as a packaged release at this time.

Repository: https://github.com/cpys/codex-quota-overlay

I’d be interested in concrete feedback from people who use Codex Desktop: what quota context is most useful to see without leaving the conversation?


r/LLMDevs • • 17h ago

Tools Culler - a local, open source software factory

Thumbnail
gallery
3 Upvotes

Culler is an MIT licensed software factory that runs containerized agents on your Mac, finds work worth doing, and presents you a queue of comprehensible review packets with their rationale.

Its agents run as five roles in the following order:

  1. Strategist proactively proposes high-leverage work based on your open-ended Vision.
  2. Arbiter judges each proposal and promotes the most effective one.
  3. Worker works the task in its own repository copy.
  4. Verifier checks the work against acceptance criteria and asks Worker to correct any gaps.
  5. Packager creates an illustrative review packet meant for a human mind.

Configure each agent role with your AI model of choice. Put your smartest model as Strategist and a workhorse as Worker.

Depending on the AI provider used for a run, Culler leverages one of three proven harnesses: Codex, Claude Code, or OpenCode. Configure agents to run with your ChatGPT or Claude subscription. Or use API keys compatible with any of the 200+ providers supported by the OpenCode catalog.

If you have already concrete work in mind, either define new tasks yourself or with the help of Collie, your Culler assistant.

Culler opts out of shipping its own memory system and instead uses Openwiki. After initializing the wiki for you, and on a schedule, Scribe pores over recent changes in the Project to maintain it.

The wiki serves three purposes:

  1. Acts as index into the repository's ground truth to enable more efficient search for agents.
  2. Acts as a knowledgebase to help the Strategist prioritize work.
  3. Provides a viewer so you can re-familiarize yourself with your project as it continually develops.

For agent observability, Culler uses AgentsView. It supports each trace format Culler uses. It allows for inspecting context, tool calls, token usage, and other metrics.

On security, Projects are never mounted into agent containers. Each container run receives only a Git copy of the repo, and only changes approved by you are merged back onto the host. Agent credentials never reach inside the container. Culler runs a proxy on the host that swaps dummy tokens for real ones when model requests are made.

Next on the Roadmap - Worksources and MCPs:

Enable smooth integration of additional services to allow agents to query/leverage data from them. Instead of seeding work only from your Vision, use connected data sources such as Github Issues, Linear, Gmail etc. to act as Worksources.

---

I built Culler with the solopreneur and project junkie in mind. Start from scratch. Or import your active/retired projects, give a description of what you imagine the project to be, and let the emergent properties of gradient descent work its magic. The user experience is designed around the premise that human attention is paramount, now and in perpetuity.

https://github.com/vinchenkov/Culler

  • Feedback/contributions - on GitHub

r/LLMDevs • • 18h ago

Discussion I’ve been annoyed with the state of memory benchmarks and how easy it is to present dishonest results, so I made a deterministic, LLM-agnostic doc2query benchmark myself.

4 Upvotes

I started building memory systems for myself early this year, and that expanded into other areas of AI infrastructure. I pretty much exclusively engineered solutions to solve my own problems, and the results turned into quality systems that started to hold up more and more. Somewhere in there, when I was trying to improve my agent's memory, I ran more memory benchmarks than any one person should have to sit through, and I realized how easy the numbers are to move. Pick a friendlier k, a smaller pool, a kinder judge model or a stronger answer model, and the same system suddenly looks a lot better. What annoyed me the most was that oftentimes that wasn't disclosed. That, and there's a lot of chest-thumping over numbers that don't really mean that much, yet some still define their worth by the high number.

So I decided I wanted to build a benchmark that didn't have variables to tweak, had no LLM involved in the scoring, and was deterministic at its core. After months of putting it off, I finally finished Parsica Recall. It only asks one thing: when an agent needs a memory, does the memory system put the right record in front of it?

My rules:

- The judge never calls a model. Every score is an id match, a phrase match or a token overlap, so two people running the same inputs get the same bytes. `--verify` runs the whole thing twice and compares the hashes.

- You don't get to pick k. The window comes from the size of the pool, and the chance line (what a random retriever would score) is printed next to every number.

- Every run writes out each question's full window with ids, ranks and scores, so any number can be re-derived.

- It doesn't publish anyone's scores, mine included. The only numbers in the repo are calibration, plain BM25 and an oracle, so you can see where the floor and the ceiling are. BM25 gets 87.58 gold@15 on the current set, so that's the floor to beat. I don't really think comparing benchmark scores is a good indicator that a system is actually useful and practical, so I've never felt the need to advertise numbers for my own systems.

The coolest part of this, and the thing I wanted the most, is that you can build a benchmark out of your own agent's memories, doc2query style. It reads an export of your store and writes the questions an agent would later ask about those records, with the right records pinned by id. That’s the one step that uses a model. It works with Anthropic, OpenAI or Google using your own API key, or any endpoint that speaks OpenAI’s chat format, including a local model, and it gives you a cost estimate first and won’t do a live run without a spending cap. Scoring never touches a model, and the benchmark itself runs fully offline. Fair warning though, building a set sends a sample of your records to whichever API you pick (unless it’s a local model), so read the doc2query guide guide before you point it at anything sensitive.

It should work with pretty much any memory system. You can hand it a results file from anything in any language, write a small Python adapter, or run it read-only against a live store.

pip install parsica-recall

parsica-recall run --adapter bm25 --verify

It's MIT, standard library only, Python 3.11+. It's also a beta, and it stays a beta until it's actually been used and peer reviewed, which is a big part of why I'm posting it here. If the methodology is wrong somewhere, I really want to know. Not in this release yet: the bigger standard tracks, scoring against the sealed half of the questions, and a full-stack mode where an answer model reads the window. But please let me know if you have thoughts or ideas. I would love to collaborate or take feedback. As far as I know, nothing like this really exists, so it's important to me that it eventually gets a green light from other users in the community.

https://github.com/Antaris-Analytics-LLC/Parsica-Recall


r/LLMDevs • • 21h ago

Discussion Valid JSON isn't always grounded: How are you handling semantic validation on streamed LLM outputs?

4 Upvotes

In most structured-output setups, schema enforcement stops at syntax and types (valid JSON, string/float matching). In practice, the trickier failure mode is semantically invalid but syntactically valid output—like a model returning a source_id that wasn't in the retrieved context, or a score outside domain bounds.
I put together a minimal, zero-dependency Python prototype (neuroformatting) to test separating structural checks from context-aware semantic rules on streamed chunks:
https://github.com/furkandemirtass/neuroformatting
Curious how others here structure this layer in production:
Do you run semantic/grounding checks inside the stream consumer before committing output, or strictly post-hoc?
When a grounding check fails, do you trigger a targeted repair prompt with the exact validation error, or fall back deterministically?


r/LLMDevs • • 12h ago

Great Resource 🚀 What actually happens inside an LLM? I built TokenPrint to visualize it in 3D.

6 Upvotes

LLMs shouldn't have to remain black boxes.

I've been building TokenPrint, an open-source 3D debugger for exploring transformer internals during real model execution.

Explore attention heads, hidden states, Logit Lens, activation patching, ablation and more through interactive 3D visualizations.

Built with PyTorch, Hugging Face Transformers, Three.js and FastAPI, with a hosted backend on Hugging Face Spaces.

I'm looking for feedback and contributors interested in LLMs, interpretability, model architectures and visualization.

Explore:

GitHub: https://github.com/Sudharsanselvaraj/Token-Print

Website: https://tokenprint.in/

Backend: https://huggingface.co/spaces/sudharsan112321/tokenprint-backend

What transformer architecture or interpretability feature should I explore next?


r/LLMDevs • • 3h ago

Tools Sharing my free, drop-in Code Mode to cut ~60% of Your Rust Agent’s Token Cost

Thumbnail guthub.com
5 Upvotes

‼️MODERATORS‼️ Really sorry--I misspelled 'github' in the post url (can a mod please change this?!). Real url is https://github.com/legend-of-alan/ghostmode

Hey all, I'm trying to share this across a few places and hopefully my free tool saves you some money if you're building an AI agent or harness in Rust!

TLDR; Code Mode is a feature that Anthropic announced a few days ago that can save >60% of token costs for agent tool calls. Since token cost dominates agentic software, I built a Rust tool called Ghost Mode to open source this technique (and also because it's fun to build stuff!). You can find it at the link above! It's a drop-in for your existing MCP servers/clients with minimal boilerplate.

Details;

So I don’t know if you saw the news from Anthropic, but they released this feature called ‘Code Mode’ for Claude Code. Basically it ‘batches’ LLM tool calls into scripts instead of constantly asking the LLM for the next tool. They claimed 95% token savings (Haven't benchmarked it myself yet, but independent research claim more like a 60% savings, depending on workload). Up to now, this seems mostly used by bigger coding agent products like Claude Code and Codex.

I immediately became obsessed, because I wanted it to run AI-powered RPG games & private servers on an engine I’m building. So I took a look around, and there’s not much in Rust. There’s a few libraries like with TypeScript and other backends, but they’re like 5x slower (execution) to 100x slower (startup) compared to the (admittedly self-reported) claims of modern options like Pydantic’s Monty (Python backend written on Rust).

So I went temporarily insane, didn’t sleep, and built/released my own in the days I had off… I call it Ghost Mode. It’s MIT-licensed, because I wanted to share it with others. To use it, just run:

“cargo add ghostmode”

Give it a try if you’re looking to save tokens in your new Rust agents (or if you just want to flatter me by testing it). If anybody tries it and likes it, I think we can build a really powerful Code Mode implementation together (extend it for other languages, add new features, etc.).
I hope you get something useful out of it.

And if you get nothing out of this, just keep an eye for code mode in your code harnesses soon!