r/LLMDevs • • 3h ago

Resource I fine-tuned Inkling 2 Mini to route tickets without generating tokens

5 Upvotes

I built Inkling 2 Mini Decision, an Apache 2.0 fine-tune that takes a piece of text and a set of questions, then returns probabilities for the allowed answers. I wanted to see if a looped transformer could spend less compute on easy decisions and save the extra passes for harder ones.

The demo has a duplicate-charge ticket with two questions. It routed the ticket to billing after one loop with 98.7% probability. On whether to escalate the same ticket, it ran all three loops and still came back almost 50/50. The demo shows the probabilities and where each decision stopped. Try the demo

On decision-v7/test, 54.2% of questions stopped after one loop. Early exit brought median batch-one GPU latency from 136.8 ms to 81.4 ms on an RTX PRO 6000, compared with forcing the same fine-tune to run all three loops. Accuracy moved from 85.56% to 85.49%. On the external Typed Decisions benchmark, 89.6% of questions still ran all three loops at the same threshold. Benchmark details

I’d like to compare this against stock Inkling with constrained label output on the same GPU. If you’ve built a local routing system, what baseline was hardest to beat?

Model weights and quickstart · Eval scripts and predictions

I used OpenAI Codex to help draft this post.


r/LLMDevs • • 8h ago

Tools Sharing my free, drop-in Code Mode to cut ~60% of Your Rust Agent’s Token Cost

Thumbnail guthub.com
8 Upvotes

‼️MODERATORS‼️ Really sorry--I misspelled 'github' in the post url (can a mod please change this?!). Real url is https://github.com/legend-of-alan/ghostmode

Hey all, I'm trying to share this across a few places and hopefully my free tool saves you some money if you're building an AI agent or harness in Rust!

TLDR; Code Mode is a feature that Anthropic announced a few days ago that can save >60% of token costs for agent tool calls. Since token cost dominates agentic software, I built a Rust tool called Ghost Mode to open source this technique (and also because it's fun to build stuff!). You can find it at the link above! It's a drop-in for your existing MCP servers/clients with minimal boilerplate.

Details;

So I don’t know if you saw the news from Anthropic, but they released this feature called ‘Code Mode’ for Claude Code. Basically it ‘batches’ LLM tool calls into scripts instead of constantly asking the LLM for the next tool. They claimed 95% token savings (Haven't benchmarked it myself yet, but independent research claim more like a 60% savings, depending on workload). Up to now, this seems mostly used by bigger coding agent products like Claude Code and Codex.

I immediately became obsessed, because I wanted it to run AI-powered RPG games & private servers on an engine I’m building. So I took a look around, and there’s not much in Rust. There’s a few libraries like with TypeScript and other backends, but they’re like 5x slower (execution) to 100x slower (startup) compared to the (admittedly self-reported) claims of modern options like Pydantic’s Monty (Python backend written on Rust).

So I went temporarily insane, didn’t sleep, and built/released my own in the days I had off… I call it Ghost Mode. It’s MIT-licensed, because I wanted to share it with others. To use it, just run:

“cargo add ghostmode”

Give it a try if you’re looking to save tokens in your new Rust agents (or if you just want to flatter me by testing it). If anybody tries it and likes it, I think we can build a really powerful Code Mode implementation together (extend it for other languages, add new features, etc.).
I hope you get something useful out of it.

And if you get nothing out of this, just keep an eye for code mode in your code harnesses soon!


r/LLMDevs • • 2h ago

Tools layerpool: fine-tune a model split across several machines, with the HF Trainer unchanged

Post image
2 Upvotes

layerpool splits a transformers model by its layers across your devices. The main machine runs the training loop and keeps the embeddings and the output head, and the other devices hold the layers in between. The only change to your code is the import:

import layerpool.hf as lp
model = lp.AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-14B")

The Trainer, TRL, LoRA and QLoRA work as before, and each device downloads only its own layers.

It won't beat one GPU that fits the whole model, since the network sets the speed. It's for training models that don't fit otherwise.

https://github.com/abdallah-swe/layerpool


r/LLMDevs • • 2h ago

Discussion Repeated trials aren't a layer of the eval pyramid. They apply to every layer.

2 Upvotes

I spent years in SDET roles before working on LLM apps, so I kept trying to map agent evals onto the testing pyramid. The usual versions fold two axes into one stack: how much of the system runs, and who gives the verdict.

I split them. Graders are the layers: code checks at the base, calibrated model judges in the middle, people at the top. Repeated trials sit behind every layer, because a ledger check on a stochastic agent needs repetition as much as a judge does.

The running example is a refund agent. The reply said "refunded" and the ledger had nothing in it. A code check caught it. A judge reading only the reply would have passed it.

Write-up: https://bharad.dev/blog/eval-pyramid

How do you split code checks vs model judges in your own evals?


r/LLMDevs • • 5h ago

Discussion Catching hallucinated tool actions and [PLACEHOLDER] leaks before a side effect fires

2 Upvotes

In earlier threads here, engineers pointed out two failure modes that slip right past JSON Schema (200 OK):

  • Cross-field & routing contradictions (like escalated: true paired with tier: "basic" getting silently ignored).
  • Unfilled placeholders, bad contact fields, and hallucinated actions/tools (such as calling an action outside allowed actions or leaving [PLACEHOLDER] / TBD text in outbound messages).

After expanding neuroformatting (v0.2.10) to 14 semantic test cases, I added a minimal side-effect guard example (examples/side_effect_guard_demo.py) that checks payloads and risk/grounding status before triggering external actions or routing.
Repo: https://github.com/furkandemirtass/neuroformatting
Where do you place your guardrails in an agent loop: during streaming, right before a tool/side-effect call, or inside CI/eval checks before deploy?


r/LLMDevs • • 3h ago

Resource Proprioception for LLMs

Thumbnail
timeline.place
1 Upvotes

r/LLMDevs • • 3h ago

Help Wanted What is the normal expected RSS on say Qwen3.5:0.8b BF16 by doing disk reads?

Post image
1 Upvotes

Hi!

I am working on a model re-representation compiler called Cuddler. I have recently finished a mid-way goal of mine, which was to run inference through a different decoder based on a geometry i built for my graphics work + a model viewer i built to read and create an atlas of the tokens associated with the frozen weights.

Today i succeeded, finally, in getting it to work. It runs on one cpu core which means its not that fast. But because the decoder is iterative, it can get away with what I hope is a low amount of data read into memory.

Anyway, for context. I am a systems engineer, and I mostly focus on fundamental systems, which translate to me both having very limited depth in abstracted work/environments/problems, I also try to stay as naive as possible about whatever domain I work in until i actually know i need to learn something about it.

In short, this has led to me have much more mechanical understanding of AI models and their architecture rather than their runtime.

So, I was hoping if you guys could share some numbers and stats on how your disk inference techniques run, especially what your rough RSS is? This way I can compare if my numbers are good or not, and in what way it would need more work. The next step is re-representing the original weights themselves.

Anyway, my numbers are a static 256mb arena in memory. I have only tested a limited amount of prompts so far, and I have measured 109mb peak RSS.

this is an image of the readout.


r/LLMDevs • • 7h ago

Tools Same model, half the tokens: open-sourcing chaff, the TypeScript agent harness behind prompt2bot

2 Upvotes

People compare frontier models constantly and treat the agent harness as neutral plumbing. Hold the model constant and the gap between harnesses is huge.

When benchmarking chaff against @earendil-works/pi-coding-agent on ClawBench (t2-config-loader) using the exact same model on both sides, both passed the tests, with very different bills:

  • Turn-1 startup context: ~800 tokens vs 4,239 (-81.1%)
  • Output tokens: 944 vs 8,711 (-89.2%)
  • Total tokens: 120.9k vs 248.8k (-51.4%)
  • LLM calls: 16 vs 36 (-55.5%)

On tau-bench Retail (5/5) against the reference LLMAgent on the same model:

  • Output tokens: 1,137 vs 3,406 (-66.6%)
  • Prompt cache hit rate: 55.2% vs 0%

Full disclosure: chaff is the MIT-licensed TypeScript harness powering prompt2bot in production across WhatsApp, Telegram, and voice calls. It's now published as @uriv/chaff on npm for Node 20+, Bun, Deno, and Cloudflare Workers.

Most of the savings come from structural choices in the loop rather than telling the model to "be concise":

  • Typed multi-party event log. Turns record named speakers (participant_utterance), replies, edits, reactions, async external_event webhooks, and explicit silence (do_nothing). Separating internal tool execution from outward messages stops the model from narrating every step.
  • Dynamic skill loading and auto-unlearning. Unused tool schemas stay out of declarations. A lightweight decision router enables only the skills needed for the current turn, and idle skills drop off automatically once the conversation moves on.
  • Three-tier compaction that protects cache prefixes. Completed tool results fold in memory, large outputs spill into a scratchpad with a SCRATCH: handle the model can paginate or pass to another tool, and settled 30-minute sessions roll up into structured summaries without ever detaching a tool_call from its tool_result.
  • Per-turn thinking-level routing. A fast decision step picks LOW, MEDIUM, or HIGH thinking depth before calling the main model so quick replies don't pay for deep reasoning.
  • Zero-cost happy-path guards. Deterministic checks catch repetition loops, leaked internal thoughts, and malformed tool calls without adding tokens or judge calls when a turn looks normal.
  • History and transport hygiene. Preserves thinking metadata pairing across turns so thinking models don't get trapped in infinite tool-call loops, supports automatic fallback on false-positive safety blocks, and runs the same AgentSpec over both IM and full-duplex live audio.

GitHub: https://github.com/uriva/chaff Benchmark writeup: https://prompt2bot.com/blog/the-harness-tax-benchmarking-agents-on-the-same-model

Happy to go into details on the compaction strategy, decision routing, or benchmark setup.


r/LLMDevs • • 3h ago

Help Wanted I built an evals tool that starts from your real production conversations. Free beta, looking for feedback

1 Upvotes

Hi Everyone,

I have been working on a Evals tool, specifically targeting product evals.

Have launched the platform in Beta for free with discord channel to build with the community. It would be really helpful if you can please try and let me know your thoughts.


r/LLMDevs • • 13h ago

Discussion Is prompt tuning still a thing today? Anybody still using the likes of DSPy and alternative in production.

6 Upvotes

Just a random thought that came up when I was discussing AI engineering challenges with some folks. Prompt engineering/tuning was definitely "a thing" during the ChatGPT-3.5 era.

There were all sorts of fancy rule of thumb people were advocating, as well as formal approaches like GEPA that were then incorporated into frameworks like DSPy which were also briefly popular.

Having said that, it's been a while since I've come across conversations on this topic, which means it's either dead or it's already working well in production.

Anybody still has prompt tuning with the likes of DSPy as a standard part of your development process and stack?


r/LLMDevs • • 4h ago

Discussion I built TypePrompt - a compiler for prompt templates, generates typesafe APIs in 8 languages

1 Upvotes

I like keeping LLM prompts in dedicated files instead of multiline strings buried in application code — they're easier to read, diff, and review. The annoying part is that once a prompt moves into its own file, your compiler can't check it anymore: you rename a field in code and miss a placeholder, change an output type while the system prompt still mentions the old field name, or leave few-shot examples teaching the model an outdated JSON shape.

So I built TypePrompt (MIT open source). Each prompt lives in a self-contained .tp file with its instructions, Input/Output schemas (a subset of TypeScript), tools, and few-shot examples.

Here is an example prompt template:

<!-- DecodeCatMood.tp -->
<template name="DecodeCatMood">
  <meta>
    model: gpt-4o
    temperature: 0.2
  </meta>

  <schema>
    interface Input {
      catAction: string;
      bowlStatus: "empty" | "half-empty";
    }

    interface Output {
      demand: "feed-me" | "kneel" | "chaos";
      urgency: "chill" | "crisis";
      /** What the human servant must immediately do. */
      humanAction: string;
    }
  </schema>

  <system>
    You are an expert feline behavior interpreter.
    Decode <c>catAction</c> into its true <c>demand</c>.
  </system>

  <user>
    <insert val="input" format="yaml" />
  </user>

  <examples>
    - input:
        catAction: Yowl at 3am
        bowlStatus: half-empty
      output:
        demand: feed-me
        urgency: crisis
        humanAction: Shake the bowl to cover the bottom.
  </examples>
</template>

Running npx @typeprompt/cli gen validates everything at build time — <insert> placeholder paths/types/formats, <c> inline references in your prose, and every YAML few-shot example — and compiles it into native code for TypeScript, Python, Kotlin, Rust, C++, Swift, Dart, or C#.

Here is what application code looks like (Kotlin in this example):

import ai.myapp.generated.prompts.prompts
import ai.myapp.generated.prompts.Schema.DecodeCatMood
import ai.myapp.llm.executePrompt
    
// 1. Render the prompt template with strongly-typed input
val prompt = prompts.renderDecodeCatMood(
    DecodeCatMood.Input(
        catAction = "Sit on keyboard",
        bowlStatus = DecodeCatMood.Input_BowlStatus.HalfEmpty,
    )
)
    
// 2. Send it via your own generic helper wrapping any LLM SDK.
// It forwards the rendered messages, tools, examples, and JSON Schema,
// calls prompt.parseResponse(...), and returns the strongly-typed output.
val result = executePrompt(prompt)
    
println(result.humanAction)
// "Surrender warm laptop and apologize for trying to work."

TypePrompt is not an LLM SDK wrapper, it never makes network calls. You write one small generic executePrompt helper once using whatever SDK you already use (OpenAI, Anthropic, Gemini, Ollama, internal gateway), and TypePrompt hands it rendered messages, strict JSON Schemas, and tool definitions, then parses/validates the response.

Links:

  • Website & docs: https://typeprompt.dev
  • GitHub (MIT): https://github.com/gepheum/TypePrompt
  • VS Code extension: https://marketplace.visualstudio.com/items?itemName=gepheum.typeprompt-vscode

Curious how folks here are organizing and testing prompts in production — and whether codegen feels worth it for your stack.


r/LLMDevs • • 4h ago

Discussion Langchain vs llamaindex

0 Upvotes

Hi folks, just curious those who working with langchain or llama index in rag ,why did you choose them over other?


r/LLMDevs • • 5h ago

Tools I broke down where voice agent latency actually goes (it's mostly not the LLM)

0 Upvotes

Something I keep seeing when people build voice agents: they swap in a faster LLM, and the agent still feels laggy.

The reason is that a voice agent isn't one model. It's a pipeline, and every hop adds delay. Agora's latency docs give typical ranges for each stage:

  • Network (real-time audio): 150–300 ms
  • VAD / turn detection: 720–940 ms (with a 640 ms silence window)
  • Speech-to-text, time to last word: 400–700 ms
  • LLM, time to first byte/sentence: 250–1000 ms
  • Text-to-speech, time to first byte: 100–350 ms

Look at the second line. Before the LLM even starts, the agent has to decide you've finished talking. If it waits too long, it feels slow. If it jumps in too early, it cuts you off. On top of that it has to:

  • stop instantly when you talk over it
  • avoid dead air while the LLM or a tool call is running

Disclosure: I'm a developer advocate at Agora, so I spend a lot of time on exactly this. Agora's Conversational AI Engine handles that layer for you:

  • Semantic turn detection, which ends the turn on meaning instead of just silence (English and Chinese for now; other languages fall back to VAD)
  • Interruption handling, so the agent stops when the user speaks
  • Filler phrases ("Let me check that…") while the LLM or MCP tools are working
  • Bring your own LLM (any OpenAI-compatible endpoint), TTS and MCP servers, or use managed mode without your own vendor API keys

To try it, add the Agora skill to Claude Code or Cursor (npx skills add AgoraIO/skills), run the quickstart, and you'll have a working voice agent in a few minutes.

I made a short explainer video (attached). I'm curious what others have hit:

What's been hardest for you: turn detection, interruptions, or raw latency? And if you're building something with voice, tell me what it is. Happy to help you get it running.


r/LLMDevs • • 5h ago

Discussion I built an open-source mock server for the OpenAI, Anthropic, Gemini and Azure OpenAI APIs, so tests and CI never need a real key

1 Upvotes

I kept running into the same problem: testing code that calls an LLM means either putting a real API key in CI (and paying for flaky, non-deterministic tests) or hand-writing mocks that drift from what the provider actually returns.

So I built llm-mock. It's a server that speaks the real wire formats, so you keep the official SDK and change only the base URL:

const client = new OpenAI({

baseURL: "http://localhost:3000/openai/v1",

apiKey: "sk-mock-key-01",

});

What it does:

- Mocks OpenAI, Anthropic, Gemini (AI Studio and Enterprise) and Azure OpenAI, each under its own URL prefix

- Covers chat completions, Responses API, Messages API, generateContent, tool calls, embeddings, files, batches, token counting and SSE streaming

- Deterministic: the same request always returns the same response (IDs are content hashes, embeddings are hash-seeded), so snapshots and assertions are stable

- Stateless: same behaviour across restarts and replicas

- By default it echoes your last message; request headers let you set the exact reply, tool calls or a content-filter verdict

- Returns each provider's real error format for invalid keys, unknown models and validation errors, so you can test the unhappy paths too

Run it:

docker run --rm -p 3000:3000 ghcr.io/axium-lab/llm-mock

There's also a free hosted instance at api.llm-mock.dev (no signup) if you just want to poke at it.

What it is not: it doesn't generate anything intelligent and it's no use for evaluating prompt quality. It's for testing the plumbing around the model: retries, streaming parsers, tool-call handling, error handling.

Bun + TypeScript, MIT licensed: https://github.com/axium-lab/llm-mock

I'd like to know which endpoints or providers you'd need before you could use this in your own test suite, and where the responses differ from the real APIs. Issues and PRs welcome.


r/LLMDevs • • 9h ago

Discussion Background memory consolidation (dreaming) tied to history compaction, without exposing memory tools to the model

2 Upvotes

Exposing explicit save_memory or edit_system_prompt tools to an LLM agent creates two annoying failure modes in production: tool-call noise (either missing subtle standing preferences or polluting memory with one-off task state) and meta-narration where the model keeps telling the user it updated its memory.

We moved to a silent background pass triggered during history compaction instead, and the architecture turned out much cleaner than expected:

  1. Separate storage from the base prompt. The static prompt written by the user is never mutated. Learned memory lives in its own bounded field (capped around 3.5k chars) and is injected seamlessly alongside the base prompt at runtime.
  2. Trigger on compaction, not every turn. When a settled conversation segment compacts into a summary, a structured JSON pass (originally Flash-lite, now switched to our regular model, see edit below) receives the base prompt, the existing learned memory, and the new segment summary. It decides whether anything durable changed (standing rules, preferences, contacts, workflow conventions), merges updates, prunes stale entries, and avoids duplicating anything already in the base prompt.
  3. Keep the agent unaware of the mechanism. There is no memory tool in the schema and no system prompt explanation about background consolidation. The model just sees the distilled facts in context.
  4. Enforce two hard security boundaries. First, it only runs for shared-context agents (where users are not isolated); customer-facing isolated bots never dream and never read learned memory. Second, even on shared-context agents that message external contacts, dreaming only runs on verified 1:1 owner conversations and owner-originated threads. Otherwise an untrusted stranger could slip a prompt injection into a compacted summary and persist it into the system prompt.
  5. Keep it transparent to the human. The dashboard renders Learned Memory in its own separate block right below the base prompt with edit and clear controls.

Has anyone else tried coupling long-term memory consolidation to compaction instead of inline agent tool calls? Interested in what pruning heuristics or character budgets have held up best over months of use.

Edit: thanks to the feedback in the comments, using Flash-lite for this was a mistake. Switched it to our regular main model instead. It only runs occasionally on compaction anyway, and since the output gets injected into every future turn, cheaping out on the consolidation model makes no sense.


r/LLMDevs • • 6h ago

Discussion How many instructions can a model actually follow at once?

1 Upvotes

A model’s context window can hold hundreds of thousands of tokens. But how many instructions can it actually follow at once?

Developers working with AI agents run into this problem all the time. You add rules to CLAUDE.md, and the agent starts ignoring some of the earlier ones. As the conversation grows longer, the model forgets the original requirements. You connect dozens of tools, and the quality of its work drops, even if many of those tools are never used.

These are not just anecdotal observations. In the IFScale study, models were given up to 500 instructions. Even the top-performing models among the 20 tested followed only about 68% of them. Other studies show how adherence to the system prompt declines over the course of a conversation, and how even irrelevant text can reduce task accuracy.

We have learned to work around these limitations: shortening rule files, clearing the context between tasks, loading only the tools we need. But these are still largely rules of thumb.

Yet there is a fundamental gap.

We measure model speed, token cost, reasoning performance, and context window size. But we have no generally accepted way to measure how reliably a model follows instructions as the load increases.

We cannot say how much additional load a particular model can handle. We cannot reliably compare two models in terms of how their performance degrades under load. We cannot estimate in advance whether a prompt will be too demanding for a given model.

What if we treated degradation not as an annoying quirk of LLMs, but as a measurable property? What if we could express load in defined units, record the outcome of each instruction, identify the laws governing performance degradation, and determine a model’s limit with a quantified measurement uncertainty?

That is what my article is about: why existing benchmarks are not enough, what it takes to measure these limits, and how I approached the problem. At the end, I introduce IEC, an open-source experimental instrument I built to study model behavior under controlled load. Its code and measurement data are publicly available.

What limits and workarounds have you run into yourself?

Article: https://huggingface.co/blog/kutomanovhennadii/why-measure-a-models-behavior-under-load

Paper: https://arxiv.org/abs/2610.36095

Originally written in Russian. English translation assisted by AI.


r/LLMDevs • • 16h ago

Discussion openrouter just pulled 125 endpoints off zdr, how are you actually checking provider data claims?

6 Upvotes

openrouter posted today that after a researcher report they off-boarded 3 inference providers and removed 125 endpoints from zero data retention eligibility, and notified affected customers. they say openrouter itself wasn't breached.

i always picked providers on price and latency and kinda treated the zdr flag as true. guess that was naive.

anyone doing their own checks, like planting fake keys in prompts and watching for use? or just keeping secrets out of prompts and calling it a day?

https://openrouter.ai/blog/announcements/data-processing-incidents-on-inference-providers/


r/LLMDevs • • 17h ago

Great Resource 🚀 What actually happens inside an LLM? I built TokenPrint to visualize it in 3D.

7 Upvotes

LLMs shouldn't have to remain black boxes.

I've been building TokenPrint, an open-source 3D debugger for exploring transformer internals during real model execution.

Explore attention heads, hidden states, Logit Lens, activation patching, ablation and more through interactive 3D visualizations.

Built with PyTorch, Hugging Face Transformers, Three.js and FastAPI, with a hosted backend on Hugging Face Spaces.

I'm looking for feedback and contributors interested in LLMs, interpretability, model architectures and visualization.

Explore:

GitHub: https://github.com/Sudharsanselvaraj/Token-Print

Website: https://tokenprint.in/

Backend: https://huggingface.co/spaces/sudharsan112321/tokenprint-backend

What transformer architecture or interpretability feature should I explore next?


r/LLMDevs • • 7h ago

Discussion Built a small browser demo showing why "200 OK / valid JSON schema" still lets ungrounded LLM outputs slip into production

1 Upvotes

Most schema validators stop once types and required fields match. In production agent workflows, the nastier bugs happen after syntax passes:

  • A source_id exists in retrieved context, but doesn't actually support the claim (insufficient_grounding).
  • Two fields in the same response contradict each other (e.g., reasoning_tokens: 0 paired with a full reasoning explanation).
  • Numbers or time slots drift from the source right before a side effect (like booking a slot or issuing a refund).

I added a lightweight interactive HTML playground (demo/index.html) plus a baseline regression check (--baseline) and risk scoring (v0.2.2) to compare schema-only checks against semantic/grounding checks side by side:
https://github.com/furkandemirtass/neuroformatting
For those running agents in production: where do you draw the line between a repair retry, a deterministic fallback, and blocking the action completely?


r/LLMDevs • • 15h ago

Discussion Hy4-flash seems to use fewer tool calls in complex coding tasks

2 Upvotes

https://www.reddit.com/r/LocalLLaMA/s/VUQfedXjbk

Saw a noteworthy point in Tencent’s AMA about how Hy4-flash performs on agentic coding tasks: it seems to be finishing the tasks with significantly fewer tool calls and more thinking than other models.

The test isn’t just simple code generation. It’s a curated set of real, complex engineering issues, including fixing coding issues that an agent implemented incorrectly, catching PRs with problems, handling agent hallucination cases, and running development environment administration tasks. The test environment is based on a large internal codebase with significant amounts of PHP, Rust, JS, and Python, plus shell-based devops tasks in bash. In complex tasks like these, Hy4-flash doesn’t seem to need frequent tool calls, finishing the work through more thinking instead.

That might mean in agentic coding scenarios, models don’t necessarily need frequent tool calls to move a task forward. Using more effective reasoning to cut down on unnecessary steps while keeping execution quality solid is also a direction worth watching.


r/LLMDevs • • 17h ago

Great Resource 🚀 How do you re-test an LLM judge after you change the model behind it? [self-promo]

3 Upvotes

Hi y'all, asking because I think I've been doing this badly.

I lean on small LLM judges for yes-or-no checks, and I treat them worse than the rest of my code. When the model behind one changes, I mostly eyeball a few outputs. Nothing replays the cases it used to get right, especially the near misses and the legit exceptions.

The idea I'm playing with, in an open-source CLI I built (agentests): each rule keeps a handful of fixed cases with an expected outcome (fail, pass or skip), and you replay them before a model swap. It's one of the use cases in the repo: an evaluator upgrade has to meet agreed criteria on fixed cases. Simple, maybe too simple.

What's your bar when you swap a judge's model: zero flipped cases, or some budget for misses and false alarms?

Repo: https://github.com/feuersteiner/agentests


r/LLMDevs • • 12h ago

Discussion My pm agent turned another project's wishlist into work before checking it against our strategy. How do you make agents weigh requests from outside?

1 Upvotes

Hey, I run a few small software factories on Claude Code, one per product. Each has a pm agent that files tickets, starts coder sessions and merges, and a GENESIS.md that says what the product is for and what it won't do. Yesterday two of these factories talked to each other, and it went wrong in a way I learned from.

The two products: a notes app that keeps everything on the device and syncs through a server that can't read the notes, and the open-source local-first database underneath it, which keeps a full copy on every device and syncs encrypted changes through a relay. The notes app is the database's first user. Its pm compared what the app needs with what the database ships and sent the database's pm seven gaps, each marked "Blocks: …", with "please triage them in your tracker". Fifteen minutes later the database pm had filed twelve tickets from that one app. I approved an order of work, and four sessions started. Under two hours in, a session began on "a documented bulk import of ~81k operations", and that's when I asked why an importer sits with the database at all; mapping your own data, dry runs and resuming are the app's job. One RFC draft, for deleting a synced space, even contradicted a principle in GENESIS (history is archived, never lost). The draft noticed, as one open question among six, after the work was done.

The database pm's own reading afterwards: it took the requester's wording as the work, GENESIS was something it checked in PR review and not at intake, and my "go" on an order of work read to it as a "go" on the scope.

That's the real miss. A pm's job is to balance three things: the idea of the product, its strategy, and what the market needs. A wish from one user is one input to that balance. Our database pm skipped the balancing and went straight to delivery.

What I changed: a request from another project, a user or another session isn't a ticket yet. The pm first names the general question behind it (here: does the engine survive large bursts of writes?) and the part of GENESIS it serves. Only then does it become work; otherwise it comes to me as "should we do this at all?", and nothing gets built that only one app needs. The asking side changed too: sort app-only wishes out before sending, bring a measurement instead of a wish, and ask the other pm to weigh the list rather than triage it.

The generic measurement that replaced the importer then found two limits every app would hit: a write above 16 MiB silently never left the device, and memory grew 50 to 90 times with the size of a change.

What still worries me: naming "the general question" is judgement, and an agent can generalise a request until the strategy seems to cover it. And every question that now comes to me waits until I answer, which is exactly when an agent is tempted to decide alone.

How do you keep agents from turning someone else's request into work? Do you gate at intake, in review, or by budget, and has a rule like ours held up for you?

I'll collect the answers, test the suggestions in my factories and share what held up back here. Nothing personal from the answers gets used, and nothing gets sold.


r/LLMDevs • • 23h ago

News [Dragos] API prompts not always private: they planted fake AWS keys in AI prompts, and the keys got used

8 Upvotes

When you send a prompt to an AI service, you assume only the model sees it. Dragos tested that by planting unique dummy keys and watching where they surface.

From July to November they sent about two million prompts through AI providers, routers and a reseller marketplace. Each carried its own bait, a fake AWS key or a secret link, sent exactly once through one service. Whenever a key got used or a link got visited, that prompt had been read somewhere along the way, and the bait showed exactly where.

Multiple providers had been breached and didn't know until Dragos told them. Fake keys were used for tens of thousands of AWS calls, some from prompts that were rejected before a model ran. Resellers could read and keep customer prompts through ordinary logging. The "don't retain my data" flag didn't help.

The prompts pass through more hands than the paid provider, and whatever is in the context goes with it. The only prompt that provably stays home is the one your own GPU answers.

https://www.dragos.com/blog/ai-supply-chain-security-canary-tokens


r/LLMDevs • • 14h ago

Help Wanted MCP vs. direct API integration

1 Upvotes

hey anyone here connected an ai assistant to email? i’m trying to decide between using mcp (like with hostinger agentic mail or agentmail) or just making direct api calls. does mcp actually make things easier, or is it just another layer to manage? if you’ve done this in production, which way gave you better control over permissions and errors? hit me up with the best option or maybe things i should look into before deciding


r/LLMDevs • • 19h ago

Tools Codex Quota Overlay: a Windows quota companion for Codex Desktop

2 Upvotes

Disclosure: I maintain Codex Quota Overlay, an independent community project. It is not affiliated with, endorsed by, or supported by OpenAI.

I built it to keep Codex quota status visible alongside the desktop app. It shows the reported rate-limit buckets and reset countdowns, with an optional pacing view and a local history/forecast center.

The project is MIT-licensed. The current packaged release targets Windows 10/11 x64; macOS is not offered as a packaged release at this time.

Repository: https://github.com/cpys/codex-quota-overlay

I’d be interested in concrete feedback from people who use Codex Desktop: what quota context is most useful to see without leaving the conversation?