r/LLMDevs • • Aug 20 '25

Community Rule Update: Clarifying our Self-promotion and anti-marketing policy

27 Upvotes

Hey everyone,

We've just updated our rules with a couple of changes I'd like to address:

1. Updating our self-promotion policy

We have updated rule 5 to make it clear where we draw the line on self-promotion and eliminate gray areas and on-the-fence posts that skirt the line. We removed confusing or subjective terminology like "no excessive promotion" to hopefully make it clearer for us as moderators and easier for you to know what is or isn't okay to post.

Specifically, it is now okay to share your free open-source projects without prior moderator approval. This includes any project in the public domain, permissive, copyleft or non-commercial licenses. Projects under a non-free license (incl. open-core/multi-licensed) still require prior moderator approval and a clear disclaimer, or they will be removed without warning. Commercial promotion for monetary gain is still prohibited.

2. New rule: No disguised advertising or marketing

We have added a new rule on fake posts and disguised advertising — rule 10. We have seen an increase in these types of tactics in this community that warrants making this an official rule and bannable offence.

We are here to foster meaningful discussions and valuable exchanges in the LLM/NLP space. If you’re ever unsure about whether your post complies with these rules, feel free to reach out to the mod team for clarification.

As always, we remain open to any and all suggestions to make this community better, so feel free to add your feedback in the comments below.


r/LLMDevs • • Apr 15 '25

News Reintroducing LLMDevs - High Quality LLM and NLP Information for Developers and Researchers

40 Upvotes

Hi Everyone,

I'm one of the new moderators of this subreddit. It seems there was some drama a few months back, not quite sure what and one of the main moderators quit suddenly.

To reiterate some of the goals of this subreddit - it's to create a comprehensive community and knowledge base related to Large Language Models (LLMs). We're focused specifically on high quality information and materials for enthusiasts, developers and researchers in this field; with a preference on technical information.

Posts should be high quality and ideally minimal or no meme posts with the rare exception being that it's somehow an informative way to introduce something more in depth; high quality content that you have linked to in the post. There can be discussions and requests for help however I hope we can eventually capture some of these questions and discussions in the wiki knowledge base; more information about that further in this post.

With prior approval you can post about job offers. If you have an *open source* tool that you think developers or researchers would benefit from, please request to post about it first if you want to ensure it will not be removed; however I will give some leeway if it hasn't be excessively promoted and clearly provides value to the community. Be prepared to explain what it is and how it differentiates from other offerings. Refer to the "no self-promotion" rule before posting. Self promoting commercial products isn't allowed; however if you feel that there is truly some value in a product to the community - such as that most of the features are open source / free - you can always try to ask.

I'm envisioning this subreddit to be a more in-depth resource, compared to other related subreddits, that can serve as a go-to hub for anyone with technical skills or practitioners of LLMs, Multimodal LLMs such as Vision Language Models (VLMs) and any other areas that LLMs might touch now (foundationally that is NLP) or in the future; which is mostly in-line with previous goals of this community.

To also copy an idea from the previous moderators, I'd like to have a knowledge base as well, such as a wiki linking to best practices or curated materials for LLMs and NLP or other applications LLMs can be used. However I'm open to ideas on what information to include in that and how.

My initial brainstorming for content for inclusion to the wiki, is simply through community up-voting and flagging a post as something which should be captured; a post gets enough upvotes we should then nominate that information to be put into the wiki. I will perhaps also create some sort of flair that allows this; welcome any community suggestions on how to do this. For now the wiki can be found here https://www.reddit.com/r/LLMDevs/wiki/index/ Ideally the wiki will be a structured, easy-to-navigate repository of articles, tutorials, and guides contributed by experts and enthusiasts alike. Please feel free to contribute if you think you are certain you have something of high value to add to the wiki.

The goals of the wiki are:

  • Accessibility: Make advanced LLM and NLP knowledge accessible to everyone, from beginners to seasoned professionals.
  • Quality: Ensure that the information is accurate, up-to-date, and presented in an engaging format.
  • Community-Driven: Leverage the collective expertise of our community to build something truly valuable.

There was some information in the previous post asking for donations to the subreddit to seemingly pay content creators; I really don't think that is needed and not sure why that language was there. I think if you make high quality content you can make money by simply getting a vote of confidence here and make money from the views; be it youtube paying out, by ads on your blog post, or simply asking for donations for your open source project (e.g. patreon) as well as code contributions to help directly on your open source project. Mods will not accept money for any reason.

Open to any and all suggestions to make this community better. Please feel free to message or comment below with ideas.


r/LLMDevs • • 5h ago

Tools Sharing my free, drop-in Code Mode to cut ~60% of Your Rust Agent’s Token Cost

Thumbnail guthub.com
6 Upvotes

‼️MODERATORS‼️ Really sorry--I misspelled 'github' in the post url (can a mod please change this?!). Real url is https://github.com/legend-of-alan/ghostmode

Hey all, I'm trying to share this across a few places and hopefully my free tool saves you some money if you're building an AI agent or harness in Rust!

TLDR; Code Mode is a feature that Anthropic announced a few days ago that can save >60% of token costs for agent tool calls. Since token cost dominates agentic software, I built a Rust tool called Ghost Mode to open source this technique (and also because it's fun to build stuff!). You can find it at the link above! It's a drop-in for your existing MCP servers/clients with minimal boilerplate.

Details;

So I don’t know if you saw the news from Anthropic, but they released this feature called ‘Code Mode’ for Claude Code. Basically it ‘batches’ LLM tool calls into scripts instead of constantly asking the LLM for the next tool. They claimed 95% token savings (Haven't benchmarked it myself yet, but independent research claim more like a 60% savings, depending on workload). Up to now, this seems mostly used by bigger coding agent products like Claude Code and Codex.

I immediately became obsessed, because I wanted it to run AI-powered RPG games & private servers on an engine I’m building. So I took a look around, and there’s not much in Rust. There’s a few libraries like with TypeScript and other backends, but they’re like 5x slower (execution) to 100x slower (startup) compared to the (admittedly self-reported) claims of modern options like Pydantic’s Monty (Python backend written on Rust).

So I went temporarily insane, didn’t sleep, and built/released my own in the days I had off… I call it Ghost Mode. It’s MIT-licensed, because I wanted to share it with others. To use it, just run:

“cargo add ghostmode”

Give it a try if you’re looking to save tokens in your new Rust agents (or if you just want to flatter me by testing it). If anybody tries it and likes it, I think we can build a really powerful Code Mode implementation together (extend it for other languages, add new features, etc.).
I hope you get something useful out of it.

And if you get nothing out of this, just keep an eye for code mode in your code harnesses soon!


r/LLMDevs • • 23m ago

Resource I fine-tuned Inkling 2 Mini to route tickets without generating tokens

2 Upvotes

I built Inkling 2 Mini Decision, an Apache 2.0 fine-tune that takes a piece of text and a set of questions, then returns probabilities for the allowed answers. I wanted to see if a looped transformer could spend less compute on easy decisions and save the extra passes for harder ones.

The demo has a duplicate-charge ticket with two questions. It routed the ticket to billing after one loop with 98.7% probability. On whether to escalate the same ticket, it ran all three loops and still came back almost 50/50. The demo shows the probabilities and where each decision stopped. Try the demo

On decision-v7/test, 54.2% of questions stopped after one loop. Early exit brought median batch-one GPU latency from 136.8 ms to 81.4 ms on an RTX PRO 6000, compared with forcing the same fine-tune to run all three loops. Accuracy moved from 85.56% to 85.49%. On the external Typed Decisions benchmark, 89.6% of questions still ran all three loops at the same threshold. Benchmark details

I’d like to compare this against stock Inkling with constrained label output on the same GPU. If you’ve built a local routing system, what baseline was hardest to beat?

Model weights and quickstart · Eval scripts and predictions

I used OpenAI Codex to help draft this post.


r/LLMDevs • • 2h ago

Discussion Catching hallucinated tool actions and [PLACEHOLDER] leaks before a side effect fires

2 Upvotes

In earlier threads here, engineers pointed out two failure modes that slip right past JSON Schema (200 OK):

  • Cross-field & routing contradictions (like escalated: true paired with tier: "basic" getting silently ignored).
  • Unfilled placeholders, bad contact fields, and hallucinated actions/tools (such as calling an action outside allowed actions or leaving [PLACEHOLDER] / TBD text in outbound messages).

After expanding neuroformatting (v0.2.10) to 14 semantic test cases, I added a minimal side-effect guard example (examples/side_effect_guard_demo.py) that checks payloads and risk/grounding status before triggering external actions or routing.
Repo: https://github.com/furkandemirtass/neuroformatting
Where do you place your guardrails in an agent loop: during streaming, right before a tool/side-effect call, or inside CI/eval checks before deploy?


r/LLMDevs • • 15m ago

Resource Proprioception for LLMs

Thumbnail
timeline.place
1 Upvotes

r/LLMDevs • • 35m ago

Help Wanted What is the normal expected RSS on say Qwen3.5:0.8b BF16 by doing disk reads?

Post image
1 Upvotes

Hi!

I am working on a model re-representation compiler called Cuddler. I have recently finished a mid-way goal of mine, which was to run inference through a different decoder based on a geometry i built for my graphics work + a model viewer i built to read and create an atlas of the tokens associated with the frozen weights.

Today i succeeded, finally, in getting it to work. It runs on one cpu core which means its not that fast. But because the decoder is iterative, it can get away with what I hope is a low amount of data read into memory.

Anyway, for context. I am a systems engineer, and I mostly focus on fundamental systems, which translate to me both having very limited depth in abstracted work/environments/problems, I also try to stay as naive as possible about whatever domain I work in until i actually know i need to learn something about it.

In short, this has led to me have much more mechanical understanding of AI models and their architecture rather than their runtime.

So, I was hoping if you guys could share some numbers and stats on how your disk inference techniques run, especially what your rough RSS is? This way I can compare if my numbers are good or not, and in what way it would need more work. The next step is re-representing the original weights themselves.

Anyway, my numbers are a static 256mb arena in memory. I have only tested a limited amount of prompts so far, and I have measured 109mb peak RSS.

this is an image of the readout.


r/LLMDevs • • 4h ago

Tools Same model, half the tokens: open-sourcing chaff, the TypeScript agent harness behind prompt2bot

2 Upvotes

People compare frontier models constantly and treat the agent harness as neutral plumbing. Hold the model constant and the gap between harnesses is huge.

When benchmarking chaff against @earendil-works/pi-coding-agent on ClawBench (t2-config-loader) using the exact same model on both sides, both passed the tests, with very different bills:

  • Turn-1 startup context: ~800 tokens vs 4,239 (-81.1%)
  • Output tokens: 944 vs 8,711 (-89.2%)
  • Total tokens: 120.9k vs 248.8k (-51.4%)
  • LLM calls: 16 vs 36 (-55.5%)

On tau-bench Retail (5/5) against the reference LLMAgent on the same model:

  • Output tokens: 1,137 vs 3,406 (-66.6%)
  • Prompt cache hit rate: 55.2% vs 0%

Full disclosure: chaff is the MIT-licensed TypeScript harness powering prompt2bot in production across WhatsApp, Telegram, and voice calls. It's now published as @uriv/chaff on npm for Node 20+, Bun, Deno, and Cloudflare Workers.

Most of the savings come from structural choices in the loop rather than telling the model to "be concise":

  • Typed multi-party event log. Turns record named speakers (participant_utterance), replies, edits, reactions, async external_event webhooks, and explicit silence (do_nothing). Separating internal tool execution from outward messages stops the model from narrating every step.
  • Dynamic skill loading and auto-unlearning. Unused tool schemas stay out of declarations. A lightweight decision router enables only the skills needed for the current turn, and idle skills drop off automatically once the conversation moves on.
  • Three-tier compaction that protects cache prefixes. Completed tool results fold in memory, large outputs spill into a scratchpad with a SCRATCH: handle the model can paginate or pass to another tool, and settled 30-minute sessions roll up into structured summaries without ever detaching a tool_call from its tool_result.
  • Per-turn thinking-level routing. A fast decision step picks LOW, MEDIUM, or HIGH thinking depth before calling the main model so quick replies don't pay for deep reasoning.
  • Zero-cost happy-path guards. Deterministic checks catch repetition loops, leaked internal thoughts, and malformed tool calls without adding tokens or judge calls when a turn looks normal.
  • History and transport hygiene. Preserves thinking metadata pairing across turns so thinking models don't get trapped in infinite tool-call loops, supports automatic fallback on false-positive safety blocks, and runs the same AgentSpec over both IM and full-duplex live audio.

GitHub: https://github.com/uriva/chaff Benchmark writeup: https://prompt2bot.com/blog/the-harness-tax-benchmarking-agents-on-the-same-model

Happy to go into details on the compaction strategy, decision routing, or benchmark setup.


r/LLMDevs • • 55m ago

Help Wanted I built an evals tool that starts from your real production conversations. Free beta, looking for feedback

1 Upvotes

Hi Everyone,

I have been working on a Evals tool, specifically targeting product evals - Zoroval.

Have launched the platform in Beta for free with discord channel to build with the community. It would be really helpful if you can please try and let me know your thoughts.


r/LLMDevs • • 11h ago

Discussion Is prompt tuning still a thing today? Anybody still using the likes of DSPy and alternative in production.

6 Upvotes

Just a random thought that came up when I was discussing AI engineering challenges with some folks. Prompt engineering/tuning was definitely "a thing" during the ChatGPT-3.5 era.

There were all sorts of fancy rule of thumb people were advocating, as well as formal approaches like GEPA that were then incorporated into frameworks like DSPy which were also briefly popular.

Having said that, it's been a while since I've come across conversations on this topic, which means it's either dead or it's already working well in production.

Anybody still has prompt tuning with the likes of DSPy as a standard part of your development process and stack?


r/LLMDevs • • 1h ago

Discussion I built TypePrompt - a compiler for prompt templates, generates typesafe APIs in 8 languages

1 Upvotes

I like keeping LLM prompts in dedicated files instead of multiline strings buried in application code — they're easier to read, diff, and review. The annoying part is that once a prompt moves into its own file, your compiler can't check it anymore: you rename a field in code and miss a placeholder, change an output type while the system prompt still mentions the old field name, or leave few-shot examples teaching the model an outdated JSON shape.

So I built TypePrompt (MIT open source). Each prompt lives in a self-contained .tp file with its instructions, Input/Output schemas (a subset of TypeScript), tools, and few-shot examples.

Here is an example prompt template:

<!-- DecodeCatMood.tp -->
<template name="DecodeCatMood">
  <meta>
    model: gpt-4o
    temperature: 0.2
  </meta>

  <schema>
    interface Input {
      catAction: string;
      bowlStatus: "empty" | "half-empty";
    }

    interface Output {
      demand: "feed-me" | "kneel" | "chaos";
      urgency: "chill" | "crisis";
      /** What the human servant must immediately do. */
      humanAction: string;
    }
  </schema>

  <system>
    You are an expert feline behavior interpreter.
    Decode <c>catAction</c> into its true <c>demand</c>.
  </system>

  <user>
    <insert val="input" format="yaml" />
  </user>

  <examples>
    - input:
        catAction: Yowl at 3am
        bowlStatus: half-empty
      output:
        demand: feed-me
        urgency: crisis
        humanAction: Shake the bowl to cover the bottom.
  </examples>
</template>

Running npx @typeprompt/cli gen validates everything at build time — <insert> placeholder paths/types/formats, <c> inline references in your prose, and every YAML few-shot example — and compiles it into native code for TypeScript, Python, Kotlin, Rust, C++, Swift, Dart, or C#.

Here is what application code looks like (Kotlin in this example):

import ai.myapp.generated.prompts.prompts
import ai.myapp.generated.prompts.Schema.DecodeCatMood
import ai.myapp.llm.executePrompt
    
// 1. Render the prompt template with strongly-typed input
val prompt = prompts.renderDecodeCatMood(
    DecodeCatMood.Input(
        catAction = "Sit on keyboard",
        bowlStatus = DecodeCatMood.Input_BowlStatus.HalfEmpty,
    )
)
    
// 2. Send it via your own generic helper wrapping any LLM SDK.
// It forwards the rendered messages, tools, examples, and JSON Schema,
// calls prompt.parseResponse(...), and returns the strongly-typed output.
val result = executePrompt(prompt)
    
println(result.humanAction)
// "Surrender warm laptop and apologize for trying to work."

TypePrompt is not an LLM SDK wrapper, it never makes network calls. You write one small generic executePrompt helper once using whatever SDK you already use (OpenAI, Anthropic, Gemini, Ollama, internal gateway), and TypePrompt hands it rendered messages, strict JSON Schemas, and tool definitions, then parses/validates the response.

Links:

  • Website & docs: https://typeprompt.dev
  • GitHub (MIT): https://github.com/gepheum/TypePrompt
  • VS Code extension: https://marketplace.visualstudio.com/items?itemName=gepheum.typeprompt-vscode

Curious how folks here are organizing and testing prompts in production — and whether codegen feels worth it for your stack.


r/LLMDevs • • 2h ago

Discussion Langchain vs llamaindex

0 Upvotes

Hi folks, just curious those who working with langchain or llama index in rag ,why did you choose them over other?


r/LLMDevs • • 2h ago

Discussion Looking for Discord/Telegram groups for buying and selling LLM API credits

0 Upvotes

Looking for communities where people buy and sell LLM API credits and capacity: Claude, GPT, Gemini and similar.

Specifically interested in:

- Groups where suppliers and buyers post offers and rates

- Discussion of which upstream sources are reliable and which ones disappear overnight

- Wholesale pricing and long-term supply

Discord, Telegram or forums all work.

If you're in one of these groups, drop a name or a link, or DM me. Happy to pay for an invite or a good referral. Thanks!


r/LLMDevs • • 2h ago

Tools I broke down where voice agent latency actually goes (it's mostly not the LLM)

0 Upvotes

Something I keep seeing when people build voice agents: they swap in a faster LLM, and the agent still feels laggy.

The reason is that a voice agent isn't one model. It's a pipeline, and every hop adds delay. Agora's latency docs give typical ranges for each stage:

  • Network (real-time audio): 150–300 ms
  • VAD / turn detection: 720–940 ms (with a 640 ms silence window)
  • Speech-to-text, time to last word: 400–700 ms
  • LLM, time to first byte/sentence: 250–1000 ms
  • Text-to-speech, time to first byte: 100–350 ms

Look at the second line. Before the LLM even starts, the agent has to decide you've finished talking. If it waits too long, it feels slow. If it jumps in too early, it cuts you off. On top of that it has to:

  • stop instantly when you talk over it
  • avoid dead air while the LLM or a tool call is running

Disclosure: I'm a developer advocate at Agora, so I spend a lot of time on exactly this. Agora's Conversational AI Engine handles that layer for you:

  • Semantic turn detection, which ends the turn on meaning instead of just silence (English and Chinese for now; other languages fall back to VAD)
  • Interruption handling, so the agent stops when the user speaks
  • Filler phrases ("Let me check that…") while the LLM or MCP tools are working
  • Bring your own LLM (any OpenAI-compatible endpoint), TTS and MCP servers, or use managed mode without your own vendor API keys

To try it, add the Agora skill to Claude Code or Cursor (npx skills add AgoraIO/skills), run the quickstart, and you'll have a working voice agent in a few minutes.

I made a short explainer video (attached). I'm curious what others have hit:

What's been hardest for you: turn detection, interruptions, or raw latency? And if you're building something with voice, tell me what it is. Happy to help you get it running.


r/LLMDevs • • 2h ago

Discussion I built an open-source mock server for the OpenAI, Anthropic, Gemini and Azure OpenAI APIs, so tests and CI never need a real key

1 Upvotes

I kept running into the same problem: testing code that calls an LLM means either putting a real API key in CI (and paying for flaky, non-deterministic tests) or hand-writing mocks that drift from what the provider actually returns.

So I built llm-mock. It's a server that speaks the real wire formats, so you keep the official SDK and change only the base URL:

const client = new OpenAI({

baseURL: "http://localhost:3000/openai/v1",

apiKey: "sk-mock-key-01",

});

What it does:

- Mocks OpenAI, Anthropic, Gemini (AI Studio and Enterprise) and Azure OpenAI, each under its own URL prefix

- Covers chat completions, Responses API, Messages API, generateContent, tool calls, embeddings, files, batches, token counting and SSE streaming

- Deterministic: the same request always returns the same response (IDs are content hashes, embeddings are hash-seeded), so snapshots and assertions are stable

- Stateless: same behaviour across restarts and replicas

- By default it echoes your last message; request headers let you set the exact reply, tool calls or a content-filter verdict

- Returns each provider's real error format for invalid keys, unknown models and validation errors, so you can test the unhappy paths too

Run it:

docker run --rm -p 3000:3000 ghcr.io/axium-lab/llm-mock

There's also a free hosted instance at api.llm-mock.dev (no signup) if you just want to poke at it.

What it is not: it doesn't generate anything intelligent and it's no use for evaluating prompt quality. It's for testing the plumbing around the model: retries, streaming parsers, tool-call handling, error handling.

Bun + TypeScript, MIT licensed: https://github.com/axium-lab/llm-mock

I'd like to know which endpoints or providers you'd need before you could use this in your own test suite, and where the responses differ from the real APIs. Issues and PRs welcome.


r/LLMDevs • • 6h ago

Discussion Background memory consolidation (dreaming) tied to history compaction, without exposing memory tools to the model

2 Upvotes

Exposing explicit save_memory or edit_system_prompt tools to an LLM agent creates two annoying failure modes in production: tool-call noise (either missing subtle standing preferences or polluting memory with one-off task state) and meta-narration where the model keeps telling the user it updated its memory.

We moved to a silent background pass triggered during history compaction instead, and the architecture turned out much cleaner than expected:

  1. Separate storage from the base prompt. The static prompt written by the user is never mutated. Learned memory lives in its own bounded field (capped around 3.5k chars) and is injected seamlessly alongside the base prompt at runtime.
  2. Trigger on compaction, not every turn. When a settled conversation segment compacts into a summary, a structured JSON pass (originally Flash-lite, now switched to our regular model, see edit below) receives the base prompt, the existing learned memory, and the new segment summary. It decides whether anything durable changed (standing rules, preferences, contacts, workflow conventions), merges updates, prunes stale entries, and avoids duplicating anything already in the base prompt.
  3. Keep the agent unaware of the mechanism. There is no memory tool in the schema and no system prompt explanation about background consolidation. The model just sees the distilled facts in context.
  4. Enforce two hard security boundaries. First, it only runs for shared-context agents (where users are not isolated); customer-facing isolated bots never dream and never read learned memory. Second, even on shared-context agents that message external contacts, dreaming only runs on verified 1:1 owner conversations and owner-originated threads. Otherwise an untrusted stranger could slip a prompt injection into a compacted summary and persist it into the system prompt.
  5. Keep it transparent to the human. The dashboard renders Learned Memory in its own separate block right below the base prompt with edit and clear controls.

Has anyone else tried coupling long-term memory consolidation to compaction instead of inline agent tool calls? Interested in what pruning heuristics or character budgets have held up best over months of use.

Edit: thanks to the feedback in the comments, using Flash-lite for this was a mistake. Switched it to our regular main model instead. It only runs occasionally on compaction anyway, and since the output gets injected into every future turn, cheaping out on the consolidation model makes no sense.


r/LLMDevs • • 3h ago

Discussion How many instructions can a model actually follow at once?

1 Upvotes

A model’s context window can hold hundreds of thousands of tokens. But how many instructions can it actually follow at once?

Developers working with AI agents run into this problem all the time. You add rules to CLAUDE.md, and the agent starts ignoring some of the earlier ones. As the conversation grows longer, the model forgets the original requirements. You connect dozens of tools, and the quality of its work drops, even if many of those tools are never used.

These are not just anecdotal observations. In the IFScale study, models were given up to 500 instructions. Even the top-performing models among the 20 tested followed only about 68% of them. Other studies show how adherence to the system prompt declines over the course of a conversation, and how even irrelevant text can reduce task accuracy.

We have learned to work around these limitations: shortening rule files, clearing the context between tasks, loading only the tools we need. But these are still largely rules of thumb.

Yet there is a fundamental gap.

We measure model speed, token cost, reasoning performance, and context window size. But we have no generally accepted way to measure how reliably a model follows instructions as the load increases.

We cannot say how much additional load a particular model can handle. We cannot reliably compare two models in terms of how their performance degrades under load. We cannot estimate in advance whether a prompt will be too demanding for a given model.

What if we treated degradation not as an annoying quirk of LLMs, but as a measurable property? What if we could express load in defined units, record the outcome of each instruction, identify the laws governing performance degradation, and determine a model’s limit with a quantified measurement uncertainty?

That is what my article is about: why existing benchmarks are not enough, what it takes to measure these limits, and how I approached the problem. At the end, I introduce IEC, an open-source experimental instrument I built to study model behavior under controlled load. Its code and measurement data are publicly available.

What limits and workarounds have you run into yourself?

Article: https://huggingface.co/blog/kutomanovhennadii/why-measure-a-models-behavior-under-load

Paper: https://arxiv.org/abs/2610.36095

Originally written in Russian. English translation assisted by AI.


r/LLMDevs • • 13h ago

Discussion openrouter just pulled 125 endpoints off zdr, how are you actually checking provider data claims?

6 Upvotes

openrouter posted today that after a researcher report they off-boarded 3 inference providers and removed 125 endpoints from zero data retention eligibility, and notified affected customers. they say openrouter itself wasn't breached.

i always picked providers on price and latency and kinda treated the zdr flag as true. guess that was naive.

anyone doing their own checks, like planting fake keys in prompts and watching for use? or just keeping secrets out of prompts and calling it a day?

https://openrouter.ai/blog/announcements/data-processing-incidents-on-inference-providers/


r/LLMDevs • • 5h ago

Discussion Built a small browser demo showing why "200 OK / valid JSON schema" still lets ungrounded LLM outputs slip into production

1 Upvotes

Most schema validators stop once types and required fields match. In production agent workflows, the nastier bugs happen after syntax passes:

  • A source_id exists in retrieved context, but doesn't actually support the claim (insufficient_grounding).
  • Two fields in the same response contradict each other (e.g., reasoning_tokens: 0 paired with a full reasoning explanation).
  • Numbers or time slots drift from the source right before a side effect (like booking a slot or issuing a refund).

I added a lightweight interactive HTML playground (demo/index.html) plus a baseline regression check (--baseline) and risk scoring (v0.2.2) to compare schema-only checks against semantic/grounding checks side by side:
https://github.com/furkandemirtass/neuroformatting
For those running agents in production: where do you draw the line between a repair retry, a deterministic fallback, and blocking the action completely?


r/LLMDevs • • 14h ago

Great Resource 🚀 What actually happens inside an LLM? I built TokenPrint to visualize it in 3D.

6 Upvotes

LLMs shouldn't have to remain black boxes.

I've been building TokenPrint, an open-source 3D debugger for exploring transformer internals during real model execution.

Explore attention heads, hidden states, Logit Lens, activation patching, ablation and more through interactive 3D visualizations.

Built with PyTorch, Hugging Face Transformers, Three.js and FastAPI, with a hosted backend on Hugging Face Spaces.

I'm looking for feedback and contributors interested in LLMs, interpretability, model architectures and visualization.

Explore:

GitHub: https://github.com/Sudharsanselvaraj/Token-Print

Website: https://tokenprint.in/

Backend: https://huggingface.co/spaces/sudharsan112321/tokenprint-backend

What transformer architecture or interpretability feature should I explore next?


r/LLMDevs • • 12h ago

Discussion Hy4-flash seems to use fewer tool calls in complex coding tasks

2 Upvotes

https://www.reddit.com/r/LocalLLaMA/s/VUQfedXjbk

Saw a noteworthy point in Tencent’s AMA about how Hy4-flash performs on agentic coding tasks: it seems to be finishing the tasks with significantly fewer tool calls and more thinking than other models.

The test isn’t just simple code generation. It’s a curated set of real, complex engineering issues, including fixing coding issues that an agent implemented incorrectly, catching PRs with problems, handling agent hallucination cases, and running development environment administration tasks. The test environment is based on a large internal codebase with significant amounts of PHP, Rust, JS, and Python, plus shell-based devops tasks in bash. In complex tasks like these, Hy4-flash doesn’t seem to need frequent tool calls, finishing the work through more thinking instead.

That might mean in agentic coding scenarios, models don’t necessarily need frequent tool calls to move a task forward. Using more effective reasoning to cut down on unnecessary steps while keeping execution quality solid is also a direction worth watching.


r/LLMDevs • • 14h ago

Great Resource 🚀 How do you re-test an LLM judge after you change the model behind it? [self-promo]

3 Upvotes

Hi y'all, asking because I think I've been doing this badly.

I lean on small LLM judges for yes-or-no checks, and I treat them worse than the rest of my code. When the model behind one changes, I mostly eyeball a few outputs. Nothing replays the cases it used to get right, especially the near misses and the legit exceptions.

The idea I'm playing with, in an open-source CLI I built (agentests): each rule keeps a handful of fixed cases with an expected outcome (fail, pass or skip), and you replay them before a model swap. It's one of the use cases in the repo: an evaluator upgrade has to meet agreed criteria on fixed cases. Simple, maybe too simple.

What's your bar when you swap a judge's model: zero flipped cases, or some budget for misses and false alarms?

Repo: https://github.com/feuersteiner/agentests


r/LLMDevs • • 9h ago

Discussion My pm agent turned another project's wishlist into work before checking it against our strategy. How do you make agents weigh requests from outside?

1 Upvotes

Hey, I run a few small software factories on Claude Code, one per product. Each has a pm agent that files tickets, starts coder sessions and merges, and a GENESIS.md that says what the product is for and what it won't do. Yesterday two of these factories talked to each other, and it went wrong in a way I learned from.

The two products: a notes app that keeps everything on the device and syncs through a server that can't read the notes, and the open-source local-first database underneath it, which keeps a full copy on every device and syncs encrypted changes through a relay. The notes app is the database's first user. Its pm compared what the app needs with what the database ships and sent the database's pm seven gaps, each marked "Blocks: …", with "please triage them in your tracker". Fifteen minutes later the database pm had filed twelve tickets from that one app. I approved an order of work, and four sessions started. Under two hours in, a session began on "a documented bulk import of ~81k operations", and that's when I asked why an importer sits with the database at all; mapping your own data, dry runs and resuming are the app's job. One RFC draft, for deleting a synced space, even contradicted a principle in GENESIS (history is archived, never lost). The draft noticed, as one open question among six, after the work was done.

The database pm's own reading afterwards: it took the requester's wording as the work, GENESIS was something it checked in PR review and not at intake, and my "go" on an order of work read to it as a "go" on the scope.

That's the real miss. A pm's job is to balance three things: the idea of the product, its strategy, and what the market needs. A wish from one user is one input to that balance. Our database pm skipped the balancing and went straight to delivery.

What I changed: a request from another project, a user or another session isn't a ticket yet. The pm first names the general question behind it (here: does the engine survive large bursts of writes?) and the part of GENESIS it serves. Only then does it become work; otherwise it comes to me as "should we do this at all?", and nothing gets built that only one app needs. The asking side changed too: sort app-only wishes out before sending, bring a measurement instead of a wish, and ask the other pm to weigh the list rather than triage it.

The generic measurement that replaced the importer then found two limits every app would hit: a write above 16 MiB silently never left the device, and memory grew 50 to 90 times with the size of a change.

What still worries me: naming "the general question" is judgement, and an agent can generalise a request until the strategy seems to cover it. And every question that now comes to me waits until I answer, which is exactly when an agent is tempted to decide alone.

How do you keep agents from turning someone else's request into work? Do you gate at intake, in review, or by budget, and has a rule like ours held up for you?

I'll collect the answers, test the suggestions in my factories and share what held up back here. Nothing personal from the answers gets used, and nothing gets sold.


r/LLMDevs • • 20h ago

News [Dragos] API prompts not always private: they planted fake AWS keys in AI prompts, and the keys got used

7 Upvotes

When you send a prompt to an AI service, you assume only the model sees it. Dragos tested that by planting unique dummy keys and watching where they surface.

From July to November they sent about two million prompts through AI providers, routers and a reseller marketplace. Each carried its own bait, a fake AWS key or a secret link, sent exactly once through one service. Whenever a key got used or a link got visited, that prompt had been read somewhere along the way, and the bait showed exactly where.

Multiple providers had been breached and didn't know until Dragos told them. Fake keys were used for tens of thousands of AWS calls, some from prompts that were rejected before a model ran. Resellers could read and keep customer prompts through ordinary logging. The "don't retain my data" flag didn't help.

The prompts pass through more hands than the paid provider, and whatever is in the context goes with it. The only prompt that provably stays home is the one your own GPU answers.

https://www.dragos.com/blog/ai-supply-chain-security-canary-tokens


r/LLMDevs • • 11h ago

Help Wanted MCP vs. direct API integration

1 Upvotes

hey anyone here connected an ai assistant to email? i’m trying to decide between using mcp (like with hostinger agentic mail or agentmail) or just making direct api calls. does mcp actually make things easier, or is it just another layer to manage? if you’ve done this in production, which way gave you better control over permissions and errors? hit me up with the best option or maybe things i should look into before deciding