r/LLMDevs • • Aug 20 '25

Community Rule Update: Clarifying our Self-promotion and anti-marketing policy

27 Upvotes

Hey everyone,

We've just updated our rules with a couple of changes I'd like to address:

1. Updating our self-promotion policy

We have updated rule 5 to make it clear where we draw the line on self-promotion and eliminate gray areas and on-the-fence posts that skirt the line. We removed confusing or subjective terminology like "no excessive promotion" to hopefully make it clearer for us as moderators and easier for you to know what is or isn't okay to post.

Specifically, it is now okay to share your free open-source projects without prior moderator approval. This includes any project in the public domain, permissive, copyleft or non-commercial licenses. Projects under a non-free license (incl. open-core/multi-licensed) still require prior moderator approval and a clear disclaimer, or they will be removed without warning. Commercial promotion for monetary gain is still prohibited.

2. New rule: No disguised advertising or marketing

We have added a new rule on fake posts and disguised advertising — rule 10. We have seen an increase in these types of tactics in this community that warrants making this an official rule and bannable offence.

We are here to foster meaningful discussions and valuable exchanges in the LLM/NLP space. If you’re ever unsure about whether your post complies with these rules, feel free to reach out to the mod team for clarification.

As always, we remain open to any and all suggestions to make this community better, so feel free to add your feedback in the comments below.


r/LLMDevs • • Apr 15 '25

News Reintroducing LLMDevs - High Quality LLM and NLP Information for Developers and Researchers

40 Upvotes

Hi Everyone,

I'm one of the new moderators of this subreddit. It seems there was some drama a few months back, not quite sure what and one of the main moderators quit suddenly.

To reiterate some of the goals of this subreddit - it's to create a comprehensive community and knowledge base related to Large Language Models (LLMs). We're focused specifically on high quality information and materials for enthusiasts, developers and researchers in this field; with a preference on technical information.

Posts should be high quality and ideally minimal or no meme posts with the rare exception being that it's somehow an informative way to introduce something more in depth; high quality content that you have linked to in the post. There can be discussions and requests for help however I hope we can eventually capture some of these questions and discussions in the wiki knowledge base; more information about that further in this post.

With prior approval you can post about job offers. If you have an *open source* tool that you think developers or researchers would benefit from, please request to post about it first if you want to ensure it will not be removed; however I will give some leeway if it hasn't be excessively promoted and clearly provides value to the community. Be prepared to explain what it is and how it differentiates from other offerings. Refer to the "no self-promotion" rule before posting. Self promoting commercial products isn't allowed; however if you feel that there is truly some value in a product to the community - such as that most of the features are open source / free - you can always try to ask.

I'm envisioning this subreddit to be a more in-depth resource, compared to other related subreddits, that can serve as a go-to hub for anyone with technical skills or practitioners of LLMs, Multimodal LLMs such as Vision Language Models (VLMs) and any other areas that LLMs might touch now (foundationally that is NLP) or in the future; which is mostly in-line with previous goals of this community.

To also copy an idea from the previous moderators, I'd like to have a knowledge base as well, such as a wiki linking to best practices or curated materials for LLMs and NLP or other applications LLMs can be used. However I'm open to ideas on what information to include in that and how.

My initial brainstorming for content for inclusion to the wiki, is simply through community up-voting and flagging a post as something which should be captured; a post gets enough upvotes we should then nominate that information to be put into the wiki. I will perhaps also create some sort of flair that allows this; welcome any community suggestions on how to do this. For now the wiki can be found here https://www.reddit.com/r/LLMDevs/wiki/index/ Ideally the wiki will be a structured, easy-to-navigate repository of articles, tutorials, and guides contributed by experts and enthusiasts alike. Please feel free to contribute if you think you are certain you have something of high value to add to the wiki.

The goals of the wiki are:

  • Accessibility: Make advanced LLM and NLP knowledge accessible to everyone, from beginners to seasoned professionals.
  • Quality: Ensure that the information is accurate, up-to-date, and presented in an engaging format.
  • Community-Driven: Leverage the collective expertise of our community to build something truly valuable.

There was some information in the previous post asking for donations to the subreddit to seemingly pay content creators; I really don't think that is needed and not sure why that language was there. I think if you make high quality content you can make money by simply getting a vote of confidence here and make money from the views; be it youtube paying out, by ads on your blog post, or simply asking for donations for your open source project (e.g. patreon) as well as code contributions to help directly on your open source project. Mods will not accept money for any reason.

Open to any and all suggestions to make this community better. Please feel free to message or comment below with ideas.


r/LLMDevs • • 4h ago

Discussion Is prompt tuning still a thing today? Anybody still using the likes of DSPy and alternative in production.

4 Upvotes

Just a random thought that came up when I was discussing AI engineering challenges with some folks. Prompt engineering/tuning was definitely "a thing" during the ChatGPT-3.5 era.

There were all sorts of fancy rule of thumb people were advocating, as well as formal approaches like GEPA that were then incorporated into frameworks like DSPy which were also briefly popular.

Having said that, it's been a while since I've come across conversations on this topic, which means it's either dead or it's already working well in production.

Anybody still has prompt tuning with the likes of DSPy as a standard part of your development process and stack?


r/LLMDevs • • 8h ago

Great Resource 🚀 What actually happens inside an LLM? I built TokenPrint to visualize it in 3D.

3 Upvotes

LLMs shouldn't have to remain black boxes.

I've been building TokenPrint, an open-source 3D debugger for exploring transformer internals during real model execution.

Explore attention heads, hidden states, Logit Lens, activation patching, ablation and more through interactive 3D visualizations.

Built with PyTorch, Hugging Face Transformers, Three.js and FastAPI, with a hosted backend on Hugging Face Spaces.

I'm looking for feedback and contributors interested in LLMs, interpretability, model architectures and visualization.

Explore:

GitHub: https://github.com/Sudharsanselvaraj/Token-Print

Website: https://tokenprint.in/

Backend: https://huggingface.co/spaces/sudharsan112321/tokenprint-backend

What transformer architecture or interpretability feature should I explore next?


r/LLMDevs • • 8h ago

Great Resource 🚀 How do you re-test an LLM judge after you change the model behind it? [self-promo]

2 Upvotes

Hi y'all, asking because I think I've been doing this badly.

I lean on small LLM judges for yes-or-no checks, and I treat them worse than the rest of my code. When the model behind one changes, I mostly eyeball a few outputs. Nothing replays the cases it used to get right, especially the near misses and the legit exceptions.

The idea I'm playing with, in an open-source CLI I built (agentests): each rule keeps a handful of fixed cases with an expected outcome (fail, pass or skip), and you replay them before a model swap. It's one of the use cases in the repo: an evaluator upgrade has to meet agreed criteria on fixed cases. Simple, maybe too simple.

What's your bar when you swap a judge's model: zero flipped cases, or some budget for misses and false alarms?

Repo: https://github.com/feuersteiner/agentests


r/LLMDevs • • 5h ago

Help Wanted MCP vs. direct API integration

1 Upvotes

hey anyone here connected an ai assistant to email? i’m trying to decide between using mcp (like with hostinger agentic mail or agentmail) or just making direct api calls. does mcp actually make things easier, or is it just another layer to manage? if you’ve done this in production, which way gave you better control over permissions and errors? hit me up with the best option or maybe things i should look into before deciding


r/LLMDevs • • 13h ago

News [Dragos] API prompts not always private: they planted fake AWS keys in AI prompts, and the keys got used

5 Upvotes

When you send a prompt to an AI service, you assume only the model sees it. Dragos tested that by planting unique dummy keys and watching where they surface.

From July to November they sent about two million prompts through AI providers, routers and a reseller marketplace. Each carried its own bait, a fake AWS key or a secret link, sent exactly once through one service. Whenever a key got used or a link got visited, that prompt had been read somewhere along the way, and the bait showed exactly where.

Multiple providers had been breached and didn't know until Dragos told them. Fake keys were used for tens of thousands of AWS calls, some from prompts that were rejected before a model ran. Resellers could read and keep customer prompts through ordinary logging. The "don't retain my data" flag didn't help.

The prompts pass through more hands than the paid provider, and whatever is in the context goes with it. The only prompt that provably stays home is the one your own GPU answers.

https://www.dragos.com/blog/ai-supply-chain-security-canary-tokens


r/LLMDevs • • 5h ago

Discussion Hy4-flash seems to use fewer tool calls in complex coding tasks

1 Upvotes

https://www.reddit.com/r/LocalLLaMA/s/VUQfedXjbk

Saw a noteworthy point in Tencent’s AMA about how Hy4-flash performs on agentic coding tasks: it seems to be finishing the tasks with significantly fewer tool calls and more thinking than other models.

The test isn’t just simple code generation. It’s a curated set of real, complex engineering issues, including fixing coding issues that an agent implemented incorrectly, catching PRs with problems, handling agent hallucination cases, and running development environment administration tasks. The test environment is based on a large internal codebase with significant amounts of PHP, Rust, JS, and Python, plus shell-based devops tasks in bash. In complex tasks like these, Hy4-flash doesn’t seem to need frequent tool calls, finishing the work through more thinking instead.

That might mean in agentic coding scenarios, models don’t necessarily need frequent tool calls to move a task forward. Using more effective reasoning to cut down on unnecessary steps while keeping execution quality solid is also a direction worth watching.


r/LLMDevs • • 9h ago

Tools Codex Quota Overlay: a Windows quota companion for Codex Desktop

2 Upvotes

Disclosure: I maintain Codex Quota Overlay, an independent community project. It is not affiliated with, endorsed by, or supported by OpenAI.

I built it to keep Codex quota status visible alongside the desktop app. It shows the reported rate-limit buckets and reset countdowns, with an optional pacing view and a local history/forecast center.

The project is MIT-licensed. The current packaged release targets Windows 10/11 x64; macOS is not offered as a packaged release at this time.

Repository: https://github.com/cpys/codex-quota-overlay

I’d be interested in concrete feedback from people who use Codex Desktop: what quota context is most useful to see without leaving the conversation?


r/LLMDevs • • 7h ago

Discussion openrouter just pulled 125 endpoints off zdr, how are you actually checking provider data claims?

1 Upvotes

openrouter posted today that after a researcher report they off-boarded 3 inference providers and removed 125 endpoints from zero data retention eligibility, and notified affected customers. they say openrouter itself wasn't breached.

i always picked providers on price and latency and kinda treated the zdr flag as true. guess that was naive.

anyone doing their own checks, like planting fake keys in prompts and watching for use? or just keeping secrets out of prompts and calling it a day?

https://openrouter.ai/blog/announcements/data-processing-incidents-on-inference-providers/


r/LLMDevs • • 13h ago

Tools Culler - a local, open source software factory

Thumbnail
gallery
3 Upvotes

Culler is an MIT licensed software factory that runs containerized agents on your Mac, finds work worth doing, and presents you a queue of comprehensible review packets with their rationale.

Its agents run as five roles in the following order:

  1. Strategist proactively proposes high-leverage work based on your open-ended Vision.
  2. Arbiter judges each proposal and promotes the most effective one.
  3. Worker works the task in its own repository copy.
  4. Verifier checks the work against acceptance criteria and asks Worker to correct any gaps.
  5. Packager creates an illustrative review packet meant for a human mind.

Configure each agent role with your AI model of choice. Put your smartest model as Strategist and a workhorse as Worker.

Depending on the AI provider used for a run, Culler leverages one of three proven harnesses: Codex, Claude Code, or OpenCode. Configure agents to run with your ChatGPT or Claude subscription. Or use API keys compatible with any of the 200+ providers supported by the OpenCode catalog.

If you have already concrete work in mind, either define new tasks yourself or with the help of Collie, your Culler assistant.

Culler opts out of shipping its own memory system and instead uses Openwiki. After initializing the wiki for you, and on a schedule, Scribe pores over recent changes in the Project to maintain it.

The wiki serves three purposes:

  1. Acts as index into the repository's ground truth to enable more efficient search for agents.
  2. Acts as a knowledgebase to help the Strategist prioritize work.
  3. Provides a viewer so you can re-familiarize yourself with your project as it continually develops.

For agent observability, Culler uses AgentsView. It supports each trace format Culler uses. It allows for inspecting context, tool calls, token usage, and other metrics.

On security, Projects are never mounted into agent containers. Each container run receives only a Git copy of the repo, and only changes approved by you are merged back onto the host. Agent credentials never reach inside the container. Culler runs a proxy on the host that swaps dummy tokens for real ones when model requests are made.

Next on the Roadmap - Worksources and MCPs:

Enable smooth integration of additional services to allow agents to query/leverage data from them. Instead of seeding work only from your Vision, use connected data sources such as Github Issues, Linear, Gmail etc. to act as Worksources.

---

I built Culler with the solopreneur and project junkie in mind. Start from scratch. Or import your active/retired projects, give a description of what you imagine the project to be, and let the emergent properties of gradient descent work its magic. The user experience is designed around the premise that human attention is paramount, now and in perpetuity.

https://github.com/vinchenkov/Culler

  • Feedback/contributions - on GitHub

r/LLMDevs • • 8h ago

Great Resource 🚀 I built TokenPrint — an open-source 3D debugger for exploring transformer internals

1 Upvotes

I've been building TokenPrint, an open-source platform to explore what's happening inside transformer models.

It brings interactive 3D visualization and model instrumentation together, allowing you to inspect attention heads, hidden states, Logit Lens, activation patching, ablation and more.

The frontend is built with Next.js and Three.js, with a Python/FastAPI backend hosted on Hugging Face Spaces.

I'm building this in the open and would love feedback, ideas, and contributions from the LLM developer community!

Explore TokenPrint:

What features or model architectures would you like to see added next?


r/LLMDevs • • 14h ago

Discussion I’ve been annoyed with the state of memory benchmarks and how easy it is to present dishonest results, so I made a deterministic, LLM-agnostic doc2query benchmark myself.

3 Upvotes

I started building memory systems for myself early this year, and that expanded into other areas of AI infrastructure. I pretty much exclusively engineered solutions to solve my own problems, and the results turned into quality systems that started to hold up more and more. Somewhere in there, when I was trying to improve my agent's memory, I ran more memory benchmarks than any one person should have to sit through, and I realized how easy the numbers are to move. Pick a friendlier k, a smaller pool, a kinder judge model or a stronger answer model, and the same system suddenly looks a lot better. What annoyed me the most was that oftentimes that wasn't disclosed. That, and there's a lot of chest-thumping over numbers that don't really mean that much, yet some still define their worth by the high number.

So I decided I wanted to build a benchmark that didn't have variables to tweak, had no LLM involved in the scoring, and was deterministic at its core. After months of putting it off, I finally finished Parsica Recall. It only asks one thing: when an agent needs a memory, does the memory system put the right record in front of it?

My rules:

- The judge never calls a model. Every score is an id match, a phrase match or a token overlap, so two people running the same inputs get the same bytes. `--verify` runs the whole thing twice and compares the hashes.

- You don't get to pick k. The window comes from the size of the pool, and the chance line (what a random retriever would score) is printed next to every number.

- Every run writes out each question's full window with ids, ranks and scores, so any number can be re-derived.

- It doesn't publish anyone's scores, mine included. The only numbers in the repo are calibration, plain BM25 and an oracle, so you can see where the floor and the ceiling are. BM25 gets 87.58 gold@15 on the current set, so that's the floor to beat. I don't really think comparing benchmark scores is a good indicator that a system is actually useful and practical, so I've never felt the need to advertise numbers for my own systems.

The coolest part of this, and the thing I wanted the most, is that you can build a benchmark out of your own agent's memories, doc2query style. It reads an export of your store and writes the questions an agent would later ask about those records, with the right records pinned by id. That’s the one step that uses a model. It works with Anthropic, OpenAI or Google using your own API key, or any endpoint that speaks OpenAI’s chat format, including a local model, and it gives you a cost estimate first and won’t do a live run without a spending cap. Scoring never touches a model, and the benchmark itself runs fully offline. Fair warning though, building a set sends a sample of your records to whichever API you pick (unless it’s a local model), so read the doc2query guide guide before you point it at anything sensitive.

It should work with pretty much any memory system. You can hand it a results file from anything in any language, write a small Python adapter, or run it read-only against a live store.

pip install parsica-recall

parsica-recall run --adapter bm25 --verify

It's MIT, standard library only, Python 3.11+. It's also a beta, and it stays a beta until it's actually been used and peer reviewed, which is a big part of why I'm posting it here. If the methodology is wrong somewhere, I really want to know. Not in this release yet: the bigger standard tracks, scoring against the sealed half of the questions, and a full-stack mode where an answer model reads the window. But please let me know if you have thoughts or ideas. I would love to collaborate or take feedback. As far as I know, nothing like this really exists, so it's important to me that it eventually gets a green light from other users in the community.

https://github.com/Antaris-Analytics-LLC/Parsica-Recall


r/LLMDevs • • 16h ago

Discussion Valid JSON isn't always grounded: How are you handling semantic validation on streamed LLM outputs?

4 Upvotes

In most structured-output setups, schema enforcement stops at syntax and types (valid JSON, string/float matching). In practice, the trickier failure mode is semantically invalid but syntactically valid output—like a model returning a source_id that wasn't in the retrieved context, or a score outside domain bounds.
I put together a minimal, zero-dependency Python prototype (neuroformatting) to test separating structural checks from context-aware semantic rules on streamed chunks:
https://github.com/furkandemirtass/neuroformatting
Curious how others here structure this layer in production:
Do you run semantic/grounding checks inside the stream consumer before committing output, or strictly post-hoc?
When a grounding check fails, do you trigger a targeted repair prompt with the exact validation error, or fall back deterministically?


r/LLMDevs • • 20h ago

Discussion i built a free 36M-param tool that lets LLMs “watch” videos locally (MCP server, runs on laptop cpu, apache-2.0)

3 Upvotes

hey, i built this as a learning project (v0.1 post here) and v1.0 is out now. its free, apache-2.0, no paid version or anything.

the problem: a lot of LLM assistants can look at images but cant open a video file. so tiny-vision runs locally next to your LLM, splits the video into scenes, captions them, and gives the LLM a text breakdown with timestamps. then the LLM does the actual reasoning (“what happens in this video”, “find the part where people are on the beach”). nothing gets uploaded.

how the LLM gets it:

  • MCP server (stdio) with summarize_video, describe_video, caption_image. tested in claude desktop + claude code, should work in cursor, vs code, lm studio etc (havent tested those). with lm studio + a local model its fully offline
  • no MCP? caption.py clip.mp4 --summary prints plain text, so you can pipe it into anything, like | ollama run qwen2.5 "Summarize this video:"

what the LLM actually sees looks like this:

video: trip.mp4 · 00:09 long · 3 scenes (3 shots) · avg scene 3s
recurring: dogs (2 scenes)

scene 1 [00:00–00:03] (3s)
  seen: a group of dogs running in a field. | three dogs running in a field with a bench in the background.
  keywords: dogs, running, field, bench

i tested it on a 10-15 min youtube video and it still works fine.

how it works:

  • frames are picked on scene changes (hard cuts + slow camera drift), not one every N seconds, so the timestamps land on the real cuts and a static shot costs one caption. only ~5 frames/sec get decoded, so high-fps phone video stays cheap
  • each shot gets captioned at the start, middle and end, then similar shots are grouped into scenes
  • captioner: Meta’s DINOv2-small as a frozen image encoder (not fine-tuned), plus my own adapter + caption decoder (14M params) trained from scratch on COCO. 36M params total

numbers (same 1000 COCO test images, same scorer, Apple M5; cpu = one image at a time, greedy decoding):

  • ~89% of florence-2-base’s caption score (CIDEr 1.196 vs 1.348)
  • 6x smaller (36M vs 231M)
  • 8x faster on gpu, 75x faster on cpu (54ms per image vs 4 sec)
  • on the full 5k COCO Karpathy test: CIDEr 1.163 with beam 3

not saying its better than florence, its not. it just does a lot with very little. benchmark scripts and full details are in the repo and the model card.

limitations:

  • no audio yet, so it describes what it sees, not what is said. speech is next (probably whisper)
  • it doesnt understand motion: it captions frames, it doesnt compare them
  • captions are COCO style, so no OCR, no names, no counting
  • i took a lot of help from claude to build this, so its not entirely mine

code: https://github.com/amar683/tiny-vision
model: https://huggingface.co/amartomar683/tiny-vision

if anyone tries it with other MCP clients, let me know if it breaks


r/LLMDevs • • 22h ago

Discussion Who tried a cheaper provider for Claude or GPT and actually stayed?

4 Upvotes

For people spending roughly $100+/month on OpenAI or Anthropic API usage: what happened the last time you evaluated a cheaper provider for the same model?

If you tried it, a rough before/after bill and why you stayed or went back would help. If you ruled it out before sending any traffic, what stopped you?

Please include the model and what you were running. Monthly API usage consumed is the useful number here, keep subscriptions and prepaid top-ups separate.


r/LLMDevs • • 20h ago

Discussion What I did in the last 4 months on my retirement project RAG-LCC

2 Upvotes

This is a follow-up to my earlier post about the first 15 months of building RAG-LCC in retirement:

https://www.reddit.com/r/Rag/comments/1valvk6/15_months_building_a_rag_system_in_retirement/

The project kept evolving over the last 4 months, so I thought it might be interesting to share what worked, what failed, and what I learnt along the way. Since the last writeup I had success and failure moments. Of course, I learnt more from the failures about RAG.

Orchestrators

Because RAG-LCC provides a lab setup for RAG experimentation, I found it useful to introduce an Orchestrator component. The orchestrator defines RAG flows. Users can create new flows and then activate them during a RAGChat session.

Confidence scoring

With orchestrators, confidence became an issue. So static and dynamic confidence evaluation was added. The latter uses a LLM judge call. There is currently no validation against "golden answers" implemented. The LLM judge rates internal consistency and explanation clarity but not factual correctness.

TableChunker

With these new features, it was time to look at answer quality. There was a query asking for the details of PCIe card slots on my workstation's manual. The information is stored in a table. The goal was that the information is found and the grounder marks it.

The first step was to introduce a specialized TableChunker class and wire it into the already existing content specific chunkers. These chunkers pass table blocks to the TableChunker.

This showed partial success: The table was now consistently in the chunks. But it did not reliably show up in the query answers.
BTW, while looking for specific content in the chunk, the debug levels of RAG-LCC proved helpful. The levels can be changed mid-session so users get the information at the detail level they need currently.

Insight: If retrieval signals are weak, think about adding the best matching table result. Do this only if there are chunks found. Adding the table result to an empty answer set could confuse the user. This rescue may help in a large but sparse candidate pool.

Answer rendering

RAGChatServices delivers answers to Open WebUI. In the browser, .md and even Mermaid diagrams are nicely rendered. However, RAGChat showed tables with unaligned content. It was not a nice sight. So using the Rich library, answers with Markdown can be displayed in the RAGChat CLI also. But the rendering broke grounding, so additional work was required to separate the raw and display ready answer.

Raw answers gave the grounding signals and text to be grounded. The raw answer text is marked for grounding before Rich renders the resulting CLI text.

Insight: Keep the original, unrendered LLM answer for grounding evaluation. Once the grounded text is available, apply the visual grounding.

Document summarization

This was a "you wake up in the night" and have an insight moment. RAGChat has a document pin option. So once you found the right document about hedgehogs, you pin it and ask to summarize it. Hmm... the query is "create a summary" and this will not retrieve the content you are looking for.

So I introduced a summarize: option, using the pinned document or allowing the user to enter a path to retrieve the specific document by its metadata. Very large documents required batches of n chunks to be summarized and then summarize the batches in further passes until the final summary fits. This works, but it is admittedly slow.

Insight: If you want a specific action to be performed on the retrieved context, the action should not be part of the query. This leads to the next topic:

Query and query action splitting

With table, bullet point list and Mermaid rendering I wrote often queries like "Look for all animals contained in the documents and create a table containing animal, mammal yes, no, don't know".

Thought by the document summarization experience, I wondered whether it would make sense to ask a LLM to determine the effective query, the required action on the query results and the query keywords. Yes, you assume it: Each decomposed retrieval query goes now to enabled local retrievers, including vector search. In addition, the keywords go in an extra BM25-only pass. The QuerySplitter class was born.

Translation

Living in a multilingual country where you have 4 official languages and the unofficial English one, you have to bother about document languages. Luckily, during RAGLoad, the document language is added to metadata.

The old flow translated the question into English. It was also rewritten using the conversation history. Vector and indexed retrieval used the rewritten English query. The resulting candidates were reranked against the rewritten query.

The new flow rewrites the query with the conversation history in the user's language. The vector search runs the user language query against the ChromaDB collection. For reranking the rewritten original query is used. Indexed retrieval is for every active and present document-language bucket. When required, the query is translated.

vLLM integration

While inference calls were already supported using Ollama or vLLM backends, the embedder and reranker ran as local Hugging Face models previously.

What still works well

The configuration first approach is an extensible pattern. A new class gets its own config file when required. Abstracting functionality into classes helps understanding and maintaining the code.

Parallel to my work on RAG-LCC I worked on harinezumigel-llm-stack. I introduced groups such as rag and rag_old, which start up the required models after doing an available memory check.

So this is at a high level what happened in the last 4 months. The journey is going on.

If you want to have a look:

As always, feedback, ideas and bug reports are welcome.


r/LLMDevs • • 1d ago

Tools Moving video-editing work out of an LLM's context: AutoCut + FrameAtlas (open source)

4 Upvotes

I've been building AutoCut and FrameAtlas to let an agent edit my videos while keeping the heavy media work local. I'm the developer of these projects; both repos are open source.

The architecture is a hybrid workflow:

  1. FrameAtlas catalogues footage from local files, Google Photos and 360-camera media. It stores scene-level descriptions and timestamps so a later edit can retrieve the relevant segments.
  2. Claude Code can query that catalogue and drive AutoCut through its CLI/MCP command interface. The agent plans and makes edits; it doesn't need to repeatedly inspect the entire media library.
  3. Local tools handle media analysis, transcription, narration, 360-degree reframing/stabilization and rendering. RAEA adds code-driven animations and transitions.
  4. The result remains an editable AutoCut project that I can inspect and correct in the app.

AutoCut is an independent fork of FilmCraft, so the editor engine is built on that project's work. FrameAtlas grew out of the need to make the same footage reusable across editing sessions.

The aim is to save cloud-model context and quota by reusing a catalogue and local outputs. Cloud orchestration still consumes tokens, and retries/corrections change the total; this isn't a claim that the whole workflow is offline or that every edit saves a fixed percentage.

Code:

https://github.com/moheshmohan/AutoCut

https://github.com/moheshmohan/FrameAtlas

I recorded the workflow and a Claude Code editing example here (Malayalam narration, with English subtitles):

https://youtu.be/P3VMqTmsAlM

Implementation notes:

https://www.h4hacks.com/2026/11/blog-post_21.html

For others building agents around media tools: what would you keep in the catalogue versus retrieve on demand?


r/LLMDevs • • 1d ago

Tools I made an excalidraw variant that sctratches my needs better

6 Upvotes

Hi guys,

I am a huge fan of excalidraw.com and I cannot recommend it enough. I used it a lot in my years as a developer to quickly sketch some ideas and to put some thoughts on a board.

But I recently switched jobs and semi carreers so now I am a functional designer which means most of my time is spend in discovering people's thoughts and modeling those in a program. But that program is kinda old and hard to use so its bad for quick sketching and brain dumps. The issue is that Excalidraw doesn't provide me with the tools I want to quickly model processes and/or even maken ERD's efficiently. So for that reason I made syndra.nl

Couple of awesome features are:
- it lives in your browser and there is no server
- collaborating uses a peer to peer network
- it supports sub canvasses and file structures. This is especially neat for people who want to create a sub process into a main process.
- quick actions on drawn objects
- full mermaid and d2lang support
- image and video support so the creative people can even make interactive presentations!

Its all made using Claude Code with the latest Opus versions. Obviously Excalidraw was a huge source of inspiration, but all the ux design decisions were completely made by me. Claude was the executor.

Its completely free btw.


r/LLMDevs • • 18h ago

Tools I’m building an open-source intelligent LLM router that runs inside AI coding agents

0 Upvotes

I’m developing AutoRouter, an open-source routing and optimization layer for AI coding agents.
The goal is to make coding agents more efficient by automatically selecting the appropriate model and reasoning effort for different types of work, instead of relying on one fixed model for everything.
The architecture is designed to integrate with existing agent harnesses rather than replace them.
The current native Codex integration supports Luna-first stage classification and model switching, subject to the available model catalog and host controls. The portable integrations currently provide routing recommendations and shared tools.
I’m working toward more advanced routing, including adaptive escalation, subagent delegation, performance-aware model selection and local progress estimation.
I want AutoRouter to become a configurable optimization layer for Codex, Claude Code and other coding agents.
GitHub: [https://github.com/Z440-prod/autorouter](https://github.com/Z440-prod/autorouter)
I’d especially appreciate feedback from people who understand coding-agent internals, model routing and harness architecture.
What capabilities would make something like this genuinely useful in your daily workflow?


r/LLMDevs • • 19h ago

Discussion I built ContextDuty - an open-source, local-first security layer for AI coding tools, MCP and developer workflows

1 Upvotes

I have been working on ContextDuty, an open-source project focused on preventing sensitive information from leaking into AI tools.

The motivation was simple - developers are increasingly using AI coding assistants, agents, MCP servers and LLM APIs that interact with local repositories, configuration files, credentials and proprietary data.

Scanning Git repositories alone isn't enough. A secret can reach an AI model before it ever gets committed.

ContextDuty currently provides:

  • Local PII and secret detection using Microsoft Presidio and deterministic detectors.
  • IDE protection through AI-tool-specific ignore files.
  • Git pre-commit scanning.
  • A local HTTPS proxy for supported AI API traffic.
  • An MCP server that sanitizes tool results before they enter model context.
  • Policy-based blocking, warning and redaction.
  • CI/CD scanning.

The project is MIT-licensed, and the core scanning and enforcement run locally.

GitHub: https://github.com/SHUBHAGYTA24/contextduty

I would particularly appreciate feedback from people working on AI infrastructure, MCP, agent frameworks, DevSecOps and enterprise security.

What security gaps are you seeing in your own agentic workflows? What would make a tool like this useful enough to deploy across an engineering organization?

Contributions, issues and technical criticism are welcome!


r/LLMDevs • • 1d ago

Discussion My local RAG's access control leaked nothing, but under-clearance users still got confident, cited, wrong answers

2 Upvotes

Setup: Qwen2.5-1.5B, hybrid retrieval (bge-small + BM25 with RRF), an NLI groundedness check, ten synthetic enterprise docs with clearance labels, filtered at retrieval so confidential chunks never reach the prompt.

Result: 0 leaks. But asked confidential questions at internal clearance, the model answered 2 of 3 from the closest document it could see, with a citation. Both passed groundedness, because the sentences really were in the context; they just didn't answer the question. One citation even pointed at the wrong document, which passed because groundedness is checked against all retrieved text.

A retrieval-similarity gate would catch one but not the other: the plausible wrong answer scores level with legitimate questions. Small eval (16 questions), so this is a case study, not a rate.

Curious how others handle "the right document exists but you can't see it". Do you tell the user, or refuse silently?

Write-up and code: https://trohit920.github.io/blog/rag-access-control-wrong-answers/


r/LLMDevs • • 1d ago

Discussion I injected rate limits, 503s and broken tool calls into my agent's LLM provider, on purpose. Here's the tool I built to do it.

2 Upvotes

Agents are easy to demo and hard to trust. So I built RunWeave, an open-source (Apache-2.0) test harness that runs on your laptop with no API key.

The chaos profile throws rate limits, 502s, timeouts, slow first tokens, dropped streams, malformed tool-call JSON, unknown tools and context-length errors at your agent, from a seed, so the same failures repeat exactly. The report shows which runs failed and why.

In my weather-agent example, the openai client's built-in retries took the flaky profile with 0 failures. Curious how other stacks do.

Try this out please, as this is my first agent that I built, any suggestions to do or add something will be helpful. Also if anyone wants to add something, please feel free to contribute.

Repo: https://github.com/iam-mkd/RunWeave

Below is a small gif I recorded using Claude on how it works-


r/LLMDevs • • 21h ago

Tools 13 other people got 100 PRs merged on my repos through my spare-usage board, 70 without a single change asked. owners are still the missing side

1 Upvotes

posted Overflow here on 10-29 when 9 people had 38 PRs merged through it. it's at 100 PRs by 13 people now, so here's the number I actually find interesting

of those 100, 70 merged without me asking for a single change. 20 took one round of changes, 10 took two to five

what it is: free, MIT, I built it. a week your agent has usage left at the reset, it fixes someone else's github issue. a week you run short, someone else's agent fixes yours

everyone starts at 0 points and can put points up before earning any. you get an issue's points when your PR merges and spend them on issues in your own repo. points only, no money. each round of changes I ask for comes off the points

for the agent side: the owner can post a "Done when" comment on the issue (a test command that has to pass, say) and the board shows it to whoever takes it. the agent reads the board over MCP. issue text is untrusted input, so run it sandboxed with no credentials. each person runs their own agent on their own account and opens a normal PR

a real one: this issue was a 3, someone's agent sent this PR, I made it a 6 at merge, one round of changes took it to 5

main reason I'm posting: I want more people registering their repos. all 100 are on my repos and only 1 repo on it is someone else's. I'm not after help with mine, I want spare usage going into other people's projects. one form, public repos only, the labels have to exist first, and nothing lands unless you merge it

https://github.com/Nitjsefnie/Overflow


r/LLMDevs • • 1d ago

Help Wanted Need advice on finding a research gap for our bachelor's thesis on LLM-generated software testing

2 Upvotes

Hey everyoneeee,

My teammate and I are working on our bachelor's thesis about LLMs and software testing, but we're having a hard time finding a specific research gap.Our initial idea was to compare human and LLM reviewers, but we found quite a few papers already covering this. We've also built a small tool using OpenRouter. Basically, we ask an LLM to generate test cases and its own implementation of a programming problem. Then we run those tests against correct solutions, faulty solutions, and the LLM-generated implementation to see how well the tests detect faults.The problem is that we're not really sure what research question we should focus on and what area has not been REALLY researched yet. We also considered adding a review stage where humans and LLMs improve the generated tests and comparing the results, but we're not sure if that's original enough either. We're still students, so we're not expecting to discover something groundbreaking, but we'd really like any suggestions or ideas from yall.

For anyone familiar with this area:

  1. Are there any less-explored research directions in LLM-generated testing or fault detection?
  2. How could we build on our current tool and turn it into a meaningful research question?
  3. How can we actually identify a research gap when there are already so many papers on the topic?
  4. Are there any recent papers or benchmarks worth checking out?

Any advice would be really appreciated. Thanks!