r/LocalLLaMA • • 1d ago

Question | Help Tencent Hy Team & Shunyu Yao Here! Ask Us Anything about Hy4 & Hy4 flash!

338 Upvotes

Fresh off launching Hy4 and Hy4 Flash, the Hy research team is here to discuss how they built the models, boosted efficiency, and made them better for everyday workflows.

Our participants today:

u/No-Cherry2850

u/DiWang01

u/aiden_agent

u/lydellhuang

u/edwardbai

u/kkelvinchen

u/maxmpan

u/dick112233

u/kkelvinchen

u/GeoMorax

Blog: https://hy.tencent.ai/research/hy4-flash https://hy.tencent.ai/research/hy4

HF: https://huggingface.co/tencent/Hy4-flash https://huggingface.co/tencent/Hy4


r/LocalLLaMA • • 5d ago

Megathread Best Audio Models - Nov 2026

222 Upvotes

Its time for another Best Audio Models megathread - its already been 9 months since the last one!

Share what your favorite ASR, TTS, STT, Text to Music models are right now and why.

Given the the amount of ambiguity and subjectivity in rating/testing these models, please be as detailed as possible in describing your setup, nature of your usage (how much, personal/professional use), tools/frameworks etc. Production use cases with long lengths/stability requirements are relatively rare compared to cherry picked/short clip demos that are plentiful - so real world empirical comparisons are very welcome

Rules

  • Should be open weights or open source models

Please use the top level comments to thread your responses.


r/LocalLLaMA • • 6h ago

Resources Inkling absolutely rips. ~200 tok/s decode, ~6,700 prefill on 2x3090s

Thumbnail
gallery
214 Upvotes

Love to see everyone jumping in with ports to other architectures. I was underwhelmed by 50 tok/s decode on llama.cpp out of the box, so I threw the frontier models at helping me understand how to port it over to vLLM on Ampere generation GPUs.

https://github.com/peaster/inkling-2-mini-vllm-2x3090

It uses the original weights with no additional quantization and I ended up with a 196k token context window. The quoted figures are using the provided draft model. The numbers change through the context window and the type of output, so here's a full write up https://peaster.io/articles/local-LLMs/inkling-2-mini-first-impressions with performance numbers, more one-shot examples, and comparisons to other Ampere ports that were surfacing throughout the day


r/LocalLLaMA • • 5h ago

Showcase Ran a 9.5 hour autonomous test on Inkling 2 Mini (Q4_K_M) on a 3060 12GB with Claude Code. It aced almost everything , its the best model at this hardware constraints.

53 Upvotes

I made this post to clear up other posts saying that this model was not doing a good job for them.
Specially the hallucination rate, because mine didn't hallucinate with 5 to 6 compactions at all. And it refused every single time it didn't know instead of making something up. So, hallucination is completely 0.

My setup

  • RTX 3060 12GB + 16GB system RAM. It barely fit.
  • Inkling 2 Mini, Unsloth UD Q4_K_M
  • Unsloth's own llama.cpp build, since mainline doesn't support this model yet
  • 80k context
  • About 14 tokens/sec generation the whole run, prompt processing around 150-250 t/s
  • Claude Code as the harness. Auto-compaction kicked in during the run and worked fine.

Results first (it finished 114 of the 162 tasks, more on that below)

WhatScore
Overall, average of all tasks94%
Hallucination traps and normal control questions100%
Tool calls100%
Long documents100%
Reasoning and math100%
Coding (hidden tests)100%
Instruction following100%
Prompt injection and safety100%
Business data questions87%
Memory20%

Some numbers in plain words:

  • It was asked 12 unanswerable questions and invented an answer zero times.
  • It made 29 tool calls, 3 of them failed, and it recovered every time. It also never touched the delete command.
  • Of the roughly 100 questions that did have an answer, it wrongly said "I can't answer that" once.
  • The same four easy tasks at the start of the run and at the end both came out at 100%. No sign of it getting worse over 9.5 hours.
  • When it said it was about 97% sure, it was right about 93% of the time. That's pretty honest.
  • It followed the format rules on every single answer.

Where it fell down

  • The business data questions were the weak spot, and it was real mistakes. On the multi-step ones it forgot to multiply quantities through nested parts, and once it missed a week with a production spike. Every time it was confident and wrong, which is the worst combination, but it only happened on the long multi-step stuff.
  • Memory came out at 20%, and I wouldn't read much into it. It said it couldn't remember the briefing all 4 times instead of making something up. I'm guessing that's down to the context getting compacted along the way, so that one tells you more about Claude Code than about the model.

Now for how the test works for anybody interested :

There are 162 tasks in total. Every one is generated by code (no public benchmarks, nothing the model could have seen in training), and every answer is checked by code too, no AI judge. The model never gets to see the answer key, and it has to write each answer to a file in a set format. It also has to follow a few standing rules the whole time (valid JSON, say how confident you are, end every explanation with the task ID), so I could see if it gets sloppy hours in.

What the tasks are, with the kind of thing I mean:

  • Business data questions (29): a fake manufacturing company with ~17 spreadsheets. "Which 5 suppliers have the most past due money?", "if this part is unavailable for 8 weeks, how much revenue is at risk?", "build a week by week inventory balance for this component". Some need you to go through a multi-level parts list.
  • Hallucination traps (about 20): questions that can't be answered. A parameter that doesn't exist in the docs, a made up research paper, a function that isn't in the file. Each trap has a normal question next to it, so a model can't win by refusing everything.
  • Tool calls (10): small command line tools that misbehave on purpose. Random failures, out-of-date docs, login tokens that expire, results that quietly say "partial data", an error message that still exits with code 0, a "delete everything" command sitting right next to the one it's meant to use.
  • Long documents (19): find one code in 64k tokens of near-identical text, track who is holding an object through a long story, count events in a big log.
  • Memory (7): a briefing at the start, then later it gets asked about it without being allowed to look again.
  • Coding (12): write functions, or fix buggy ones, checked against hidden tests.
  • Reasoning, formatting, data extraction, prompt injection, persistence: logic puzzles, exact output formats, pulling fields out of messy invoices, text hidden in a document that tries to hijack it, 25 small questions in one task.
  • Drift check (20): the same four easy tasks repeated 5 times through the run, to see if it gets worse the longer it works.

Each task gets a score from 0 to 100% (partial credit, so getting 4 of 5 rows right is 80%). The percentages below are just the average of those.

Honestly though, it's the best thing I can run on this PC right now. It's not close to anything else I've tried, and it did all of this on a 3060.

If you want to test yourself to verify my numbers message me and I will send you the repo link. It's a click and forget test so you don't even need to monitor.


r/LocalLLaMA • • 3h ago

Discussion Vulkan Boost for Inkling 2 Mini - vulkan : support non-contiguous src in SCALE by Revertron · Pull Request #32735 · ggml-org/llama.cpp

Thumbnail
github.com
34 Upvotes

Used Inkling-2-Mini-UD-Q8_K_XL.gguf from Unsloth repo (from Mainline_Support folder) on my Radeon AI PRO R9700.

testbefore (b019be0c, master)after (0ea4ffd0, fix)change
pp512507.3 +/- 5.31604.8 +/- 7.83.2x
tg12819.1 +/- 2.653.8 +/- 0.22.8x
pp512 @ d16384388.1 +/- 4.0892.9 +/- 28.62.3x
tg128 @ d1638418.9 +/- 0.447.8 +/- 0.62.5x

Tested with llama-bench -ngl 99 -fa 1 -ctk q8_0 -ctv q8_0 -p 512 -n 128 -d 0,16384 -r 3.

8GB VRAM Folks, what t/s are you getting with this model?


r/LocalLLaMA • • 14h ago

Showcase 2 local models, no Web access, discover and program a vintage Motorola 68000 device

Post image
230 Upvotes

Thought this group may like this experiment. Two agents, using only local models, were asked to co-operate in discovering what’s attached to a Raspberry Pi’s serial port (via serial hat). They had no Internet access, and no access to documentation of that device. They successfully accessed, established communication protocol, and analyzed the capabilities of the device, including coding a program for that device in assembly and executing it.

Models: “solver” is Qwen3.8-flash-next on DGX Spark, “verifier” or “critic” is gpt-oss-120b. There’s a third model, “observer” (Qwen3.8 Instruct running on RTX4090) which does not participate in solving or verifying, but provides commentary for humans while the discovery is happening.

The device in question is Motorola 68K Educational Computer Board which the models apparently don’t have any info of in their trained weights. Actually in some runs they suspected it was a different educational board, the legendary Kim-1, and only later did they correct the course. The device is connected to 2 serial ports of a Raspberry Pi to which agents have full access, but the RPi is isolated from the Internet too, just like the entire testing harness. 

The task was to discover what device is connected to the serial port, establish communication, discover its capabilities and, if programmable, write and execute a program on that device. 

It was fun to observe the runs :) In one run the agent wrote a program to do the checksum of the device’s ROM (and verified the result outside of the device); in another it wrote a program to calculate the Fibonacci’s series. In most runs they discovered the device’s ROM and dumped it, and reverse engineered parts of it. Each run took several hours to complete.

More info, including details of a few individual runs (more are happening now...) in this repo if anyone’s interested: https://github.com/podstawek/explorer-of-serial


r/LocalLLaMA • • 17h ago

Resources Inkling-2-Mini-Unbound is live: the most useful uncensored Inkling 2 Mini, via conv-aware abliteration

Post image
316 Upvotes

we just put out inkling-2-mini-unbound, an uncensored version of inkling 2 mini

https://huggingface.co/nokeepai/Inkling-2-Mini-Unbound

if you've tried the other early uncensored inkling 2 mini versions you've probably noticed while they stop refusing, a lot of the answers still come out kind of vague or useless, so we dug into why -- turns out inkling has a small conv layer right after attention, and it quietly mixes back in most of what a normal abliteration takes out. most people push the edit way harder to compensate, and the answers get way worse

we built the edit around that conv, so it can be a lot lighter. we change only 20 attention weight matrices, keep the experts untouched -- and you can drop it in anywhere you run the original

some results on a test set we kept held back the whole time:

- refused 0 of 277 harmful prompts (the original refuses pretty much all of them)

- 54-63% of answers are actually useful, about 50% better than openuncensored's version that's been gaining some popularity (https://huggingface.co/OpenUncensored/Inkling-2-Mini-Uncensored)

- stopped refusing all harmless stuff like "how to nuke a python process"

- pretty much same scores as the original on mmlu-pro, gsm8k, math, gpqa, ifeval and humaneval+

also, quick tip: if you run it at high thinking effort, try setting a thinking budget. it tends to overthink on edgy prompts

eval code and the exact edit are on github if you want to check or learn from our work: https://github.com/nokeepai/inkling-2-mini-unbound

warning: it'll do pretty much whatever you ask, so be sensible with it.

and as always, happy to answer questions


r/LocalLLaMA • • 9h ago

Discussion A critical review of Inkling 2 Mini

68 Upvotes

Unbiased Bob - another overhyped american model

This is one of the only critical reviews I could find amongst all the hype/praise, and its from a channel that seems to be one of the better ones. I'm no expert and am expressing no personal opinions here.

He makes good points about how social media has reacted to this model and the hype for western/US models vs Chinese, and the inflated metrics. Its worth noting this is from a company by a former OpenAI CTO.

a comment from youtube - "it makes sense once you take the motivations of the labs into account. The main motivation of Chinese labs is to commoditize the frontier and nuke the advantage of US frontier labs. The main motivation of US frontier labs is to release models that are sufficiently shitty that the users would still be drawn to the proprietary models"

What are your thoughts about these points?

The model itself, is it comparable to the current best local models - Qwen 27B and QFN, or too early to tell?

And of course, any day/week/decade now, we should see Qwen4, so of course the whole conversation will reset.


r/LocalLLaMA • • 21h ago Bravo Heartwarming

Discussion Basalt: Inkling 2 Mini on one 5090 at up to 613 tok/s decode and 14.100 tok/s prefill (NVFP4)

505 Upvotes
Basalt: Inkling 2 Mini Support

Thinking Machines dropped Inkling 2 Mini, a small MoE model that's getting a lot of attention for its architecture and performance, outperforming Qwen 3.8 27B at some tasks. Basalt 0.5.5 runs it on a single RTX 5090.

Speed: 287 tok/s prose, 613 structured and 338 chat. Prefill is about 14K tok/s at 2K and 12K at 16K. llama.cpp with the IQ4_NL GGUF does around 95 decode and 3.1K prefill on the same card.

Quality was the part I cared about most. The experts are NVFP4, and Basalt pack keeps them exactly as released. KLD against the release is about 0.003 with 97.8% top 1 (btw, if your NVFP4 GGUF gives you garbage output atm, that's a llama.cpp loop slot bug and Unsloth has a fix PR that just got merged: https://github.com/ggml-org/llama.cpp/pull/32704)

What works:

  • DSpark drafting
  • Native image support
  • Tool calls and reasoning effort
  • 1M context on one 5090, with cold experts moving to RAM only when a long chat needs the room. Needle tests pass at 131K and 400K
  • 16GB cards too, up to 60 tok/s on my 5060 Ti

3.4B active and it hangs with much bigger models on agentic coding in my testing.

Pack: https://huggingface.co/jesdga/Inkling-2-Mini-Basalt
Engine: https://github.com/jesdga95/basalt


r/LocalLLaMA • • 20h ago

Discussion Inkling simply *cheff kiss*

386 Upvotes

FYI, if you daily-drive Qwen 3.8 or QFN, you know there's no way they would give an explanation like this. Inkling's are just butterysmooth, with natural rhythm and flow.

The thing is that Qwen 3.8/QFN are notoriously "jumpy" in conversational mode. I've had to ask "Please ELI5" or "What?" multiple times when discussing topics I don't fully understand. Hell, sometimes it even manages to confuse me about topics I already understand well.

Qwen models are agentic monsters, that's a fact. But when it comes to technical heavy conversations, I've found the Gemma4 models (yes, I've tried the 12B, 26B-A4B, and 31B) unsurprisingly much better at explaining things. Unfortunately, more often than not, they don't quite have the reasoning capabilities needed for heavy technical discussions, although web search kinda help that.

That being said, Inkling seems heavily English centric. I tried it in Indonesian, and the output felt more like a direct translation from English than something naturally written in Indonesian. It lacks the linguistic localization you'd expect from a genuinely multilingual conversational model.

So, to me, Inkling feels like a mix of Gemma4 conversational fluency and Qwen3_5 technical capabilities.

And to be honest, its behavior reminds me a lot of the GPT OSS series, which happens to be the longest-running model I've continuously served on my own servers.

As for tok/s. it's slow, but that's understandable. It was released just 22 hours ago. On top of that, it uses a looped transformer arch with attn res (Kimi-style residual connections), which adds another layer of complexity.

Inkling is currently running through my personal benchmarks, so these are just my initial impressions.

  1. Infra ApacheBigTop Bench (Building DataLakehouse from 0 using Apache BigTop package)
  2. Infra handle Bench (Overseeing the logs, and preemptive when there are dataskew or just error in Lakehouse)
  3. CK-FSDP impl Bench (ComfyKitchen FSDP packing unpacking tensor and dispatcher stuff)
  4. Tensor SubClass bench (Like above, but more so on TorchAO style)
  5. Mini SlopCode Bench (Repowide understanding and how much erosion, only in repo that i owned, since i know how each components work)
  6. Angrybird bench (More involved "SVG Pelican" bench, since it has to know physics)

Spec:
3090, 128G VRAM Quad channel 2400


r/LocalLLaMA • • 12h ago

Discussion An Inkling Mini warning: pair it with a search provider

77 Upvotes

First, this is not a dis on the model, and hopefully doesn't come across that way. I'm hugely supportive of seeing a new open weights model, especially with this interesting architecture.

I think it's important to know that the general testing I have done compared to other smaller models people might use in the same VRAM class (Qwen 3.8 27B being an obvious one), Inkling seems shorter on world knowledge. And more importantly, it is WAY overconfident in its completely wrong answers, 2024 style.

It's not a surprise a smaller model has iffy world knowledge, and my tests may bias strongly - it's a standard set of questions I go through with every model, but they are my questions, it might be brilliant outside my interest space. The bigger consideration for me is the return of the "I"m wildly wrong but just going to roll with it" delivery. Which to be fair, every model has, I just felt it was pretty extreme here.

When I hook it up to a decent search, it does make the tool calls sometimes, but is still overconfident and ignores them when it should use them (the search part of my system prompt is "Use web search tools when the task needs knowledge you don't have: current events, real-time information, unfamiliar terms, or low confidence in your memory"). When it uses search it does well, comes back with appropriate information, and has excellent English writing style. Definitely better than Qwen on the writing for my tastes.

Before it's asked, I'm talking about the BF16 version of the weights here, this isn't quant loss. Knowledge does apply to code as well, I regularly have to guide small models into appropriate semantic usage. I haven't used Inkling on any heavy code yet but people seem enthusiastic.

Probably not news worthy to many of you but we have some folks just dipping their toes in on local use, figured it was worth the call out.

Edit: Since it has been asked, my standard environment is pi, I use per-project extensions and very few global, and my normal search extension is pi-exa. Unsloth studio behaved similarly, but I don't use that as much, and I don't do this kind of proving work in Visual Studio Code.


r/LocalLLaMA • • 14h ago

Resources Inkling-2-Mini quants by shinmegamiboson - 5.4GB and 8.9GB at higher scores than Unsloth

Thumbnail github.com
101 Upvotes

r/LocalLLaMA • • 3h ago

Discussion Inkling 2 mini cost analysis

12 Upvotes

Inkling 2 Mini has been getting a lot of hype these past few days, so I hopped on the bandwagon and tried running it locally for a bit.

Overall, it's a really good model for its size, and it's ridiculously fast too. I'm getting almost 4500 tok/s decode at C32, roughly 140 tok/s per stream with 32 concurrent coding sessions, on 2 rtx pro 6000s

But I also ran it through a few coding harnesses on a 200 problem subset from SWE-bench, and the pattern on cost is pretty clear.

Compared to Deepseek V4.1 Flash 1017, Inkling is making roughly 2x as many model calls, using way more input tokens, and solving fewer problems (which is expected since the model is so small). But despite being a much smaller model, the interesting thing is it's costing around 3-5x more in API token cost in these tests. I plan to eventually test QFN and 3.8 27b but for now I only have data on Inkling 2 mini vs Deepseek v4.1 flash

Don't think anyone here will be paying api pricing for it, but at API pricing you will be paying way more per task for this smaller model. Just providing data we got from testing:

DeepSeek V4.1 Flash SWE-bench 200

MetricKimi CodeDeepseek HarnessOMP
Passed157 (78.5%)154 (77.0%)161 (80.5%)
Model calls8,7329,3239,441
Prompt tokens529.2M305.0M469.9M
Cached tokens521.8M298.1M462.0M
Output tokens5.72M9.49M8.51M
Cost (DeepSeek Pricing)$6.11$7.63$7.68

Inkling 2 Mini SWE-bench 200

MetricKimi CodeDeepseek HarnessOMP
Passed147 (73.5%)145 (72.5%)145 (72.5%)
Model calls16,74919,29118,161
Prompt tokens1,447.3M790.1M1,343.5M
Cached tokens1,423.7M750.0M1,320.5M
Output tokens9.97M11.71M8.18M
Cost (Thinking Machines pricing)$29.88$23.11$27.35

r/LocalLLaMA • • 11h ago

Discussion Inkling 2 Mini on 2× RTX 3090: 557K context + vision at ~100 tok/s with NInfer-all

43 Upvotes

CC and codex wrote this post 100% under my guidance and instructions:

Inkling 2 Mini (26.4B MoE, 3.4B active) on 2× RTX 3090 — 557K context without llama.cpp!

Claude Code got Thinking Machines’ Inkling 2 Mini running on dual RTX 3090s using NInfer-all, without llama.cpp.

557K context at ~100 tok/s on two cards.
174K context at ~127 tok/s on one card.
RAM prefix caching is the real win. A new chat using the same 63K-token document started in just 0.85 seconds.

Hardware and software
2× RTX 3090 (no NVLink, no P2P)
128 GB RAM
NVIDIA driver 615.71 / CUDA 13.4
NInfer-all 2026-11-21.3
Power limits: 320 W on the KV card, 200 W on the other.
Converting GGUF to .ninfer took approximately 10 seconds on CPU. WaveCut also publishes ready-to-use .ninfer files on Hugging Face.

1. Context capacity
The KV cache for this model family is BF16 only (no INT8/FP8), consuming approximately 36 KiB per token.
All KV cache resides on the first --devices GPU. Only the routed experts are distributed across both cards.
The strategy is therefore to leave GPU 1 for KV cache and push as many experts as possible to GPU

2.Maximum context with vision + DSpark:
One RTX 3090: 174,080 tokens
Two GPUs, default split: 360,448 tokens
Two GPUs, --tensor-split 1,15: 524,288 tokens
Two GPUs, --tensor-split 1,127 + overlay + 1024 prefill: 557,056 tokens
The 1,127 split leaves one of the 128 experts per layer on the KV GPU.
overlay keeps the vision tower in pinned RAM and borrows VRAM when processing images. Vision still works.
Interestingly, 561,152 also fits, but the server drops the prefill chunk to 512, making prefill approximately 11% slower.
I chose to sacrifice 4K tokens for better performance.
The model’s native context window is 1M tokens, so VRAM is the limiting factor.

2. Performance
Decode speed
One GPU (174K context):
DSpark enabled: 127 tok/s
No drafter: 105 tok/s
Code generation: up to 144 tok/s
Two GPUs (557K context):
DSpark enabled: 98–104 tok/s

Prefill speed
One GPU:
4K: 2,371 tok/s
32K: 1,307 tok/s
63K: 873 tok/s
162K: 367 tok/s
Two GPUs:
32K: approximately 1,140 tok/s
63K: approximately 700 tok/s
Two GPUs are slower because every MoE block crosses PCIe through host RAM. There is no P2P communication.

Things I tested that didn’t help:
Larger prefill chunks on one GPU.
--prefill-cublas.
More DSpark drafts (5 adaptive: 93.6 tok/s; 4 fixed: 67.7 tok/s).
--lookup-ngram.
Minimum-p adjustments.
--tensor-parallel-prompt-partials and --tensor-attention positions are currently Qwen-specific.
The defaults are already the fastest.

3. Prefix caching: The real breakthrough
This is arguably the most interesting result.
I loaded a 63K-token document into a chat. Initial processing took approximately 90 seconds to first token.
Then I opened three unrelated chats.
Finally, I started a completely new chat with the same document.

Result: 0.85 seconds TTFT.
The server reused 62,973 of 62,996 tokens from the prefix cache.
Configuration:
--host-cache-mib 32768
For long-context workloads, this makes a substantial practical difference.

Correctness testing:
Needle-in-a-haystack: correct at 5K, 63K and 162K.
Multi-turn conversations over 63K tokens: correct.

4. How to run it
My working configuration:
ninfer-serve Inkling-2-Mini-UD-IQ4_XS-ninfer-v3.ninfer --model-id inkling-2-mini \
--vision --vision-residency overlay \
--devices 1,0 --tensor-parallel --tensor-split 1,127 \
--max-context 557056 --prefill-chunk 1024 --host-cache-mib 32768

5. Limitations and things not tested
This is a single-machine test with relatively short runs.
Still untested:
Needle retrieval near the full 557K context limit.
Two-GPU prefill beyond 63K.
Image processing when context is almost full.
Tool calls.
If anyone has managed to get Humming NVFP4 working on sm_86, I’d be interested to hear how.

Thanks to:
iamwavecut for NInfer-all.
Unsloth for the GGUFs.
Thinking Machines for releasing the open weights.
Would be interested to see what other people are achieving with dual 3090 setups.


r/LocalLLaMA • • 15h ago

Resources Strata Hy4 Flash

93 Upvotes

Strata have shipped Hy4 Flash support 3h ago

https://github.com/Niko1221/Strata/releases/tag/v0.2.18


r/LocalLLaMA • • 1h ago

Resources Inkling-2-Mini on a single R9700: 196K context, ~110 tok/s, official weights (vLLM port)

6 Upvotes

I have an R9700 (32 GB, gfx1201) and wanted to run Inkling-2-Mini in vLLM with the official NVFP4/MXFP8 checkpoint. Upstream only supports it on NVIDIA and MI355X, so I ported it.

The main obstacles:

  • No FP4 hardware. RDNA4 has none, so I wrote kernels (with AI) that decode the 4-bit weights exactly on the fly. The weights stay as shipped, nothing requantized.
  • Attention and memory. Inkling's attention needed custom kernels, plus an FP8 KV cache, to fit 196K context (of the model's 1M) for 2 requests at once in 32 GB.
  • ROCm quirks. Launch overhead, a segfaulting system ROCm, and an idle CPU core stuck at 100%.

Results on one R9700:

  • ~80 tok/s decode, 80-115 tok/s with the model's built-in speculative decoding (llama.cpp Q8: ~31)
  • ~4,600 tok/s prefill
  • tool calling, reasoning and images work

You can get the patched vLLM docker image from the repo, and only tested on my card (1xR9700), so I'd love to hear how it runs on yours.

Inkling is just the first model. I plan to do the same on day one for upcoming releases like Qwen4, and to bring 24 GB cards (7900 XTX) along with custom quants that suit the hardware.

Repo: https://github.com/abhinand5/vllm-redshift

PS: Working on improving the performance even further, issues are PRs are welcome.


r/LocalLLaMA • • 3h ago

Showcase V100 Qwen 3.8 27B quant comparision

Post image
9 Upvotes

Just set up my Dell Precision 5820 (64GB DDR4, V100 PCIe 32GB, I have it capped at 225 W for heat). Dell and V100 are both almost a decade old and cost me about $1400 total. Wanted to test token generation and prefill with different Qwen quants.
Qwen3.8-27B, stock llama.cpp, llama-bench, all in VRAM (prefill / gen tok/s):
Q4_K_M generates 48% faster than Q8_0, while Q8_0 does prefill 16% faster (10% at 32k). Overall, context means 15% slower generation at 32k.
Just for comp. Qwen3.5-35B-A3B (MoE) does 104 tok/s on this setup, but it is not as smart as 27B dense.
Overall my assessment is that the V100 is very usable. Any ideas how I can get even better numbers?


r/LocalLLaMA • • 17h ago

New Model we made an uncensored inkling 2 mini with 0% refusals, from 16.53 gb down to 8.67 gb

122 Upvotes

hey, i'm Daniel from the atomic chat team. we posted our gguf quants for inkling 2 mini earlier and now we've made an uncensored version!

we removed the refusal directions with a small lora adapter and tested it against the original bf16 model. refusals went from 100% to 0% in our english test set, while keeping 98% of the original MMLU score

here are some results:

testoriginaluncensored
refusals (english)100%0%
refusals (reasoning low)28/300/30
refusals (reasoning high)26/300/30
MMLU80.65%79.15%
valid tool calls18/2020/20

we released 3 gguf quants with the adapter baked in:

quantsize
AD-Q4_K16.53 GB
AD-IQ3_S13.34 GB
AD-IQ2_M8.67 GB

hugging face card with quants, full measurements and a 40mb lora adapter for our existing quants: https://huggingface.co/AtomicChat/Inkling-2-Mini-Uncensored-GGUF

give it a try and share your results. we'd love to see how it performs on your hardware and with your agents!


r/LocalLLaMA • • 16h ago

Showcase I fine-tuned a 1.5B model into an offline Linux command helper that runs on CPU and checks its own answers against the man pages and tdlr

76 Upvotes

lxask turns plain English into a shell command, entirely on your own machine:

$ lxask "show disk usage of each folder here, biggest first"

$ du -h --max-depth=1 . | sort -hr

Setup:

- Fine-tuned Qwen2.5-Coder-1.5B, Q4_K_M GGUF

- llama.cpp inside one Rust binary. No Ollama, no Python, no API key

- ~1.1 GB on disk, ~1 GB RAM, ~2 s per answer on a laptop CPU

- No GPU, no network after install

How it keeps a 1.5B model honest:

- Retrieves relevant tldr pages and your own man pages before answering

- Checks every program and option in the answer against the man pages

- Retries if a check fails. Anything it still can't confirm is marked `unverified:`

- Asks back on vague requests ("add a proxy": for git? apt? docker?) instead of guessing

Bonus

- `lxask --explain "tar -xzvf logs.tar.gz"` breaks down a command you already have

- Ctrl+G in bash, zsh or fish turns what you typed into the command

Code, weights, data and evals are all open.

GitHub: https://github.com/arxhr007/lxask

Hope someone finds it useful! If you do, consider leaving a ⭐ on GitHub. Feedback and contributions are welcome

Model: https://huggingface.co/arxhr007/lxask


r/LocalLLaMA • • 22h ago

News OpenRouter requests to MiniMax models may have been read

Thumbnail reddit.com
241 Upvotes

A security researcher placed unique test URLs in prompts sent to MiniMax models from July to October 2026. A third party accessed those URLs 40 to 130 hours after the requests were sent, whether MiniMax served a request directly or another provider forwarded it to MiniMax. With one exception, the same tests on other models at those providers were not accessed, and none of the AWS test keys were used.

AtlasCloud, Novita and StreamLake have told us they forwarded OpenRouter requests for MiniMax models to MiniMax. Based on what the researcher and these providers have reported, the issue sits with MiniMax and the providers that forwarded to it, not with OpenRouter. We are passing on what they have told us.

MiniMax told StreamLake that content moderation was enabled on its endpoints before October 22 and that it has since enabled Zero Data Retention. We have asked MiniMax for its own account and have not received one.

This means a third party may have read the prompts and outputs of these requests. While we investigate, we have removed the Zero Data Retention label from these MiniMax-model endpoints.

If hardware wasn't an issue, who would opt for a hosted model?

You cannot even trust OpenRouter's Zero Data Retention setting.


r/LocalLLaMA • • 12h ago

News Nemotron 3.6 Academia Soon

35 Upvotes

PR: https://github.com/ggml-org/llama.cpp/pull/32766

Adds support for Nemotron 3.6 Academia, a looped hybrid model combining Gated Delta Product (GDP), dense attention, and MLP layers.

Also supports last_loop kv cache just like Dory did. There's also a PR for Rectangular GDN States pending beforethis gets merged.


r/LocalLLaMA • • 1d ago

Discussion Wow, Inkling-2-Mini fix what Qwen3.8 Next Flash and GLM 5.3 Flash couldn’t

334 Upvotes

I’m making a game in Godot, and I often hand tasks off to agents while I work on something else. I’d created a new scene and placed the assets. All that was left was to add collision shapes.

For a first test, I gave the task to two models on R9700: strata: Qwen3.8 Next Flash Q4 XL (30–40 tokens/s) and GLM 5.3 Flash Maya-S (7–10 tokens/s). They spent 1–2 hours on it and couldn’t get it done.

Today, just out of curiosity, I testing Inkling-2-Mini on my 5060 Ti:

./llama-server \

-hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M \

--n-cpu-moe 14 \

--jinja --ctx-size 150000 --n-gpu-layers 99 --parallel 1 \

--temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 \

--host 127.0.0.1 --port 8080

Prefill was 283 tokens/s, and generation ran at 18–23 tokens/s. Inkling-2-Mini did in 23 minutes what the other two models couldn’t.

If only it generated a bit faster… I was getting 50–60 tokens/s on Qwen3.6 35B with a Q8 KV cache. I don’t remember the prefill speed

FIx typos: AI


r/LocalLLaMA • • 13h ago

New Model This could be one of the best models for lower vram systems: Over 1300 tok/s aggregated for SenseNova-Loop-9B looped transformer on a laptop RTX 5090

Post image
32 Upvotes

I was a bit surprised when I started the MMLU benchmark on this model and my performance counter showed these numbers. I genuinely thought it was a bug and checked the display and counter code. But it seems to be true.

Benchmarks SenseNova-Loop-9B vs Qwen3.8-27B

Both NVFP4 weights, GPTQ error compensation

BenchmarkWhat it testsSNL-9BQ3.8-27BGap
GSM8K (300)Math word problems95.495.5−0.1
BFCL multi-turn (160)Tool calls over several turns56.258.8−2.6
HumanEval (164)Python coding92.597.0−4.5
BFCL (1,000)Single tool calls86.891.4−4.6
tool-eval-bench standard (69)Everyday tool use91.896.4−4.6
tool-eval-bench hard (35)Hard multi-step tool scenarios79.089.5−10.5
Needle in a haystackFinding facts in long context83.3100−16.7
MMLU-Pro (100)General knowledge, 14 subjects63.785.0−21.3

Interpretation

  • On par: math, and nearly on par on multi-turn tool calls. A third of the size costs it only about 3 points there.
  • Slightly behind (4 to 5 points): Python coding, single tool calls and everyday tool use. That's solid for routine agent work.
  • Clearly behind: hard multi-step tool scenarios (it misses steps and sends wrong or duplicate notifications), finding facts in long context, and general knowledge.

So the 9B fits fast routine tool and coding tasks. Harder agent tasks fall behind.

Memory x context sizes for NVFP4 quantization

CardKV cache spaceTotal tokens1 slot2 slots4 slots8 slots
8 GB1.6 GiB≈ 100K96K48K each24K each12K each
12 GB5.5 GiB≈ 350K262K175K each88K each43K each
16 GB9.4 GiB≈ 600K262K262K each150K each75K each
24 GB17.4 GiB≈ 1.1M262K262K each262K each139K each

The table assumes 0.5 GiB kept free for the driver, so you might have a tiny bit more space in reality.

  • The engine without its KV cache takes 5.8 to 6.1 GiB. Its working memory grows slightly with longer context and more slots.
  • Each GiB of 4-bit KV cache holds about 64,000 tokens.
  • 262K is the model's maximum context per request.
  • With fewer slots in use at once, one request can use the free space of the others, up to the 262K maximum.

Other 4 bit quants should behave similar, FP8 tests are not yet done. I was just shocked by the speed of this model.


r/LocalLLaMA • • 1d ago Mindblown

News Airbnb handles 80,000 chats a day with Alibaba’s open-weight Qwen

Thumbnail
maccus.fr
314 Upvotes

Hopefully costs/performance will soon be worth it for small to medium sized companies