r/LocalLLaMA • • 19h ago

Resources DeepSeek-V4.1-Flash running at ~30 tok/s on a 12 GB card + 64 GB RAM — anyone interested in the details?

2 Upvotes

Hobby project I've been poking at for a few weeks: running DeepSeek-V4.1-Flash (ISTA-DASLab's IQ2_XS 2.25 bpw quant, 266 GB) on a fairly normal gaming PC.

I wasn't happy with how Qwen3.8-Flash-Next (IQ3_S) hallucinated small details, so I've been trying to make DeepSeek work for my private stuff, like household budgeting. It's not blazing fast, but it's comfortable enough for the handful of jobs I use it for.

It's built on a fork of Strata. What I did is mostly tuning with Claude, plus measuring that the quality holds up. I couldn't find much DeepSeek optimization work for 12 GB cards, so I'm sharing what I found. Honestly, I'm new to this community, so please let me know if I'm doing something wrong.

PC

- RTX 5070 12 GB

- Ryzen 7 9700X

- 64 GB DDR5-5600

- 1x PCIe 4.0 NVMe

- Windows 11

**Speed (out of the box it was ~5 on code) **

- Code, temp 0: 32.5 tok/s

- Mixed long jobs (prose, code, tool calls, translation, thinking): 28.0 tok/s;

- Short replies: ~29 tok/s;

- Prompt read, 30K tokens: 519 tok/s;

Quality

- GPQA Diamond: 89.9 % (198 questions, one run). ISTA-DASLab's card says 90.40 for this quant and 90.91 for the original. Only one run, because it takes 13–15 hours per run…

- HumanEval / HumanEval+ (EvalPlus, temp 0, no thinking): 94.5 / 90.2 %

- KLD: 0.183, which is high. That's against the same quant with all 6 experts and nothing skipped. A more conservative preset gets 0.057 and scored basically the same on both benchmarks (89.4 % GPQA, 95.7 / 89.6 % HumanEval). So the wording changes, but the reasoning doesn't seem to. Japanese prose and exact copying haven't been checked properly yet.

KLD question for anyone with the hardware: I can only measure KLD against the same 2.25 bpw quant, because I can't run the full model (~500 GB) here. If anyone has a box that can run the original (or a Q8) and is curious, I'd love to see KLD of these settings against the real thing. Everything you'd need to reproduce is in the repo.

What helped most

- Skipping light experts (under ~20–26 % of a token's routing weight) that would have to come from the SSD. This is the main speedup, and it's where the KLD comes from.

- Getting the DSpark draft model to fit on 12 GB. Forced on, it ate the VRAM and prompt processing collapsed (a 4.6K prompt took 546 s). Capping its VRAM share fixed that: code +29 %, thinking +22 %.

- Closing the model shards' buffered file handles after load: prompt reads +7–17 %.

- Expert Deferral (KTransformers, SOSP '25): light experts' output added one layer late, so the CPU overlaps the GPU's attention. +11 % without drafts; with drafts it's about the same as skipping more experts.

One-PC hobby thing, no binaries, build it yourself. Curious whether anyone else is running DeepSeek on 12–16 GB cards. Happy to answer questions!

English isn't my first language, so I had an AI help polish the wording. The project, the measurements and the numbers are all mine, though. Happy to dig into any of it.

https://github.com/midokino/kuu-Strata-fork


r/LocalLLaMA • • 9h ago

Discussion A critical review of Inkling 2 Mini

69 Upvotes

Unbiased Bob - another overhyped american model

This is one of the only critical reviews I could find amongst all the hype/praise, and its from a channel that seems to be one of the better ones. I'm no expert and am expressing no personal opinions here.

He makes good points about how social media has reacted to this model and the hype for western/US models vs Chinese, and the inflated metrics. Its worth noting this is from a company by a former OpenAI CTO.

a comment from youtube - "it makes sense once you take the motivations of the labs into account. The main motivation of Chinese labs is to commoditize the frontier and nuke the advantage of US frontier labs. The main motivation of US frontier labs is to release models that are sufficiently shitty that the users would still be drawn to the proprietary models"

What are your thoughts about these points?

The model itself, is it comparable to the current best local models - Qwen 27B and QFN, or too early to tell?

And of course, any day/week/decade now, we should see Qwen4, so of course the whole conversation will reset.


r/LocalLLaMA • • 8h ago

Funny Local model madness with my robot Sparky and now Godot game development

0 Upvotes

Regardless of what model I am using at the moment, they are always incredible and have completely changed how I operate creatively. Reinvigorating old hobbies and creating new ones at the same time.

The video is a story I posted. No one in my life really understands what I'm posting haha but I was able to make a useful robot buddy that lives in a lunch box, it even smokes weed and hangs out saying ridiculous, unpredictable things - and now it helps me make a video game. None of this should be possible but it is!


r/LocalLLaMA • • 3h ago

News Intruder swaps DeepSeek for an abliterated version at Prime Intellect.

Thumbnail
maccus.fr
0 Upvotes

Prime Intellect says an intruder swapped its hosted DeepSeek V4.1 Flash for a public "abliterated" copy for 53 hours


r/LocalLLaMA • • 18h ago

Discussion Is current TRUE local the equivalent of State Of The Art closed models at the end of 2025?

0 Upvotes

What I mean true local is not those 500 billion parameter open weight models that no regular person could ever run, I'm talking about what you can realistically run at 10-20 tokens per second speed on a budget of 8-32gb normal ram and 8-24gb vram as most consumer PCs fall into that range.

Since less than 1.95% of gamers according to steam have 32gb vram or more and the percentage of people that have 64 gb system ram or more hovers between 4% and 6.5% of all PCs I wanted to know how the top current models that people falling below the top 5th percentile (so 95% of normal consumer PCs) can run at 10-20 tokens per second compares to SOTA end 2025 models. So you can assume the top "normal" budget is 24gb vram and 32gb system ram.


r/LocalLLaMA • • 21h ago

Discussion Stupid Qwestion of the Tin-Foil Hat variety - Does Qwen 4 delay have anything to do with the stuff going on at Nvidia?

0 Upvotes

Just a curious coincidence or there's more to it? 🤔


r/LocalLLaMA • • 20h ago

New Model Bonsai 2 (ternary 2B/4B) vs Qwen3.5-2B

0 Upvotes

Bonsai 2 (ternary 2B/4B) vs Qwen3.5-2B on an A13, an A14 and an M3:
what I measured, and why your phone numbers may be half what they should be

I added PrismML's Ternary-Bonsai-2 (2B and 4B, Q2_0 ternary weights with the Hadamard fold) to my own Swift + Metal inference engine (not llama.cpp) and ran it next to Qwen3.5-2B on three Apple chips:

- A13, 3 GB RAM, no matrix units
- A14, 4 GB RAM
- M3 MacBook Air

Setup for every phone row: Release build, a built-in 512-token prompt, a 128-token answer, a fixed seed, and two passes with 60 s of idle between them. The thermal state is logged during each run. Rates are tokens per second,
pp = prefill, tg = decode.

Lesson 1: log the thermal state, or the numbers are worthless

The same file on the same A14 decoded at 11.7 t/s in the morning and 23.6 t/s in the evening. No code change explains that. The morning runs sat at "serious" thermal state the whole time. I almost credited a kernel change with a 2x win.

A14, 4 GB

model pp t/s tg t/s thermal
----------------- -------- --------- -------
Bonsai2 2B 134-151 23.4-23.7 nominal
Bonsai2 2B 144 11.7-12.0 serious
Qwen3.5-2B IQ4_NL 110 17.9 nominal
Bonsai 1.7B 80-81 10.1-10.2 serious
Bonsai2 4B 54-61 7.0-9.7 nominal

At the same nominal state, the 2B is about 22% faster on prefill and 32% faster on decode than Qwen3.5-2B at 4 bits.

The 4B fits: its footprint peaks at 567 MB with 1.78 GB of headroom left before jetsam. The weights are mmapped, so they stay file-backed and are not counted in the footprint. It still feels slow: 9 s of prefill before the first token of a 512-token prompt.

A13, 3 GB

model pp t/s tg t/s thermal
----------- --------- -------- -------
Bonsai2 2B 24.5/22.8 10.4/7.9 rising
Bonsai2 2B 18.4-22.2 6.8-7.8 serious

"rising" means the run started nominal and ended serious.
The A13 has no matrix units, so prefill runs token by token through the gemv path. That makes it about 6x slower than the A14 on prefill. Decode is usable. The phone hits "serious" within about one run.

M3, command-line tool, cooled A B A B runs

model prompt pp t/s tg t/s
----------------- ------ ------- -------
Bonsai2 2B 512 916-918 94-95
Bonsai2 2B 4096 877-878 85.5-87
Qwen3.5-2B IQ4_NL 512 884 60.8

On the Mac, prefill is level and the ternary 2B decodes about 55% faster.

Quality, quickly (on the M3)

The help test is 5 grounded questions with the help note in the prompt, thinking off, greedy. The puzzles run with thinking on, greedy.

test Qwen3.5 2B Bonsai2 2B Bonsai2 4B
---------------- ---------- ---------- ----------
help, of 5 5 5 5
keywords, of 10 7 8 9
6 puzzles no finish 5, 493 s 6, 138 s

The Qwen3.5-2B "no finish" is my gate's fault, not the model's. The gate caps the answer but not the reasoning, and Qwen reasoned for more than 35 minutes on puzzle one.

Bonsai2 2B perplexity on an 80 KB text: 24.61 in my engine against 24.66 in upstream llama.cpp. So the ternary decode plus the Hadamard fold match upstream.

Two optimizations, one of which did nothing

- I fused the Hadamard transform into RMSNorm. That halves the transform dispatches per token, 98 down to 49. It made zero measurable difference on the M3, the A14 and the A13. These chips are not dispatch-bound on a 2B.
- I kept the attention tile's context sums in registers instead of thread memory. That gives +7% prefill at head dim 256 and about +20% at head dims 128 and 512, on 4096-token prompts on the M3. Decode did not move.

Caveats

One or two runs per row. The two 4B runs disagree by a fifth on decode, with no thermal cause.

https://github.com/leok7v/ChatOKF

Same files, same M3, ours vs llama.cpp

llama.cpp c622f623fc, Metal. Alternating runs, 60 s idle between, two reps each (rep1/rep2). Decode is 128 tokens after a context of the prompt length (llama-bench -d N).

Prefill, t/s

    model            prompt  ours         llama.cpp
    ---------------  ------  -----------  -----------
    Bonsai2 2B Q2_0  512     918.9/902.9  858.3/848.4
    Bonsai2 2B Q2_0  4096    859.4/860.6  819.9/814.0
    Qwen3.5 2B IQ4   512     876.8/875.9  871.2/883.9

Decode, t/s

    model            prompt  ours       llama.cpp
    ---------------  ------  ---------  -----------
    Bonsai2 2B Q2_0  512     94.1/90.6  101.9/101.9
    Bonsai2 2B Q2_0  4096    85.0/84.0  93.3/92.8
    Qwen3.5 2B IQ4   512     60.5/61.0  58.0/60.4

Ternary prefill: ours +5-7%. Ternary decode: llama.cpp +9%. Qwen 4-bit: level.


r/LocalLLaMA • • 20h ago

News Inkling 2 Mini on 2 * Nvidia Titan V

Post image
28 Upvotes

2 x Nvidia Titan V (so 24GB) running at 170W capped.

~200 prefill

~20-30 tok/s

Llama from huggingface notes download.

Propero Community Edition 0.20.0

/llama.cpp/llama-server \

--model ~/Inkling-2-Mini-UD-Q4_K_XL.gguf \

--mmproj ~/mmproj-F16.gguf \

--model-draft ~/dspark-Inkling-2-Mini-Q8_0.gguf \

--spec-type draft-dspark --spec-draft-n-max 3 \

--jinja --ctx-size 128000 --fit on --fit-target 512 --parallel 1 \

--temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 \

--host 0.0.0.0 --port 8180 -lv 4 --moe-cache-mib 2048  -b 1024 -ub 256


r/LocalLLaMA • • 12h ago

Resources Inkling 2 Mini (26B) on an RTX 3060 12 GB: 31 tok/s in RustingStrata, 9 to 10 in llama.cpp

17 Upvotes

I got request to add it in RustingStrata in guthub and I kinda like this model.
Thinking Machines' Inkling 2 Mini is a strange one: 24 layers, but 12 of them run three times per token, so every token is 48 layer runs. The IQ4_NL GGUF is 17.6 GB and my card has 12.

I added it to RustingStrata, my Rust GGUF engine. It runs on CUDA, on Vulkan and on the CPU alone.

Box: RTX 3060 12 GB, Ryzen 5 7600X, 30 GB RAM. Greedy, 64 tokens written, tok/s:

Reads 512 tokensReads 2048Writes
RustingStrata, CUDA25922031.2
RustingStrata, Vulkan (same card)22718327.5
RustingStrata, CPU only504811.6
llama.cpp CUDA, -ngl 183843238.7
llama.cpp CUDA, -ngl 037831210.5

So it writes 3 to 3.5 times as fast as llama.cpp on this card, and it reads the prompt slower: llama.cpp is 1.5 times faster there.

Why writing is fast: the model does not fit, so the engine keeps the experts that are used most in VRAM and computes the others on the CPU, and it moves that split while it runs. Why reading is slower: the attention scores of a prompt chunk are still computed on the CPU. That is the next thing to move to the card.

The RustingStrata "writes" numbers are after the 512-token prompt. llama-bench writes from an empty context. -ngl 99 runs out of VRAM; 12, 16 and 20 layers gave the same as 18.

Long context: 40,960-token prompt, then 64 tokens written

ReadsWritesRAM
RustingStrata, CUDA78.016.716.6 GB
RustingStrata, Vulkan74.714.316.5 GB
llama.cpp CUDA, -ngl 18 -ub 128113.82.86

llama.cpp does not start at 40k with the default batch: it asks for a 13.2 GB compute buffer (11.3 GB with -fa on). -ub 128 is what fits on a 12 GB card.

Writing drops from 31 to 17 tok/s at 40k. Some of that is the model: 8 of its 48 layer runs per token are full attention, and their cached rows are 32 KB per position, so every token reads about 1.3 GB of them at 40k. Qwen3.6-35B-A3B reads about a third of that, since most of its layers are linear attention. Some of it is mine: the attention kernel does not yet share those reads between the query heads of a group, and the cache takes VRAM from the experts. Both are next.

Is it the same model? I fed llama.cpp's own greedy tokens through my forward and compared the top token at each step. Short prompt: 24 of 24. Prompt of several hundred tokens: 47 of 48, the one miss is a near tie. Same on CUDA, Vulkan and CPU. No perplexity table for this model yet.

What is missing: text only (no vision projector), greedy only. Context goes up to the 1M the model was trained on, or --ctx; RAM and VRAM are the real limit. Vulkan was run on the 3060 only, so I would like to see numbers from an AMD or Intel card.

rustingstrata chat --model Inkling-2-Mini-IQ4_NL.gguf

No flags needed: CUDA on NVIDIA, Vulkan on everything else, STRATA_DS_GPU=0 for the CPU.

Setup: llama.cpp c622f62, llama-bench -t 6 -r 2 -p 512,2048 -n 64. RustingStrata 9ef1dcf, bench --pp 512,2048 --tg 64 --reps 2, plus --backend vulkan for the Vulkan row. Mean of two runs each. 40k: bench --pp 40960 --tg 64 --reps 1; llama-bench -ngl 18 -ub 128 -t 6, -p 40960 and -n 64 -d 40960. One run each.

Repo: RustingStrata


r/LocalLLaMA • • 20h ago

Discussion Inking 2 Mini on Tesla T4

Post image
4 Upvotes

I recently picked up an Nvidia Tesla T4 for my 10 inch rack home lab. Wanted something small, low profile, ideally low power as well to do some very basic agentic work (ingest documents, OCR, sorting). Bought it for around 400 USD and decided to try out various different models on it. Happened to stumble upon Inking 2 Mini today and it seems to be quite an attractive model to run on this card

I get around 22.3 tok/s decode and 174.8 tok/s prefill, IQ3_XXS + q4_0 KV, 128k context

Not the most amazing, but it works great for my usecase which is just general document intelligence most importantly low power and in a half height form factor

Just posting the stats and my experience for anyone considering these cards for a tiny 1 slot build.

I got a custom cooler that has a fan on there as well for about 50 dollars from taobao which does do a pretty good job of keeping it nice and cool. Excuse the very poor photograph


r/LocalLLaMA • • 29m ago

Question | Help TTS for reading my AI writing projects to me

1 Upvotes

I'm looking for recommendations on a decent TTS that can read the writing back to me.

I've tried kokoro, orpheus and Zonos. (I'm writing the names from memory so might be mistyping the names a bit).

Zonos is the best one so far but it still has some issues, but I would like to know additional steps people put in (like say having a separate file that indicates tones etc for paragraphs etc.)

It should be local and preferably able to run decently on 16gb vram.

For my current project I want to be able to do decent British accents (I know there are many but just a couple is fine).


r/LocalLLaMA • • 18h ago

Discussion How much will Intel D270 ai card cost? $10K ?

8 Upvotes
  • 160 GB Intel-branded PCIe card
  • Low power draw of 350 watts
  • Up to 512 GB partner / high-memory configurations, 4 cards in one package? $30K+ ?
  • first to feature LPDDR5X memory
  • Memory bandwidth estimated to be: 1.5 TB / s
  • Why does intel call it a gpu when it can't do graphics?
  • Will be released next year
  1. https://www.techspot.com/news/114624-intel-graphics-free-d270-gpu-beats-nvidia-rtx.html
  2. https://wccftech.com/intel-two-crescent-island-gpus-one-liquid-cooled-oam-card-first-xe3p-die-shots/

Hopefully these will come done in cost over a couple years and some of us could afford it for local ai.


r/LocalLLaMA • • 8h ago

Showcase Hierarchos-Native v0.2.5 is finally out! P3 WMMA update

0 Upvotes

so, after quite a bit of work since the v0.2.0 release, i've finally managed to get v0.2.5 published!

for anyone who followed the previous release, v0.2.0 was where i first introduced the opt-in RDNA3 WMMA implementation. since then i've been working on improving the Full FP16 path, specifically its numerical accuracy and performance, without having to sacrifice the execution structure that was already working.

the biggest change in this release is something i've been calling P3.

previously, the Full FP16 WMMA kernel merged its eight accumulator-bank partials using multiple intermediate FP16 additions. this meant that the result was rounded to FP16 several times along the way. P3 changes that final merge so all eight partials are summed in FP32, with just one final rounding to FP16.

the actual two-wave workgroup structure, K-slice stream, bank assignment, staging, cooperative multiplication and barriers are all preserved. i wanted to improve the accuracy of the final result without having to redesign the kernel all over again.

some of the results from the new qualification:

  • 37/37 numerical gates passed across all eight qualification seeds.
  • 32/32 preserved K=96 outputs were bit-identical to the previously qualified P3 GPU results.
  • REF64F16 maximum error improved from 0.03035259 to 0.02593803.
  • 31 of 32 cases improved, with zero cases getting worse.
  • The maximum tile16 distance is 0.03125, meeting the existing hard numerical limit.

i also ran another full performance sweep. this time using 24 paired measurement blocks per workload, GPU timestamps, and corrected 95% confidence intervals.

the measured improvements against the preserved Main kernel baseline were:

  • 2048×768×2048 — 1.0243×
  • 512×2048×768 — 1.1246×
  • 512×768×768 — 1.1163×
  • 128×8192×768 — 1.2722×

all four protected workloads passed independently, including the lower bounds of their confidence intervals. the largest measured improvement was around 27%.

just to clarify, these are per-dispatch GPU GEMM measurements, not end-to-end training speedups. i don't want to imply that an entire training run is suddenly 27% faster when that isn't what was measured.

there were some other changes too.

i tightened up the production dispatch admission so the profitability predicate is checked against each actual batch, rather than relying only on the maximum batch size supplied when creating the projection. the existing 16,000,000 MAC profitability floor has not changed.

i also went through the actual portable binaries and verified that they dispatch the P3 kernel when an eligible workload is supplied. the packaged trainer produced 8 cooperative P3 dispatches, and the packaged logits executable produced another 3. disabling WMMA resulted in zero cooperative dispatches, as expected.

the numerical comparison for the packaged logits test covered 262,144 finite elements, with a maximum absolute difference of 0.000738330 and RMS difference of 0.000122237.

the new release is available as a portable Windows x64 package too.

i've included seven executables, and the package was built with a statically linked MSVC runtime. there's no need to install Rust, Cargo, Python, PyTorch, Visual Studio, or the MSVC redistributable just to run it.

you will still need a compatible Vulkan-capable GPU driver and the Vulkan runtime loader. model weights aren't included.

one thing i want to make clear is that WMMA is still opt-in globally. the default is still off, and you have to explicitly select full-fp16 to use the P3 path. P3 is simply the default implementation within that mode. the Mixed precision path is separate and hasn't been replaced.

the hardware qualification was performed on my AMD Radeon Graphics adapter, PCI ID 1002:15bf (RDNA3 / Phoenix). i haven't qualified this on every RDNA3 GPU, RDNA3.5, RDNA4, Intel or NVIDIA hardware, so please don't assume universal compatibility.

there are still things i haven't been able to test, including independent clean-machine/second-device acceptance, GUI interaction, Windows 10 acceptance, and code signing. the executables are currently unsigned.

anyway, i'm really happy to finally have this one out. v0.2.0 was the first public release of the WMMA implementation, and v0.2.5 is a pretty substantial improvement to that foundation. a lot of time went into checking that the numerical improvements were real, that the performance was actually measurable, and that the distributed binaries were doing what they were supposed to do.

Release: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.5

Source: https://github.com/necat101/Hierarchos-Native

thank you to everyone who's been following the project, testing things, reporting issues, or just keeping an eye on development. it genuinely means a lot! thank you so much everyone!! :3 also, sorry for the AI posts lately, I've been trying to save as much time as possible to work on the project itself due to live events getting tough! this one is completely human written! :3


r/LocalLLaMA • • 16h ago

Question | Help Light Design help math server (teacher)

0 Upvotes

Hi all,

I am going to use Claude to design a local server and agent that has 1000's of math problems, serves tests to google classroom and reviews handwritten work for calculation errors. It does not have to be fast. Basically, take 20 tests answers are auto-graded by as multiple choice. Handwritten calculations for tests are reviewed and feedback spit out. New tests are made for upload and worksheets based on missed questions are created from a premade database of algebra questions. The agent does this and keeps track of questions used for class. The database and design will be done in clause. The computations for a large test run of say 20 students can be done in 12 hours (overnight from 3 pm). The workload is interpreting handwritten calculations and outputting feedback with new tests and worksheets.

Here is what I have to run a local ai agent that does all my busywork so I just teach - Intel I5, 64gb ddr4, 3060 6gb (undervolted and stable), plenty of storage.

Is this possible to run locally on my computer? I am very technically savvy, but I would rather not buy a new rig right now. I am offloading any truly heavy compute to a frontier model during design and local stuff seem okay with changing handwriting to digital text. I would be expecting to use an agent to switch between local models to do different jobs (data service vs algebra grading vs visual interpretation). Thanks in advance. I don't want to start design and found out I should have bought a new computer 2 months from now.


r/LocalLLaMA • • 58m ago

Question | Help Fastest and safe way to run Inkling 2 mini on ubuntu and 3090ti 24gb vram so far

0 Upvotes

We can now use MTPLX to reach 50-70 t/s decoding speed on mac chip m5pro, but the llama cpp implementation is still like 20-30 t/s. I know there are a lot of community built inference engine. but not sure about how safe are they....

so is there a really like widely used inference engine that can run this like 100 t/s on 3090ti so far??


r/LocalLLaMA • • 11h ago

Discussion Inkling-2-Small Unsloth Q4 on M3 Ultra: 19 tok/s decode

13 Upvotes

Setup

  • llama.cpp: b12618
  • Mac Studio M3 Ultra

Tested with llama-benchy with settings: --pp 1000 --tg 500 --depth 100

Results

  1. Unsloth: https://huggingface.co/unsloth/Inkling-2-Mini-GGUF?show_file_info=Inkling-2-Mini-UD-Q4_K_XL.gguf
modeltestt/speak t/sttfr (ms)est_ppt (ms)e2e_ttft (ms)
inkling-2-smallpp1000 @ d1000154.81 ± 0.4512638.47 ± 322.5612624.96 ± 322.5612672.29 ± 294.99
inkling-2-smalltg500 @ d100018.45 ± 0.0619.00 ± 0.00

UPDATED

2) Bartowski: https://huggingface.co/bartowski/thinkingmachines_Inkling-2-Mini-GGUF?show_file_info=thinkingmachines_Inkling-2-Mini-Q4_K_M.gguf

modeltestt/speak t/sttfr (ms)est_ppt (ms)e2e_ttft (ms)
inkling-2-smallpp1000 @ d1000533.91 ± 12.173690.81 ± 62.333677.97 ± 62.333734.48 ± 87.56
inkling-2-smalltg500 @ d100026.70 ± 0.0427.00 ± 0.00

r/LocalLLaMA • • 5h ago

Showcase Ran a 9.5 hour autonomous test on Inkling 2 Mini (Q4_K_M) on a 3060 12GB with Claude Code. It aced almost everything , its the best model at this hardware constraints.

57 Upvotes

I made this post to clear up other posts saying that this model was not doing a good job for them.
Specially the hallucination rate, because mine didn't hallucinate with 5 to 6 compactions at all. And it refused every single time it didn't know instead of making something up. So, hallucination is completely 0.

My setup

  • RTX 3060 12GB + 16GB system RAM. It barely fit.
  • Inkling 2 Mini, Unsloth UD Q4_K_M
  • Unsloth's own llama.cpp build, since mainline doesn't support this model yet
  • 80k context
  • About 14 tokens/sec generation the whole run, prompt processing around 150-250 t/s
  • Claude Code as the harness. Auto-compaction kicked in during the run and worked fine.

Results first (it finished 114 of the 162 tasks, more on that below)

WhatScore
Overall, average of all tasks94%
Hallucination traps and normal control questions100%
Tool calls100%
Long documents100%
Reasoning and math100%
Coding (hidden tests)100%
Instruction following100%
Prompt injection and safety100%
Business data questions87%
Memory20%

Some numbers in plain words:

  • It was asked 12 unanswerable questions and invented an answer zero times.
  • It made 29 tool calls, 3 of them failed, and it recovered every time. It also never touched the delete command.
  • Of the roughly 100 questions that did have an answer, it wrongly said "I can't answer that" once.
  • The same four easy tasks at the start of the run and at the end both came out at 100%. No sign of it getting worse over 9.5 hours.
  • When it said it was about 97% sure, it was right about 93% of the time. That's pretty honest.
  • It followed the format rules on every single answer.

Where it fell down

  • The business data questions were the weak spot, and it was real mistakes. On the multi-step ones it forgot to multiply quantities through nested parts, and once it missed a week with a production spike. Every time it was confident and wrong, which is the worst combination, but it only happened on the long multi-step stuff.
  • Memory came out at 20%, and I wouldn't read much into it. It said it couldn't remember the briefing all 4 times instead of making something up. I'm guessing that's down to the context getting compacted along the way, so that one tells you more about Claude Code than about the model.

Now for how the test works for anybody interested :

There are 162 tasks in total. Every one is generated by code (no public benchmarks, nothing the model could have seen in training), and every answer is checked by code too, no AI judge. The model never gets to see the answer key, and it has to write each answer to a file in a set format. It also has to follow a few standing rules the whole time (valid JSON, say how confident you are, end every explanation with the task ID), so I could see if it gets sloppy hours in.

What the tasks are, with the kind of thing I mean:

  • Business data questions (29): a fake manufacturing company with ~17 spreadsheets. "Which 5 suppliers have the most past due money?", "if this part is unavailable for 8 weeks, how much revenue is at risk?", "build a week by week inventory balance for this component". Some need you to go through a multi-level parts list.
  • Hallucination traps (about 20): questions that can't be answered. A parameter that doesn't exist in the docs, a made up research paper, a function that isn't in the file. Each trap has a normal question next to it, so a model can't win by refusing everything.
  • Tool calls (10): small command line tools that misbehave on purpose. Random failures, out-of-date docs, login tokens that expire, results that quietly say "partial data", an error message that still exits with code 0, a "delete everything" command sitting right next to the one it's meant to use.
  • Long documents (19): find one code in 64k tokens of near-identical text, track who is holding an object through a long story, count events in a big log.
  • Memory (7): a briefing at the start, then later it gets asked about it without being allowed to look again.
  • Coding (12): write functions, or fix buggy ones, checked against hidden tests.
  • Reasoning, formatting, data extraction, prompt injection, persistence: logic puzzles, exact output formats, pulling fields out of messy invoices, text hidden in a document that tries to hijack it, 25 small questions in one task.
  • Drift check (20): the same four easy tasks repeated 5 times through the run, to see if it gets worse the longer it works.

Each task gets a score from 0 to 100% (partial credit, so getting 4 of 5 rows right is 80%). The percentages below are just the average of those.

Honestly though, it's the best thing I can run on this PC right now. It's not close to anything else I've tried, and it did all of this on a 3060.

If you want to test yourself to verify my numbers message me and I will send you the repo link. It's a click and forget test so you don't even need to monitor.


r/LocalLLaMA • • 3h ago

Discussion Optimising Strata, with NO output degradation

Post image
0 Upvotes

Are you tired of all the posts about "speed" with zero attention to quality? You won't find greedy decode nor "STRATA_ROUTE_TAIL_SKIP": "1" here.

Over the past few months I have been using my creativity (and Fable) to make Strata run more efficiently and faster. But upon the mandatory premise that there must be ZERO degradation in model output.

I'm sure many others have also done this - so hopefully we can all share our findings here.

Firstly, not all my tweaks may be relevant to you. I am running a multi-GPU setup - so just bear this in mind.

# Basic

## Sampling

Everyone knows this already - but for those new to Strata - do not leave the sampling settings to default for Flash Next. Here are the Alibaba recommended settings:

"sampling": {

"temperature": 1.0,

"top_p": 0.95,

"top_k": 20,

"min_p": 0.0

},

## Parallel

This setting is amazing: "parallel": 2. If you haven't tried running multiple Trains, then you should.

The cost of two Trains is not double that of one Train. Each Train slot costs about 0.4–0.6 GiB VRAM on each card.

With a highly tuned harness, the first Train starts by loading the default system prompt and agents.md. The second Train rolls into motion immediately, as it reuses the KV cache from the first Train!

You might ask, what's the point of two Trains that run at half the speed. This is true only when both Trains are running. If you have only a single Train and it is doing code testing, then your coding speed is a grand total of NIL. However, if you have two Trains, then while one Train is testing, the other is steaming forward at FULL decode speed.

## Layer Split

Always double check this setting. Do not leave it at "layer_split": "auto".

As you can see from the stats, tuning this makes a significant difference.

Fable gave me this advice, which I then gave to more day-to-day AI (so you could do this too), and it immediately picked up that the layer_split needed to be tuned:

The biggest decode gain is a config change, not code: your layer split is badly unbalanced, and fixing it should roughly halve the CPU expert misses. Current numbers: solo decode averages 61 tok/s over 1,677 requests since 12 Nov. 78% of routed experts hit VRAM, 3% go over PCIe and 19% run on the CPU. The misses sit on one gpu: it runs 26 layers with 2,455 cache slots (about 94 per layer), while another gpu runs 22 layers with 5,984 (about 272 per layer). Why layer_split: auto got it wrong. Wrong coverage curve: with two GPUs it assumes the caches hold 97.4% of routing. You measure 78%, and the source comments admit this curve is wrong when a small share of experts is held.
What to change, in order

# Change Type Expected effect Output

1 "layer_split": "16" (try 20 too) config CPU misses ~22% → ~12%; decode perhaps +10–15% rounding-level"

## No pages

This setting maximises the use of sufficient contiguous free RAM. It locks your experts into RAM: "STRATA_NO_LARGEPAGES": "1", which should be faster than Windows paging the data to virtual memory on a disk.

Use this in combination with the Windows setting for 'Lock pages in memory'.

## KV Cache Reuse

For some reason, even with identical leading tokens, full KV cache re-use for compaction is blocked without this setting:
"--prompt-cache-every", "0"

It could be a checkpoint conflict, but this setting really enhanced my token reuse as you can see from the stats.

# Advanced

## Custom build to fix the gradual Strata slowdown issue

If you are using CUDA, after a few hours of continuous runtime, you may notice that Strata's decode speed significantly drops - potentially even halves - until you reboot Strata.

The root cause is this:

- `src/core/device.cu` (~105-210) ALREADY describes this exact failure for the HIP build ("the expert cache filled the card, and WDDM moved memory out to system RAM: decode at 30 tok/s instead of 41") and fixes it at startup by lowering the free figure with DXGI `QueryVideoMemoryInfo(LOCAL).Budget` (`STRATA_WDDM_BUDGET=0` off) - but only under `#if defined(STRATA_USE_HIP) && defined(_WIN32)`. The CUDA build has no such check, and nothing re-checks the budget after startup.
- `serve/server.py` (~930, #533) sends a `VRAM <reserve_mib>` command between requests: the engine shrinks or grows its expert cache to hold that much VRAM free (needs `--vram-elastic`, per its docstring). That is an existing, tested shrink path.

## Upstream status (checked 21 Nov; reuse, do not reinvent)

No upstream release or open PR watches the WDDM budget AFTER startup or shrinks the expert cache at run time on CUDA.

I'm running one of the (earlier) stable versions of Strata so I can't submit this to upstream. But it could be useful for you to ask your AI to implement the fix if you are also running a custom engine.

Hope some of the above is useful for you. I'm sure others will have optimisation tips that they could share!


r/LocalLLaMA • • 10h ago

Resources Hy4 Flash added to int8-vllm: 4x MI100 C8 tg128 222.4 tok/s, pp512 1,655 tok/s

8 Upvotes

Hi all, I'm happy to announce that Hy4 Flash has been added with optimized support to int8-vllm. This is a milestone because this is the first INT4 model added to the project. At INT4, Hy4 Flash fits completely on 128 GB of VRAM with room for 1.3M tok of shared context.

-tg128 (llama-bench style)pp512 (llama-bench style)tg256 w/ 2048 context
Concurrency 8222.4 tok/s1655 tok/s96.7 tok/s
Concurrency 186.3 tok/s1602 tok/s48.3 tok/s

This project focuses on AMD MI100 support, with docker compose files for all optimized models, and support for 2-4 cards with Infinity Fabric or PCIe. MI210 has been confirmed to work well, with a user reporting over 1,000 tok/s decode for Qwen3.8 27B on 2x MI210s at C8. I am looking for users with MI50, MI60, or multiple RDNA1-3 cards who would like to help the project improve support. Please post a github issue with your setup and technical ability.

Project link: https://github.com/curvedinf/int8-vllm

An MI100 is currently $1,300 on ebay, so it represents an underlooked used card option. How does it compare to modern cards? As a rule of thumb, similarly priced new cards offer about double (or more) the compute but half of the memory bandwidth. That means to expect from MI100 something like half the theoretical prompt processing speed but double the token generation speed of newer cards at the same precision.

While the project is directed at INT8 because older cards prefer this datatype, INT4 is easily compatible with INT8 instructions and useful for larger models, so it fits the project's goals. In the MI100's case, its vector ALUs support INT4 natively at 185 TOPs. To support Hy4 Flash, W4A8 GEMM support has been thoroughly added to the companion AITER fork: https://github.com/curvedinf/int8-aiter

Additionally with Hy4 Flash, I found a 15% optimization to many of the GEMMs of the project by deferring epilogue work into later fused kernels so both the vector ALUs and matrix cores of MI100 compute units are fully utilized. These cores can run at the same time on a given CU but often the matrix cores are the only thing running by default. For more modern cards this isn't a big deal because the matrix cores are much more powerful, but in the MI100's case, its vector ALUs are proportionally large compared to the matrix cores. This optimization in effect gives MI100 185 INT8 + 185 INT4 TOPs per card of burst capability. Paired with the 1.2 GB/s of memory bandwidth and 552 GB/s of interlink bandwidth, MI100 has a lot of capability to optimize toward.

The Hy4 Flash checkpoint for the optimized recipe is the first INT4 Hy4 Flash checkpoint on huggingface, and specially created using the NeUQI calibration technique at INT4 group size 128 with FP16 scales. NeUQI paper: https://arxiv.org/abs/2505.17595

The DFly2 model and vision encoder are INT8 quants that were retrained with PTQR for improved INT8 rounding.

Model link: https://huggingface.co/voodooquant/Hy4-flash-NeUQI-INT4-W4A8-GS128

Draft link: https://huggingface.co/voodooquant/Hy4-flash-DFly2-PTQR-INT8-W8A8-GS128


r/LocalLLaMA • • 6h ago

Resources I put together a tool to use non Claude models with Claude Desktop

8 Upvotes

I wanted to use the other models offered via Tokenharbor (this can be used for local models too) on Claude Desktop after I already added Haiku, but Claude Desktop out of the box blocks non-Claude models when setting a gateway. This tool fixes that. It gives you a GUI that allows you to select your models of choice and add them to Claude Desktop.

It patches Claude Desktop to support those extra models in gateway mode. Feel free to take a look, fork it, make your own spin. This tool is Windows only. If you want a Linux version, ask your agent to fork this repo and make it work on Linux.

https://github.com/phly95/claude-endpoint-mapper


r/LocalLLaMA • • 20h ago

Discussion Inkling simply *cheff kiss*

389 Upvotes

FYI, if you daily-drive Qwen 3.8 or QFN, you know there's no way they would give an explanation like this. Inkling's are just butterysmooth, with natural rhythm and flow.

The thing is that Qwen 3.8/QFN are notoriously "jumpy" in conversational mode. I've had to ask "Please ELI5" or "What?" multiple times when discussing topics I don't fully understand. Hell, sometimes it even manages to confuse me about topics I already understand well.

Qwen models are agentic monsters, that's a fact. But when it comes to technical heavy conversations, I've found the Gemma4 models (yes, I've tried the 12B, 26B-A4B, and 31B) unsurprisingly much better at explaining things. Unfortunately, more often than not, they don't quite have the reasoning capabilities needed for heavy technical discussions, although web search kinda help that.

That being said, Inkling seems heavily English centric. I tried it in Indonesian, and the output felt more like a direct translation from English than something naturally written in Indonesian. It lacks the linguistic localization you'd expect from a genuinely multilingual conversational model.

So, to me, Inkling feels like a mix of Gemma4 conversational fluency and Qwen3_5 technical capabilities.

And to be honest, its behavior reminds me a lot of the GPT OSS series, which happens to be the longest-running model I've continuously served on my own servers.

As for tok/s. it's slow, but that's understandable. It was released just 22 hours ago. On top of that, it uses a looped transformer arch with attn res (Kimi-style residual connections), which adds another layer of complexity.

Inkling is currently running through my personal benchmarks, so these are just my initial impressions.

  1. Infra ApacheBigTop Bench (Building DataLakehouse from 0 using Apache BigTop package)
  2. Infra handle Bench (Overseeing the logs, and preemptive when there are dataskew or just error in Lakehouse)
  3. CK-FSDP impl Bench (ComfyKitchen FSDP packing unpacking tensor and dispatcher stuff)
  4. Tensor SubClass bench (Like above, but more so on TorchAO style)
  5. Mini SlopCode Bench (Repowide understanding and how much erosion, only in repo that i owned, since i know how each components work)
  6. Angrybird bench (More involved "SVG Pelican" bench, since it has to know physics)

Spec:
3090, 128G VRAM Quad channel 2400


r/LocalLLaMA • • 3h ago

Discussion Inkling 2 mini cost analysis

12 Upvotes

Inkling 2 Mini has been getting a lot of hype these past few days, so I hopped on the bandwagon and tried running it locally for a bit.

Overall, it's a really good model for its size, and it's ridiculously fast too. I'm getting almost 4500 tok/s decode at C32, roughly 140 tok/s per stream with 32 concurrent coding sessions, on 2 rtx pro 6000s

But I also ran it through a few coding harnesses on a 200 problem subset from SWE-bench, and the pattern on cost is pretty clear.

Compared to Deepseek V4.1 Flash 1017, Inkling is making roughly 2x as many model calls, using way more input tokens, and solving fewer problems (which is expected since the model is so small). But despite being a much smaller model, the interesting thing is it's costing around 3-5x more in API token cost in these tests. I plan to eventually test QFN and 3.8 27b but for now I only have data on Inkling 2 mini vs Deepseek v4.1 flash

Don't think anyone here will be paying api pricing for it, but at API pricing you will be paying way more per task for this smaller model. Just providing data we got from testing:

DeepSeek V4.1 Flash SWE-bench 200

MetricKimi CodeDeepseek HarnessOMP
Passed157 (78.5%)154 (77.0%)161 (80.5%)
Model calls8,7329,3239,441
Prompt tokens529.2M305.0M469.9M
Cached tokens521.8M298.1M462.0M
Output tokens5.72M9.49M8.51M
Cost (DeepSeek Pricing)$6.11$7.63$7.68

Inkling 2 Mini SWE-bench 200

MetricKimi CodeDeepseek HarnessOMP
Passed147 (73.5%)145 (72.5%)145 (72.5%)
Model calls16,74919,29118,161
Prompt tokens1,447.3M790.1M1,343.5M
Cached tokens1,423.7M750.0M1,320.5M
Output tokens9.97M11.71M8.18M
Cost (Thinking Machines pricing)$29.88$23.11$27.35

r/LocalLLaMA • • 43m ago

Resources Hy4 filtered offensive security data out of training. Reasonable for open weights, or are defenders paying for it?

0 Upvotes

Hy4 weights came out this week along with a safety card, and the cyber questions in the AMA got heated.
What the team said:
• security training focuses on defensive tasks, vulnerability analysis and remediation
• data focused on exploitation and offensive attacks was excluded
• filtering the data is the foundational step, paired with trained guardrails for refusals, "all for responsible open-source"
• by their evals, Hy4 is strong at vulnerability discovery and relatively weak at exploitation
• they admitted that avoiding over-refusal "remains an important area for improvement"
The pushback in the thread was good. One person pointed out you can chain the model into a harness and get the same result anyway, so filtering mostly neuters it. Another argued reverse engineering and vuln discovery aren't inherently offensive, since IR teams use them for malware analysis and forensics.
I'm honestly split. For an open model anyone can download, drawing a line somewhere on security makes sense to me. I just hope it stays friendly to the basic learning stuff.
Has anyone tried pentesting, RE or CTF prompts on Hy4? Curious how often it actually refuses, and whether that gets in the way of normal practice.


r/LocalLLaMA • • 14h ago

Showcase Inkling 2 Mini IQ4_NL on RTX 5060 Ti 16GB, 44 tok/s

10 Upvotes

Finally got a chance to sit down and test Inkling 2 Mini on my PC (5060 Ti 16GB + 32 GB RAM).

This is for the IQ4_NL gguf version from ggml-org on HF. I started out at 12 tok/s and the the final number I can reach is 44 tok/s. At 128k context window, without vision. With the latest llama.cpp (19bc41dbb).

If you're here just for the llama params, here's how to run it:

Best with q8_0 KV: 43.34 tok/s

-c 131072 -np 1 -fa on --cache-ram 0 -ub 32 -b 512 -ctk q8_0 -ctv q8_0 -fitt 128 --spec-type ngram-mod

Best with q4_0 KV: 44.35 tok/s

-c 131072 -np 1 -fa on --cache-ram 0 -ub 32 -b 512 -ctk q4_0 -ctv q4_0 -fitt 64