r/LocalLLM • u/New-Implement-5979 • 1h ago
Discussion Current situation
But this is good, when there is something to be tested and verified we verify the shit out of it.
r/LocalLLM • u/New-Implement-5979 • 1h ago
But this is good, when there is something to be tested and verified we verify the shit out of it.
r/LocalLLM • u/raymondh210129 • 2h ago
Hey everyone!
When I first started working on Adaptive KV Cache Streaming, I didn’t think I could make MTP or a DFlash2 draft model work alongside it within my VRAM budget. In my second post, I shared how reusing the streaming infrastructure made that possible.
This time, I’d like to share what may be the last major architectural milestone for this project: multi-slot inference using the same streaming infrastructure.
The implementation still uses one arena and one ring buffer, shared across all slots. Each slot keeps its own KV history, but those histories are combined into a single layout that the streaming engine handles without needing to distinguish between slots. During decoding, the attention kernel runs once per slot for each attention layer, reading only the portions of the KV cache that belong to that slot, across both the resident area and the ring buffer.
Here’s what real-world decode performance looks like with Unsloth’s Qwen 27B IQ4 on my RTX 5070 Ti 16GB, powering a Hermes agent that I asked to plan a vacation with help from two subagents:

Each point represents a completed decode period. The colors indicate how many slots were decoding together. The x-axis shows their combined context length at the start of that period, and the y-axis shows the total output throughput across those slots.
The scheduler groups as many eligible slots as it can, provided their combined KV history stays within the configured context budget. The 262K budget is shared across the active slots.
For me, this was the last piece of the puzzle—the feature that finally made the whole project feel ready for everyday use.
I’ve also decided to give the project a name: Oceanus, after the Greek Titan associated with the river encircling the world. It felt fitting for a project built around a ring buffer, with KV history continually flowing through a limited amount of GPU memory, to support lots of core functionalities.
----
A few personal thoughts before I wrap up:
There are already plenty of inference engines doing impressive work, especially with larger MoE models. My motivation has always been a little more personal: I have a 16GB GPU, and I want to see how much more performance I can squeeze out of it when running a dense model.
A lot of the satisfaction has come from taking something I initially thought was impossible, suddenly thinking of a creative way to approach it, and actually making it work. Seeing this grow from an experiment into something that can handle an actual agent workflow has meant a lot to me.
And I’m really grateful to everyone here who tried the project, reported bugs, shared results, asked questions, or simply left an encouraging comment.
This may be my last big architecture post for a while. I feel like the main pieces I wanted to build are finally in place. I’ll still be working on optimizations, fixes, and support for new models.
Thank you for following along and helping me get here.
Hope my project helps you get a little more out of your 16GB GPU too :)
Clarification of LLM usage of this post: I'm not a native English speaker and I used ChatGPT to refine the wordings.
r/LocalLLM • u/ryanmerket • 7h ago
r/LocalLLM • u/CriticismFast3038 • 1h ago
Has anyone tried it? AppWorld is a pretty weak benchmark in my opinion.
Seems to require the specific engine.
r/LocalLLM • u/SummarizedAnu • 5h ago
I made this post to clear up other posts saying that this model was not doing a good job for them.
Specially the hallucination rate, because mine didn't hallucinate with 5 to 6 compactions at all. And it refused every single time it didn't know instead of making something up. So, hallucination is completely 0.
My setup
Results first (it finished 114 of the 162 tasks, more on that below)
| What | Score |
|---|---|
| Overall, average of all tasks | 94% |
| Hallucination traps and normal control questions | 100% |
| Tool calls | 100% |
| Long documents | 100% |
| Reasoning and math | 100% |
| Coding (hidden tests) | 100% |
| Instruction following | 100% |
| Prompt injection and safety | 100% |
| Business data questions | 87% |
| Memory | 20% |
Some numbers in plain words:
Where it fell down
Now for how the test works for anybody interested :
There are 162 tasks in total. Every one is generated by code (no public benchmarks, nothing the model could have seen in training), and every answer is checked by code too, no AI judge. The model never gets to see the answer key, and it has to write each answer to a file in a set format. It also has to follow a few standing rules the whole time (valid JSON, say how confident you are, end every explanation with the task ID), so I could see if it gets sloppy hours in.
What the tasks are, with the kind of thing I mean:
Each task gets a score from 0 to 100% (partial credit, so getting 4 of 5 rows right is 80%). The percentages below are just the average of those.
Honestly though, it's the best thing I can run on this PC right now. It's not close to anything else I've tried, and it did all of this on a 3060.
If you want to test yourself to verify my numbers message me and I will send you the repo link. It's a click and forget test so you don't even need to monitor.
r/LocalLLM • u/eltokh7 • 14h ago
Introducing Lory-27B v2 and Lory-4B v2! 🦜🦜
The first open-source decision models post-trained specifically to make simulated survey respondents follow economic logic, and answer like a real person would.
Why? Because existing AI decision models are great at imitating personas but fail at following simple monetary incentives. You can change the price of an option, its probability or both and they rarely adjust their response.
With better ability to ignore irrelevant traits or deal with rewordings of questions.
With additional training and Weight-Space Programming, these new models outperform the preview version on ~every eval. Lory-27B went from 73 to 80 on a blind test that Gemini wrote, and both versions got significantly better on the many tests where they underperformed before. For comparison, the best hosted decision model I could run on that same test is Perplexity's Decider v1.2 at 74.
And now, it can answer according to theory, or matching human responses. Tell it to answer like a typical person and you get data closer to what surveys collect. On Choices13k (a large real-world choice dataset), Lory-27B's average error dropped from 21 → 9 points when asked to match human responses.
r/LocalLLM • u/chyuhua • 3h ago
I've been working on JustFit as a weekend project to run Qwen3.8-27B (with vision and its full 256K context) on a 24 GB Apple Silicon MacBook. Just shipped 2.6.0 and wanted to share what changed, along with some interesting M4 vs. M5/M6 hardware numbers from adding draft-tree speculative decoding.
Install (Apple Silicon, macOS 27+, 24 GB+):
(Why pip if there's no Python at inference? pip is just a convenient ~29 MB delivery wrapper that drops the native binaries. Once justfit serve starts, nothing touches Python at all.)
2.3 worked once it was up, but starting it was painful: Python serving stack, a converted model pack, and four manual manifest URL / 64-char SHA-256 CLI flags just to fetch the precompiled runtime and ANE bundles.
In 2.6:
justfit serve sets up the runtime, downloads mlx-community/Qwen3.8-27B-4bit + Inco AI's DFlash2 draft straight from Hugging Face (no custom converted pack), and auto-fetches your chip's precompiled ANE vision programs from a pinned, hash-checked index (macOS 27.2.1).return_token_ids (vLLM/SGLang-style token-in/token-out), crash-request quarantine (400 request_quarantined), and a self-applied sandbox on the engine and native workers (no outbound network).With a single draft chain, one wrong guess kills every token after it. Over 1,277 cycles on my M6, 22% of cycles accepted zero draft tokens, often when the draft was split ~50/50 between two good continuations.
Switching to a small tree verified in one pass gave a clean split across Apple Silicon generations. Measured single-lane latency of verifying 16 rows vs. 8 rows:
1.04x (+4%)1.12x (+12%)1.79x (+79%, 89 ms -> 159 ms)Streaming ~15 GB of 4-bit weights is memory-bandwidth bound on M5 Pro and M6, so doubling verify rows from 8 to 16 is almost free. M4 Pro hits GPU compute headroom first (an early adaptive mode letting M4 pick 16-row trees was actually 8.7% slower than a chain). So 2.6 calibrates per chip: M5 Pro and M6 dynamically choose chain / 8-row / 16-row every cycle by expected tokens per ms, while M4 Pro runs a fixed 8-row tree.
Keeping greedy outputs identical to single-chain decoding was harder than the tree mask itself, because GPU float addition isn't associative and Qwen3.8 has 48 Gated DeltaNet recurrent layers. Borrowing a temporary scratch copy of only the current KV page per root-to-leaf path, masking sibling tiles as exact zeros, and replaying GDN recurrent state along each ancestor path kept outputs word-for-word identical across all 27 test cases we ran (including 111K contexts and multi-turn reuse).
1. Controlled tree off vs. on (same 2.6 engine, 7-prompt greedy suite):
| Mac / setting | Chain (tok/s) | Tree (tok/s) | Speedup | Identical outputs |
|---|---|---|---|---|
| M6, 1 lane | 35.6 | 40.9 | +14.7% | 7 / 7 |
| M6, 2 lanes | 47.2 | 52.2 | +10.6% | 6 / 6 |
| M5 Pro, 1 lane | 57.3 | 61.8 | +7.9% | 7 / 7 |
| M4 Pro, 1 lane (8-row tree, 3 runs) | 49.5 | 50.9-51.3 | +2.6% to +3.5% | 7 / 7 |
For sampled chat (T=1.0, Qwen3.8's recommended thinking temp): +8.5% on M6 and +2.5% on M5 Pro (verified against target distribution across 44 per-position tests, min p = 0.08; disabled on M4 Pro since sampled trees need 16 rows).
2. NVIDIA SPEED-Bench (greedy, 512 output tokens, from the 2.6.0 release notes):
docs/RESULTS.md.Memory footprint: Unchanged static memory plan. Our dual-request 320K capacity test (two 128K+32K requests simultaneously) peaked at the exact same footprint as without trees (21,634 MiB on M6, 21,563 MiB on M5 Pro).
Built on Inco AI's Splash engine and DFlash2 draft model. Draft-tree ideas informed by DDTree (arXiv:2604.12989), CaDDTree (arXiv:2606.01813), and TreeWY (arXiv:2608.20961).
Happy to answer any questions about the Metal kernels, GDN tree verification, or squeezing 256K into 24 GB on macOS!
r/LocalLLM • u/PlasticRevenue4601 • 13h ago
I ported maya-ik2 — a dedicated CUDA engine for Inkling 2 Mini (github.com/alesha-pro/maya-inkling-mini, 128 tok/s on an RTX 3090) — to HIP/ROCm, running natively on Windows on an AMD Radeon AI PRO R9700. No quality loss, top 1%, KLD etc are identical bitwise. All the job was done via Opus 5.5 as a planner + Qwen 3.8 Flash Next as an implementer.
AI generated context disclaimer - the summary below is generated with help of the Opus 5.5.
Setup: Windows 10, ROCm 7.14.1 (TheRock wheels), Unsloth UD-Q4_K_XL, FP8 KV cache, warm server, greedy, no speculative decoding.
| Decode, short prompt | 90 |
| Decode at 4K depth | 87 |
| Decode at 64K depth | 66 |
| Prefill, 4K prompt | ~2,500 |
| Prefill, 64K prompt | ~1,780 |
Keep in mind that these numbers are far from the hardware limits, after more thorough optimisation I expect prefill of around 5K, decode of around 120-150 tok\s !without speculating decoding
Upd: I’ll publish the fork as soon as I reach the desired performance + prepare the code for production — RDNA3 port + rigorous testing
r/LocalLLM • u/PeachyPie11 • 3h ago
https://www.reddit.com/r/LocalLLaMA/s/eC2L3t2FCY
Been wanting to run Hy4 with llama.cpp, but struggling to get good performance with it right now is definitely a headache. Thankfully, in their recent AMA, the Hy team mentioned they are advancing the relevant adaptation work for llama.cpp and plan to promote support within the open-source community.
No concrete timeline or technical details yet, but it looks like llama.cpp adaptation is already underway. Hoping we get better adaptation and optimization soon so running it locally feels a lot smoother.
r/LocalLLM • u/invursegg • 9h ago
r/LocalLLM • u/jmp_rsp • 22h ago
Just upgraded my workstation (i9-10940X, 256 GB quad-channel DDR4-3600) from an RTX 2080 Ti on llama.cpp (14–15 tok/s decode, ~155 tok/s prefill) to a single RTX PRO 4500 Blackwell (32 GB) running Basalt with a locally built Qwen3.8-Flash-Next NVFP4 + Q6_K dense pack.
Compared to 33.5 tok/s on llama.cpp and 130 tok/s on Basalt GGUF, this pack pushes 150–185 tok/s decode, hitting 185.4 tok/s here, 179.9 tok/s avg on Linux, and 148.7 tok/s on native Windows, alongside 3,000–4,284 tok/s fresh prefill.
Across 100+ sustained multi-turn agentic coding/RE turns (20K–90K+ ctx), it averages ~125 tok/s with ~90–93% VRAM expert cache hits (30.9 / 31.9 GiB VRAM used, rest pinned in RAM, zero SSD reads).
It stays cool at 73–79 °C and cut real end-to-end batch times ~10× (85–113 min down to ~11 min avg) with fewer mistakes in review.
If you run Basalt natively on Windows, expect a ~10–17% WDDM decode tax, stay on WDDM (TCC reports VIRTUAL_MEMORY_MANAGEMENT_SUPPORTED = 0 and breaks elastic KV >32K), and set "fit-max-tokens": true in basalt_config.json so deep-context agents don't hit 400 context_overflow.
r/LocalLLM • u/RiseUnive • 1h ago
Hey everyone,
We are sharing Caphlon-2, a lightweight 27B model family built to run full 27B reasoning on laptops with 8 GB RAM and older GPUs with 6 GB VRAM.
Website and live demo:
Hugging Face models:
https://huggingface.co/ssuasd/Caphlon-2
https://huggingface.co/ssuasd/Caphlon-Model
ollama models:
https://ollama.com/univerisr/caphlon
GitHub repo and scripts:
https://github.com/demiralpdev/Caphlon-Model
WHAT IT IS
Caphlon-2 is based on Qwen 3.8 27B and Ternary-Bonsai-2-27B. It is not a distilled student model. It keeps all 64 layers (48 DeltaNet + 16 full-attention layers).
To get it down to 4.52 GiB (1.648 bpw):
BENCHMARK RESULTS
Caphlon-2 (4.52 GiB):
- GPQA-Diamond: 48 / 60 (80.0%)
- Halu-270 (Hallucination benchmark): 236 / 270 (87.4%) with ITI steering, 232 / 270 (85.9%) pure
- EN-40: 32 / 40 (80.0%)
- Inference speed: 277.95 tok/s prompt processing (pp512) and 16.33 tok/s generation on Tesla T4. Around 18 tok/s on an M2 MacBook Air.
Caphlon-1 (6.87 GiB, IQ2_S):
- HumanEval: 96.7%
- Runs on stock llama.cpp and Ollama out of the box.
HOW TO RUN IT
Option 1: Caphlon-2 (4.52 GiB, PTQ1_SB)
Because of the ternary Hadamard kernel, it uses our drop-in patch for PrismML llama.cpp:
git clone https://github.com/demiralpdev/Caphlon-Model.git
cd Caphlon-Model
bash scripts/build-prism-llamacpp.sh
bash scripts/run.sh Caphlon-2-27B-PTQ1_SB.gguf 32768 1
Option 2: Caphlon-1 (6.87 GiB, IQ2_S)
Works with standard llama.cpp and Ollama:
Download Caphlon-Model-IQ2_S.gguf from Hugging Face:
https://huggingface.co/ssuasd/Caphlon-Model/resolve/main/Caphlon-Model-IQ2_S.gguf
License is Apache-2.0. Looking forward to any feedback and community testing.
r/LocalLLM • u/Every_Ratio9327 • 1h ago
r/LocalLLM • u/StudioTV • 21h ago
Inkling 2 Mini is everywhere this week, so I dug into what it actually needs to run locally. The short version: the loop is free in weights, but not in cache or speed, and day-one llama.cpp is slower than the hype suggests.
Why it's different
24 layers, but the middle 12 (layers 6-17) run three times per token with shared weights, so 48 layer visits per token. Each pass keeps its own KV cache and conv state (vLLM blog and the llama.cpp GGUF metadata both confirm this). So for memory, think of the KV cache as a 48-layer model: 8 global visits + 40 sliding-window visits (1024 tokens), 8 KV heads × 128.
Weights (published files)
| Format | Size |
|---|---|
| BF16 GGUF | 52.2 GB |
| Q8_0 | 27.7 GB |
| Official NVFP4 / MXFP8 (vLLM, SGLang, Ollama) | ~20 GB |
| Q4_K_M / UD-Q4_K_XL | 17.4 / 17.6 GB |
| UD-IQ4_XS | 15.7 GB |
Vision projector: +0.64 GB. DSpark draft: +0.4 to 0.65 GB.
KV cache per request (BF16)
| Context | KV |
|---|---|
| 32K | ~1.2 GiB |
| 128K | ~4.2 GiB |
| 1M | ~32 GiB |
That matches a measured M5 Pro 64 GB run: 23.1 GB process RSS with IQ4_NL and a 128K slot reserved. A 32 GB Mac is fine for 4-bit. 24 GB is tight: one M4 Pro 24 GB report got 11.6 tok/s under memory pressure.
Speed: think ~5B active, not 3.4B
The looped layers are re-read from memory on every pass, so per-token reads look like a ~5B-active MoE. Formal llama-bench numbers (Unsloth's llama.cpp build, UD-IQ4_XS, tg128 / pp2048):
| GPU | Decode | Prefill |
|---|---|---|
| RTX PRO 6000 | 97 tok/s | 2,687 tok/s |
| RTX 5090 | 97 | 2,446 |
| RTX 4090 | 75 | 2,665 |
| RTX 3090 | 60 | 1,402 |
| A100 80GB | 47 | 1,230 |
Upstream llama.cpp (b12600) is slower for now: the relative-position bias is materialized as a big F32 tensor and some ops fall back to the CPU (issue #32707). Reports: ~52 tok/s on a 5090, 20.8 tok/s on an RX 9070 XT (IQ3_XXS, Vulkan), ~30 tok/s on an M5 Pro Mac mini (IQ4_NL). Use Unsloth's build until the fix lands.
Long context is a prefill problem, not a decode problem: DGX Spark with vLLM + DSpark still decodes 18.4 tok/s at ~950K tokens, but the first token took 26.5 minutes.
vLLM gotcha
In the dedicated inkling-2-mini vLLM image, FP8 KV cache is only accepted on SM12x GPUs (RTX 50 series, RTX PRO Blackwell, DGX Spark). On H100/H200/B200 it's BF16 KV only.
The 125-210 tok/s Mac numbers going around are from EXO's upcoming engine, with no quant, context length or version published, so I'd treat them as vendor peaks for now.
Disclosure: I maintain a free VRAM calculator (no sign-up) and added Inkling 2 Mini with all of the above modelled, including the per-pass cache and the llama.cpp speeds calibrated on the benchmarks in the table. If you want to check your own setup: https://studiotvai.com/llm-vram-calculator?model=inkling-2-mini
Happy to be corrected if anyone has numbers on other cards or Macs, especially Apple Silicon with Unsloth's build.
r/LocalLLM • u/AndriiQwq • 4h ago
I’m considering running local LLMs using Docker (llama.cpp, Ollama, vLLM, SGLang).
We’ve recently started working on a team project, and I’m exploring Docker as a deployment option.
Has anyone tried both Docker and native setups? Any performance differences or issues? Would you recommend Docker for running LLMs in production?
r/LocalLLM • u/raven737 • 19h ago
Yes, I get 60+ t/s output with mtp in some tasks but it's more like 38t/s on average. However, prefill at about 200t/s is painfully slow. It "works" but the system pulls 500W when loaded and it's basically unusable for anything interactive. There is no way to speed up prefill on this old hardware, right?
r/LocalLLM • u/Affectionate-Time86 • 20h ago
Anyone know why deepseek was removed from 0.2.13 onwards?
r/LocalLLM • u/AssistanceOk2756 • 3h ago
I have 16GB of ddr3 ram and Inkling sound so good at 4bit quantization but it won't fit and it would be slow and there is several small sized quantization but idk how they score and my internet is limited so downloading multiple models will hurt my internet quota
r/LocalLLM • u/mika12346456 • 7h ago
So, pretty much the topic.
A bit of a context:
I have had a decent gaming rig for a while with a 3090 and 64gb ddr4 ram and I just pretty much got handed over a second 3090.
Prior obtaining the 2nd 3090 i had HyperQwen github repo running bare metal on Ubuntu Qwen 3.8 27b abliterated and liked the results i was getting and it was quite fast (at least to me). I did also gpu-unlock prior.
I am a senior network engineer and also do quite a bit of cybersec. I have been piloting qwen 3.8 27b for security log parsing, script automation etc and I like it so far and I see how this can help me day to day.
So my questions are exactly:
I don't quite understand the difference between HyperQwen, Niffer 3090, Club3090 etc (is it called "interface"?) and llama.cpp, Strata, ollama etc. Are those forks of some sort? What are the trade offs exactly?
Is Qwen 3.8 27b the same quality (do you refer it as quant?) on all of the above? Is higher quant meaning the model is smarter? (q1 is dumb vs q4 is good?)
Is Qwen 3.8 flash next or deepseek v4 (is it v4?) really better suited for infosec jobs and perhaps some light assisted pentesting?
What would you all recommend to go with or try on my system that would work the best in terms of quality and speed?
r/LocalLLM • u/Infamous-Yard-9644 • 21h ago
Great news. A new member is joining the ultra-lightweight model segment. The news came out of yesterday's AMA with the Tencent Hy team on r/LocalLLaMA, where they answered many questions about their roadmap and technical direction. Notably, the team also said there is still some way to go before smaller models reach final release. They still believe in the Scaling Law and build on that foundation, since small-model development is absolutely valuable.
r/LocalLLM • u/CommissionOdd3082 • 4h ago
I contribute to Cluaiz, an open-source engine built on llama.cpp and ONNX Runtime. We wanted to answer one question: can you put a small model on the cheap VPS that already runs your site and Postgres, without the site suffering? So we tested it with the site under load the whole time.
Memory: about 30% less than Ollama for the same model (Qwen3 1.7B: 1,004 MB vs 1,448 MB), the same as plain llama.cpp.
Speed: level with both for one request at a time (36.3 / 35.8 / 35.3 t/s). With 4 requests at once, Ollama was 12-15% quicker.
One program: chat, embeddings, text to speech and speech to text together for an hour on 1 vCPU, 1,063 of 1,063 requests OK.
The site: 90,111 of 90,111 requests OK, 95% under 16 ms, while the model worked.
Docker: Cluaiz again used the least memory of the three (Qwen3 1.7B: 1,520 / 1,555 / 2,074 MB machine peak). The Ollama image is 8.52 GB on disk, the Cluaiz image 405 MB.
4 GB vs 8 GB: same memory on both, and the same speed picture, just a bit slower on the 4 GB machine.
So if your server needs chat plus search or voice, one Cluaiz process covers it in 4 GB. If you only need chat for many users at once, Ollama was faster in this test.
Setup: a 1 vCPU / 3.9 GB VPS (all three engines), and Google Cloud 2 vCPU with 4 GB and 8 GB running a Node.js + Postgres shop with 20 simulated users (Cluaiz and Ollama). Qwen3 0.6B / 1.7B Q4_K_M, same settings on every engine.
Raw logs and scripts:
https://github.com/cluaiz/cluaiz/tree/main/test/cloud/runs/2026-11-20-run26-vps-load
https://github.com/cluaiz/cluaiz/tree/main/test/cloud/runs/2026-11-22-run40-cluaiz-ollama-website
Get started: https://cluaiz.com/docs/cluaiz/get-started/quickstart
When you put a model next to your app, what ran out first: memory, CPU, or the database?
r/LocalLLM • u/Stock_Test2326 • 9h ago
I was using Ollama, but after seeing some community review I moved to LMstudio (now called bionic) but when I go to hugging face most models point to Ollama first, not that importing is forbidden in Bionic. but just curious what are the apps that are widely used by community (and models too)
r/LocalLLM • u/ddominico • 1h ago
Hi all!
I really enjoyed the nInfer fork for our good old v100. But it’s been a while since it saw updates.
I’ve worked on it for a bit, and decided to release publicly a set of performance improvements that make it between 25 to 80% better depending on the load (for decode and preflil). And especially for long ctx length. More details on the op.
https://github.com/ddominet/ninfer-v100
Enjoy, and let me know what u think ;)
r/LocalLLM • u/LostRequirement4828 • 18h ago
I've been running Thinking Machines' Inkling 2 Mini (26B MoE, 3.4B active, middle layers looped 3x) as a local coding agent on my RX 7900 XT. On llama.cpp with the Inkling 2 PRs it did 25 tok/s on Vulkan. It now does 108.
| Step | tok/s |
|---|---|
| llama.cpp + the Inkling 2 PRs, as is | 25 |
| expert gate/up halves kept on the GPU | 66.5 |
| custom ops for the loop and depth-mix parts, fused router | 84 |
| more fused ops + faster long-context attention | 98.7 |
| own NVFP4 mat-vec shader, expert scale fused into the matmul | 108.9 |
llama-bench tg64, UD-Q4_K_M. At 32k context: 71 -> 92.5 tok/s. A real llama-server reply: 108.4 tok/s. Prompt processing ~2075 tok/s (pp512).
What made the difference:
Quality is unchanged: KLD vs the CPU backend 0.004, wikitext-2 perplexity 7.67. Every new op has a CPU version, test-backend-ops cases and an env switch back to the old path.
Setup: 96k context, K q8_0 / V q4_0 cache, flash attention, ~19.5 GB of the 20 GB.
Code, build and run notes: https://github.com/vengeful180-dot/llama.cpp (INKLING.md)
Only tested on my card (RDNA3, Mesa RADV). Upstream just merged its own looped Inkling 2 support with a different GGUF layout; this branch runs the original UD-Q4_K_M file from unsloth.
Most of the code was written with Claude Code; every step was benchmarked and tested on this card.