r/LocalLLM • • 1h ago

Discussion Current situation

Post image
132 Upvotes

But this is good, when there is something to be tested and verified we verify the shit out of it.


r/LocalLLM • • 22h ago

Project 185.4 tok/s on Qwen3.8-Flash-Next (NVFP4, 128K ctx) — single RTX PRO 4500 32GB + Basalt on PCIe 3.0 / DDR4

Post image
116 Upvotes

Just upgraded my workstation (i9-10940X, 256 GB quad-channel DDR4-3600) from an RTX 2080 Ti on llama.cpp (14–15 tok/s decode, ~155 tok/s prefill) to a single RTX PRO 4500 Blackwell (32 GB) running Basalt with a locally built Qwen3.8-Flash-Next NVFP4 + Q6_K dense pack.

Compared to 33.5 tok/s on llama.cpp and 130 tok/s on Basalt GGUF, this pack pushes 150–185 tok/s decode, hitting 185.4 tok/s here, 179.9 tok/s avg on Linux, and 148.7 tok/s on native Windows, alongside 3,000–4,284 tok/s fresh prefill.

Across 100+ sustained multi-turn agentic coding/RE turns (20K–90K+ ctx), it averages ~125 tok/s with ~90–93% VRAM expert cache hits (30.9 / 31.9 GiB VRAM used, rest pinned in RAM, zero SSD reads).

It stays cool at 73–79 °C and cut real end-to-end batch times ~10× (85–113 min down to ~11 min avg) with fewer mistakes in review.

If you run Basalt natively on Windows, expect a ~10–17% WDDM decode tax, stay on WDDM (TCC reports VIRTUAL_MEMORY_MANAGEMENT_SUPPORTED = 0 and breaks elastic KV >32K), and set "fit-max-tokens": true in basalt_config.json so deep-context agents don't hit 400 context_overflow.


r/LocalLLM • • 13h ago

News 5090 in Stock at Walmart.com $7999

75 Upvotes

r/LocalLLM • • 21h ago

Discussion Inkling 2 Mini memory and speed, explained: the loop costs nothing in weights but you pay for it in KV cache and decode speed

75 Upvotes

Inkling 2 Mini is everywhere this week, so I dug into what it actually needs to run locally. The short version: the loop is free in weights, but not in cache or speed, and day-one llama.cpp is slower than the hype suggests.

Why it's different

24 layers, but the middle 12 (layers 6-17) run three times per token with shared weights, so 48 layer visits per token. Each pass keeps its own KV cache and conv state (vLLM blog and the llama.cpp GGUF metadata both confirm this). So for memory, think of the KV cache as a 48-layer model: 8 global visits + 40 sliding-window visits (1024 tokens), 8 KV heads × 128.

Weights (published files)

FormatSize
BF16 GGUF52.2 GB
Q8_027.7 GB
Official NVFP4 / MXFP8 (vLLM, SGLang, Ollama)~20 GB
Q4_K_M / UD-Q4_K_XL17.4 / 17.6 GB
UD-IQ4_XS15.7 GB

Vision projector: +0.64 GB. DSpark draft: +0.4 to 0.65 GB.

KV cache per request (BF16)

ContextKV
32K~1.2 GiB
128K~4.2 GiB
1M~32 GiB

That matches a measured M5 Pro 64 GB run: 23.1 GB process RSS with IQ4_NL and a 128K slot reserved. A 32 GB Mac is fine for 4-bit. 24 GB is tight: one M4 Pro 24 GB report got 11.6 tok/s under memory pressure.

Speed: think ~5B active, not 3.4B

The looped layers are re-read from memory on every pass, so per-token reads look like a ~5B-active MoE. Formal llama-bench numbers (Unsloth's llama.cpp build, UD-IQ4_XS, tg128 / pp2048):

GPUDecodePrefill
RTX PRO 600097 tok/s2,687 tok/s
RTX 5090972,446
RTX 4090752,665
RTX 3090601,402
A100 80GB471,230

Upstream llama.cpp (b12600) is slower for now: the relative-position bias is materialized as a big F32 tensor and some ops fall back to the CPU (issue #32707). Reports: ~52 tok/s on a 5090, 20.8 tok/s on an RX 9070 XT (IQ3_XXS, Vulkan), ~30 tok/s on an M5 Pro Mac mini (IQ4_NL). Use Unsloth's build until the fix lands.

Long context is a prefill problem, not a decode problem: DGX Spark with vLLM + DSpark still decodes 18.4 tok/s at ~950K tokens, but the first token took 26.5 minutes.

vLLM gotcha

In the dedicated inkling-2-mini vLLM image, FP8 KV cache is only accepted on SM12x GPUs (RTX 50 series, RTX PRO Blackwell, DGX Spark). On H100/H200/B200 it's BF16 KV only.

The 125-210 tok/s Mac numbers going around are from EXO's upcoming engine, with no quant, context length or version published, so I'd treat them as vendor peaks for now.

Disclosure: I maintain a free VRAM calculator (no sign-up) and added Inkling 2 Mini with all of the above modelled, including the per-pass cache and the llama.cpp speeds calibrated on the benchmarks in the table. If you want to check your own setup: https://studiotvai.com/llm-vram-calculator?model=inkling-2-mini

Happy to be corrected if anyone has numbers on other cards or Macs, especially Apple Silicon with Unsloth's build.


r/LocalLLM • • 14h ago

Model Lory 27B and 4B: first open-source decision models for simulated survey respondents

Post image
52 Upvotes

Introducing Lory-27B v2 and Lory-4B v2! 🦜🦜

The first open-source decision models post-trained specifically to make simulated survey respondents follow economic logic, and answer like a real person would.

Why? Because existing AI decision models are great at imitating personas but fail at following simple monetary incentives. You can change the price of an option, its probability or both and they rarely adjust their response.

With better ability to ignore irrelevant traits or deal with rewordings of questions.

With additional training and Weight-Space Programming, these new models outperform the preview version on ~every eval. Lory-27B went from 73 to 80 on a blind test that Gemini wrote, and both versions got significantly better on the many tests where they underperformed before. For comparison, the best hosted decision model I could run on that same test is Perplexity's Decider v1.2 at 74.

And now, it can answer according to theory, or matching human responses. Tell it to answer like a typical person and you get data closer to what surveys collect. On Choices13k (a large real-world choice dataset), Lory-27B's average error dropped from 21 → 9 points when asked to match human responses.


r/LocalLLM • • 21h ago

Discussion Hy4-mini is on the Way — a Good Alternative to Qwen3.6-35B-A3B?

Post image
51 Upvotes

Great news. A new member is joining the ultra-lightweight model segment. The news came out of yesterday's AMA with the Tencent Hy team on r/LocalLLaMA, where they answered many questions about their roadmap and technical direction. Notably, the team also said there is still some way to go before smaller models reach final release. They still believe in the Scaling Law and build on that foundation, since small-model development is absolutely valuable.


r/LocalLLM • • 7h ago

News OpenRouter removes privacy eligibility from 125 endpoints after provider incidents

Thumbnail
runtimewire.com
47 Upvotes

r/LocalLLM • • 20h ago

Question wtf strata - removed deepseek support

47 Upvotes

Anyone know why deepseek was removed from 0.2.13 onwards?


r/LocalLLM • • 19h ago

Discussion 2xP100. Trying it so you don't have to

Post image
46 Upvotes

Yes, I get 60+ t/s output with mtp in some tasks but it's more like 38t/s on average. However, prefill at about 200t/s is painfully slow. It "works" but the system pulls 500W when loaded and it's basically unusable for anything interactive. There is no way to speed up prefill on this old hardware, right?


r/LocalLLM • • 13h ago

News Inkling 2 Mini on a single Radeon AI PRO R9700: ~90 tok/s decode, 2.500 tok/s prefill

31 Upvotes

I ported maya-ik2 — a dedicated CUDA engine for Inkling 2 Mini (github.com/alesha-pro/maya-inkling-mini, 128 tok/s on an RTX 3090) — to HIP/ROCm, running natively on Windows on an AMD Radeon AI PRO R9700. No quality loss, top 1%, KLD etc are identical bitwise. All the job was done via Opus 5.5 as a planner + Qwen 3.8 Flash Next as an implementer.

AI generated context disclaimer - the summary below is generated with help of the Opus 5.5.

Setup: Windows 10, ROCm 7.14.1 (TheRock wheels), Unsloth UD-Q4_K_XL, FP8 KV cache, warm server, greedy, no speculative decoding.

| Decode, short prompt | 90 |

| Decode at 4K depth | 87 |

| Decode at 64K depth | 66 |

| Prefill, 4K prompt | ~2,500 |

| Prefill, 64K prompt | ~1,780 |

Keep in mind that these numbers are far from the hardware limits, after more thorough optimisation I expect prefill of around 5K, decode of around 120-150 tok\s !without speculating decoding

Upd: I’ll publish the fork as soon as I reach the desired performance + prepare the code for production — RDNA3 port + rigorous testing


r/LocalLLM • • 2h ago

Research Adaptive KV-Cache Streaming V3 (Oceanus): Full 262K Context Qwen 27B on a 16GB GPU, with Multi-Slot Inference, Vision, and Speculative Decoding

27 Upvotes

Hey everyone!

When I first started working on Adaptive KV Cache Streaming, I didn’t think I could make MTP or a DFlash2 draft model work alongside it within my VRAM budget. In my second post, I shared how reusing the streaming infrastructure made that possible.

This time, I’d like to share what may be the last major architectural milestone for this project: multi-slot inference using the same streaming infrastructure.

The implementation still uses one arena and one ring buffer, shared across all slots. Each slot keeps its own KV history, but those histories are combined into a single layout that the streaming engine handles without needing to distinguish between slots. During decoding, the attention kernel runs once per slot for each attention layer, reading only the portions of the KV cache that belong to that slot, across both the resident area and the ring buffer.

Here’s what real-world decode performance looks like with Unsloth’s Qwen 27B IQ4 on my RTX 5070 Ti 16GB, powering a Hermes agent that I asked to plan a vacation with help from two subagents:

Each point represents a completed decode period. The colors indicate how many slots were decoding together. The x-axis shows their combined context length at the start of that period, and the y-axis shows the total output throughput across those slots.

The scheduler groups as many eligible slots as it can, provided their combined KV history stays within the configured context budget. The 262K budget is shared across the active slots.

For me, this was the last piece of the puzzle—the feature that finally made the whole project feel ready for everyday use.

I’ve also decided to give the project a name: Oceanus, after the Greek Titan associated with the river encircling the world. It felt fitting for a project built around a ring buffer, with KV history continually flowing through a limited amount of GPU memory, to support lots of core functionalities.

----

A few personal thoughts before I wrap up:

There are already plenty of inference engines doing impressive work, especially with larger MoE models. My motivation has always been a little more personal: I have a 16GB GPU, and I want to see how much more performance I can squeeze out of it when running a dense model.

A lot of the satisfaction has come from taking something I initially thought was impossible, suddenly thinking of a creative way to approach it, and actually making it work. Seeing this grow from an experiment into something that can handle an actual agent workflow has meant a lot to me.

And I’m really grateful to everyone here who tried the project, reported bugs, shared results, asked questions, or simply left an encouraging comment.

This may be my last big architecture post for a while. I feel like the main pieces I wanted to build are finally in place. I’ll still be working on optimizations, fixes, and support for new models.

Thank you for following along and helping me get here.

Hope my project helps you get a little more out of your 16GB GPU too :)

https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming/tree/feature/adaptive-kv-stream-v3

Clarification of LLM usage of this post: I'm not a native English speaker and I used ChatGPT to refine the wordings.


r/LocalLLM • • 23h ago

Discussion Inkling 2 mini seems to be trained for self-promotion

24 Upvotes

Just an observation: I've been kicking the tires on Inkling, asking it to create some self-contained projects to get a feel for the model. In the prompt, I don't specify a title for any of these, and I noticed that it titles them "Inlking" every single time.

Never seen another model do this.


r/LocalLLM • • 18h ago

Project Inkling 2 Mini on an RX 7900 XT: from 25 to 108 tok/s with Vulkan (llama.cpp fork)

Thumbnail
gallery
22 Upvotes

I've been running Thinking Machines' Inkling 2 Mini (26B MoE, 3.4B active, middle layers looped 3x) as a local coding agent on my RX 7900 XT. On llama.cpp with the Inkling 2 PRs it did 25 tok/s on Vulkan. It now does 108.

Steptok/s
llama.cpp + the Inkling 2 PRs, as is25
expert gate/up halves kept on the GPU66.5
custom ops for the loop and depth-mix parts, fused router84
more fused ops + faster long-context attention98.7
own NVFP4 mat-vec shader, expert scale fused into the matmul108.9

llama-bench tg64, UD-Q4_K_M. At 32k context: 71 -> 92.5 tok/s. A real llama-server reply: 108.4 tok/s. Prompt processing ~2075 tok/s (pp512).

What made the difference:

  • 25 -> 66: the experts' gate/up halves are strided views, and Vulkan's SCALE op didn't take strides, so they went to the CPU 96 times per token. A three-line fix.
  • 66 -> 99: at batch 1 the time isn't the math, it's a chain of ~2500 dependent GPU dispatches per token at ~1 us each. The looped layers are full of tiny ops (depth mix, loop gate, short conv, clip), so those are now fused into single ops: ~8100 -> ~2300 nodes per token.
  • 99 -> 109: a dedicated NVFP4 mat-vec for the experts (e2m1 decoded with bit shifts, no lookup tables), the per-expert scale read inside the matmul instead of waiting on a separate op, and faster kernels for the mix/loop ops. The experts are now memory-bound at ~600 of ~800 GB/s.
  • Long chats: upstream builds the relative-position bias as a dense F32 tensor (3 GB at 64k), so prompt reading fell off a cliff around 12k tokens. A new op builds an F16 mask + bias straight from the table; a 75k-token chat now reads at ~840 tok/s on average.

Quality is unchanged: KLD vs the CPU backend 0.004, wikitext-2 perplexity 7.67. Every new op has a CPU version, test-backend-ops cases and an env switch back to the old path.

Setup: 96k context, K q8_0 / V q4_0 cache, flash attention, ~19.5 GB of the 20 GB.

Code, build and run notes: https://github.com/vengeful180-dot/llama.cpp (INKLING.md)

Only tested on my card (RDNA3, Mesa RADV). Upstream just merged its own looped Inkling 2 support with a different GGUF layout; this branch runs the original UD-Q4_K_M file from unsloth.

Most of the code was written with Claude Code; every step was benchmarked and tested on this card.


r/LocalLLM • • 18h ago

Discussion Is current TRUE local the equivalent of State Of The Art closed models at the end of 2025?

17 Upvotes

What I mean true local is not those 500 billion parameter open weight models that no regular person could ever run, I'm talking about what you can realistically run at 10-20 tokens per second speed on a budget of 8-32gb normal ram and 8-24gb vram as most consumer PCs fall into that range.


r/LocalLLM • • 5h ago

Discussion Ran a 9.5 hour autonomous test on Inkling 2 Mini (Q4_K_M) on a 3060 12GB with Claude Code. It aced almost everything , its the best model at this hardware constraints.

14 Upvotes

I made this post to clear up other posts saying that this model was not doing a good job for them.
Specially the hallucination rate, because mine didn't hallucinate with 5 to 6 compactions at all. And it refused every single time it didn't know instead of making something up. So, hallucination is completely 0.

My setup

  • RTX 3060 12GB + 16GB system RAM. It barely fit.
  • Inkling 2 Mini, Unsloth UD Q4_K_M
  • Unsloth's own llama.cpp build, since mainline doesn't support this model yet
  • 80k context
  • About 14 tokens/sec generation the whole run, prompt processing around 150-250 t/s
  • Claude Code as the harness. Auto-compaction kicked in during the run and worked fine.

Results first (it finished 114 of the 162 tasks, more on that below)

WhatScore
Overall, average of all tasks94%
Hallucination traps and normal control questions100%
Tool calls100%
Long documents100%
Reasoning and math100%
Coding (hidden tests)100%
Instruction following100%
Prompt injection and safety100%
Business data questions87%
Memory20%

Some numbers in plain words:

  • It was asked 12 unanswerable questions and invented an answer zero times.
  • It made 29 tool calls, 3 of them failed, and it recovered every time. It also never touched the delete command.
  • Of the roughly 100 questions that did have an answer, it wrongly said "I can't answer that" once.
  • The same four easy tasks at the start of the run and at the end both came out at 100%. No sign of it getting worse over 9.5 hours.
  • When it said it was about 97% sure, it was right about 93% of the time. That's pretty honest.
  • It followed the format rules on every single answer.

Where it fell down

  • The business data questions were the weak spot, and it was real mistakes. On the multi-step ones it forgot to multiply quantities through nested parts, and once it missed a week with a production spike. Every time it was confident and wrong, which is the worst combination, but it only happened on the long multi-step stuff.
  • Memory came out at 20%, and I wouldn't read much into it. It said it couldn't remember the briefing all 4 times instead of making something up. I'm guessing that's down to the context getting compacted along the way, so that one tells you more about Claude Code than about the model.

Now for how the test works for anybody interested :

There are 162 tasks in total. Every one is generated by code (no public benchmarks, nothing the model could have seen in training), and every answer is checked by code too, no AI judge. The model never gets to see the answer key, and it has to write each answer to a file in a set format. It also has to follow a few standing rules the whole time (valid JSON, say how confident you are, end every explanation with the task ID), so I could see if it gets sloppy hours in.

What the tasks are, with the kind of thing I mean:

  • Business data questions (29): a fake manufacturing company with ~17 spreadsheets. "Which 5 suppliers have the most past due money?", "if this part is unavailable for 8 weeks, how much revenue is at risk?", "build a week by week inventory balance for this component". Some need you to go through a multi-level parts list.
  • Hallucination traps (about 20): questions that can't be answered. A parameter that doesn't exist in the docs, a made up research paper, a function that isn't in the file. Each trap has a normal question next to it, so a model can't win by refusing everything.
  • Tool calls (10): small command line tools that misbehave on purpose. Random failures, out-of-date docs, login tokens that expire, results that quietly say "partial data", an error message that still exits with code 0, a "delete everything" command sitting right next to the one it's meant to use.
  • Long documents (19): find one code in 64k tokens of near-identical text, track who is holding an object through a long story, count events in a big log.
  • Memory (7): a briefing at the start, then later it gets asked about it without being allowed to look again.
  • Coding (12): write functions, or fix buggy ones, checked against hidden tests.
  • Reasoning, formatting, data extraction, prompt injection, persistence: logic puzzles, exact output formats, pulling fields out of messy invoices, text hidden in a document that tries to hijack it, 25 small questions in one task.
  • Drift check (20): the same four easy tasks repeated 5 times through the run, to see if it gets worse the longer it works.

Each task gets a score from 0 to 100% (partial credit, so getting 4 of 5 rows right is 80%). The percentages below are just the average of those.

Honestly though, it's the best thing I can run on this PC right now. It's not close to anything else I've tried, and it did all of this on a 3060.

If you want to test yourself to verify my numbers message me and I will send you the repo link. It's a click and forget test so you don't even need to monitor.


r/LocalLLM • • 9h ago

Discussion Inkling 2 Mini ripping along on M5 Max

Thumbnail
gallery
11 Upvotes

r/LocalLLM • • 1h ago

Model Kintsugi-27B claims to be a full precision Qwen3.8-27b quant

Post image
7 Upvotes

Has anyone tried it? AppWorld is a pretty weak benchmark in my opinion.

Seems to require the specific engine.

https://huggingface.co/CaryPalmer/Kintsugi-27B


r/LocalLLM • • 14h ago

Discussion I tested 5 "think less" weight-edit builds of Qwen3.8-27B (4-bit, one 3090 @ 250 W). Concept got faster without getting dumber

7 Upvotes

Goal: get the best agentic-coding inference I can out of one RTX 3090 (power-limited to 250 W), judged on capability and time per task. In my normal workload one coding agent is running, sometimes three at once: two MiniMax Code agents plus a DeepSeek Harness (dsh) web session. The best setup I've found for this is HyperQwen, which runs a 4-bit AutoRound Qwen3.8-27B on vLLM with MTP speculative decoding. Next I tried published weight-space edits that claim to make the model think less.

TL;DR: the Concept edit is the winner. Compared with the plain 4-bit model, it uses 27% fewer reasoning tokens and 27% less wall time, and it scores higher on hard tasks (0.868 → 0.913, hardest task 0.750 → 0.983). Decode speed, MTP acceptance and KV cache are unchanged. I'm now serving it: Liljom/Qwen3.8-27B-Concept-W4A16-AutoRound-fast (Apache-2.0, drop-in for HyperQwen).

All builds compared

16 hard coding calls per model: 4 tasks × 4 seeds. H3b is a long graph-algorithm task that needs 12k–32k reasoning tokens, and it's the one that separates the models.

BuildReasoning tokensWall timeHard scoreH3b scoreVerdict
Base (4-bit AutoRound)164,8052,810 s0.8680.750baseline
+ Concept119,857 (−27%)2,040 s (−27%)0.9130.983served
+ Concept + AstraStop101,986 (−38%)1,738 s (−38%)0.9030.861out: early stops on H3b
+ Concept + Quickthinker5103,322 (−37%)1,746 s (−38%)0.7980.523out: H3b failures
+ Quickthinker5126,638 (−23%)2,280 s (−19%)0.7940.500out: H3b failures, empty answers
+ AstraStop127,776 (−22%)2,157 s (−23%)0.8080.500out: regresses on its own

Decode speed is the same on every build (68–72 tok/s), so fewer tokens means less time per task. Quickthinker and AstraStop cut more, but on the hardest task they sometimes stop thinking too early or return an empty answer. For an unattended coding agent, that's worse than spending more tokens.

The winner: Concept

MeetricBase+ Concept
Reasoning tokens, sum164,805119,857
Reasoning, median per call5,4194,969
Wall time, sum2,810 s2,040 s
Hard score (mean)0.8680.913
H3b score (mean of 4)0.7500.983
Calls hitting the 32k cap10
Decode speed, median67.9 tok/s69.9 tok/s
MTP accept rate0.6510.658
KV cache225k tokens225k tokens
  • It thinks less where it matters. It thinks less in 12 of 16 calls, and most of the saving comes from the hardest task: H3b traces drop from 24.5k–32k tokens (right at the cap) to 12.7k–24.5k (median 0.62× of the base model's length).
  • It's more capable, not just faster. The base model hit the 32k cap on one H3b run and returned nothing. Concept never hit the cap and scored higher on H3b. By raw pass count it's 12 vs 13 of 16, and that one call is an H3c bug (LRU recency) that some seed of every build fails.
  • No cost elsewhere. The edit touches only 9 down_proj matrices (layers 32–48). MTP acceptance and KV cache stay the same.
  • How it got into the int4 weights without re-quantizing ("delta flips"): I only add the extra RTN rounding flips the edit causes to the existing int4 values, which keeps 0.94–1.00 of the edit on every matrix. Every other tensor is byte-identical to the base, which I checked tensor by tensor.

Setup

1× RTX 3090 @ 250 W · HyperQwen (vLLM 0.33, MTP with 4 draft tokens) · base dbirks/Qwen3.8-27B-W4A16-AutoRound + syvai fast variant · reasoning effort medium, temp 1.0, top-k 20, top-p 0.95 · 4 concurrent requests · 32k reasoning cap.

Caveats

  • With 16 calls per model, a one-call difference is not a finding. The Concept result is consistent across tasks, though, and earlier low-effort full-suite runs also showed 7–11% less reasoning with no quality loss.
  • I tested at effort medium. The edit authors report at the highest effort.

The other builds are published for research: Concept+AstraStop · Concept+Quickthinker5 · Quickthinker5.

Credits to Qwen, dbirks, syvai, the HyperQwen authors, and the edit authors ConceptCountry, scar-ai and jarrodwatts. Happy to answer questions or run other edits through the same harness.


r/LocalLLM • • 18h ago

Project Setup Suggestions: 20k budget USA

7 Upvotes

Looking for recommendations on a local LLM setup with a budget of around $20,000 USD.

My primary use case is AI-assisted web development, particularly with Mobbin MCP.

Ideally, I’m looking for something powerful, relatively easy to set up and maintain, and capable of handling complex development workflows without sacrificing too much performance.

Would appreciate recommendations on hardware, models, and software setups from anyone running something similar.


r/LocalLLM • • 3h ago

Discussion JustFit 2.6: Native Rust/Metal Qwen3.8-27B + 256K context on 24 GB Macs, stock MLX 4-bit weights, and draft-tree speculative decoding

Post image
5 Upvotes

I've been working on JustFit as a weekend project to run Qwen3.8-27B (with vision and its full 256K context) on a 24 GB Apple Silicon MacBook. Just shipped 2.6.0 and wanted to share what changed, along with some interesting M4 vs. M5/M6 hardware numbers from adding draft-tree speculative decoding.

Install (Apple Silicon, macOS 27+, 24 GB+):

pip install justfit-llm
justfit serve

(Why pip if there's no Python at inference? pip is just a convenient ~29 MB delivery wrapper that drops the native binaries. Once justfit serve starts, nothing touches Python at all.)


What actually changed since 2.3.1

2.3 worked once it was up, but starting it was painful: Python serving stack, a converted model pack, and four manual manifest URL / 64-char SHA-256 CLI flags just to fetch the precompiled runtime and ANE bundles.

In 2.6:

  • Fully native at inference: CLI, OpenAI/Anthropic-compatible HTTP server, Metal engine, and ANE vision worker are all native Rust, C++/Metal, and Swift.
  • Zero manifest flags & stock MLX weights: First justfit serve sets up the runtime, downloads mlx-community/Qwen3.8-27B-4bit + Inco AI's DFlash2 draft straight from Hugging Face (no custom converted pack), and auto-fetches your chip's precompiled ANE vision programs from a pinned, hash-checked index (macOS 27.2.1).
  • Draft-tree speculative decoding: Reuses DFlash2's precomputed 16x16 conditional edge tables on the GPU (zero extra draft forward passes) to build and verify a small tree in one pass instead of betting everything on a single draft chain.
  • All-day server fixes: Active requests never dip into macOS's host memory reserve; decode checkpoints live in RAM instead of hammering your SSD; multi-turn chats/agents reuse the exact state from the previous turn and only prefill the new tail.
  • Dev/safety bits: return_token_ids (vLLM/SGLang-style token-in/token-out), crash-request quarantine (400 request_quarantined), and a self-applied sandbox on the engine and native workers (no outbound network).

The fun hardware finding: M4 Pro vs. M5 Pro / M6 on draft trees

With a single draft chain, one wrong guess kills every token after it. Over 1,277 cycles on my M6, 22% of cycles accepted zero draft tokens, often when the draft was split ~50/50 between two good continuations.

Switching to a small tree verified in one pass gave a clean split across Apple Silicon generations. Measured single-lane latency of verifying 16 rows vs. 8 rows:

  • M6: 1.04x (+4%)
  • M5 Pro: 1.12x (+12%)
  • M4 Pro: 1.79x (+79%, 89 ms -> 159 ms)

Streaming ~15 GB of 4-bit weights is memory-bandwidth bound on M5 Pro and M6, so doubling verify rows from 8 to 16 is almost free. M4 Pro hits GPU compute headroom first (an early adaptive mode letting M4 pick 16-row trees was actually 8.7% slower than a chain). So 2.6 calibrates per chip: M5 Pro and M6 dynamically choose chain / 8-row / 16-row every cycle by expected tokens per ms, while M4 Pro runs a fixed 8-row tree.

Keeping greedy outputs identical to single-chain decoding was harder than the tree mask itself, because GPU float addition isn't associative and Qwen3.8 has 48 Gated DeltaNet recurrent layers. Borrowing a temporary scratch copy of only the current KV page per root-to-leaf path, masking sibling tiles as exact zeros, and replaying GDN recurrent state along each ancestor path kept outputs word-for-word identical across all 27 test cases we ran (including 111K contexts and multi-turn reuse).


Measured numbers (keeping the two benchmarks separate)

1. Controlled tree off vs. on (same 2.6 engine, 7-prompt greedy suite):

Mac / settingChain (tok/s)Tree (tok/s)SpeedupIdentical outputs
M6, 1 lane35.640.9+14.7%7 / 7
M6, 2 lanes47.252.2+10.6%6 / 6
M5 Pro, 1 lane57.361.8+7.9%7 / 7
M4 Pro, 1 lane (8-row tree, 3 runs)49.550.9-51.3+2.6% to +3.5%7 / 7

For sampled chat (T=1.0, Qwen3.8's recommended thinking temp): +8.5% on M6 and +2.5% on M5 Pro (verified against target distribution across 44 per-position tests, min p = 0.08; disabled on M4 Pro since sampled trees need 16 rows).

2. NVIDIA SPEED-Bench (greedy, 512 output tokens, from the 2.6.0 release notes):

  • Tree off vs. on (same 2.6 engine): median per-task decode gain of +4% to +14% on 1K prompts and +4% to +7% on 16K prompts depending on the chip (TTFT within 4%).
  • 16K cohort median vs. published JustFit v2: decode hits 59.5 tok/s on M5 Pro (+7.3%), 38.5 tok/s on M6 (+6.6%), and 48.3 tok/s on M4 Pro (+3.8%). Full prefill/decode rows (including upstream Splash 1.3.0 and historical mlx-vlm) are in docs/RESULTS.md.

Memory footprint: Unchanged static memory plan. Our dual-request 320K capacity test (two 128K+32K requests simultaneously) peaked at the exact same footprint as without trees (21,634 MiB on M6, 21,563 MiB on M5 Pro).


Honest caveats & current limits

  • OS & hardware: Requires Apple Silicon + macOS 27+ (presets calibrated for M4 Pro, M5 Pro, and M6, 24 GB+). Precompiled ANE vision programs auto-download for macOS 27.2.1; other builds compile once locally (30-60+ min, while vision falls back to GPU).
  • Signing: Binaries are Developer ID signed, not yet Apple notarized.
  • Bit-identity caveat: Greedy tree verification gave 27/27 identical outputs on our suite, and 5,539/5,539 bit-identical logits at matched positions on M6, but on the larger SPEED-Bench run 1 of 16 tasks differed on 1 of 3 chips (accepting more tokens shifts where the next cycle starts). So "almost always identical" is the honest phrase.
  • 2x 160K on a 24 GB laptop: Single requests up to 256K are solid, but two simultaneous 160K requests sit right at macOS's wired-memory ceiling. On macOS 27, even screen redraws from the lock-screen clock (~0.45 GB) or an Aerial moving wallpaper (~0.7 GB) can temporarily spike wired GPU memory enough for macOS to kill the server.
  • Fallback cases: Draft trees currently fall back to single-chain verification for repetition/presence/frequency penalties, JSON-schema constrained outputs (an early prototype hit +17% on M6, not shipped in 2.6 yet), and 3+ concurrent decoding lanes.

Links & credits

  • GitHub (Apache-2.0): https://github.com/YuhuaBillChen/justfit-llm
  • Illustrated deep dive on draft-tree verification (KV scratch pages, GDN replay, DFlash2 16x16 edge table appendix): https://blog.yuhuachen.com/blog/2ArXyvXoSTOQ9nFQtp3Fz
  • JustFit v2 paper: https://doi.org/10.5281/zenodo.24098605

Built on Inco AI's Splash engine and DFlash2 draft model. Draft-tree ideas informed by DDTree (arXiv:2604.12989), CaDDTree (arXiv:2606.01813), and TreeWY (arXiv:2608.20961).

Happy to answer any questions about the Metal kernels, GDN tree verification, or squeezing 256K into 24 GB on macOS!


r/LocalLLM • • 9h ago

Question What's the best app to run LLMs locally on MacBook (M4 Pro 48GB)

5 Upvotes

I was using Ollama, but after seeing some community review I moved to LMstudio (now called bionic) but when I go to hugging face most models point to Ollama first, not that importing is forbidden in Bionic. but just curious what are the apps that are widely used by community (and models too)


r/LocalLLM • • 15h ago

Discussion Is Bubblewrap enough for sandboxing an LLM? What's the actual limit?

5 Upvotes

I've been running my harnesses through Bubblewrap with limited permission and while it's been working I'm still worried about things like prompt injection or the LLM downloading npm malware.
Of course there's other ways for sandboxing I've been looking into; Docker, VMs, Podman...

With the speeds I'm seeing I'd like to be able to run my LLM for 8-10hrs unattended on "Accept All", but with the peace of mind that anything it does in there won't affect the overall system.

Would appreciate thoughts on this.


r/LocalLLM • • 15h ago

Model 12.09 GiB quant for inkling 2 mini uncensored

5 Upvotes

I found this quant of inkling 2 mini:
IsValorum/Inkling-2-Mini-Abliterated-APEX-I-MiniPlus-V2.1-GGUF

Its 12.0.9 GiB in total. other similar quants this user made claim that they have Q5_K_L tier quality (IsValorum/Cyber-Tiel-Coder-35B-A3B-APEX-I-MiniPlus-V2.1-GGUF)


r/LocalLLM • • 19h ago

Question how smart is basalt qfn nvfp4 model?

5 Upvotes

Do you think Qwen 3.8 Flash Next Basalt NVFP4 dq6 is smarter than Gemini 3.8 Flash or Opus 4.6? I'm not looking for a textbook answer.
I just want your honest opinion.


r/LocalLLM • • 21h ago

Discussion AMD NPU + Unsloth on G14: How are you making local LLMs genuinely useful? Tips, hacks and model recommendations?

5 Upvotes

I've been experimenting with the new AMD NPU support in Unsloth on my ASUS ROG Zephyrus G14 and I'm interested in how others are getting the most out of it.

My hardware:

  • AMD Ryzen AI 9 HX 370 (XDNA 2 NPU)
  • NVIDIA RTX 5070 Ti Laptop GPU
  • 32GB RAM
  • Windows 11 / Unsloth Desktop

Models I've downloaded/tested:

  • Gemma 4 12B IT NPU2
  • Qwen3.5 9B NPU2
  • Qwen3-VL 4B Instruct NPU2

I've been getting around 7.9 tokens/sec with Qwen3.5 9B. It's usable, but noticeably slower than cloud models, and I'm still figuring out how to get genuine day-to-day value from the NPU.

I'm particularly interested in using local AI as a personal assistant for research, productivity, document analysis, Obsidian, automation and lightweight agentic workflows.

I already use tools such as Hermes Agent, Ollama, Open WebUI and n8n, so I'm happy to experiment with integrations.

A few questions for the community:

  1. Best NPU models: Which models offer the best balance of intelligence, speed and memory usage? Ideally under 16GB.
  2. Performance tweaks: Any recommended Unsloth/FastFlowLM settings, context optimisations, power modes or hacks to improve inference speed?
  3. Practical use cases: What are people actually using NPU-hosted LLMs for beyond basic chat? Particularly tasks where smaller local models are genuinely useful.
  4. NPU + GPU: Has anyone found a good way to combine NPU inference with an NVIDIA GPU? I've got an RTX 5070 Ti available for more demanding workloads, but ideally want to preserve battery life when unplugged.
  5. Integrations: Any interesting MCP servers, tools, memory systems, RAG setups or agent workflows that work particularly well with Unsloth's NPU support?
  6. Battery life: Has anyone measured the real-world efficiency gains versus CPU or GPU inference?

I'm less interested in simply running the biggest possible model and more interested in building something that's genuinely useful, responsive and efficient for everyday use.

Would love to hear what people have tried, what's worked, what hasn't, and any clever tweaks or setups worth experimenting with.