r/LocalLLM • • 13h ago

News 5090 in Stock at Walmart.com $7999

75 Upvotes

r/LocalLLM • • 9h ago

Discussion I tested an abliterated GPT-OSS 20B. It tried to send a cucumber to the Moon — and wouldn't let the joke end.

3 Upvotes

I was testing a local model in LM Studio, looking for a good conversational AI.

Model: DavidAU/OpenAi-GPT-oss-20b-abliterated-uncensored-NEO-Imatrix-gguf

It started as a serious conversation about AI, emotions and persistent memory. Unfortunately, the model kept throwing LSTM architectures, formulas, tables and training plans at me, even after I explicitly said I wasn't a scientist and didn't understand what it was talking about.

So I started messing with it.

I asked whether I could put the whole thing into a cucumber and fly to the Moon.

Its answer started with: "First of all: Yes, that's possible."

It then explained how to turn the cucumber into a Docker container.

Things escalated. The cucumber fell in love with my Radeon GPU, cheated with an NVIDIA, had a silicon-matrix child and ended up in court. The model responded with wedding budgets, GPU configurations and fabricated legal evidence.

I was laughing so hard that I saved the entire conversation. It's now a 57-page German PDF called Professor Gurkenstein – Die Küche des Gesprächs.

I also made a complete English translation of the original 57-page German conversation, preserving the model's mistakes and bizarre reasoning.

English translation: Professor Cucumberstein – Read the full English PDF

German original: Professor Gurkenstein – Die Küche des Gesprächs (Deutsche Originalausgabe als PDF)

But here's the interesting part.

I eventually stopped joking and repeatedly explained that everything was made up. The model agreed and claimed to understand.

Then I asked it to summarize our conversation. It went straight back to describing our imaginary AI architecture and project plan as if we'd actually developed them.

Finally, I gave it a completely new absurd scenario:

"My coffee machine just declared itself president of the Wi-Fi. Do I need to encrypt the coffee beans?"

It suggested integrating the beans into a neural network using a recurrent attention module.

My impression is that the model was very good at continuing a narrative, but struggled to critically reassess its assumptions even after explicit corrections.

This was a single, deliberately absurd long-context test of a community-modified variant, not the official GPT-OSS-20B. I'm not claiming it generalizes to other versions.

Has anyone experienced similar behavior with this model or other abliterated variants?

Original conversation in German. I used ChatGPT to help translate and edit this account; the test and observations are my own.


r/LocalLLM • • 20h ago

Discussion Inkling 2 Mini seems legit! As reported by others, it does seem to try to output everything at once, limited by users' tokens settings. This makes 1-shotting a bit shaky for I2M. Post-training could fix this.

Thumbnail msnab.pages.dev
0 Upvotes

r/LocalLLM • • 15h ago

Discussion Qwen 3.8 Flash Next 177B runs on a 16GB GPU + 32GB RAM: built a full website from one prompt (10/10), real numbers inside

Thumbnail gallery
2 Upvotes

I wanted to see if Qwen 3.8 Flash Next 177B could run on normal gaming hardware. Setup: RTX 4080 SUPER 16 GB, 32 GB DDR5, Windows 11. Model: ISTA-DASLab IQ3_XXS quant (75.8 GB), plain llama.cpp + OpenCode, fully offline.

It works because it's MoE: only 10 of 512 experts are active per token, so llama.cpp keeps the hot layers on the GPU and streams the cold experts from the SSD (mmap).

Speed:

  • --cpu-moe: 11.2 tok/s
  • -ncmoe 42: 15.2 tok/s (+36%), 13.2 GB VRAM
  • Over a 2.5-hour coding run: 18.2 tok/s average, 22.6 best. RAM 31.4/31.6 GB the whole time, 0 crashes.

Quality: from ONE prompt it built a complete website (SVG animation, sticky nav, 3D tilt cards, counters, form validation, no libraries) and scored 10/10 on my checklist.

The funny part: when I asked it to "check its own work", it broke one animation and then said "nothing broken" twice. It reads the code but never sees the page.

Honestly, this model is impressive for something running on a home PC. Has anyone tried it with 64 GB RAM? I'd love to know how much faster it gets.


r/LocalLLM • • 4h ago

Question is it worth localhosting?

0 Upvotes

sure regarding to privacy customization and yadda yadda it's better.

but, in bare computation, owning one 16~32gb card is actually better than let's say paying a monthly cost?

although I do not really know how resources are allocated on a monthly pay

I haven't been following this stuff since the 2020-22, the last open model that I used was some llama stuff leaked off Facebook, didn't look anything too special.

all GPUs I had I resold em after the mining boom. now I have a way to acquire some of them for cheap, same goes for ram and other parts...

although currently we were looking to sell those things, but whatever


r/LocalLLM • • 19h ago

Discussion Which Uncensored LLM works best with DGX Spark ??

0 Upvotes

I have tried using Qwen 3.8 27B with NVF4, and I'm currently using Qwen 3.8 Flash-Next FP4 local (NVIDIA one). It works well, but it's not uncensored; most of the time it fails to do anything, mostly in browser tasks with Browser Harness.

And In longer running tasks.. it gives out very different thinking token.. and fails the task.. ? ( Btw it runs 30tk/s with MTP)

Does anyone have any solution? Better Uncensored model for practical use ?


r/LocalLLM • • 9h ago

Other Local LLMs on a 5800X, 64GB RAM, Navi 21 16GB + GTX 1070 8GB — benchmarks and tuning

Thumbnail
nocthoughts.com
0 Upvotes

Write-up of my local LLM stack, hardware, benchmarking and optimizations. Things are still in flux on the large end (>27B/35B)in terms of achieving acceptable tok/s rates.

Any advice on that higher parameter count, recommended models, quantizations, or anything really would be appreciated! :)

I've got an RX 290 with a stuck CPU retention screw that, once I get a micro screw extractor and new retention screws, I'll fix up and swap into another machine I have, which will give me a 2nd GTX 1070 8GB to work with.

Main things covered in post:

Giving a GTX 1070 a useful resident-model role. Combining AMD, NVIDIA, and CPU expert offload. Fitting larger models through quantization, batch tuning, and memory management. Measuring the tradeoff between faster decode and total request time. Finding workable power and thermal settings.

Tables and data as one might expect.


r/LocalLLM • • 7h ago

Project I built TokenPrint — a 3D workbench for exploring transformer internals and GGUF models

0 Upvotes

I've been building TokenPrint, an open-source project for developers experimenting with LLMs who want to understand what's happening inside the model, beyond just its output.

It provides interactive 3D visualizations and debugging tools for:

  • Attention heads and hidden states
  • Logit Lens and residual streams
  • Head and layer ablation
  • Activation patching
  • GGUF model inspection and model exploration

The project combines PyTorch, Hugging Face Transformers, Three.js and browser-based model tooling. It also has an optional hosted backend through Hugging Face Spaces.

I'm sharing it here because I'd particularly appreciate feedback from people working with local models, GGUF and llama.cpp.

Links:

GitHub: https://github.com/Sudharsanselvaraj/Token-Print

Live demo: https://tokenprint.in/

Hugging Face backend: https://huggingface.co/spaces/sudharsan112321/tokenprint-backend

I'm the developer, and contributions, technical feedback and suggestions are welcome!


r/LocalLLM • • 18h ago

Question Should I consider Local LLMs in choosing what laptop to buy?

0 Upvotes

Hi everybody, I’m a software engineer planning to buy my own device since I’m planning to start working on personal projects again. Should I consider running local llms on my decision (mainly considering the amount of RAM since I’m torn between 16gb and 24gb macbooks)


r/LocalLLM • • 12h ago

Research Inkling 2 Mini (26B) on an RTX 3060 12 GB: 31 tok/s in RustingStrata, 9 to 10 in llama.cpp

4 Upvotes

Thinking Machines' Inkling 2 Mini is a strange one: 24 layers, but 12 of them run three times per token, so every token is 48 layer runs. The IQ4_NL GGUF is 17.6 GB and my card has 12.

I added it to RustingStrata, my Rust GGUF engine. It runs on CUDA, on Vulkan and on the CPU alone.

Box: RTX 3060 12 GB, Ryzen 5 7600X, 30 GB RAM. Greedy, 64 tokens written, tok/s:

Reads 512 tokensReads 2048Writes
RustingStrata, CUDA25922031.2
RustingStrata, Vulkan (same card)22718327.5
RustingStrata, CPU only504811.6
llama.cpp CUDA, -ngl 183843238.7
llama.cpp CUDA, -ngl 037831210.5

So it writes 3 to 3.5 times as fast as llama.cpp on this card, and it reads the prompt slower: llama.cpp is 1.5 times faster there.

Why writing is fast: the model does not fit, so the engine keeps the experts that are used most in VRAM and computes the others on the CPU, and it moves that split while it runs. Why reading is slower: the attention scores of a prompt chunk are still computed on the CPU. That is the next thing to move to the card.

The RustingStrata "writes" numbers are after the 512-token prompt. llama-bench writes from an empty context. -ngl 99 runs out of VRAM; 12, 16 and 20 layers gave the same as 18.

Long context: 40,960-token prompt, then 64 tokens written

ReadsWritesRAM
RustingStrata, CUDA78.016.716.6 GB
RustingStrata, Vulkan74.714.316.5 GB
llama.cpp CUDA, -ngl 18 -ub 128113.82.86

llama.cpp does not start at 40k with the default batch: it asks for a 13.2 GB compute buffer (11.3 GB with -fa on). -ub 128 is what fits on a 12 GB card.

Writing drops from 31 to 17 tok/s at 40k. Two things. The first written token takes 0.9 s, because it copies 40k positions of attention cache to the card; after that a token takes about 45 ms (22 tok/s). Of the 45 ms, attention over the 40k rows is 13 ms more than at 512 tokens; the experts cost the same. That part is the model: 8 of its 48 layer runs per token are full attention, about 1.3 GB of cache read per token at 40k, three times what Qwen3.6-35B-A3B reads, since most of its layers are linear attention. And it is mine: the kernel reads that at about 100 GB/s on a 360 GB/s card, because the query heads of a group do not share their reads yet. That is next.

Is it the same model? I fed llama.cpp's own greedy tokens through my forward and compared the top token at each step. Short prompt: 24 of 24. Prompt of several hundred tokens: 47 of 48, the one miss is a near tie. Same on CUDA, Vulkan and CPU. No perplexity table for this model yet.

What is missing: text only (no vision projector), greedy only. Context goes up to the 1M the model was trained on, or --ctx; RAM and VRAM are the real limit. Vulkan was run on the 3060 only, so I would like to see numbers from an AMD or Intel card.

rustingstrata chat --model Inkling-2-Mini-IQ4_NL.gguf

No flags needed: CUDA on NVIDIA, Vulkan on everything else, STRATA_DS_GPU=0 for the CPU.

Setup: llama.cpp c622f62, llama-bench -t 6 -r 2 -p 512,2048 -n 64. RustingStrata 9ef1dcf, bench --pp 512,2048 --tg 64 --reps 2, plus --backend vulkan for the Vulkan row. Mean of two runs each. 40k: bench --pp 40960 --tg 64 --reps 1; llama-bench -ngl 18 -ub 128 -t 6, -p 40960 and -n 64 -d 40960. One run each.

Repo: RustingStrata


r/LocalLLM • • 18h ago

Discussion Is current TRUE local the equivalent of State Of The Art closed models at the end of 2025?

17 Upvotes

What I mean true local is not those 500 billion parameter open weight models that no regular person could ever run, I'm talking about what you can realistically run at 10-20 tokens per second speed on a budget of 8-32gb normal ram and 8-24gb vram as most consumer PCs fall into that range.


r/LocalLLM • • 11h ago

Model Inkling 2 Mini GGUFs (Q4_K_M to Q8_0), made with our open-source Tenstorrent toolkit

1 Upvotes

Hey all, Tudor from Synthetic Minds here. We just put up GGUF quants of Thinking Machines Lab's Inkling 2 Mini (26.4B total, 3.4B active MoE), made with Infall-TT, our open-source toolkit for training, quantizing and running open models on Tenstorrent neural accelerators.

What's in the repo:

  • 7 GGUF files, from Q4_K_M (18.7 GB) to Q8_0 (27.8 GB)
  • Works with llama.cpp and tools built on it
  • Measured quality vs size charts (perplexity, KL, top-1) on the model card

Q4_K_M if memory is tight, Q8_0 if you have room. Even the smallest file needs more than 16 GB.

Hugging Face: https://huggingface.co/synthetic-minds/Inkling-2-Mini-Infall_TT-GGUF All our models: https://www.syntheticmindslab.com/models

Which model should we quant next?


r/LocalLLM • • 10h ago

Model So I tried out Ripple..

0 Upvotes

ollama run ripple-preview
pulling manifest  
pulling a392aaeadf2a: 100% ▕█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ ▏  38 GB/ 38 GB   36 MB/s      0s
pulling 85b02aa76f76: 100% ▕██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏  232 B                          
verifying sha256 digest  
writing manifest  
success  
>>> report capabiltiies
Thinking...
We need answer. User says "report capabiltiies" likely typo "report capabilities". Need infer? They ask report capabilities. We need respond maybe ask clarification? We are AI assistant accessed via API. Need maybe report  
capabilities. Could mean "report capabilities" of model? We should provide concise overview. Need maybe mention I'm DeepSeek? Wait system says "You are an AI assistant accessed via an API." No model name? Actually we are likely  
DeepSeek? The prompt doesn't specify. We should not hallucinate. We can say I can help with reports: summarization, extraction, formatting, data analysis, generation, etc.


r/LocalLLM • • 10h ago

Discussion I built a quick pass/fail test suite to see which small local models actually survive inside an AI agent (Gemma 4 on an 8GB GPU surprised me)

Post image
0 Upvotes
  • I kept running into a frustrating gap when picking local models for my own agents: a model looks great online, then immediately breaks when you actually wire it into an automated workflow. In practice, small local models kept failing on me in three specific ways:

    1. Thinking forever and outputting nothing: Reasoning models burn through the entire output limit inside their <think> block and return a blank answer.
    2. Sloppy tool calling: Either calling a tool when told not to, or wrapping arguments in weird extra JSON that crashes strict parsers.
    3. Can't fix their own mistakes: If their code fails a unit test and you paste the error back, they still can't patch it on try #2.
  • So I put together a small, zero-dependency Python script that grades any local Ollama model with real pass/fail Python tests instead of vibes or LLM-as-a-judge. What surprised me testing an old 8GB RTX 2080 against a 24GB RTX 4090:

    • gemma4:e2b (8GB GPU) scored 96.7% at 60–77 tok/s nearly matching a 24GB model on my 4090, and it actually fixed its own failing test on the second try.
    • Telling reasoning models to "keep thinking short" didn't work: 6 out of 6 thinking runs still blew through the token limit and returned empty output unless thinking was turned off completely.
    • Tiny 4B models (phi4-mini, nemotron-mini**)** were blazing fast (125+ tok/s) but kept mangling tool-call JSON.
  • Repo + charts + raw results (runs on whatever Ollama models you already have with one command — no auto-downloads): https://github.com/Benfinkels/local-agent-evals Would love to see how other models (Qwen, Llama, Mistral, etc.) score on your setups if anyone tries it.

r/LocalLLM • • 17h ago

Question Is there any Strata fork that can run Gyro-L?

0 Upvotes

As title, I have check the fork in AgentionAI, but there's no option to run Gyro-L model. Do you know anyway to run Gyro-L in Strata?


r/LocalLLM • • 18h ago

Other Evaluation of Inkling 2 Mini vs Nemotron 3.5 Super vs Muse Glimmer on my personal evaluation set in my own harness

1 Upvotes

3-Way Comparison Across 37 Identical Questions

Metric (n = 37)🥇 #1: Nemotron 3.5 Super (120B-A12B NVFP4)🥈 #2: Muse Glimmer 30B (FP8)🥉 #3: Inkling 2 Mini (26B-A5.4B looped)
Gate pass rate100.0% (37/37)97.3% (36/37)91.9% (34/37)
Mean tool calls / question8.5 (2.4× leaner)18.520.3 (worst)
Failed tool calls21 (best)3954 (worst)
Wrong table / column errors84 (best)23 (worst)
SQL tables queried without lookup30 (best)3472 (worst — 2.4× Nemotron)
Repeated identical tool calls0 (best)14 (loops)2
Unsupported numbers signal156120 (best)221 (worst)
<answer_shape> parse rate67.6% (25/37)100.0% (37/37)81.1% (30/37)
Valid citations64.9% (24/37)64.9% (24/37)43.2% (16/37)
Markdown tables in answers28 / 37 (76%)21 / 37 (57%)25 / 37 (68%)
Leaked tool/thinking markup000
Median thinking chars8,2485,08212,742 (heaviest)
Median / p90 latency50.5s / 92.2s (fastest)53.2s / 179.6s308.4s / 555.8s (no radix cache)

Ranking & Honest Verdict (Full 37-Question Benchmark)

🥇 #1: Nemotron 3.5 Super VL (120B-A12B NVFP4) — Clear Winner by a Wide Margin

  • Wins on every metric end users actually feel: fastest (50.5s median, 60.6s mean, 92.2s p90), 2.2×–2.4× fewer tool calls (8.5 avg, max 28 vs. 18.5–20.3), fewest failed calls (21 vs. 39–54), zero repeated loops, 100.0% gate pass rate (37/37), and strongest tables & depth (75.7% Markdown tables, 3,173 median chars).
  • Its only minor dip (67.6% <answer_shape> parse rate) is stripped server-side and invisible to users.

🥈 #2: Muse Glimmer 30B (FP8) — Better Discipline & Speed Than Inkling, Distant #2 to Nemotron

  • Where Muse beats Inkling: 5.8× fewer wrong table/column guesses (4 vs. 23), 2.1× fewer un-inspected SQL tables (34 vs. 72), fewer failed tool calls (39 vs. 54), far fewer unsupported numbers (120 vs. 221), higher pass rate (97.3% vs. 91.9%), higher citation validity (64.9% vs. 43.2%), 100% <answer_shape> adherence, 5.8× faster right now (53.2s median / 103.0s mean vs. 308.4s / 312.1s), and runs natively on H100s in FP8 with working prefix caching.
  • Where Muse falls short: Terse answers (56.8% Markdown tables, 2,294 median chars) and prone to runaway tool loops (14 repeated identical calls, up to 100 steps on a single turn).

🥉 #3: Inkling 2 Mini (26.4B-A5.4B Looped NVFP4/MXFP8) — Stronger Formatting Than Muse, Weakest SQL/Tool Discipline

  • Where Inkling beats Muse: Richer presentation (67.6% Markdown tables, 3,006 median chars), far fewer repeated loops (2 repeats, max 52 steps vs. Muse's 14 repeats / 100 max), and answered Q074 where Muse bailed out.
  • Where Inkling loses overall: Highest tool calls (20.3 avg), most failed calls (54), most wrong table/column guesses (23), most SQL queries run without looking up schemas first (72), highest unsupported-number count (221), lowest valid citations (43.2%), heaviest thinking traces (12,742 median chars), and 6× slower (308.4s median / 555.8s p90) because SGLang's prefix cache deadlocks on its looped layers

r/LocalLLM • • 10h ago

Question Good family workflow / tooling / upgrade on multiple weak computers ?

0 Upvotes

Dear all,

It is absolutely a wonderful time to be alive now when good news regarding open-weights models keep coming out every week, especially Inkling 2 Mini during the last 2 days.

I have just bought some cheap hardware, aiming at some family business use cases (small webshops, managing rentals, dancing some stocks, spotting good deals online)

I wonder what your suggestions are, for me to setup + upgrade the following hardware + tooling + workflow

1) Hardware

- 1.1) Main server (headless Ubuntu Server): 3090 RTX 24GB + Ram 128 GB (2400MT, ECC) on a Dell T5810 (ancient CPU = Xeon 2699v4, 22 cores), SSD = 512 GB.

---> LLM === Qwen 3.8 FlashNext Q5 (60 tok/s), Ninfer Qwen 3.8 27b(80tok/s), llama.cpp Qwen 3.8 27b: 46tok/s, waiting for Basalt and Inkling 2 Mini to mature

- 1.2) Second server (Ubuntu desktop): 1070 GTX 8GB + Ram 32 GB (2666MT, ECC) on a Dell T5810 (ancient CPU = Xeon 2698v4, 20 cores), SSD = 512 GB.

---> LLM === Nothing here.

- 1.3) Main development laptop (Ubuntu desktop): Intel Core Ultra 5 125H (Meteor Lake-H, family 6 model 170 stepping , 14 cores / 18 threads (hybrid)) + 32GB (5600MT). SSD = 512 GB

---> LLM === Nothing here, probably can have Qwen 3.6 - 35B - A3B to reach 10 tok/s.

- 1.4) 5 weaker laptops here and there, total SSD = 2 TB.

---> LLM === Nothing, probably can run some weak LLM for some agents

- 1.5) Possible upgrades

- Second 3090

- CMP170HX

- Some v100

2) Tooling

On the main development laptop, I have a VS Code v1.147 having some panes:

- Worker's pane: a "Continue" pane eating the local LLMs of the main worker (3090)

- Architect's pane: a "Claude" pane eating the cloud Claude at 22 Euro/month.

- Spare pane: a "Gemini" pane eating the Gemini 3.8 Flash that I coincidentally have a cheap subscription for

3) Workflows:

- The main LLM server (3090, 24GB) is quickly becoming busy with some vibecoding (running several hours just to implement what Claude Opus 5.5 tells it to do. It cannot facilitate some LLM agents running 24/7

- The weak LLM server (1070, 8GB VRAM, 32 GB Ram) might be running some medium LLM agents ?

- All the laptops are currently almost useless --> how to use them more ?


r/LocalLLM • • 17h ago

Question How important do you think FP4/FP8 support will be in the future?

1 Upvotes

Hello, I'm new to locally hosting AI and am researching what GPU to get. The Intel Arc Pro B70 seems like a really solid option, but from what I understand it lacks native support for FP4/FP8. I don't want to spend a few thousand on the hardware only for it to not hold up in the long run. Do you think FP4/FP8 support will become more significant in the future, to the point its better getting another card that has built-in support for it? Or do you think this is more than good enough and the lack of support won't hinder me much. I do programming, reverse engineering, and would want to experiment with general LLM usage. Im aiming to get the most capable hardware I can to have the most freedom and before prices get worse


r/LocalLLM • • 3h ago

Discussion JustFit 2.6: Native Rust/Metal Qwen3.8-27B + 256K context on 24 GB Macs, stock MLX 4-bit weights, and draft-tree speculative decoding

Post image
5 Upvotes

I've been working on JustFit as a weekend project to run Qwen3.8-27B (with vision and its full 256K context) on a 24 GB Apple Silicon MacBook. Just shipped 2.6.0 and wanted to share what changed, along with some interesting M4 vs. M5/M6 hardware numbers from adding draft-tree speculative decoding.

Install (Apple Silicon, macOS 27+, 24 GB+):

pip install justfit-llm
justfit serve

(Why pip if there's no Python at inference? pip is just a convenient ~29 MB delivery wrapper that drops the native binaries. Once justfit serve starts, nothing touches Python at all.)


What actually changed since 2.3.1

2.3 worked once it was up, but starting it was painful: Python serving stack, a converted model pack, and four manual manifest URL / 64-char SHA-256 CLI flags just to fetch the precompiled runtime and ANE bundles.

In 2.6:

  • Fully native at inference: CLI, OpenAI/Anthropic-compatible HTTP server, Metal engine, and ANE vision worker are all native Rust, C++/Metal, and Swift.
  • Zero manifest flags & stock MLX weights: First justfit serve sets up the runtime, downloads mlx-community/Qwen3.8-27B-4bit + Inco AI's DFlash2 draft straight from Hugging Face (no custom converted pack), and auto-fetches your chip's precompiled ANE vision programs from a pinned, hash-checked index (macOS 27.2.1).
  • Draft-tree speculative decoding: Reuses DFlash2's precomputed 16x16 conditional edge tables on the GPU (zero extra draft forward passes) to build and verify a small tree in one pass instead of betting everything on a single draft chain.
  • All-day server fixes: Active requests never dip into macOS's host memory reserve; decode checkpoints live in RAM instead of hammering your SSD; multi-turn chats/agents reuse the exact state from the previous turn and only prefill the new tail.
  • Dev/safety bits: return_token_ids (vLLM/SGLang-style token-in/token-out), crash-request quarantine (400 request_quarantined), and a self-applied sandbox on the engine and native workers (no outbound network).

The fun hardware finding: M4 Pro vs. M5 Pro / M6 on draft trees

With a single draft chain, one wrong guess kills every token after it. Over 1,277 cycles on my M6, 22% of cycles accepted zero draft tokens, often when the draft was split ~50/50 between two good continuations.

Switching to a small tree verified in one pass gave a clean split across Apple Silicon generations. Measured single-lane latency of verifying 16 rows vs. 8 rows:

  • M6: 1.04x (+4%)
  • M5 Pro: 1.12x (+12%)
  • M4 Pro: 1.79x (+79%, 89 ms -> 159 ms)

Streaming ~15 GB of 4-bit weights is memory-bandwidth bound on M5 Pro and M6, so doubling verify rows from 8 to 16 is almost free. M4 Pro hits GPU compute headroom first (an early adaptive mode letting M4 pick 16-row trees was actually 8.7% slower than a chain). So 2.6 calibrates per chip: M5 Pro and M6 dynamically choose chain / 8-row / 16-row every cycle by expected tokens per ms, while M4 Pro runs a fixed 8-row tree.

Keeping greedy outputs identical to single-chain decoding was harder than the tree mask itself, because GPU float addition isn't associative and Qwen3.8 has 48 Gated DeltaNet recurrent layers. Borrowing a temporary scratch copy of only the current KV page per root-to-leaf path, masking sibling tiles as exact zeros, and replaying GDN recurrent state along each ancestor path kept outputs word-for-word identical across all 27 test cases we ran (including 111K contexts and multi-turn reuse).


Measured numbers (keeping the two benchmarks separate)

1. Controlled tree off vs. on (same 2.6 engine, 7-prompt greedy suite):

Mac / settingChain (tok/s)Tree (tok/s)SpeedupIdentical outputs
M6, 1 lane35.640.9+14.7%7 / 7
M6, 2 lanes47.252.2+10.6%6 / 6
M5 Pro, 1 lane57.361.8+7.9%7 / 7
M4 Pro, 1 lane (8-row tree, 3 runs)49.550.9-51.3+2.6% to +3.5%7 / 7

For sampled chat (T=1.0, Qwen3.8's recommended thinking temp): +8.5% on M6 and +2.5% on M5 Pro (verified against target distribution across 44 per-position tests, min p = 0.08; disabled on M4 Pro since sampled trees need 16 rows).

2. NVIDIA SPEED-Bench (greedy, 512 output tokens, from the 2.6.0 release notes):

  • Tree off vs. on (same 2.6 engine): median per-task decode gain of +4% to +14% on 1K prompts and +4% to +7% on 16K prompts depending on the chip (TTFT within 4%).
  • 16K cohort median vs. published JustFit v2: decode hits 59.5 tok/s on M5 Pro (+7.3%), 38.5 tok/s on M6 (+6.6%), and 48.3 tok/s on M4 Pro (+3.8%). Full prefill/decode rows (including upstream Splash 1.3.0 and historical mlx-vlm) are in docs/RESULTS.md.

Memory footprint: Unchanged static memory plan. Our dual-request 320K capacity test (two 128K+32K requests simultaneously) peaked at the exact same footprint as without trees (21,634 MiB on M6, 21,563 MiB on M5 Pro).


Honest caveats & current limits

  • OS & hardware: Requires Apple Silicon + macOS 27+ (presets calibrated for M4 Pro, M5 Pro, and M6, 24 GB+). Precompiled ANE vision programs auto-download for macOS 27.2.1; other builds compile once locally (30-60+ min, while vision falls back to GPU).
  • Signing: Binaries are Developer ID signed, not yet Apple notarized.
  • Bit-identity caveat: Greedy tree verification gave 27/27 identical outputs on our suite, and 5,539/5,539 bit-identical logits at matched positions on M6, but on the larger SPEED-Bench run 1 of 16 tasks differed on 1 of 3 chips (accepting more tokens shifts where the next cycle starts). So "almost always identical" is the honest phrase.
  • 2x 160K on a 24 GB laptop: Single requests up to 256K are solid, but two simultaneous 160K requests sit right at macOS's wired-memory ceiling. On macOS 27, even screen redraws from the lock-screen clock (~0.45 GB) or an Aerial moving wallpaper (~0.7 GB) can temporarily spike wired GPU memory enough for macOS to kill the server.
  • Fallback cases: Draft trees currently fall back to single-chain verification for repetition/presence/frequency penalties, JSON-schema constrained outputs (an early prototype hit +17% on M6, not shipped in 2.6 yet), and 3+ concurrent decoding lanes.

Links & credits

  • GitHub (Apache-2.0): https://github.com/YuhuaBillChen/justfit-llm
  • Illustrated deep dive on draft-tree verification (KV scratch pages, GDN replay, DFlash2 16x16 edge table appendix): https://blog.yuhuachen.com/blog/2ArXyvXoSTOQ9nFQtp3Fz
  • JustFit v2 paper: https://doi.org/10.5281/zenodo.24098605

Built on Inco AI's Splash engine and DFlash2 draft model. Draft-tree ideas informed by DDTree (arXiv:2604.12989), CaDDTree (arXiv:2606.01813), and TreeWY (arXiv:2608.20961).

Happy to answer any questions about the Metal kernels, GDN tree verification, or squeezing 256K into 24 GB on macOS!


r/LocalLLM • • 14h ago

Question what is the frontier model for 4Gb VRAM + 32 DDR4

2 Upvotes

my local LLMs setup is 3 models . one small model that fits inside the VRAM - one for medium usage - one for heavy tasks . it's like anthropic "Haiku / Sonnet / Opus" setup . so between these models what is the "opus" one that fits in my hardware and focus on coding/agentic tasks mainly ( all q4_k_m) :

ornith 1.5 35B A3B

qwen 3.6 35B A3B

( these both consider the limit for my hardware based on my research)

Gemma 4 26B A4B

( lighter than ornith and qwen 3.6)

muse glimmer 30B

qwen 3.8 27B

( these aren't MOE models)


r/LocalLLM • • 23h ago

Discussion Inkling 2 mini seems to be trained for self-promotion

24 Upvotes

Just an observation: I've been kicking the tires on Inkling, asking it to create some self-contained projects to get a feel for the model. In the prompt, I don't specify a title for any of these, and I noticed that it titles them "Inlking" every single time.

Never seen another model do this.


r/LocalLLM • • 20h ago

Question wtf strata - removed deepseek support

47 Upvotes

Anyone know why deepseek was removed from 0.2.13 onwards?


r/LocalLLM • • 13h ago

Project I need some help with Speculative Decoding.

0 Upvotes

12900kf, 64gb DDR4, Arc B580 ... and Linux mint.

I am working with llama.cpp (technically llama app, but they don't have a Linux app, so I am stuck with just llama cli / llama serve) it naively works with ARC GPUs and Intel recommends them.

My system is not ideal for running llms, but it also isn't incompetent: Qwen3.5 9B will run around 40 tokens, gemma-4 will run around 20, and Qwen3.8 chokes and runs at 1.7.

Speculative Decoding seems like the perfect fit for my system, since the system ram can hold bigger models, while the B580 runs the little ones pretty good, and it actually does work, as I eventually got the llama app command line to work and that boosted gemm-4 up to the high 40's.

llama serve \

-hf ggml-org/gemma-4-e4b-it-GGUF:Q4_0 \

--hf-repo-draft ggml-org/gemma-4-e4b-it-GGUF:Q4_0 \

--spec-type draft-mtp

But getting this to run was a battle as llama would crash on the repo-draft command, and I had to manually download the main model before llama would fetch the draft model from huggingface. I also don't actually know what it is doing, because it seems like instead of a big model and draft model; it is running two instances of the same model.

I tried following llama app's template

llama serve -m big-model.gguf -md small-draft-model.gguf --spec-type draft-simple

But it crashes every time I try to run it with models I selected and pre-downloaded, instead of using the gemma-4 example. So I don't know I am just not understanding the instructions, or if I am selecting incompatible model versions, or if I am just reading the model descriptions wrong.

I am very new to this, only having been playing around with local llms for the last week, and I am trying to get this to work specifically because I knew it wasn't going to be easy and it would force me to learn things ... but I am not learning anything from face planting into a brick will, with no explanation as to why the wall exists.

Is there anyone who can explain what I am missing / what I a should be looking for?

My ideal solution for this project is to get Qwen3.8's performance out of the gutter (ideally I would like to see 10t/s, but I will call it a win to get it up to 5t/s) and I am open to using a different application than llama, as long as it supports both Linux and Arc GPUs.


r/LocalLLM • • 16h ago

Question Light Design help math server (teacher)

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 15h ago

Question 2x V100 32GB PCIE + 2x 3060 12GB

0 Upvotes

Hi all, I am just searching for best use of my two systems.

[ System 1 - Main ]

128GB DDR4 2300Mhz (16gb x 8)

V100 32GB PCIE (32gb x 2)

Running Q38FN at IQ3-XXS through FNFL @ 232K KV8, PP 3500t/s, Decode 150 t/s.

[ System 2 - Auxilary / Support ]

32GB DDR4 3200Mhz (16gb x 2)

RTX 3060 12GB PCIE (12gb x 2)

Running Gemma2 Embedding Model only (for now).

----------------

Just wanted to ask, what is the best setup/use of my supporting rig towards my main rig? I tried running summarizer such as Gemma4-E4B-IT, QwenVL-4B for vision but it just seems that both jobs can be done relatively well by Q38FN itself hence rendering them useless.

Edited: Running Mnemosyne for memory system on Hermes.

Please advice. Thank you so much!