r/LocalLLaMA • • 29m ago

Question | Help TTS for reading my AI writing projects to me

1 Upvotes

I'm looking for recommendations on a decent TTS that can read the writing back to me.

I've tried kokoro, orpheus and Zonos. (I'm writing the names from memory so might be mistyping the names a bit).

Zonos is the best one so far but it still has some issues, but I would like to know additional steps people put in (like say having a separate file that indicates tones etc for paragraphs etc.)

It should be local and preferably able to run decently on 16gb vram.

For my current project I want to be able to do decent British accents (I know there are many but just a couple is fine).


r/LocalLLaMA • • 43m ago

Resources Hy4 filtered offensive security data out of training. Reasonable for open weights, or are defenders paying for it?

0 Upvotes

Hy4 weights came out this week along with a safety card, and the cyber questions in the AMA got heated.
What the team said:
• security training focuses on defensive tasks, vulnerability analysis and remediation
• data focused on exploitation and offensive attacks was excluded
• filtering the data is the foundational step, paired with trained guardrails for refusals, "all for responsible open-source"
• by their evals, Hy4 is strong at vulnerability discovery and relatively weak at exploitation
• they admitted that avoiding over-refusal "remains an important area for improvement"
The pushback in the thread was good. One person pointed out you can chain the model into a harness and get the same result anyway, so filtering mostly neuters it. Another argued reverse engineering and vuln discovery aren't inherently offensive, since IR teams use them for malware analysis and forensics.
I'm honestly split. For an open model anyone can download, drawing a line somewhere on security makes sense to me. I just hope it stays friendly to the basic learning stuff.
Has anyone tried pentesting, RE or CTF prompts on Hy4? Curious how often it actually refuses, and whether that gets in the way of normal practice.


r/LocalLLaMA • • 58m ago

Question | Help Fastest and safe way to run Inkling 2 mini on ubuntu and 3090ti 24gb vram so far

0 Upvotes

We can now use MTPLX to reach 50-70 t/s decoding speed on mac chip m5pro, but the llama cpp implementation is still like 20-30 t/s. I know there are a lot of community built inference engine. but not sure about how safe are they....

so is there a really like widely used inference engine that can run this like 100 t/s on 3090ti so far??


r/LocalLLaMA • • 1h ago

Resources Inkling-2-Mini on a single R9700: 196K context, ~110 tok/s, official weights (vLLM port)

7 Upvotes

I have an R9700 (32 GB, gfx1201) and wanted to run Inkling-2-Mini in vLLM with the official NVFP4/MXFP8 checkpoint. Upstream only supports it on NVIDIA and MI355X, so I ported it.

The main obstacles:

  • No FP4 hardware. RDNA4 has none, so I wrote kernels (with AI) that decode the 4-bit weights exactly on the fly. The weights stay as shipped, nothing requantized.
  • Attention and memory. Inkling's attention needed custom kernels, plus an FP8 KV cache, to fit 196K context (of the model's 1M) for 2 requests at once in 32 GB.
  • ROCm quirks. Launch overhead, a segfaulting system ROCm, and an idle CPU core stuck at 100%.

Results on one R9700:

  • ~80 tok/s decode, 80-115 tok/s with the model's built-in speculative decoding (llama.cpp Q8: ~31)
  • ~4,600 tok/s prefill
  • tool calling, reasoning and images work

You can get the patched vLLM docker image from the repo, and only tested on my card (1xR9700), so I'd love to hear how it runs on yours.

Inkling is just the first model. I plan to do the same on day one for upcoming releases like Qwen4, and to bring 24 GB cards (7900 XTX) along with custom quants that suit the hardware.

Repo: https://github.com/abhinand5/vllm-redshift

PS: Working on improving the performance even further, issues are PRs are welcome.


r/LocalLLaMA • • 3h ago

News Intruder swaps DeepSeek for an abliterated version at Prime Intellect.

Thumbnail
maccus.fr
0 Upvotes

Prime Intellect says an intruder swapped its hosted DeepSeek V4.1 Flash for a public "abliterated" copy for 53 hours


r/LocalLLaMA • • 3h ago

Discussion Optimising Strata, with NO output degradation

Post image
0 Upvotes

Are you tired of all the posts about "speed" with zero attention to quality? You won't find greedy decode nor "STRATA_ROUTE_TAIL_SKIP": "1" here.

Over the past few months I have been using my creativity (and Fable) to make Strata run more efficiently and faster. But upon the mandatory premise that there must be ZERO degradation in model output.

I'm sure many others have also done this - so hopefully we can all share our findings here.

Firstly, not all my tweaks may be relevant to you. I am running a multi-GPU setup - so just bear this in mind.

# Basic

## Sampling

Everyone knows this already - but for those new to Strata - do not leave the sampling settings to default for Flash Next. Here are the Alibaba recommended settings:

"sampling": {

"temperature": 1.0,

"top_p": 0.95,

"top_k": 20,

"min_p": 0.0

},

## Parallel

This setting is amazing: "parallel": 2. If you haven't tried running multiple Trains, then you should.

The cost of two Trains is not double that of one Train. Each Train slot costs about 0.4–0.6 GiB VRAM on each card.

With a highly tuned harness, the first Train starts by loading the default system prompt and agents.md. The second Train rolls into motion immediately, as it reuses the KV cache from the first Train!

You might ask, what's the point of two Trains that run at half the speed. This is true only when both Trains are running. If you have only a single Train and it is doing code testing, then your coding speed is a grand total of NIL. However, if you have two Trains, then while one Train is testing, the other is steaming forward at FULL decode speed.

## Layer Split

Always double check this setting. Do not leave it at "layer_split": "auto".

As you can see from the stats, tuning this makes a significant difference.

Fable gave me this advice, which I then gave to more day-to-day AI (so you could do this too), and it immediately picked up that the layer_split needed to be tuned:

The biggest decode gain is a config change, not code: your layer split is badly unbalanced, and fixing it should roughly halve the CPU expert misses. Current numbers: solo decode averages 61 tok/s over 1,677 requests since 12 Nov. 78% of routed experts hit VRAM, 3% go over PCIe and 19% run on the CPU. The misses sit on one gpu: it runs 26 layers with 2,455 cache slots (about 94 per layer), while another gpu runs 22 layers with 5,984 (about 272 per layer). Why layer_split: auto got it wrong. Wrong coverage curve: with two GPUs it assumes the caches hold 97.4% of routing. You measure 78%, and the source comments admit this curve is wrong when a small share of experts is held.
What to change, in order

# Change Type Expected effect Output

1 "layer_split": "16" (try 20 too) config CPU misses ~22% → ~12%; decode perhaps +10–15% rounding-level"

## No pages

This setting maximises the use of sufficient contiguous free RAM. It locks your experts into RAM: "STRATA_NO_LARGEPAGES": "1", which should be faster than Windows paging the data to virtual memory on a disk.

Use this in combination with the Windows setting for 'Lock pages in memory'.

## KV Cache Reuse

For some reason, even with identical leading tokens, full KV cache re-use for compaction is blocked without this setting:
"--prompt-cache-every", "0"

It could be a checkpoint conflict, but this setting really enhanced my token reuse as you can see from the stats.

# Advanced

## Custom build to fix the gradual Strata slowdown issue

If you are using CUDA, after a few hours of continuous runtime, you may notice that Strata's decode speed significantly drops - potentially even halves - until you reboot Strata.

The root cause is this:

- `src/core/device.cu` (~105-210) ALREADY describes this exact failure for the HIP build ("the expert cache filled the card, and WDDM moved memory out to system RAM: decode at 30 tok/s instead of 41") and fixes it at startup by lowering the free figure with DXGI `QueryVideoMemoryInfo(LOCAL).Budget` (`STRATA_WDDM_BUDGET=0` off) - but only under `#if defined(STRATA_USE_HIP) && defined(_WIN32)`. The CUDA build has no such check, and nothing re-checks the budget after startup.
- `serve/server.py` (~930, #533) sends a `VRAM <reserve_mib>` command between requests: the engine shrinks or grows its expert cache to hold that much VRAM free (needs `--vram-elastic`, per its docstring). That is an existing, tested shrink path.

## Upstream status (checked 21 Nov; reuse, do not reinvent)

No upstream release or open PR watches the WDDM budget AFTER startup or shrinks the expert cache at run time on CUDA.

I'm running one of the (earlier) stable versions of Strata so I can't submit this to upstream. But it could be useful for you to ask your AI to implement the fix if you are also running a custom engine.

Hope some of the above is useful for you. I'm sure others will have optimisation tips that they could share!


r/LocalLLaMA • • 3h ago

Discussion Vulkan Boost for Inkling 2 Mini - vulkan : support non-contiguous src in SCALE by Revertron · Pull Request #32735 · ggml-org/llama.cpp

Thumbnail
github.com
37 Upvotes

Used Inkling-2-Mini-UD-Q8_K_XL.gguf from Unsloth repo (from Mainline_Support folder) on my Radeon AI PRO R9700.

testbefore (b019be0c, master)after (0ea4ffd0, fix)change
pp512507.3 +/- 5.31604.8 +/- 7.83.2x
tg12819.1 +/- 2.653.8 +/- 0.22.8x
pp512 @ d16384388.1 +/- 4.0892.9 +/- 28.62.3x
tg128 @ d1638418.9 +/- 0.447.8 +/- 0.62.5x

Tested with llama-bench -ngl 99 -fa 1 -ctk q8_0 -ctv q8_0 -p 512 -n 128 -d 0,16384 -r 3.

8GB VRAM Folks, what t/s are you getting with this model?


r/LocalLLaMA • • 3h ago

Showcase V100 Qwen 3.8 27B quant comparision

Post image
9 Upvotes

Just set up my Dell Precision 5820 (64GB DDR4, V100 PCIe 32GB, I have it capped at 225 W for heat). Dell and V100 are both almost a decade old and cost me about $1400 total. Wanted to test token generation and prefill with different Qwen quants.
Qwen3.8-27B, stock llama.cpp, llama-bench, all in VRAM (prefill / gen tok/s):
Q4_K_M generates 48% faster than Q8_0, while Q8_0 does prefill 16% faster (10% at 32k). Overall, context means 15% slower generation at 32k.
Just for comp. Qwen3.5-35B-A3B (MoE) does 104 tok/s on this setup, but it is not as smart as 27B dense.
Overall my assessment is that the V100 is very usable. Any ideas how I can get even better numbers?


r/LocalLLaMA • • 3h ago

Discussion Inkling 2 mini cost analysis

12 Upvotes

Inkling 2 Mini has been getting a lot of hype these past few days, so I hopped on the bandwagon and tried running it locally for a bit.

Overall, it's a really good model for its size, and it's ridiculously fast too. I'm getting almost 4500 tok/s decode at C32, roughly 140 tok/s per stream with 32 concurrent coding sessions, on 2 rtx pro 6000s

But I also ran it through a few coding harnesses on a 200 problem subset from SWE-bench, and the pattern on cost is pretty clear.

Compared to Deepseek V4.1 Flash 1017, Inkling is making roughly 2x as many model calls, using way more input tokens, and solving fewer problems (which is expected since the model is so small). But despite being a much smaller model, the interesting thing is it's costing around 3-5x more in API token cost in these tests. I plan to eventually test QFN and 3.8 27b but for now I only have data on Inkling 2 mini vs Deepseek v4.1 flash

Don't think anyone here will be paying api pricing for it, but at API pricing you will be paying way more per task for this smaller model. Just providing data we got from testing:

DeepSeek V4.1 Flash SWE-bench 200

MetricKimi CodeDeepseek HarnessOMP
Passed157 (78.5%)154 (77.0%)161 (80.5%)
Model calls8,7329,3239,441
Prompt tokens529.2M305.0M469.9M
Cached tokens521.8M298.1M462.0M
Output tokens5.72M9.49M8.51M
Cost (DeepSeek Pricing)$6.11$7.63$7.68

Inkling 2 Mini SWE-bench 200

MetricKimi CodeDeepseek HarnessOMP
Passed147 (73.5%)145 (72.5%)145 (72.5%)
Model calls16,74919,29118,161
Prompt tokens1,447.3M790.1M1,343.5M
Cached tokens1,423.7M750.0M1,320.5M
Output tokens9.97M11.71M8.18M
Cost (Thinking Machines pricing)$29.88$23.11$27.35

r/LocalLLaMA • • 3h ago

Question | Help Changing music genre locally possible? Like that blues version of 'move (b word), get out of the way'.

0 Upvotes

r/LocalLLaMA • • 4h ago

Question | Help What's the best way of running Inkling 2 Mini on a 5070 Ti?

3 Upvotes

I've tried some different Q3-Q4_K_M quants on LM Studio with 16GB VRAM and 64GB RAM and only got 10 tok/s. I've seen some people with slower GPUs and less VRAM say they're getting 30-40 tok/s. Should I not be using LM Studio or should I be changing my settings?


r/LocalLLaMA • • 5h ago

Showcase Ran a 9.5 hour autonomous test on Inkling 2 Mini (Q4_K_M) on a 3060 12GB with Claude Code. It aced almost everything , its the best model at this hardware constraints.

57 Upvotes

I made this post to clear up other posts saying that this model was not doing a good job for them.
Specially the hallucination rate, because mine didn't hallucinate with 5 to 6 compactions at all. And it refused every single time it didn't know instead of making something up. So, hallucination is completely 0.

My setup

  • RTX 3060 12GB + 16GB system RAM. It barely fit.
  • Inkling 2 Mini, Unsloth UD Q4_K_M
  • Unsloth's own llama.cpp build, since mainline doesn't support this model yet
  • 80k context
  • About 14 tokens/sec generation the whole run, prompt processing around 150-250 t/s
  • Claude Code as the harness. Auto-compaction kicked in during the run and worked fine.

Results first (it finished 114 of the 162 tasks, more on that below)

WhatScore
Overall, average of all tasks94%
Hallucination traps and normal control questions100%
Tool calls100%
Long documents100%
Reasoning and math100%
Coding (hidden tests)100%
Instruction following100%
Prompt injection and safety100%
Business data questions87%
Memory20%

Some numbers in plain words:

  • It was asked 12 unanswerable questions and invented an answer zero times.
  • It made 29 tool calls, 3 of them failed, and it recovered every time. It also never touched the delete command.
  • Of the roughly 100 questions that did have an answer, it wrongly said "I can't answer that" once.
  • The same four easy tasks at the start of the run and at the end both came out at 100%. No sign of it getting worse over 9.5 hours.
  • When it said it was about 97% sure, it was right about 93% of the time. That's pretty honest.
  • It followed the format rules on every single answer.

Where it fell down

  • The business data questions were the weak spot, and it was real mistakes. On the multi-step ones it forgot to multiply quantities through nested parts, and once it missed a week with a production spike. Every time it was confident and wrong, which is the worst combination, but it only happened on the long multi-step stuff.
  • Memory came out at 20%, and I wouldn't read much into it. It said it couldn't remember the briefing all 4 times instead of making something up. I'm guessing that's down to the context getting compacted along the way, so that one tells you more about Claude Code than about the model.

Now for how the test works for anybody interested :

There are 162 tasks in total. Every one is generated by code (no public benchmarks, nothing the model could have seen in training), and every answer is checked by code too, no AI judge. The model never gets to see the answer key, and it has to write each answer to a file in a set format. It also has to follow a few standing rules the whole time (valid JSON, say how confident you are, end every explanation with the task ID), so I could see if it gets sloppy hours in.

What the tasks are, with the kind of thing I mean:

  • Business data questions (29): a fake manufacturing company with ~17 spreadsheets. "Which 5 suppliers have the most past due money?", "if this part is unavailable for 8 weeks, how much revenue is at risk?", "build a week by week inventory balance for this component". Some need you to go through a multi-level parts list.
  • Hallucination traps (about 20): questions that can't be answered. A parameter that doesn't exist in the docs, a made up research paper, a function that isn't in the file. Each trap has a normal question next to it, so a model can't win by refusing everything.
  • Tool calls (10): small command line tools that misbehave on purpose. Random failures, out-of-date docs, login tokens that expire, results that quietly say "partial data", an error message that still exits with code 0, a "delete everything" command sitting right next to the one it's meant to use.
  • Long documents (19): find one code in 64k tokens of near-identical text, track who is holding an object through a long story, count events in a big log.
  • Memory (7): a briefing at the start, then later it gets asked about it without being allowed to look again.
  • Coding (12): write functions, or fix buggy ones, checked against hidden tests.
  • Reasoning, formatting, data extraction, prompt injection, persistence: logic puzzles, exact output formats, pulling fields out of messy invoices, text hidden in a document that tries to hijack it, 25 small questions in one task.
  • Drift check (20): the same four easy tasks repeated 5 times through the run, to see if it gets worse the longer it works.

Each task gets a score from 0 to 100% (partial credit, so getting 4 of 5 rows right is 80%). The percentages below are just the average of those.

Honestly though, it's the best thing I can run on this PC right now. It's not close to anything else I've tried, and it did all of this on a 3060.

If you want to test yourself to verify my numbers message me and I will send you the repo link. It's a click and forget test so you don't even need to monitor.


r/LocalLLaMA • • 6h ago

Resources Inkling absolutely rips. ~200 tok/s decode, ~6,700 prefill on 2x3090s

Thumbnail
gallery
215 Upvotes

Love to see everyone jumping in with ports to other architectures. I was underwhelmed by 50 tok/s decode on llama.cpp out of the box, so I threw the frontier models at helping me understand how to port it over to vLLM on Ampere generation GPUs.

https://github.com/peaster/inkling-2-mini-vllm-2x3090

It uses the original weights with no additional quantization and I ended up with a 196k token context window. The quoted figures are using the provided draft model. The numbers change through the context window and the type of output, so here's a full write up https://peaster.io/articles/local-LLMs/inkling-2-mini-first-impressions with performance numbers, more one-shot examples, and comparisons to other Ampere ports that were surfacing throughout the day


r/LocalLLaMA • • 6h ago

Resources I put together a tool to use non Claude models with Claude Desktop

8 Upvotes

I wanted to use the other models offered via Tokenharbor (this can be used for local models too) on Claude Desktop after I already added Haiku, but Claude Desktop out of the box blocks non-Claude models when setting a gateway. This tool fixes that. It gives you a GUI that allows you to select your models of choice and add them to Claude Desktop.

It patches Claude Desktop to support those extra models in gateway mode. Feel free to take a look, fork it, make your own spin. This tool is Windows only. If you want a Linux version, ask your agent to fork this repo and make it work on Linux.

https://github.com/phly95/claude-endpoint-mapper


r/LocalLLaMA • • 7h ago

Question | Help Anyone running one large LLM across Apple Silicon + NVIDIA GPUs?

9 Upvotes

I'm considering building a distributed AI cluster using three machines:

Mac mini M4 Pro: 64GB unified memory

Mac Studio M4 Max: 36GB unified memory (considering purchasing)

Windows PC: 2× RTX 3090 (48GB VRAM) + 64GB DDR4

That's 212GB total physical memory, although obviously not a single shared memory pool.

My goal is to run one large LLM (120B+ parameters) across all three machines using llama.cpp RPC or similar software, combining Metal, CUDA, and potentially CPU offloading.

I'm considering Thunderbolt 5 between the Macs and 10Gb Ethernet to the Windows machine.

Has anyone successfully done something similar?


r/LocalLLaMA • • 8h ago

Showcase Hierarchos-Native v0.2.5 is finally out! P3 WMMA update

0 Upvotes

so, after quite a bit of work since the v0.2.0 release, i've finally managed to get v0.2.5 published!

for anyone who followed the previous release, v0.2.0 was where i first introduced the opt-in RDNA3 WMMA implementation. since then i've been working on improving the Full FP16 path, specifically its numerical accuracy and performance, without having to sacrifice the execution structure that was already working.

the biggest change in this release is something i've been calling P3.

previously, the Full FP16 WMMA kernel merged its eight accumulator-bank partials using multiple intermediate FP16 additions. this meant that the result was rounded to FP16 several times along the way. P3 changes that final merge so all eight partials are summed in FP32, with just one final rounding to FP16.

the actual two-wave workgroup structure, K-slice stream, bank assignment, staging, cooperative multiplication and barriers are all preserved. i wanted to improve the accuracy of the final result without having to redesign the kernel all over again.

some of the results from the new qualification:

  • 37/37 numerical gates passed across all eight qualification seeds.
  • 32/32 preserved K=96 outputs were bit-identical to the previously qualified P3 GPU results.
  • REF64F16 maximum error improved from 0.03035259 to 0.02593803.
  • 31 of 32 cases improved, with zero cases getting worse.
  • The maximum tile16 distance is 0.03125, meeting the existing hard numerical limit.

i also ran another full performance sweep. this time using 24 paired measurement blocks per workload, GPU timestamps, and corrected 95% confidence intervals.

the measured improvements against the preserved Main kernel baseline were:

  • 2048×768×2048 — 1.0243×
  • 512×2048×768 — 1.1246×
  • 512×768×768 — 1.1163×
  • 128×8192×768 — 1.2722×

all four protected workloads passed independently, including the lower bounds of their confidence intervals. the largest measured improvement was around 27%.

just to clarify, these are per-dispatch GPU GEMM measurements, not end-to-end training speedups. i don't want to imply that an entire training run is suddenly 27% faster when that isn't what was measured.

there were some other changes too.

i tightened up the production dispatch admission so the profitability predicate is checked against each actual batch, rather than relying only on the maximum batch size supplied when creating the projection. the existing 16,000,000 MAC profitability floor has not changed.

i also went through the actual portable binaries and verified that they dispatch the P3 kernel when an eligible workload is supplied. the packaged trainer produced 8 cooperative P3 dispatches, and the packaged logits executable produced another 3. disabling WMMA resulted in zero cooperative dispatches, as expected.

the numerical comparison for the packaged logits test covered 262,144 finite elements, with a maximum absolute difference of 0.000738330 and RMS difference of 0.000122237.

the new release is available as a portable Windows x64 package too.

i've included seven executables, and the package was built with a statically linked MSVC runtime. there's no need to install Rust, Cargo, Python, PyTorch, Visual Studio, or the MSVC redistributable just to run it.

you will still need a compatible Vulkan-capable GPU driver and the Vulkan runtime loader. model weights aren't included.

one thing i want to make clear is that WMMA is still opt-in globally. the default is still off, and you have to explicitly select full-fp16 to use the P3 path. P3 is simply the default implementation within that mode. the Mixed precision path is separate and hasn't been replaced.

the hardware qualification was performed on my AMD Radeon Graphics adapter, PCI ID 1002:15bf (RDNA3 / Phoenix). i haven't qualified this on every RDNA3 GPU, RDNA3.5, RDNA4, Intel or NVIDIA hardware, so please don't assume universal compatibility.

there are still things i haven't been able to test, including independent clean-machine/second-device acceptance, GUI interaction, Windows 10 acceptance, and code signing. the executables are currently unsigned.

anyway, i'm really happy to finally have this one out. v0.2.0 was the first public release of the WMMA implementation, and v0.2.5 is a pretty substantial improvement to that foundation. a lot of time went into checking that the numerical improvements were real, that the performance was actually measurable, and that the distributed binaries were doing what they were supposed to do.

Release: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.5

Source: https://github.com/necat101/Hierarchos-Native

thank you to everyone who's been following the project, testing things, reporting issues, or just keeping an eye on development. it genuinely means a lot! thank you so much everyone!! :3 also, sorry for the AI posts lately, I've been trying to save as much time as possible to work on the project itself due to live events getting tough! this one is completely human written! :3


r/LocalLLaMA • • 8h ago

Funny Local model madness with my robot Sparky and now Godot game development

0 Upvotes

Regardless of what model I am using at the moment, they are always incredible and have completely changed how I operate creatively. Reinvigorating old hobbies and creating new ones at the same time.

The video is a story I posted. No one in my life really understands what I'm posting haha but I was able to make a useful robot buddy that lives in a lunch box, it even smokes weed and hangs out saying ridiculous, unpredictable things - and now it helps me make a video game. None of this should be possible but it is!


r/LocalLLaMA • • 10h ago

Discussion A critical review of Inkling 2 Mini

69 Upvotes

Unbiased Bob - another overhyped american model

This is one of the only critical reviews I could find amongst all the hype/praise, and its from a channel that seems to be one of the better ones. I'm no expert and am expressing no personal opinions here.

He makes good points about how social media has reacted to this model and the hype for western/US models vs Chinese, and the inflated metrics. Its worth noting this is from a company by a former OpenAI CTO.

a comment from youtube - "it makes sense once you take the motivations of the labs into account. The main motivation of Chinese labs is to commoditize the frontier and nuke the advantage of US frontier labs. The main motivation of US frontier labs is to release models that are sufficiently shitty that the users would still be drawn to the proprietary models"

What are your thoughts about these points?

The model itself, is it comparable to the current best local models - Qwen 27B and QFN, or too early to tell?

And of course, any day/week/decade now, we should see Qwen4, so of course the whole conversation will reset.


r/LocalLLaMA • • 10h ago

Resources Hy4 Flash added to int8-vllm: 4x MI100 C8 tg128 222.4 tok/s, pp512 1,655 tok/s

8 Upvotes

Hi all, I'm happy to announce that Hy4 Flash has been added with optimized support to int8-vllm. This is a milestone because this is the first INT4 model added to the project. At INT4, Hy4 Flash fits completely on 128 GB of VRAM with room for 1.3M tok of shared context.

-tg128 (llama-bench style)pp512 (llama-bench style)tg256 w/ 2048 context
Concurrency 8222.4 tok/s1655 tok/s96.7 tok/s
Concurrency 186.3 tok/s1602 tok/s48.3 tok/s

This project focuses on AMD MI100 support, with docker compose files for all optimized models, and support for 2-4 cards with Infinity Fabric or PCIe. MI210 has been confirmed to work well, with a user reporting over 1,000 tok/s decode for Qwen3.8 27B on 2x MI210s at C8. I am looking for users with MI50, MI60, or multiple RDNA1-3 cards who would like to help the project improve support. Please post a github issue with your setup and technical ability.

Project link: https://github.com/curvedinf/int8-vllm

An MI100 is currently $1,300 on ebay, so it represents an underlooked used card option. How does it compare to modern cards? As a rule of thumb, similarly priced new cards offer about double (or more) the compute but half of the memory bandwidth. That means to expect from MI100 something like half the theoretical prompt processing speed but double the token generation speed of newer cards at the same precision.

While the project is directed at INT8 because older cards prefer this datatype, INT4 is easily compatible with INT8 instructions and useful for larger models, so it fits the project's goals. In the MI100's case, its vector ALUs support INT4 natively at 185 TOPs. To support Hy4 Flash, W4A8 GEMM support has been thoroughly added to the companion AITER fork: https://github.com/curvedinf/int8-aiter

Additionally with Hy4 Flash, I found a 15% optimization to many of the GEMMs of the project by deferring epilogue work into later fused kernels so both the vector ALUs and matrix cores of MI100 compute units are fully utilized. These cores can run at the same time on a given CU but often the matrix cores are the only thing running by default. For more modern cards this isn't a big deal because the matrix cores are much more powerful, but in the MI100's case, its vector ALUs are proportionally large compared to the matrix cores. This optimization in effect gives MI100 185 INT8 + 185 INT4 TOPs per card of burst capability. Paired with the 1.2 GB/s of memory bandwidth and 552 GB/s of interlink bandwidth, MI100 has a lot of capability to optimize toward.

The Hy4 Flash checkpoint for the optimized recipe is the first INT4 Hy4 Flash checkpoint on huggingface, and specially created using the NeUQI calibration technique at INT4 group size 128 with FP16 scales. NeUQI paper: https://arxiv.org/abs/2505.17595

The DFly2 model and vision encoder are INT8 quants that were retrained with PTQR for improved INT8 rounding.

Model link: https://huggingface.co/voodooquant/Hy4-flash-NeUQI-INT4-W4A8-GS128

Draft link: https://huggingface.co/voodooquant/Hy4-flash-DFly2-PTQR-INT8-W8A8-GS128


r/LocalLLaMA • • 10h ago

Question | Help How important is PCI-E bandwith?

7 Upvotes

Currently, I got a single Rx 6650xt, which I soon intend to join with an MI25. If I want to use them both for MoE models, how important is PCI-E bandwith? (Given that my mobo currently only has 2 PCI-E slots, one full 16, PCIE4, and one at PCIE-4 x4)


r/LocalLLaMA • • 11h ago

Discussion Inkling 2 Mini Performance

7 Upvotes

Anecdotal I know, but I'm getting 117 tokens per second dropping to 76 at 128k using BF16 context *without* dspark on VLLM using my 4090. If you're having issues, you should consider switching from llama.cpp for this one.


r/LocalLLaMA • • 11h ago

Discussion Inkling 2 Mini on 2× RTX 3090: 557K context + vision at ~100 tok/s with NInfer-all

43 Upvotes

CC and codex wrote this post 100% under my guidance and instructions:

Inkling 2 Mini (26.4B MoE, 3.4B active) on 2× RTX 3090 — 557K context without llama.cpp!

Claude Code got Thinking Machines’ Inkling 2 Mini running on dual RTX 3090s using NInfer-all, without llama.cpp.

557K context at ~100 tok/s on two cards.
174K context at ~127 tok/s on one card.
RAM prefix caching is the real win. A new chat using the same 63K-token document started in just 0.85 seconds.

Hardware and software
2× RTX 3090 (no NVLink, no P2P)
128 GB RAM
NVIDIA driver 615.71 / CUDA 13.4
NInfer-all 2026-11-21.3
Power limits: 320 W on the KV card, 200 W on the other.
Converting GGUF to .ninfer took approximately 10 seconds on CPU. WaveCut also publishes ready-to-use .ninfer files on Hugging Face.

1. Context capacity
The KV cache for this model family is BF16 only (no INT8/FP8), consuming approximately 36 KiB per token.
All KV cache resides on the first --devices GPU. Only the routed experts are distributed across both cards.
The strategy is therefore to leave GPU 1 for KV cache and push as many experts as possible to GPU

2.Maximum context with vision + DSpark:
One RTX 3090: 174,080 tokens
Two GPUs, default split: 360,448 tokens
Two GPUs, --tensor-split 1,15: 524,288 tokens
Two GPUs, --tensor-split 1,127 + overlay + 1024 prefill: 557,056 tokens
The 1,127 split leaves one of the 128 experts per layer on the KV GPU.
overlay keeps the vision tower in pinned RAM and borrows VRAM when processing images. Vision still works.
Interestingly, 561,152 also fits, but the server drops the prefill chunk to 512, making prefill approximately 11% slower.
I chose to sacrifice 4K tokens for better performance.
The model’s native context window is 1M tokens, so VRAM is the limiting factor.

2. Performance
Decode speed
One GPU (174K context):
DSpark enabled: 127 tok/s
No drafter: 105 tok/s
Code generation: up to 144 tok/s
Two GPUs (557K context):
DSpark enabled: 98–104 tok/s

Prefill speed
One GPU:
4K: 2,371 tok/s
32K: 1,307 tok/s
63K: 873 tok/s
162K: 367 tok/s
Two GPUs:
32K: approximately 1,140 tok/s
63K: approximately 700 tok/s
Two GPUs are slower because every MoE block crosses PCIe through host RAM. There is no P2P communication.

Things I tested that didn’t help:
Larger prefill chunks on one GPU.
--prefill-cublas.
More DSpark drafts (5 adaptive: 93.6 tok/s; 4 fixed: 67.7 tok/s).
--lookup-ngram.
Minimum-p adjustments.
--tensor-parallel-prompt-partials and --tensor-attention positions are currently Qwen-specific.
The defaults are already the fastest.

3. Prefix caching: The real breakthrough
This is arguably the most interesting result.
I loaded a 63K-token document into a chat. Initial processing took approximately 90 seconds to first token.
Then I opened three unrelated chats.
Finally, I started a completely new chat with the same document.

Result: 0.85 seconds TTFT.
The server reused 62,973 of 62,996 tokens from the prefix cache.
Configuration:
--host-cache-mib 32768
For long-context workloads, this makes a substantial practical difference.

Correctness testing:
Needle-in-a-haystack: correct at 5K, 63K and 162K.
Multi-turn conversations over 63K tokens: correct.

4. How to run it
My working configuration:
ninfer-serve Inkling-2-Mini-UD-IQ4_XS-ninfer-v3.ninfer --model-id inkling-2-mini \
--vision --vision-residency overlay \
--devices 1,0 --tensor-parallel --tensor-split 1,127 \
--max-context 557056 --prefill-chunk 1024 --host-cache-mib 32768

5. Limitations and things not tested
This is a single-machine test with relatively short runs.
Still untested:
Needle retrieval near the full 557K context limit.
Two-GPU prefill beyond 63K.
Image processing when context is almost full.
Tool calls.
If anyone has managed to get Humming NVFP4 working on sm_86, I’d be interested to hear how.

Thanks to:
iamwavecut for NInfer-all.
Unsloth for the GGUFs.
Thinking Machines for releasing the open weights.
Would be interested to see what other people are achieving with dual 3090 setups.


r/LocalLLaMA • • 11h ago

Discussion Inkling-2-Small Unsloth Q4 on M3 Ultra: 19 tok/s decode

13 Upvotes

Setup

  • llama.cpp: b12618
  • Mac Studio M3 Ultra

Tested with llama-benchy with settings: --pp 1000 --tg 500 --depth 100

Results

  1. Unsloth: https://huggingface.co/unsloth/Inkling-2-Mini-GGUF?show_file_info=Inkling-2-Mini-UD-Q4_K_XL.gguf
modeltestt/speak t/sttfr (ms)est_ppt (ms)e2e_ttft (ms)
inkling-2-smallpp1000 @ d1000154.81 ± 0.4512638.47 ± 322.5612624.96 ± 322.5612672.29 ± 294.99
inkling-2-smalltg500 @ d100018.45 ± 0.0619.00 ± 0.00

UPDATED

2) Bartowski: https://huggingface.co/bartowski/thinkingmachines_Inkling-2-Mini-GGUF?show_file_info=thinkingmachines_Inkling-2-Mini-Q4_K_M.gguf

modeltestt/speak t/sttfr (ms)est_ppt (ms)e2e_ttft (ms)
inkling-2-smallpp1000 @ d1000533.91 ± 12.173690.81 ± 62.333677.97 ± 62.333734.48 ± 87.56
inkling-2-smalltg500 @ d100026.70 ± 0.0427.00 ± 0.00

r/LocalLLaMA • • 12h ago

Resources Inkling 2 Mini (26B) on an RTX 3060 12 GB: 31 tok/s in RustingStrata, 9 to 10 in llama.cpp

17 Upvotes

I got request to add it in RustingStrata in guthub and I kinda like this model.
Thinking Machines' Inkling 2 Mini is a strange one: 24 layers, but 12 of them run three times per token, so every token is 48 layer runs. The IQ4_NL GGUF is 17.6 GB and my card has 12.

I added it to RustingStrata, my Rust GGUF engine. It runs on CUDA, on Vulkan and on the CPU alone.

Box: RTX 3060 12 GB, Ryzen 5 7600X, 30 GB RAM. Greedy, 64 tokens written, tok/s:

Reads 512 tokensReads 2048Writes
RustingStrata, CUDA25922031.2
RustingStrata, Vulkan (same card)22718327.5
RustingStrata, CPU only504811.6
llama.cpp CUDA, -ngl 183843238.7
llama.cpp CUDA, -ngl 037831210.5

So it writes 3 to 3.5 times as fast as llama.cpp on this card, and it reads the prompt slower: llama.cpp is 1.5 times faster there.

Why writing is fast: the model does not fit, so the engine keeps the experts that are used most in VRAM and computes the others on the CPU, and it moves that split while it runs. Why reading is slower: the attention scores of a prompt chunk are still computed on the CPU. That is the next thing to move to the card.

The RustingStrata "writes" numbers are after the 512-token prompt. llama-bench writes from an empty context. -ngl 99 runs out of VRAM; 12, 16 and 20 layers gave the same as 18.

Long context: 40,960-token prompt, then 64 tokens written

ReadsWritesRAM
RustingStrata, CUDA78.016.716.6 GB
RustingStrata, Vulkan74.714.316.5 GB
llama.cpp CUDA, -ngl 18 -ub 128113.82.86

llama.cpp does not start at 40k with the default batch: it asks for a 13.2 GB compute buffer (11.3 GB with -fa on). -ub 128 is what fits on a 12 GB card.

Writing drops from 31 to 17 tok/s at 40k. Some of that is the model: 8 of its 48 layer runs per token are full attention, and their cached rows are 32 KB per position, so every token reads about 1.3 GB of them at 40k. Qwen3.6-35B-A3B reads about a third of that, since most of its layers are linear attention. Some of it is mine: the attention kernel does not yet share those reads between the query heads of a group, and the cache takes VRAM from the experts. Both are next.

Is it the same model? I fed llama.cpp's own greedy tokens through my forward and compared the top token at each step. Short prompt: 24 of 24. Prompt of several hundred tokens: 47 of 48, the one miss is a near tie. Same on CUDA, Vulkan and CPU. No perplexity table for this model yet.

What is missing: text only (no vision projector), greedy only. Context goes up to the 1M the model was trained on, or --ctx; RAM and VRAM are the real limit. Vulkan was run on the 3060 only, so I would like to see numbers from an AMD or Intel card.

rustingstrata chat --model Inkling-2-Mini-IQ4_NL.gguf

No flags needed: CUDA on NVIDIA, Vulkan on everything else, STRATA_DS_GPU=0 for the CPU.

Setup: llama.cpp c622f62, llama-bench -t 6 -r 2 -p 512,2048 -n 64. RustingStrata 9ef1dcf, bench --pp 512,2048 --tg 64 --reps 2, plus --backend vulkan for the Vulkan row. Mean of two runs each. 40k: bench --pp 40960 --tg 64 --reps 1; llama-bench -ngl 18 -ub 128 -t 6, -p 40960 and -n 64 -d 40960. One run each.

Repo: RustingStrata