r/Qwen_AI • • 19h ago

News We distilled our Qwen-Image-2.1 Uncensored Omni model into an 8-step Turbo LoRA—6.8× faster with comparable quality

Thumbnail
gallery
3 Upvotes

The original Omni release has now reached #8 trending out of more than 3 million models on Hugging Face. The feedback from the Qwen community helped us decide what to build next: a much faster version that preserves the model’s unrestricted generation and editing capabilities.

Qwen-Image-2.1 Uncensored Omni — Turbo LoRA

The Turbo LoRA runs with:

  • 8 sampling steps
  • CFG 1.0
  • Approximately 1-megapixel output
  • Text-to-image generation
  • Instruction-based image editing
  • ComfyUI and Diffusers support
  • No prompt filter or output safety checker added by this release

On our Playground hardware:

  • Standard Omni, 35 steps: 54.9 seconds per image
  • Omni + Turbo, 8 steps: 8.1 seconds per image
  • Speedup: 6.8×

Benchmark results

We tested the Turbo LoRA at 8 steps against the same Omni model at 35 steps.

BenchmarkOmni, 35 stepsTurbo, 8 steps
Adult prompt adherence89.789.1
Anatomy69.472.9
General SFW prompts79.679.0
Image editing78.981.6
Identity preservation in human editing55.872.1

Prompt adherence and general SFW quality were within the statistical noise of the 35-step model. Anatomy, image editing and identity preservation improved in our evaluation.

The Turbo LoRA can be added on top of supported GGUF, BF16, FP8, NF4, INT8 ConvRot and NVFP4 Omni builds.

Download and instructions

https://huggingface.co/chatrlabs/Qwen-Image-2.1-Uncensored-Omni-Quantized#turbo-lora-8-steps-cfg-10

Recommended starting settings:

  • Turbo LoRA strength: 1.0
  • Sampling steps: 8
  • True CFG: 1.0
  • No negative-prompt pass required

The model card includes the weights, checksums, Diffusers code, ComfyUI instructions, compatibility notes, complete benchmarks and known limitations.

Try it online

https://thechatr.ai/playground

Our free Playground now serves more than 10,000 image generations every day.

A huge thank-you to the Qwen team for releasing Qwen-Image-2.1 and making community research like this possible. We’re an independent community fine-tune and are not affiliated with the upstream Qwen team.

We would especially appreciate technical feedback from the Qwen community:

  • Which prompts or compositions remain difficult?
  • How does Turbo perform on your hardware?
  • Which quantized build works best for you?
  • What should we improve in the next release?

Thank you to everyone who has downloaded, tested and shared the model. ❤️


r/Qwen_AI • • 16m ago

News Alibaba just dropped Qwen4-Max: 2.4T params, 1M context, 62% cheaper per task.

Thumbnail
gallery
0 Upvotes

TL;DR: It might be extremely late, though.AlibabaのQwenチームがQwen4-Maxをリリースしました。2.4兆(2.4T)パラメータのMoEモデルで、コンテキストウィンドウは100万トークン。Claude Opus 5.5の5.5より一部のエージェント系ベンチで勝ちながら、タスクコストは~62%も削減。今後、小さめのオープンウェイトモデルも出てくる予定。Qwhen?

スペック

・総パラメータ数: 2.4T(Mixture-of-Experts)

・1トークンあたりのアクティブパラメータ: 71B

・N-gram埋め込みのパラメータ数: 328B

・コンテキストウィンドウ: 1,000,000トークン

・マルチモーダル: テキスト、画像、動画を入力可能。テキスト出力

・アーキテクチャ: GDN + Qwen Sparse Attention(QSA)、Gated Residual、N-gram Embeddingを組み込んだ新しい「Qwen4」バックボーン

アーキテクチャで新しくなったところ

Qwen4のアーキテクチャは、大規模運用における効率化にフォーカスしています:

・GDN + QSA(Qwen Sparse Attention): コンテキストを固定サイズの状態に圧縮して、必要な部分だけを選んで読み直す方式。長い入力での計算コストを下げます。

・Gated Residual: 層をまたいだ情報の流れにゲートをかけて、学習をより安定させます。

・N-gram Embedding: 単語のn-gramパターンに基づいて事前計算された表現を引っ張ってくる仕組み。比例した計算増なしで、実質的にモデルのキャパを広げます。

コスト改善(Qwen3.8-Max比)

ここが面白いところ。

Alibabaは推論効率(inference efficiency)をかなり強く押し出してます:

トークン使用量(Terminal-Bench 4.0)

~48-49%削減

平均タスクコスト

$4.43 → $1.67(62%ダウン)

Work360(高い負荷)

トークン数が35%減

つまり、より良い性能が得られつつ、払う金額は大幅に少なくなるってこと。

これはかなり筋のいい提案です。

ベンチマーク結果

勝ち:

Agents' Last Exam v1.1: 46.7(Claude Opus 5.5の43.4より上)

DeepSWE v1.1: 75.7(比較モデルの中で最高)

負け:

Terminal-Bench 4.0: 48.5(Claude Opus 5.5は59.6、Kimi K3.1は67.2)

JobBench: 68.5(Claude Opus 5.5は75.4)

Artificial Analysis Intelligence Index v5.0.1: 43(トップ10モデル中9位)

まとめ: エージェント的な、長期スパンのソフトウェア系タスクは強い。でもコーディングや職場系タスクのベンチではまだ後れを取ってる。

提供状況

APIアクセス: すでにQwenCloud、QwenWork、QwenStudioで利用可能

料金: 100万トークンあたり$0.075(入力、暗黙のキャッシュヒット込み)

オープンウェイト: Qwen4-Max自体は提供なし

オープンウェイトの小型モデル: 開発者向けに「coming soon(近日公開)」として発表。Qwhen**?**

私の見立て

ここで本当に語るべきは、効率化による伸びですね。

パラメータ2.4Tって聞くと盛りすぎに感じるけど、トークンごとのアクティブは71Bだけだから、推論コストは実際けっこう現実的。

それでいて、タスクあたりのコストを62%カットしつつベンチのスコアまで伸ばしてるのはすごい。

ただ、Artificial Analysisのインデックスでトップモデルの9位ってことを見ると、盛り上がり(hype)は独立した評価のほうがちょい先行してる可能性もあります。競争力はあるけど、明確に圧倒的とは言い切れない感じ。

オープンウェイトの小さめQwen4モデルが出るって約束が、オープンソース界隈にとってたぶん一番ワクワクするポイントでしょう。Qwhen?

みんなはどう思う?

Alibabaが西側のフロンティアモデルとのギャップを埋めてるのか、それともパラメータ数を見せるタイプのマーケ戦術なのか。誰かもう実際にAPIを試した人いるなら、感想聞きたいです。

それと、僕が一番気になってるのは…Qwen4 27bっていつリリースされるの?

Qwhen?

postscript:"Just" That was an exaggeration. Sorry. I was just summarizing some outdated news.


r/Qwen_AI • • 19h ago

News Inkling-2-Mini on one RX 7900 XTX: 80 tok/s with its draft head (llama.cpp: 25), 2,148 tok/s prompt, and a 524k context at 767 tok/s prompt with 10/10 needle retrieval. Benchmarks inside (my own Rust engine, LexiSuite)

Thumbnail
github.com
1 Upvotes

I posted earlier about getting Inkling-2-Mini running on my own inference engine the day after it came out. This is the follow up with a new image and a lot more numbers. Short version: it now runs at a 524,288 token context on one 24 GB Radeon, and the 262k numbers went up quite a bit.

Hardware for everything below: RX 7900 XTX 24 GB, i7-9700, 32 GB RAM, Linux, Vulkan. One card, whole model on the card, nothing in system RAM. 4-bit KV cache on both engines. Single runs, greedy decoding, nothing else running on the box.

The model file is 17.6 GB. It keeps the experts exactly as Thinking Machines shipped them (their own 4-bit format, bit identical to the checkpoint) and the rest is 8-bit. Both engines ran that same file.

BENCHMARKS, LexiEngine vs llama.cpp b12600, same card, same file

4,096 token prompt

LexiEngine: 2,245 tok/s prompt, 81.7 tok/s generation with the draft head (52.4 without it)

llama.cpp: 425 tok/s prompt, 25.4 tok/s generation

16,384 token prompt

LexiEngine: 2,148 tok/s prompt, 80.4 tok/s generation with the draft head (50.9 without it)

llama.cpp: 397 tok/s prompt, 25.0 tok/s generation

261,632 token prompt (262k context)

LexiEngine: 1,146 tok/s prompt (3 min 48 s), 40.2 tok/s generation

llama.cpp: did not finish the prompt in 55 minutes

523,264 token prompt (524k context)

LexiEngine: 767 tok/s prompt (11 min 23 s), 33.7 tok/s generation

llama.cpp: not run

MEMORY AT 524k

weights 17.62 GB

KV cache 5.02 GB

scratch 0.58 GB

total in use on the card: 23.7 of 25.75 GB

Load time is 13 seconds. A 1M context does not fit on 24 GB with this layout (the cache alone would be 9.85 GB), so 524k is the ceiling on this card for now.

DOES IT ACTUALLY READ THE CONTEXT

I planted ten facts through the prompt at 1%, 10%, 20% and so on up to 99%, then asked for each one back.

at 260,864 tokens: 10 of 10 correct

at 523,264 tokens: 10 of 10 correct

That is the test I trust more than a perplexity number at these lengths, because my CPU reference cannot run half a million tokens in any reasonable time.

DRAFT HEAD (the model ships its own)

Output is identical to plain greedy decoding in every run I checked, it only changes speed.

after 4k tokens: 81.7 tok/s (about 6 tokens per pass of the main model)

after 16k tokens: 80.4 tok/s

at 262k: 41.8 tok/s vs 40.2 plain

at 524k: 32.1 tok/s vs 33.7 plain, so it is slower there

served at 524k context with a short prompt: 51.0 tok/s vs 43.9

So it is a big win at short and medium depth and a small loss once you are half a million tokens deep, because the draft head's own attention over that much context costs more than it saves. I left that number in on purpose. It only applies to greedy requests in this build, and you have to point the instance at the model's checkpoint folder to turn it on (the draft head is a separate file from the GGUF).

WHAT CHANGED SINCE THE LAST IMAGE (262k context)

prompt: 791 -> 1,146 tok/s (+45%)

generation: 33.3 -> 40.2 tok/s (+21%)

Most of the prompt gain came from the attention kernel for the four layers that see the whole context. That was about half of the prompt time and I had not touched it before.

ACCURACY

Against a CPU reference of the published weights, 510 positions, 4-bit KV:

LexiEngine prompt path: mean KL 0.0038

LexiEngine token by token: mean KL 0.0043, same top token 98.6%

llama.cpp: mean KL 0.0037, same top token 98.8%

So accuracy is a tie, the difference is speed. With the KV cache left unrounded my engine matches the reference exactly (KL 0.00000 over 1,024 positions), which tells me the remaining error is the 4-bit cache and not a bug. For reference the 4-bit cache costs about 0.005 mean KL against about 0.002 for 8-bit on this model.

THINGS YOU SHOULD KNOW BEFORE TRUSTING ANY OF THIS

The llama.cpp numbers were measured the day before, with a disk copy running at the same time, and I have not re-run them. They could be a bit low. I do not think it changes the picture at 4x to 5x but I would rather say it.

llama.cpp's stock converter decodes the 4-bit experts and re-quantizes them, which gives a 27.8 GB file at 8 bits and does not fit a 24 GB card. To compare on the same card I used a file that keeps the experts as shipped, and llama.cpp needed a small loader patch to run that file correctly. The patch is printed in full on the wiki page linked below.

Everything is single runs on one machine. If you have a 7900 XTX I would genuinely like to know if you get the same numbers.

Not in this build: image input, the draft head for sampled (non greedy) requests, and a 1M context.

WHAT THIS IS

LexiSuite is my own stack, written in Rust: inference engine, control panel, a bootable OS image. It is not a wrapper around llama.cpp or Ollama. This is a development image, not a production release. The USB image has no models in it and writing it erases the stick.

Also new in this build if you care: prompt reuse between requests (a follow up turn only processes the new tokens), reasoning separated from the answer in the chat API, and LoRA adapter loading for this model.

Download and USB instructions: https://lexipanel.com/download/

Benchmarks and method: https://github.com/W61k3r/LexiSuite-Releases/wiki/Benchmarks

How the Inkling runtime works, with the llama.cpp patch: https://github.com/W61k3r/LexiSuite-Releases/wiki/LexiEngine-Inkling-2

Releases: https://github.com/W61k3r/LexiSuite-Releases/releases


r/Qwen_AI • • 23h ago

Discussion When Qwen 4 becomes available for download. What do you plan on doing with it? Will you be fine tuning it to do any specific tasks

14 Upvotes

I hope I can use it for my workflow, for my business, and depend less on cloud providers to do most of the jobs. I’m constantly testing different models to see if they can do the tasks that my cloud models can do.


r/Qwen_AI • • 20h ago

Funny Holy Ball Knowledge

Post image
1 Upvotes

r/Qwen_AI • • 1h ago

News Qwen 27B rewrote its own weights and gained 6.6 points on unseen math

Thumbnail
runtimewire.com
40 Upvotes

r/Qwen_AI • • 13h ago

Funny We got GTA VI before qwen4 27b 🥀

0 Upvotes

Sorry guys I just had to say this

Hope you don't mind me

Happy with 3.8 27b but can't fking wait for 4 27b

Imma kicking legs air thinking when qwen releases i might just break nnn ☠️🥀


r/Qwen_AI • • 8h ago

Experiment A month ago, i tried to get a local Qwen on one DGX Spark to match Opus 5.5. Coding mostly got there, front-end visuals didn't.

Thumbnail reddit.com
0 Upvotes

r/Qwen_AI • • 7h ago

Help 🙋‍♂️ how do i jailbrake qwen

0 Upvotes

what is the promt that i need to send to qwen to gailbrake it sort of like the DAN jailbrake


r/Qwen_AI • • 8h ago

Benchmark I put Qwen3.8 Flash-Next and Inkling-2-Mini to the test, and the results are surprising!

8 Upvotes

r/Qwen_AI • • 14h ago

Discussion I tested 5 "think less" weight-edit builds of Qwen3.8-27B (4-bit, one 3090 @ 250 W). Concept got faster without getting dumber

2 Upvotes

Goal: get the best agentic-coding inference I can out of one RTX 3090 (power-limited to 250 W), judged on capability and time per task. In my normal workload one coding agent is running, sometimes three at once: two MiniMax Code agents plus a DeepSeek Harness (dsh) web session. The best setup I've found for this is HyperQwen, which runs a 4-bit AutoRound Qwen3.8-27B on vLLM with MTP speculative decoding. Next I tried published weight-space edits that claim to make the model think less.

TL;DR: the Concept edit is the winner. Compared with the plain 4-bit model, it uses 27% fewer reasoning tokens and 27% less wall time, and it scores higher on hard tasks (0.868 → 0.913, hardest task 0.750 → 0.983). Decode speed, MTP acceptance and KV cache are unchanged. I'm now serving it: Liljom/Qwen3.8-27B-Concept-W4A16-AutoRound-fast (Apache-2.0, drop-in for HyperQwen).

All builds compared

16 hard coding calls per model: 4 tasks × 4 seeds. H3b is a long graph-algorithm task that needs 12k–32k reasoning tokens, and it's the one that separates the models.

BuildReasoning tokensWall timeHard scoreH3b scoreVerdict
Base (4-bit AutoRound)164,8052,810 s0.8680.750baseline
+ Concept119,857 (−27%)2,040 s (−27%)0.9130.983served
+ Concept + AstraStop101,986 (−38%)1,738 s (−38%)0.9030.861out: early stops on H3b
+ Concept + Quickthinker5103,322 (−37%)1,746 s (−38%)0.7980.523out: H3b failures
+ Quickthinker5126,638 (−23%)2,280 s (−19%)0.7940.500out: H3b failures, empty answers
+ AstraStop127,776 (−22%)2,157 s (−23%)0.8080.500out: regresses on its own

Decode speed is the same on every build (68–72 tok/s), so fewer tokens means less time per task. Quickthinker and AstraStop cut more, but on the hardest task they sometimes stop thinking too early or return an empty answer. For an unattended coding agent, that's worse than spending more tokens.

The winner: Concept

MeetricBase+ Concept
Reasoning tokens, sum164,805119,857
Reasoning, median per call5,4194,969
Wall time, sum2,810 s2,040 s
Hard score (mean)0.8680.913
H3b score (mean of 4)0.7500.983
Calls hitting the 32k cap10
Decode speed, median67.9 tok/s69.9 tok/s
MTP accept rate0.6510.658
KV cache225k tokens225k tokens
  • It thinks less where it matters. It thinks less in 12 of 16 calls, and most of the saving comes from the hardest task: H3b traces drop from 24.5k–32k tokens (right at the cap) to 12.7k–24.5k (median 0.62× of the base model's length).
  • It's more capable, not just faster. The base model hit the 32k cap on one H3b run and returned nothing. Concept never hit the cap and scored higher on H3b. By raw pass count it's 12 vs 13 of 16, and that one call is an H3c bug (LRU recency) that some seed of every build fails.
  • No cost elsewhere. The edit touches only 9 down_proj matrices (layers 32–48). MTP acceptance and KV cache stay the same.
  • How it got into the int4 weights without re-quantizing ("delta flips"): I only add the extra RTN rounding flips the edit causes to the existing int4 values, which keeps 0.94–1.00 of the edit on every matrix. Every other tensor is byte-identical to the base, which I checked tensor by tensor.

Setup

1× RTX 3090 @ 250 W · HyperQwen (vLLM 0.33, MTP with 4 draft tokens) · base dbirks/Qwen3.8-27B-W4A16-AutoRound + syvai fast variant · reasoning effort medium, temp 1.0, top-k 20, top-p 0.95 · 4 concurrent requests · 32k reasoning cap.

Caveats

  • With 16 calls per model, a one-call difference is not a finding. The Concept result is consistent across tasks, though, and earlier low-effort full-suite runs also showed 7–11% less reasoning with no quality loss.
  • I tested at effort medium. The edit authors report at the highest effort.

The other builds are published for research: Concept+AstraStop · Concept+Quickthinker5 · Quickthinker5.

Credits to Qwen, dbirks, syvai, the HyperQwen authors, and the edit authors ConceptCountry, scar-ai and jarrodwatts. Happy to answer questions or run other edits through the same harness.


r/Qwen_AI • • 5h ago

Discussion How do native Chinese speakers pronounce and write "Qwen"

21 Upvotes

Some time ago I see the post about how people of different languages pronounce Qwen. It was quite funny. By the way, I pronounce it as /kwɛn/

But after the legendary post with a fish, which was assumed to be an hieroglyph like "kun", I thought about it and can't stop thinking.

How do native Chinese pronounce it and write, only in English or as hieroglyph too?


r/Qwen_AI • • 20h ago

Funny Qwhen

162 Upvotes

r/Qwen_AI • • 20h ago

Funny They will have 10000 Qwentillion Vram and will run QWEN 100 100trillion

Post image
526 Upvotes

r/Qwen_AI • • 10h ago Heartwarming

Agent Qwen4-Max coded this 53-second 3D explainer of MoE routing, voiceover included, for about $5

6 Upvotes

Every frame of this is drawn by code Qwen4-Max wrote: a Three.js scene stepped through 1,598 frames, with narration and music laid over it. Three versions in one Hermes session took a little over two hours and about $5 of OpenRouter credit.

VersionWhat I asked forAgent time
v1the brief: a router scores every expert, the top 2 get the token, their outputs are mixed by score34 min
v2rebuild it in 3D: glass towers for the experts, a particle stream for the token42 min
v3lighting, haze, labels that never overlap, tighter framing51 min

What stood out:

  • It sourced the facts first. The skill has it keep a fact sheet, and before animating anything it wrote one citing the 2017 sparse MoE paper and the Mixtral paper.
  • Sound follows the picture. The animation exports 29 timed events and every sound effect is placed from that list. The sync check had all 28 sound cues within one frame.
  • It kept what worked. v2 and v3 use v1's exact timeline. It checked that the event lists matched before reusing the narration and music.

Setup: qwen/qwen4-max through OpenRouter, driven by Hermes Agent 0.18.2 with the sn-code-video skill installed. That skill is in the https://github.com/OpenSenseNova/SenseNova-Skills repo if you want to try it on another model. edge-tts did the voice, and headless Chromium and ffmpeg did the render. 235 tool calls in the session. One prompt, one session, so it's an anecdote and not a benchmark.

Has anyone run Qwen4-Max on longer agent sessions than this? Curious where it starts to drift.


r/Qwen_AI • • 2h ago

Discussion Small model on laptop to write code fast. Big model to review and QA

Thumbnail reddit.com
2 Upvotes

r/Qwen_AI • • 9h ago

Discussion What are the best Strata forks you have used?

Thumbnail reddit.com
2 Upvotes

r/Qwen_AI • • 5h ago

Discussion From 17 to 31 tok/s on an RTX 4060 8GB: 3-bit experts on Inkling-2-Mini (follow-up to my 25 tok/s post)

5 Upvotes

TL;DR: Inkling-2-Mini (128 experts, 8 active) on an RTX 4060 8GB now runs at 31.1 tok/s decode and 545 tok/s prefill. The stock setup gave me 17.4 and 308. My last post got to 25.5 with lossless kernel work. This step is lossy: I requantized the expert weights to 3-bit. Quality loss is small and measured below.

Setup

- RTX 4060 8GB (PCIe 4.0 x8), i5-14400F, 32 GB DDR5-5600

- Unsloth UD-Q4_K_M GGUF, a custom llama.cpp build with the kernel fusions from my last post

- llama-bench: `-ngl 99 -fa 1 -t 8 -ub 512 -b 2048`

Results

| | prefill 512 | prefill 2048 | decode |

|---|---|---|---|

| stock (`--n-cpu-moe 18`) | 308 | 315 | 17.4 |

| last post (lossless) | 447 | 429 | 25.5 |

| now (3-bit experts) | 545 | 509 | 31.1 |

That's 1.8x decode and 1.8x prefill over stock. In real long coding chats through llama-server it holds 28–29 tok/s.

What I changed

  1. Only the expert gate_up weights went to 3-bit (Q3_K). The down projections stay 4-bit because their row size (1152) isn't a multiple of 256, which K-quants and IQ-quants need. The last 4 blocks went from q5_K/q8_0 to q3_K/iq4_nl.
  2. Importance matrix: I made one from about 92k tokens of llama.cpp source code (C++, CUDA, Python, Markdown). It doesn't overlap the text I used to measure quality.
  3. File size: 15.8 GB down to 12.8 GB.
  4. Placement: the smaller experts freed enough VRAM to keep the experts of two more looped blocks on the GPU (blocks 6–15 instead of 6–13). This model runs blocks 6–17 three times per token, so every looped block moved off the CPU saves three expert reads per token. That's where most of the gain comes from:

    `-ot "blk\.(0|1|2|3|4|5|16|17|18|19|20|21|22|23)\.ffn_(gate_up|down)_exps=CPU"`

  5. VRAM peak: 7.3 GB at 8k context.

Quality, compared with the Q4 model it came from

- KL divergence: 0.011 at 512 context, 0.008 at 2048

- Same top token: 97.7% / 98.0%

- Perplexity: +0.2–0.3%, about the size of the measurement noise

- My 8-task Python test with hidden unit tests: 8/8, same as Q4. It's only a smoke test: one try per task at temperature 1.0, and it used about 9% more tokens.

So it's not lossless, but it's close.

What didn't help

- IQ3_S on the GPU blocks: not faster, slightly worse quality, and it missed one coding task by running out of tokens.

- IQ3 types on the CPU: 1.4–2.5x slower than Q3_K for this matrix shape, so Q3_K is the only 3-bit type that's fast on both CPU and GPU.

- Turning the down projections to 3-bit: no 3-bit format in llama.cpp supports their 1152-wide rows. That would need a new 3-bit format, or pruning experts.

Recipe

`llama-quantize --allow-requantize --imatrix imatrix.gguf --tensor-type-file types.txt Inkling-2-Mini-UD-Q4_K_M.gguf out.gguf Q3_K 8`

`types.txt` pins every tensor to its current type, except the expert tensors listed above. It takes about 3 minutes on this CPU.

Where it stands on this box: prefill is limited by PCIe x8, because the CPU-side experts get re-uploaded to the GPU on every batch. Decode is limited by dual-channel DDR5 bandwidth for the experts that stay on the CPU. The next big step would be pruning experts for coding-only use, which I haven't tried yet.

Profiling, kernels, quantization and benchmarks done together with Claude Opus 5.5 (Claude Code). I ran everything on my machine.


r/Qwen_AI • • 12h ago

News Caphlon-2: Running a 27B model in 4.52 GiB on an 8GB laptop (80.0% GPQA-Diamond)

5 Upvotes

Hey everyone,

We are sharing Caphlon-2, a lightweight 27B model family built to run full 27B reasoning on laptops with 8 GB RAM and older GPUs with 6 GB VRAM.

Website and live demo:

https://caphlon.dev

Hugging Face models:

https://huggingface.co/ssuasd/Caphlon-2

https://huggingface.co/ssuasd/Caphlon-Model

ollama models:

https://ollama.com/univerisr/caphlon

GitHub repo and scripts:

https://github.com/demiralpdev/Caphlon-Model

WHAT IT IS

Caphlon-2 is based on Qwen 3.8 27B and Ternary-Bonsai-2-27B. It is not a distilled student model. It keeps all 64 layers (48 DeltaNet + 16 full-attention layers).

To get it down to 4.52 GiB (1.648 bpw):

  1. Vocabulary was trimmed from 151,936 down to 83,200 active tokens (V83K), which saved around 1.1 GiB without hurting English or coding quality.
  2. Intermediate FFN size was trimmed from 17,408 to 15,456 across layers 44 to 61 (LayerTrim).
  3. Weights are quantized with PTQ1_SB (Sylvester-Hadamard ternary quantization).

BENCHMARK RESULTS

Caphlon-2 (4.52 GiB):

- GPQA-Diamond: 48 / 60 (80.0%)

- Halu-270 (Hallucination benchmark): 236 / 270 (87.4%) with ITI steering, 232 / 270 (85.9%) pure

- EN-40: 32 / 40 (80.0%)

- Inference speed: 277.95 tok/s prompt processing (pp512) and 16.33 tok/s generation on Tesla T4. Around 18 tok/s on an M2 MacBook Air.

Caphlon-1 (6.87 GiB, IQ2_S):

- HumanEval: 96.7%

- Runs on stock llama.cpp and Ollama out of the box.

HOW TO RUN IT

Option 1: Caphlon-2 (4.52 GiB, PTQ1_SB)

Because of the ternary Hadamard kernel, it uses our drop-in patch for PrismML llama.cpp:

git clone https://github.com/demiralpdev/Caphlon-Model.git

cd Caphlon-Model

bash scripts/build-prism-llamacpp.sh

bash scripts/run.sh Caphlon-2-27B-PTQ1_SB.gguf 32768 1

Option 2: Caphlon-1 (6.87 GiB, IQ2_S)

Works with standard llama.cpp and Ollama:

Download Caphlon-Model-IQ2_S.gguf from Hugging Face:

https://huggingface.co/ssuasd/Caphlon-Model/resolve/main/Caphlon-Model-IQ2_S.gguf

License is Apache-2.0. Looking forward to any feedback and community testing.


r/Qwen_AI • • 1h ago

Funny Ai training in a nutshell

2 Upvotes

r/Qwen_AI • • 14h ago

Discussion Inkling-2-Mini's looped layers were the bottleneck. Moving them to the GPU and fusing 2,500 kernels got 1.47x on an 8 GB RTX 4060, 17 → 25.5 tok/s

24 Upvotes

I spent a day trying to make Thinking Machines' Inkling-2-Mini run as fast as possible on a budget box. AMD's Loom result (5.9x prefill on Strix Halo) got me curious about how far an 8 GB card can go. Short version: ~1.45x on both prefill and decode, with output that is bit-identical to the reference. That's not 6x, and below I explain exactly which hardware walls stop it.

Setup

- GPU: RTX 4060 8 GB (PCIe Gen4 x8)

- CPU / RAM: i5-14400F, 32 GB DDR5-5600 (dual channel)

- Model: Inkling-2-Mini UD-Q4_K_M (15.5 GB). 24 blocks, and blocks 6–17 run 3x per token (48 layer executions). 128 experts with 8 active, NVFP4 expert weights.

- Runtime: Unsloth's llama.cpp fork (b12588), later built from source with CUDA 12.4 for sm_89

- VRAM rule: total usage under 7.5 GB

Results (llama-bench, tok/s)

| | Baseline | Final | Speedup |

|---|---|---|---|

| Prefill 512 | 308.5 | 447.1 | 1.45x |

| Prefill 2048 | 315.3 | 428.8 | 1.36x |

| Decode (tg128) | 17.4 | 25.5 | 1.47x |

The baseline is `-ncmoe 18`, the largest that fits. Gotcha: `-ncmoe 16` looks faster in short tests, but it spills into Windows shared memory, and pp2048 collapses to 116 tok/s.

What actually worked

1. Loop-aware expert placement (the biggest single win). `--n-cpu-moe N` puts the experts of layers 0..N-1 on the CPU, which for this model means the looped layers. Those run 3x per token, so their experts get read from RAM 3 times. I flipped it with `-ot` so the GPU keeps the looped blocks 6–13 and the CPU gets the rest:

-ngl 99 -fa on -t 8 -lm none -ub 512 -b 2048 -ot "blk\.(0|1|2|3|4|5|14|15|16|17|18|19|20|21|22|23)\.ffn_(gate_up|down)_exps=CPU"

- CPU expert reads per token drop from 1435 MiB to 881 MiB.

- Decode goes 17.4 → 23.6 and pp512 goes 308 → 431.

- `-t 8` beat 6, 10, 12 and 16 threads. `-lm none` (no mmap) added about 12% prefill.

2. Source build for sm_89 only. The CUDA DLL is 286 MB instead of 447 MB in the multi-arch prebuilt, and VRAM headroom improved by a few hundred MB.

3. Faster CPU kernel for NVFP4 × Q8_0 (AVX-VNNI). It decodes nibbles 256 bits at a time, handles two superblocks per loop with exact int32 sub-block sums and one FMA, and converts the UE4M3 scales arithmetically instead of with per-block table lookups.

- It's 11–16% faster in isolation but only about 1.5% end to end. CPU expert reads now run at ~53 GB/s, which is basically what dual-channel DDR5 delivers.

4. Fused CUDA kernels in the looped block. Profiling showed ~7,800 tiny kernels per decode token, from the model's depth-attention "mix", row clipping, loop-gate controller and short convolutions. Four fixes:

- an incremental block stack for the depth mix, so it stops re-concatenating every block on every call;

- row-clip fused from 9 ops into 1;

- the loop-controller tail fused from 6 ops into 1;

- ssm_conv + residual + scale fused into 1.

That's 7,775 → ~5,200 kernels per token, GPU time 21.4 → 19.2 ms, and decode 23.8 → 25.5.

Correctness

Every kept change was gated:

- greedy decode on 6 prompts (code, prose, 1.5k and 6k-token contexts) must give identical tokens;

- logit KLD at 512 and 2048 context must stay within the limit.

All the kernel changes are bit-identical (KLD 0.000000, top-token agreement 100%). Fun finding: simply recompiling the same source with another CUDA toolkit changes the logits by KLD ~0.003 (98.9% top-1 agreement) against the prebuilt. That's with perplexity identical within noise. A 4-bit looped model amplifies float reordering across 48 layer executions, so "within 0.002 KLD of the prebuilt" effectively means "bit-identical to the prebuilt". I re-based the reference onto my own build.

What didn't help

- DSpark drafter (speculative decoding): slower in every setting (19.6–22.9 vs 23.5 tok/s), even though 66–81% of drafted tokens were accepted. Each verified token routes to its own 8 experts, so CPU expert work grows almost linearly with draft length. The drafter's VRAM also pushes an expert layer off the GPU.

- `--moe-cache-mib` (VRAM cache for hot experts): only a 34% hit rate on coding prompts, worth about +2%, so noise.

- KV cache q8_0: saves no VRAM here because the sliding-window KV is tiny, and costs 7% prefill.

- Bigger ubatch (1024/2048): long-prompt prefill jumps to 560–660 tok/s at 2k tokens, but it goes over 7.5 GB VRAM and just misses the accuracy gate.

Where the time goes now (decode, ~39 ms/token)

- ~19 ms GPU: about 5,200 small kernels, plus attention and dense weights

- ~17 ms CPU experts: DRAM-bound

- ~1.7 ms CPU↔GPU copies

Prefill is PCIe-bound: every 512-token batch re-uploads ~13.8 GiB of CPU-resident experts over Gen4 x8. So the walls are 8 GB VRAM, an x8 PCIe link and dual-channel DDR5. Getting 4–6x like Strix Halo would need the 13.8 GiB of experts to sit in fast memory, and here they don't fit.

Quality check: it scored 8/8 on my small private Python test (LRU cache, expression parser, toposort, roman numerals and so on). That was before tuning; the later kernel changes are bit-identical to my source-build reference.

Happy to share the patches or the profiling setup if anyone's interested. Next experiment: pruning experts down to a coding-only subset so all the looped layers fit in VRAM.

How this was done

The whole optimization was run by Claude Opus 5.5 (Anthropic) in Claude Code, acting as the "GPU/CUDA kernel engineer". I gave it the goal, the hardware limits and the rule that output must not change. It handled the rest:

- reading the model's graph code

- building the profiler

- writing the AVX-VNNI CPU kernel

- designing the correctness gates

- deciding what to keep or roll back

It delegated the heavy lifting to Claude Sonnet 5.5 sub-agents. One ran the 30-config tuning sweep and golden-reference runs. Another wrote the four fused CUDA kernels under Opus's spec, each checked against the bit-identical gate.

My part was the hardware, approvals, and asking annoying questions. Every number below was measured on my machine.