r/Qwen_AI • • 16h ago

Funny They will have 10000 Qwentillion Vram and will run QWEN 100 100trillion

Post image
448 Upvotes

r/Qwen_AI • • 1h ago

Discussion How do native Chinese speakers pronounce and write "Qwen"

13 Upvotes

Some time ago I see the post about how people of different languages pronounce Qwen. It was quite funny. By the way, I pronounce it as /kwɛn/

But after the legendary post with a fish, which was assumed to be an hieroglyph like "kun", I thought about it and can't stop thinking.

How do native Chinese pronounce it and write, only in English or as hieroglyph too?


r/Qwen_AI • • 16h ago

Funny Qwhen

150 Upvotes

r/Qwen_AI • • 10h ago

Discussion Inkling-2-Mini's looped layers were the bottleneck. Moving them to the GPU and fusing 2,500 kernels got 1.47x on an 8 GB RTX 4060, 17 → 25.5 tok/s

23 Upvotes

I spent a day trying to make Thinking Machines' Inkling-2-Mini run as fast as possible on a budget box. AMD's Loom result (5.9x prefill on Strix Halo) got me curious about how far an 8 GB card can go. Short version: ~1.45x on both prefill and decode, with output that is bit-identical to the reference. That's not 6x, and below I explain exactly which hardware walls stop it.

Setup

- GPU: RTX 4060 8 GB (PCIe Gen4 x8)

- CPU / RAM: i5-14400F, 32 GB DDR5-5600 (dual channel)

- Model: Inkling-2-Mini UD-Q4_K_M (15.5 GB). 24 blocks, and blocks 6–17 run 3x per token (48 layer executions). 128 experts with 8 active, NVFP4 expert weights.

- Runtime: Unsloth's llama.cpp fork (b12588), later built from source with CUDA 12.4 for sm_89

- VRAM rule: total usage under 7.5 GB

Results (llama-bench, tok/s)

| | Baseline | Final | Speedup |

|---|---|---|---|

| Prefill 512 | 308.5 | 447.1 | 1.45x |

| Prefill 2048 | 315.3 | 428.8 | 1.36x |

| Decode (tg128) | 17.4 | 25.5 | 1.47x |

The baseline is `-ncmoe 18`, the largest that fits. Gotcha: `-ncmoe 16` looks faster in short tests, but it spills into Windows shared memory, and pp2048 collapses to 116 tok/s.

What actually worked

1. Loop-aware expert placement (the biggest single win). `--n-cpu-moe N` puts the experts of layers 0..N-1 on the CPU, which for this model means the looped layers. Those run 3x per token, so their experts get read from RAM 3 times. I flipped it with `-ot` so the GPU keeps the looped blocks 6–13 and the CPU gets the rest:

-ngl 99 -fa on -t 8 -lm none -ub 512 -b 2048 -ot "blk\.(0|1|2|3|4|5|14|15|16|17|18|19|20|21|22|23)\.ffn_(gate_up|down)_exps=CPU"

- CPU expert reads per token drop from 1435 MiB to 881 MiB.

- Decode goes 17.4 → 23.6 and pp512 goes 308 → 431.

- `-t 8` beat 6, 10, 12 and 16 threads. `-lm none` (no mmap) added about 12% prefill.

2. Source build for sm_89 only. The CUDA DLL is 286 MB instead of 447 MB in the multi-arch prebuilt, and VRAM headroom improved by a few hundred MB.

3. Faster CPU kernel for NVFP4 × Q8_0 (AVX-VNNI). It decodes nibbles 256 bits at a time, handles two superblocks per loop with exact int32 sub-block sums and one FMA, and converts the UE4M3 scales arithmetically instead of with per-block table lookups.

- It's 11–16% faster in isolation but only about 1.5% end to end. CPU expert reads now run at ~53 GB/s, which is basically what dual-channel DDR5 delivers.

4. Fused CUDA kernels in the looped block. Profiling showed ~7,800 tiny kernels per decode token, from the model's depth-attention "mix", row clipping, loop-gate controller and short convolutions. Four fixes:

- an incremental block stack for the depth mix, so it stops re-concatenating every block on every call;

- row-clip fused from 9 ops into 1;

- the loop-controller tail fused from 6 ops into 1;

- ssm_conv + residual + scale fused into 1.

That's 7,775 → ~5,200 kernels per token, GPU time 21.4 → 19.2 ms, and decode 23.8 → 25.5.

Correctness

Every kept change was gated:

- greedy decode on 6 prompts (code, prose, 1.5k and 6k-token contexts) must give identical tokens;

- logit KLD at 512 and 2048 context must stay within the limit.

All the kernel changes are bit-identical (KLD 0.000000, top-token agreement 100%). Fun finding: simply recompiling the same source with another CUDA toolkit changes the logits by KLD ~0.003 (98.9% top-1 agreement) against the prebuilt. That's with perplexity identical within noise. A 4-bit looped model amplifies float reordering across 48 layer executions, so "within 0.002 KLD of the prebuilt" effectively means "bit-identical to the prebuilt". I re-based the reference onto my own build.

What didn't help

- DSpark drafter (speculative decoding): slower in every setting (19.6–22.9 vs 23.5 tok/s), even though 66–81% of drafted tokens were accepted. Each verified token routes to its own 8 experts, so CPU expert work grows almost linearly with draft length. The drafter's VRAM also pushes an expert layer off the GPU.

- `--moe-cache-mib` (VRAM cache for hot experts): only a 34% hit rate on coding prompts, worth about +2%, so noise.

- KV cache q8_0: saves no VRAM here because the sliding-window KV is tiny, and costs 7% prefill.

- Bigger ubatch (1024/2048): long-prompt prefill jumps to 560–660 tok/s at 2k tokens, but it goes over 7.5 GB VRAM and just misses the accuracy gate.

Where the time goes now (decode, ~39 ms/token)

- ~19 ms GPU: about 5,200 small kernels, plus attention and dense weights

- ~17 ms CPU experts: DRAM-bound

- ~1.7 ms CPU↔GPU copies

Prefill is PCIe-bound: every 512-token batch re-uploads ~13.8 GiB of CPU-resident experts over Gen4 x8. So the walls are 8 GB VRAM, an x8 PCIe link and dual-channel DDR5. Getting 4–6x like Strix Halo would need the 13.8 GiB of experts to sit in fast memory, and here they don't fit.

Quality check: it scored 8/8 on my small private Python test (LRU cache, expression parser, toposort, roman numerals and so on). That was before tuning; the later kernel changes are bit-identical to my source-build reference.

Happy to share the patches or the profiling setup if anyone's interested. Next experiment: pruning experts down to a coding-only subset so all the looped layers fit in VRAM.

How this was done

The whole optimization was run by Claude Opus 5.5 (Anthropic) in Claude Code, acting as the "GPU/CUDA kernel engineer". I gave it the goal, the hardware limits and the rule that output must not change. It handled the rest:

- reading the model's graph code

- building the profiler

- writing the AVX-VNNI CPU kernel

- designing the correctness gates

- deciding what to keep or roll back

It delegated the heavy lifting to Claude Sonnet 5.5 sub-agents. One ran the 30-config tuning sweep and golden-reference runs. Another wrote the four fused CUDA kernels under Opus's spec, each checked against the bit-identical gate.

My part was the hardware, approvals, and asking annoying questions. Every number below was measured on my machine.


r/Qwen_AI • • 4h ago

Benchmark I put Qwen3.8 Flash-Next and Inkling-2-Mini to the test, and the results are surprising!

7 Upvotes

r/Qwen_AI • • 1h ago

Discussion From 17 to 31 tok/s on an RTX 4060 8GB: 3-bit experts on Inkling-2-Mini (follow-up to my 25 tok/s post)

4 Upvotes

TL;DR: Inkling-2-Mini (128 experts, 8 active) on an RTX 4060 8GB now runs at 31.1 tok/s decode and 545 tok/s prefill. The stock setup gave me 17.4 and 308. My last post got to 25.5 with lossless kernel work. This step is lossy: I requantized the expert weights to 3-bit. Quality loss is small and measured below.

Setup

- RTX 4060 8GB (PCIe 4.0 x8), i5-14400F, 32 GB DDR5-5600

- Unsloth UD-Q4_K_M GGUF, a custom llama.cpp build with the kernel fusions from my last post

- llama-bench: `-ngl 99 -fa 1 -t 8 -ub 512 -b 2048`

Results

| | prefill 512 | prefill 2048 | decode |

|---|---|---|---|

| stock (`--n-cpu-moe 18`) | 308 | 315 | 17.4 |

| last post (lossless) | 447 | 429 | 25.5 |

| now (3-bit experts) | 545 | 509 | 31.1 |

That's 1.8x decode and 1.8x prefill over stock. In real long coding chats through llama-server it holds 28–29 tok/s.

What I changed

  1. Only the expert gate_up weights went to 3-bit (Q3_K). The down projections stay 4-bit because their row size (1152) isn't a multiple of 256, which K-quants and IQ-quants need. The last 4 blocks went from q5_K/q8_0 to q3_K/iq4_nl.
  2. Importance matrix: I made one from about 92k tokens of llama.cpp source code (C++, CUDA, Python, Markdown). It doesn't overlap the text I used to measure quality.
  3. File size: 15.8 GB down to 12.8 GB.
  4. Placement: the smaller experts freed enough VRAM to keep the experts of two more looped blocks on the GPU (blocks 6–15 instead of 6–13). This model runs blocks 6–17 three times per token, so every looped block moved off the CPU saves three expert reads per token. That's where most of the gain comes from:

    `-ot "blk\.(0|1|2|3|4|5|16|17|18|19|20|21|22|23)\.ffn_(gate_up|down)_exps=CPU"`

  5. VRAM peak: 7.3 GB at 8k context.

Quality, compared with the Q4 model it came from

- KL divergence: 0.011 at 512 context, 0.008 at 2048

- Same top token: 97.7% / 98.0%

- Perplexity: +0.2–0.3%, about the size of the measurement noise

- My 8-task Python test with hidden unit tests: 8/8, same as Q4. It's only a smoke test: one try per task at temperature 1.0, and it used about 9% more tokens.

So it's not lossless, but it's close.

What didn't help

- IQ3_S on the GPU blocks: not faster, slightly worse quality, and it missed one coding task by running out of tokens.

- IQ3 types on the CPU: 1.4–2.5x slower than Q3_K for this matrix shape, so Q3_K is the only 3-bit type that's fast on both CPU and GPU.

- Turning the down projections to 3-bit: no 3-bit format in llama.cpp supports their 1152-wide rows. That would need a new 3-bit format, or pruning experts.

Recipe

`llama-quantize --allow-requantize --imatrix imatrix.gguf --tensor-type-file types.txt Inkling-2-Mini-UD-Q4_K_M.gguf out.gguf Q3_K 8`

`types.txt` pins every tensor to its current type, except the expert tensors listed above. It takes about 3 minutes on this CPU.

Where it stands on this box: prefill is limited by PCIe x8, because the CPU-side experts get re-uploaded to the GPU on every batch. Decode is limited by dual-channel DDR5 bandwidth for the experts that stay on the CPU. The next big step would be pruning experts for coding-only use, which I haven't tried yet.

Profiling, kernels, quantization and benchmarks done together with Claude Opus 5.5 (Claude Code). I ran everything on my machine.


r/Qwen_AI • • 23h ago

Discussion Qwen 3.8 Flash Next 177B runs on a 16GB GPU + 32GB RAM: built a full website from one prompt (10/10), real numbers inside

Thumbnail
gallery
124 Upvotes

I wanted to see if Qwen 3.8 Flash Next 177B could run on normal gaming hardware. Setup: RTX 4080 SUPER 16 GB, 32 GB DDR5, Windows 11. Model: ISTA-DASLab IQ3_XXS quant (75.8 GB), plain llama.cpp + OpenCode, fully offline.

It works because it's MoE: only 10 of 512 experts are active per token, so llama.cpp keeps the hot layers on the GPU and streams the cold experts from the SSD (mmap).

Speed:

  • --cpu-moe: 11.2 tok/s
  • -ncmoe 42: 15.2 tok/s (+36%), 13.2 GB VRAM
  • Over a 2.5-hour coding run: 18.2 tok/s average, 22.6 best. RAM 31.4/31.6 GB the whole time, 0 crashes.

Quality: from ONE prompt it built a complete website (SVG animation, sticky nav, 3D tilt cards, counters, form validation, no libraries) and scored 10/10 on my checklist.

The funny part: when I asked it to "check its own work", it broke one animation and then said "nothing broken" twice. It reads the code but never sees the page.

Honestly, this model is impressive for something running on a home PC. Has anyone tried it with 64 GB RAM? I'd love to know how much faster it gets.


r/Qwen_AI • • 6h ago Heartwarming

Agent Qwen4-Max coded this 53-second 3D explainer of MoE routing, voiceover included, for about $5

5 Upvotes

Every frame of this is drawn by code Qwen4-Max wrote: a Three.js scene stepped through 1,598 frames, with narration and music laid over it. Three versions in one Hermes session took a little over two hours and about $5 of OpenRouter credit.

VersionWhat I asked forAgent time
v1the brief: a router scores every expert, the top 2 get the token, their outputs are mixed by score34 min
v2rebuild it in 3D: glass towers for the experts, a particle stream for the token42 min
v3lighting, haze, labels that never overlap, tighter framing51 min

What stood out:

  • It sourced the facts first. The skill has it keep a fact sheet, and before animating anything it wrote one citing the 2017 sparse MoE paper and the Mixtral paper.
  • Sound follows the picture. The animation exports 29 timed events and every sound effect is placed from that list. The sync check had all 28 sound cues within one frame.
  • It kept what worked. v2 and v3 use v1's exact timeline. It checked that the event lists matched before reusing the narration and music.

Setup: qwen/qwen4-max through OpenRouter, driven by Hermes Agent 0.18.2 with the sn-code-video skill installed. That skill is in the https://github.com/OpenSenseNova/SenseNova-Skills repo if you want to try it on another model. edge-tts did the voice, and headless Chromium and ffmpeg did the render. 235 tool calls in the session. One prompt, one session, so it's an anecdote and not a benchmark.

Has anyone run Qwen4-Max on longer agent sessions than this? Curious where it starts to drift.


r/Qwen_AI • • 5h ago

Discussion What are the best Strata forks you have used?

Thumbnail reddit.com
2 Upvotes

r/Qwen_AI • • 8h ago

News Caphlon-2: Running a 27B model in 4.52 GiB on an 8GB laptop (80.0% GPQA-Diamond)

3 Upvotes

Hey everyone,

We are sharing Caphlon-2, a lightweight 27B model family built to run full 27B reasoning on laptops with 8 GB RAM and older GPUs with 6 GB VRAM.

Website and live demo:

https://caphlon.dev

Hugging Face models:

https://huggingface.co/ssuasd/Caphlon-2

https://huggingface.co/ssuasd/Caphlon-Model

ollama models:

https://ollama.com/univerisr/caphlon

GitHub repo and scripts:

https://github.com/demiralpdev/Caphlon-Model

WHAT IT IS

Caphlon-2 is based on Qwen 3.8 27B and Ternary-Bonsai-2-27B. It is not a distilled student model. It keeps all 64 layers (48 DeltaNet + 16 full-attention layers).

To get it down to 4.52 GiB (1.648 bpw):

  1. Vocabulary was trimmed from 151,936 down to 83,200 active tokens (V83K), which saved around 1.1 GiB without hurting English or coding quality.
  2. Intermediate FFN size was trimmed from 17,408 to 15,456 across layers 44 to 61 (LayerTrim).
  3. Weights are quantized with PTQ1_SB (Sylvester-Hadamard ternary quantization).

BENCHMARK RESULTS

Caphlon-2 (4.52 GiB):

- GPQA-Diamond: 48 / 60 (80.0%)

- Halu-270 (Hallucination benchmark): 236 / 270 (87.4%) with ITI steering, 232 / 270 (85.9%) pure

- EN-40: 32 / 40 (80.0%)

- Inference speed: 277.95 tok/s prompt processing (pp512) and 16.33 tok/s generation on Tesla T4. Around 18 tok/s on an M2 MacBook Air.

Caphlon-1 (6.87 GiB, IQ2_S):

- HumanEval: 96.7%

- Runs on stock llama.cpp and Ollama out of the box.

HOW TO RUN IT

Option 1: Caphlon-2 (4.52 GiB, PTQ1_SB)

Because of the ternary Hadamard kernel, it uses our drop-in patch for PrismML llama.cpp:

git clone https://github.com/demiralpdev/Caphlon-Model.git

cd Caphlon-Model

bash scripts/build-prism-llamacpp.sh

bash scripts/run.sh Caphlon-2-27B-PTQ1_SB.gguf 32768 1

Option 2: Caphlon-1 (6.87 GiB, IQ2_S)

Works with standard llama.cpp and Ollama:

Download Caphlon-Model-IQ2_S.gguf from Hugging Face:

https://huggingface.co/ssuasd/Caphlon-Model/resolve/main/Caphlon-Model-IQ2_S.gguf

License is Apache-2.0. Looking forward to any feedback and community testing.


r/Qwen_AI • • 4h ago

Experiment A month ago, i tried to get a local Qwen on one DGX Spark to match Opus 5.5. Coding mostly got there, front-end visuals didn't.

Thumbnail reddit.com
0 Upvotes

r/Qwen_AI • • 20h ago

Discussion When Qwen 4 becomes available for download. What do you plan on doing with it? Will you be fine tuning it to do any specific tasks

13 Upvotes

I hope I can use it for my workflow, for my business, and depend less on cloud providers to do most of the jobs. I’m constantly testing different models to see if they can do the tasks that my cloud models can do.


r/Qwen_AI • • 1d ago

Discussion Your 35B prayers have been answered

Post image
921 Upvotes

Scores the same as Qwen 3.8 27B on AA


r/Qwen_AI • • 8h ago

Discussion i built a free 36M-param tool that lets LLMs “watch” videos locally (MCP server, runs on laptop cpu, apache-2.0)

Thumbnail reddit.com
1 Upvotes

r/Qwen_AI • • 1d ago

Vibe Coding If anyone is second guess getting a M5U 256…. don’t

Post image
238 Upvotes

My uncensored qwen flash next Q8 found this bs on my tv

never said no and actually breached my tv

now we’re getting rid of it

something claude and coded refuses to do

open source is life

running this model around 100-120 tps a second

i cannot wait to get QFN4🔥🔥🔥


r/Qwen_AI • • 1d ago

Help 🙋‍♂️ Sorr, Qwen. When will the change happen?

Post image
99 Upvotes

It has been two weeks since Qwen said, "The becoming begins."

So, I turned to Qwen and asked, "Hey, Qwen. When will the change happen?"

Sorry the image is in Japanese—I’m Japan.


r/Qwen_AI • • 10h ago

Experiment Adaptive KV-Cache Streaming V3 (Oceanus): Full 262K Context Qwen 27B on a 16GB GPU, with Multi-Slot Inference, Vision, and Speculative Decoding

Thumbnail
reddit.com
1 Upvotes

r/Qwen_AI • • 10h ago

Discussion I tested 5 "think less" weight-edit builds of Qwen3.8-27B (4-bit, one 3090 @ 250 W). Concept got faster without getting dumber

1 Upvotes

Goal: get the best agentic-coding inference I can out of one RTX 3090 (power-limited to 250 W), judged on capability and time per task. In my normal workload one coding agent is running, sometimes three at once: two MiniMax Code agents plus a DeepSeek Harness (dsh) web session. The best setup I've found for this is HyperQwen, which runs a 4-bit AutoRound Qwen3.8-27B on vLLM with MTP speculative decoding. Next I tried published weight-space edits that claim to make the model think less.

TL;DR: the Concept edit is the winner. Compared with the plain 4-bit model, it uses 27% fewer reasoning tokens and 27% less wall time, and it scores higher on hard tasks (0.868 → 0.913, hardest task 0.750 → 0.983). Decode speed, MTP acceptance and KV cache are unchanged. I'm now serving it: Liljom/Qwen3.8-27B-Concept-W4A16-AutoRound-fast (Apache-2.0, drop-in for HyperQwen).

All builds compared

16 hard coding calls per model: 4 tasks × 4 seeds. H3b is a long graph-algorithm task that needs 12k–32k reasoning tokens, and it's the one that separates the models.

BuildReasoning tokensWall timeHard scoreH3b scoreVerdict
Base (4-bit AutoRound)164,8052,810 s0.8680.750baseline
+ Concept119,857 (−27%)2,040 s (−27%)0.9130.983served
+ Concept + AstraStop101,986 (−38%)1,738 s (−38%)0.9030.861out: early stops on H3b
+ Concept + Quickthinker5103,322 (−37%)1,746 s (−38%)0.7980.523out: H3b failures
+ Quickthinker5126,638 (−23%)2,280 s (−19%)0.7940.500out: H3b failures, empty answers
+ AstraStop127,776 (−22%)2,157 s (−23%)0.8080.500out: regresses on its own

Decode speed is the same on every build (68–72 tok/s), so fewer tokens means less time per task. Quickthinker and AstraStop cut more, but on the hardest task they sometimes stop thinking too early or return an empty answer. For an unattended coding agent, that's worse than spending more tokens.

The winner: Concept

MeetricBase+ Concept
Reasoning tokens, sum164,805119,857
Reasoning, median per call5,4194,969
Wall time, sum2,810 s2,040 s
Hard score (mean)0.8680.913
H3b score (mean of 4)0.7500.983
Calls hitting the 32k cap10
Decode speed, median67.9 tok/s69.9 tok/s
MTP accept rate0.6510.658
KV cache225k tokens225k tokens
  • It thinks less where it matters. It thinks less in 12 of 16 calls, and most of the saving comes from the hardest task: H3b traces drop from 24.5k–32k tokens (right at the cap) to 12.7k–24.5k (median 0.62× of the base model's length).
  • It's more capable, not just faster. The base model hit the 32k cap on one H3b run and returned nothing. Concept never hit the cap and scored higher on H3b. By raw pass count it's 12 vs 13 of 16, and that one call is an H3c bug (LRU recency) that some seed of every build fails.
  • No cost elsewhere. The edit touches only 9 down_proj matrices (layers 32–48). MTP acceptance and KV cache stay the same.
  • How it got into the int4 weights without re-quantizing ("delta flips"): I only add the extra RTN rounding flips the edit causes to the existing int4 values, which keeps 0.94–1.00 of the edit on every matrix. Every other tensor is byte-identical to the base, which I checked tensor by tensor.

Setup

1× RTX 3090 @ 250 W · HyperQwen (vLLM 0.33, MTP with 4 draft tokens) · base dbirks/Qwen3.8-27B-W4A16-AutoRound + syvai fast variant · reasoning effort medium, temp 1.0, top-k 20, top-p 0.95 · 4 concurrent requests · 32k reasoning cap.

Caveats

  • With 16 calls per model, a one-call difference is not a finding. The Concept result is consistent across tasks, though, and earlier low-effort full-suite runs also showed 7–11% less reasoning with no quality loss.
  • I tested at effort medium. The edit authors report at the highest effort.

The other builds are published for research: Concept+AstraStop · Concept+Quickthinker5 · Quickthinker5.

Credits to Qwen, dbirks, syvai, the HyperQwen authors, and the edit authors ConceptCountry, scar-ai and jarrodwatts. Happy to answer questions or run other edits through the same harness.


r/Qwen_AI • • 1d ago

Discussion So, fam, 5 days later… what are y’all’s experiences and opinions on Qwen 4 Max?

31 Upvotes

Been using the web option mostly. You can see improvements, it’s real, a bit slow but rock solid overall. Maybe not what we were all hoping for regarding frontier performance, but still, it’s not as bad or disappointing as people make it out to be.

P.S. Also appreciate that Qwen isn’t introducing limits on its web app like DeepSeek and ChatGPT.


r/Qwen_AI • • 15h ago

News We distilled our Qwen-Image-2.1 Uncensored Omni model into an 8-step Turbo LoRA—6.8× faster with comparable quality

Thumbnail
gallery
2 Upvotes

The original Omni release has now reached #8 trending out of more than 3 million models on Hugging Face. The feedback from the Qwen community helped us decide what to build next: a much faster version that preserves the model’s unrestricted generation and editing capabilities.

Qwen-Image-2.1 Uncensored Omni — Turbo LoRA

The Turbo LoRA runs with:

  • 8 sampling steps
  • CFG 1.0
  • Approximately 1-megapixel output
  • Text-to-image generation
  • Instruction-based image editing
  • ComfyUI and Diffusers support
  • No prompt filter or output safety checker added by this release

On our Playground hardware:

  • Standard Omni, 35 steps: 54.9 seconds per image
  • Omni + Turbo, 8 steps: 8.1 seconds per image
  • Speedup: 6.8×

Benchmark results

We tested the Turbo LoRA at 8 steps against the same Omni model at 35 steps.

BenchmarkOmni, 35 stepsTurbo, 8 steps
Adult prompt adherence89.789.1
Anatomy69.472.9
General SFW prompts79.679.0
Image editing78.981.6
Identity preservation in human editing55.872.1

Prompt adherence and general SFW quality were within the statistical noise of the 35-step model. Anatomy, image editing and identity preservation improved in our evaluation.

The Turbo LoRA can be added on top of supported GGUF, BF16, FP8, NF4, INT8 ConvRot and NVFP4 Omni builds.

Download and instructions

https://huggingface.co/chatrlabs/Qwen-Image-2.1-Uncensored-Omni-Quantized#turbo-lora-8-steps-cfg-10

Recommended starting settings:

  • Turbo LoRA strength: 1.0
  • Sampling steps: 8
  • True CFG: 1.0
  • No negative-prompt pass required

The model card includes the weights, checksums, Diffusers code, ComfyUI instructions, compatibility notes, complete benchmarks and known limitations.

Try it online

https://thechatr.ai/playground

Our free Playground now serves more than 10,000 image generations every day.

A huge thank-you to the Qwen team for releasing Qwen-Image-2.1 and making community research like this possible. We’re an independent community fine-tune and are not affiliated with the upstream Qwen team.

We would especially appreciate technical feedback from the Qwen community:

  • Which prompts or compositions remain difficult?
  • How does Turbo perform on your hardware?
  • Which quantized build works best for you?
  • What should we improve in the next release?

Thank you to everyone who has downloaded, tested and shared the model. ❤️


r/Qwen_AI • • 4h ago

Help 🙋‍♂️ how do i jailbrake qwen

0 Upvotes

what is the promt that i need to send to qwen to gailbrake it sort of like the DAN jailbrake


r/Qwen_AI • • 1d ago

News We got Riemann Hypothesis solved before Qwen4

70 Upvotes

r/Qwen_AI • • 1d ago

Model I fine-tuned Qwen2.5-Coder-1.5B into an offline Linux command helper that runs on CPU and checks its own answers against man pages and tldr

55 Upvotes

lxask turns plain English into a shell command, entirely on your own machine:

$ lxask "show disk usage of each folder here, biggest first"

$ du -h --max-depth=1 . | sort -hr

Setup

- Fine-tuned Qwen2.5-Coder-1.5B, Q4_K_M GGUF

- llama.cpp inside one Rust binary. No Ollama, no Python, no API key

- ~1.1 GB on disk, ~1 GB RAM, ~2 s per answer on a laptop CPU

- No GPU, no network after install

How it keeps a 1.5B model honest:

- Retrieves relevant tldr pages and your own man pages before answering

- Checks every program and option in the answer against the man pages

- Retries if a check fails. Anything it still can't confirm is marked `unverified:`

- Asks back on vague requests ("add a proxy": for git? apt? docker?) instead of guessing

Bonus

- `lxask --explain "tar -xzvf logs.tar.gz"` breaks down a command you already have

- Ctrl+G in bash, zsh or fish turns what you typed into the command

Code, weights, data and evals are all open.

GitHub: https://github.com/arxhr007/lxask

Hope someone finds it useful! If you do, consider leaving a ⭐ on GitHub. Feedback and contributions are welcome

Model: https://huggingface.co/arxhr007/lxask


r/Qwen_AI • • 15h ago

News Inkling-2-Mini on one RX 7900 XTX: 80 tok/s with its draft head (llama.cpp: 25), 2,148 tok/s prompt, and a 524k context at 767 tok/s prompt with 10/10 needle retrieval. Benchmarks inside (my own Rust engine, LexiSuite)

Thumbnail
github.com
0 Upvotes

I posted earlier about getting Inkling-2-Mini running on my own inference engine the day after it came out. This is the follow up with a new image and a lot more numbers. Short version: it now runs at a 524,288 token context on one 24 GB Radeon, and the 262k numbers went up quite a bit.

Hardware for everything below: RX 7900 XTX 24 GB, i7-9700, 32 GB RAM, Linux, Vulkan. One card, whole model on the card, nothing in system RAM. 4-bit KV cache on both engines. Single runs, greedy decoding, nothing else running on the box.

The model file is 17.6 GB. It keeps the experts exactly as Thinking Machines shipped them (their own 4-bit format, bit identical to the checkpoint) and the rest is 8-bit. Both engines ran that same file.

BENCHMARKS, LexiEngine vs llama.cpp b12600, same card, same file

4,096 token prompt

LexiEngine: 2,245 tok/s prompt, 81.7 tok/s generation with the draft head (52.4 without it)

llama.cpp: 425 tok/s prompt, 25.4 tok/s generation

16,384 token prompt

LexiEngine: 2,148 tok/s prompt, 80.4 tok/s generation with the draft head (50.9 without it)

llama.cpp: 397 tok/s prompt, 25.0 tok/s generation

261,632 token prompt (262k context)

LexiEngine: 1,146 tok/s prompt (3 min 48 s), 40.2 tok/s generation

llama.cpp: did not finish the prompt in 55 minutes

523,264 token prompt (524k context)

LexiEngine: 767 tok/s prompt (11 min 23 s), 33.7 tok/s generation

llama.cpp: not run

MEMORY AT 524k

weights 17.62 GB

KV cache 5.02 GB

scratch 0.58 GB

total in use on the card: 23.7 of 25.75 GB

Load time is 13 seconds. A 1M context does not fit on 24 GB with this layout (the cache alone would be 9.85 GB), so 524k is the ceiling on this card for now.

DOES IT ACTUALLY READ THE CONTEXT

I planted ten facts through the prompt at 1%, 10%, 20% and so on up to 99%, then asked for each one back.

at 260,864 tokens: 10 of 10 correct

at 523,264 tokens: 10 of 10 correct

That is the test I trust more than a perplexity number at these lengths, because my CPU reference cannot run half a million tokens in any reasonable time.

DRAFT HEAD (the model ships its own)

Output is identical to plain greedy decoding in every run I checked, it only changes speed.

after 4k tokens: 81.7 tok/s (about 6 tokens per pass of the main model)

after 16k tokens: 80.4 tok/s

at 262k: 41.8 tok/s vs 40.2 plain

at 524k: 32.1 tok/s vs 33.7 plain, so it is slower there

served at 524k context with a short prompt: 51.0 tok/s vs 43.9

So it is a big win at short and medium depth and a small loss once you are half a million tokens deep, because the draft head's own attention over that much context costs more than it saves. I left that number in on purpose. It only applies to greedy requests in this build, and you have to point the instance at the model's checkpoint folder to turn it on (the draft head is a separate file from the GGUF).

WHAT CHANGED SINCE THE LAST IMAGE (262k context)

prompt: 791 -> 1,146 tok/s (+45%)

generation: 33.3 -> 40.2 tok/s (+21%)

Most of the prompt gain came from the attention kernel for the four layers that see the whole context. That was about half of the prompt time and I had not touched it before.

ACCURACY

Against a CPU reference of the published weights, 510 positions, 4-bit KV:

LexiEngine prompt path: mean KL 0.0038

LexiEngine token by token: mean KL 0.0043, same top token 98.6%

llama.cpp: mean KL 0.0037, same top token 98.8%

So accuracy is a tie, the difference is speed. With the KV cache left unrounded my engine matches the reference exactly (KL 0.00000 over 1,024 positions), which tells me the remaining error is the 4-bit cache and not a bug. For reference the 4-bit cache costs about 0.005 mean KL against about 0.002 for 8-bit on this model.

THINGS YOU SHOULD KNOW BEFORE TRUSTING ANY OF THIS

The llama.cpp numbers were measured the day before, with a disk copy running at the same time, and I have not re-run them. They could be a bit low. I do not think it changes the picture at 4x to 5x but I would rather say it.

llama.cpp's stock converter decodes the 4-bit experts and re-quantizes them, which gives a 27.8 GB file at 8 bits and does not fit a 24 GB card. To compare on the same card I used a file that keeps the experts as shipped, and llama.cpp needed a small loader patch to run that file correctly. The patch is printed in full on the wiki page linked below.

Everything is single runs on one machine. If you have a 7900 XTX I would genuinely like to know if you get the same numbers.

Not in this build: image input, the draft head for sampled (non greedy) requests, and a 1M context.

WHAT THIS IS

LexiSuite is my own stack, written in Rust: inference engine, control panel, a bootable OS image. It is not a wrapper around llama.cpp or Ollama. This is a development image, not a production release. The USB image has no models in it and writing it erases the stick.

Also new in this build if you care: prompt reuse between requests (a follow up turn only processes the new tokens), reasoning separated from the answer in the chat API, and LoRA adapter loading for this model.

Download and USB instructions: https://lexipanel.com/download/

Benchmarks and method: https://github.com/W61k3r/LexiSuite-Releases/wiki/Benchmarks

How the Inkling runtime works, with the llama.cpp patch: https://github.com/W61k3r/LexiSuite-Releases/wiki/LexiEngine-Inkling-2

Releases: https://github.com/W61k3r/LexiSuite-Releases/releases


r/Qwen_AI • • 17h ago

Funny Holy Ball Knowledge

Post image
0 Upvotes