A few days ago I posted inferzero, a free, open-source (Apache 2.0) LLM engine written only for Intel GPUs (C++ on Level Zero, GGUF files as they are, Windows and Linux, no Python). Thanks to everyone who ran it: most of the numbers below come from your logs, on cards I don’t own.
First, which number is which, since that wasn’t clear enough last time:
- pp = prompt processing: how fast the prompt is read, in tokens per second (4,096-token prompts unless noted). This is the big number.
- tg = token generation: how fast the answer is written, in tokens per second. This is the number you feel while it types.
- tg + MTP = generation with the model’s built-in draft head; the text is identical to plain generation.
Qwen3.8 27B (IQ3_S), one GPU, measured by owners:
| GPU | pp (tok/s) | tg (tok/s) | tg + MTP (tok/s) |
|---|
| Arc Pro B70 32 GB (Linux) | 2,972 | 35.2 | 70.6 |
| Data Center GPU Max 1550, one stack (Linux) | 2,642 | – | – |
| Arc B580 12 GB (Linux, PCIe 3.0) | 1,877 | 13.7 | – |
| Arc Pro B65 32 GB (Linux) | 1,735 | 21.6 | 51.8 |
| Arc Pro B60 24 GB (Windows) | 1,726 | 23.7 | 51.6 |
| Arc A770 16 GB (Windows / Linux) | 1,223 | 22.9 | 40.4 (Linux) |
| Arc B390 iGPU (Core Ultra X7 358H) | 637 | 5.9 | 16.8 |
| Arc 140V iGPU (Core Ultra 7 258V) | 444 | 6.5 | 13.9 |
| Arc 140T iGPU (Core Ultra 9 285H) | 405 | 6.1 | 14.6 |
Smaller models: the Qwen3.5 4B on a B580 does pp 7,915 / tg 92.6; the Qwen3.6 35B-A3B (MoE) on a B70 pp 5,647 (8K prompt) / tg 95-98, on a B60 pp 3,536 / tg 82.4 (111.9 with MTP).
What’s new since the first post:
- Several chats at once (batched decoding,
serverz --slots 4): their next tokens run in one pass. Total tg for 4 chats on one B60: the 27B 70.2 tok/s (23.6 alone), the 4B 235.2 (90.9 alone). The output is bit-identical to running each chat alone. - A model that doesn’t fit the card: the 27B on a 12 GB B580 keeps ~0.5 GB in RAM, and the CPU now computes that part instead of the GPU reading it over PCIe. tg went 7.1 -> 13.7 tok/s (llama.cpp SYCL on the same card and file: 11.4).
- Alchemist (A770 / A750 / A580) runs, with the kernels sent in recorded lists on Windows: the 27B’s tg 12.7 -> 21.6 tok/s (22.9 on Linux).
- Head to head on a B580 (same files, d-dorok’s runs; pp / tg in tok/s): Qwen3.5 4B inferzero 7,915 / 92.6, llama.cpp SYCL 1,520 / 95.4, Vulkan 2,411 / 79.7. So pp is several times faster here, while tg is about even (llama.cpp SYCL is 3-9 % ahead on small models’ tg).
- Lunar Lake, the Pro B50, the Data Center GPU Max and several-GPU setups (2-4 B60s, 2 A770s, 2 B50s) have results too.
Every setup’s table, model by model, with its release and who measured it: https://rizagp.github.io/inferzero/results.html
It’s still far from what these chips can do. The 140T’s prompt GEMM runs at ~33-39 % of its XMX peak; the B65’s tg reads only ~41 % of its memory bandwidth; on Windows a model split over several GPUs waits on host memory for every sum, so two A770s are barely faster than one for a model that fits one card (the 35B-A3B does gain: tg 28.3 vs 20.4).
Testers wanted, especially:
- Arc A750, A580, A380, B570 and the laptop A-series (A370M / A770M): nothing measured yet.
- Core Ultra laptops other than the ones above: Meteor Lake 155H / 165H, Lunar Lake 256V / 268V, Arrow Lake H, Panther Lake.
- Two or more Arc cards on Windows, to test the faster multi-GPU sums (
ARC_TP_EVSUM=1). - Mismatched PCIe slots (x16 + x4), for a planned layer split.
Running it: download the release, start zero-launcher.cmd (Windows) or ./zero-launcher.sh (Linux), pick a model. For results, run-tests.cmd quick writes one zip to attach to an issue, or post bin\runz.exe <model.gguf> --bench-pp 4096 --n 128. There’s a “Hardware results” issue template.
Code help wanted: GEMM/GEMV kernels on XMX and DP4a, multi-GPU sums on Windows, Linux packaging (Nix, AUR, Fedora), the web UI. Good first issues are labelled; CONTRIBUTING.md lists where the speed is missing and how changes are checked (accuracy against an f32 reference, bit-identical logits where promised).