r/MachineLearning • • 17h ago

Discussion Is ultra-sparse MoE (200B-A3B style) a bad idea for inference? The Hy team thinks so [D]

5 Upvotes

Pulling this out of the Hy AMA thread for its own discussion, because it argues against something this sub asks for a lot. Someone asked them to push sparsity way further, something like 200B-A3B with engrams. They said no, and explained why:
• overly sparse MoE is inference-unfriendly: it eats GPU memory without using the FLOPs
• they ran scaling ladders on sparsity (they compared it to the MAI-Thinking-1 paper) and found that at fixed active params, more sparsity does improve compute efficiency
• but keeping total params fixed and cutting active params (their example was 156B-A8B or A5B) should hurt quality a lot, because active params are basically your compute budget
Their line was "increasing sparsity recovers some efficiency, but not all."
I mostly buy it for serving on datacenter GPUs. What I'm less sure about is local. A lot of us are already offloading experts to system RAM, where memory is cheap and compute is the bottleneck, so the trade-off kind of flips. They did say on-device models would get a different design, so maybe they agree.
Also a fun detail: they went back and forth between GQA and MLA for a while. MLA has better quality at the same KV cache size but was making spec decoding compute-bound, and DSA is what tipped it toward MLA.
Anyone here have numbers on ultra-sparse models running from RAM vs denser ones at the same total size?


r/MachineLearning • • 15h ago

Discussion ICLR 2027 - Deleted Reviews Are Back [D]

7 Upvotes

Has anyone else noticed that the deleted AI-generated reviews are visible again on OpenReview for ICLR 2027?


r/MachineLearning • • 16h ago

News ICLR 2027 Rebuttal Score Ranked by University [N]

Post image
60 Upvotes

Can't believe this is the third time posting it

Crawled from OpenReview, titles matched to arXiv, then affiliated with OpenAlex


r/MachineLearning • • 12h ago

Project PhiloLM: a 112M language model trained from scratch on philosophy texts [P]

Post image
20 Upvotes

I trained PhiloLM, a 112M-parameter language model from scratch using only openly available philosophy texts.

The training corpus, PhiloText, contains 0.5B tokens from philosophy encyclopedias (SEP, IEP), public-domain books, reviews, textbooks, and educational resources.

The model was trained on 1.5B tokens in 15 hours on a single Intel Arc A770 (16 GB). The architecture uses grouped-query attention, sliding-window attention, bigram embeddings, and the Muon optimizer.

Model and technical details: https://huggingface.co/gustavecortal/PhiloLM

The training dataset and interactive demo are also publicly available. Happy to discuss the architecture, training setup, or results.