r/LocalLLaMA • • 2d ago

Question | Help Tencent Hy Team & Shunyu Yao Here! Ask Us Anything about Hy4 & Hy4 flash!

Fresh off launching Hy4 and Hy4 Flash, the Hy research team is here to discuss how they built the models, boosted efficiency, and made them better for everyday workflows.

Our participants today:

u/No-Cherry2850

u/DiWang01

u/aiden_agent

u/lydellhuang

u/edwardbai

u/kkelvinchen

u/maxmpan

u/dick112233

u/kkelvinchen

u/GeoMorax

Blog: https://hy.tencent.ai/research/hy4-flash https://hy.tencent.ai/research/hy4

HF: https://huggingface.co/tencent/Hy4-flash https://huggingface.co/tencent/Hy4

345 Upvotes

442 comments sorted by

View all comments

Show parent comment

21

u/edwardbai 1d ago

Good question. We went back and forth between GQA and MLA for quite a while. MLA has significantly higher compute intensity than GQA, which made spec decoding on inference GPUs compute-bound. However, at the same KV cache size, MLA delivers better model quality than GQA. Fortunately, with the introduction of DSA, we were able to bring down MLA's compute intensity during decoding. That ultimately tipped the balance, and we decided to go with MLA.