ZAYA1-8B Matches DeepSeek-R1 on Math With Just 760M Active Params
A new 8B MoE model called ZAYA1-8B achieves DeepSeek-R1-level math performance while activating only 760M of its 8B para…
🏷️ MoE
2 articles about 'MoE'
Heterogeneous Grouped Mixture-of-Experts Architecture: A New MoE Paradigm Breaking the Uniform Expert Bottleneck
A latest arXiv paper proposes the Mixture of Heterogeneous Grouped Experts architecture, breaking the one-size-fits-all …
1