Hymba: A Hybrid-head Architecture for Small Language Models
Xin Dong, Y. Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, A. Mahabaleshwarkar, Shih-Yang Liu, Matthijs Van Keirsbilck, Min-Hung Chen, Yoshi Suhara, Y. Lin, Jan Kautz, Pavlo Molchanov
International Conference on Learning Representations