The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers
Zong-xiao Li, Chong You, Srinadh Bhojanapalli, Daliang Li, A. Rawat, Sashank J. Reddi, Kenneth Q Ye, Felix Chern, Felix X. Yu, Ruiqi Guo, Surinder Kumar
International Conference on Learning Representations