An Old Efficiency Hack Finds Its Place in AI Innovation

The paper Grouped Query Experts shows that a mixture-of-experts routing strategy applied to the attention layer of a language model matches standard quality while activating only about half the query heads — bringing the "committee of specialists" idea to a part of the architecture it had not touche