
What is mixture of experts? Why a 1.6 trillion parameter model uses only 48 billion per token
Mixture of experts lets a model hold huge numbers of parameters while using a small slice for each token. The idea dates to 2017, and it explains how today's trillion-parameter open models are built and priced.










