Professional team engaged in a meeting at a modern office table, discussing business strategies
ExplainersIndependently verified

What is mixture of experts? Why a 1.6 trillion parameter model uses only 48 billion per token

By Zain·

Photo: Edmond Dantès / Pexels

Mixture of experts lets a model hold huge numbers of parameters while using a small slice for each token. The idea dates to 2017, and it explains how today's trillion-parameter open models are built and priced.

Mixture of experts is the design trick behind many of the largest AI models now being released. It lets a model contain a vast number of parameters while using only a small share of them for any single word, or token. That is how Meituan's LongCat-2.5 can list 1.6 trillion parameters and still run with about 48 billion active per token.

This explainer covers what the term means, where it came from, and what the numbers on a model card actually tell you.

The basic idea

A standard "dense" language model uses all of its parameters for every token it reads or writes. Double the parameters, and you roughly double the compute per token.

A mixture-of-experts model splits part of the network into many smaller sub-networks, called experts. A small router, called a gating network, looks at each token and chooses which experts should handle it. The rest sit idle for that token. The Switch Transformer authors put it this way: "Mixture of Experts (MoE) defies this and instead selects different parameters for each incoming example."

The result, in their words, is a model with a very large parameter count but "a constant computational cost," because each input activates only part of the network.

Where it came from

The modern form comes from a 2017 paper by Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton and Jeff Dean, "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer." Its abstract reports over 1,000 times improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. The layer contained up to thousands of feed-forward experts, and the largest model reported had up to 137 billion parameters.

In 2021, William Fedus, Barret Zoph and Noam Shazeer simplified the routing in the Switch Transformer. They pre-trained models with up to a trillion parameters and reported a 4 times speedup over T5-XXL, plus up to 7 times faster pre-training for smaller models using the same compute. They also reported training techniques that allowed large sparse models to be trained in lower precision formats for the first time.

“Mixture of Experts (MoE) defies this and instead selects different parameters for each incoming example.”

Fedus, Zoph and Shazeer, Switch Transformers, arXiv 2101.03961, 2021
Electric blue wires connected to network adapter plugged in socket on shabby brown wall of building on street with shadow
Networking cables inside a data center, where experts on different chips exchange tokens. Photo: Nothing Ahead / Pexels

Reading a model card

Two numbers now appear on most open-weight cards. Total parameters tell you how much the model stores. Active parameters tell you how much it computes per token.

Meituan's LongCat-2.5 shows the gap. Its card lists 1.6 trillion total and about 48 billion active, so roughly 3 percent runs for each token. Its compute cost per token is closer to a mid-sized dense model than to a 1.6 trillion dense one, which is why vendors can offer low prices and fast generation.

But the saving is only on compute. Every expert still has to sit in memory, because the router might need any of them on the next token. A 1.6 trillion parameter model needs far more memory than a 48 billion parameter dense one, even if it computes less. That is why Meituan's W4A4 version, which stores expert weights and runs activations in 4-bit precision, reports 43.3 percent higher prefill throughput and 42.77 percent higher decode throughput than an 8-bit baseline. Smaller numbers take less memory and move faster.

What goes wrong

Routing is hard. If the router sends too many tokens to a few favorite experts, the others go untrained and the savings vanish. Papers spend pages on load balancing for this reason. Experts also sit on different chips, so tokens must travel between them, which makes the network connections between chips as important as the chips. Our piece on the Huawei Horinger data center shows how power and interconnect limit capacity, and the IQuest Q1 coverage is another example of a sparse open model with a headline benchmark claim.

A larger count of total parameters is also not a quality score. Active parameters, training data, and post-training work all matter. A trillion-parameter label is not a trillion-parameter brain.

A worked example

Take LongCat-2.5's published figures: 1.6 trillion total parameters and about 48 billion active. Dividing gives 3 percent, which is the share of the model doing work on each token. That is the compute story.

Now the memory story, by our own back-of-envelope arithmetic. Storing 1.6 trillion parameters at 16 bits, which is 2 bytes each, takes about 3.2 terabytes. At 4 bits, 0.5 bytes each, it is about 0.8 terabytes. Those are rough figures for the weights alone, ignoring the working memory that grows with context length, and real deployments differ. Even at 4 bits, the model needs many accelerators' worth of memory, which is why Meituan's sample setup uses hundreds of chips split across image processing, prompt ingestion and generation.

Compare that scale with the 2017 starting point of 137 billion parameters. The idea has scaled by a factor of more than ten in less than a decade, and the Switch Transformer paper sits in the middle at a trillion. What changed is not the concept but the engineering: routing stability, lower precision, and fast interconnects. If you are choosing a model for your own use, the practical test is simple. Ask what hardware the vendor says it needs, then check whether the price per token matches the active parameter count or the total.

Our take

Mixture of experts is the reason open-weight models can be big, cheap to run and still competitive on some benchmarks. It is a genuine efficiency idea with seven years of published evidence behind it, not marketing. The marketing risk is in how vendors quote the numbers.

When you see a new model, ask for both figures: total parameters for the memory bill, active parameters for the compute bill. If a card gives only the bigger one, it is selling scale. If it gives only the smaller one, it is selling speed. Neither number alone tells you whether the model is good.

Frequently asked questions

What is mixture of experts in AI?

A neural network design where many sub-networks, called experts, hold different parameters and a gating network picks a few of them for each token. Only the chosen experts run, so compute per token is much lower than the total parameter count suggests.

What are active parameters?

The number of parameters used to process a single token. Meituan's LongCat-2.5 lists about 48 billion active out of 1.6 trillion total, so roughly 3 percent runs per token.

Who invented mixture of experts for modern language models?

The sparsely-gated layer was introduced in a 2017 paper by Shazeer, Mirhoseini, Maziarz, Davis, Le, Hinton and Dean. The 2021 Switch Transformer by Fedus, Zoph and Shazeer simplified it and scaled to a trillion parameters.

Does mixture of experts save memory?

No. Every expert must be stored because any may be needed for the next token. The saving is in compute per token, which is why 4-bit formats such as W4A4 matter for serving.

Is a model with more total parameters better?

Not by itself. Active parameters, training data and post-training also drive quality. Total parameters mostly tell you the memory cost.

What goes wrong with mixture of experts?

Routing must be balanced so no experts are left untrained, and tokens must move between chips holding different experts, which makes interconnect speed as important as chip speed.

Sources

What each one is, and whose it is.

  1. PaperIndependent of the vendorNot peer reviewed, preprint
  2. PaperIndependent of the vendorNot peer reviewed, preprint
  3. 3

    LongCat-2.5 model card, Meituan LongCat, Hugging Face (November 15, 2026)

    Model card
  4. 4

    LongCat-2.5-W4A4 model card, Meituan LongCat, Hugging Face (November 15, 2026)

    Model card