Ai2 Launches Olmo-core 3 for Trillion-Parameter MoE Training

Allen Institute for AI has released Olmo-core 3, a training framework designed to make mixture-of-experts large language models more efficient at scale. The framework scales the expert pool from eight to 128 while selecting only four experts per token, enabling models exceeding one trillion parameters. In benchmarks on Nvidia B3000 GPUs, it processed 52,000 tokens per second on a 47 billion-parameter model, roughly 2.7 times the throughput of Nvidia's Megatron-core. Key techniques include expert parallelism, layer splitting across GPU groups, a distributed optimizer, and MXFP8 number formatting. The project is available open-source on GitHub.