#llm-pretraining

The latest AI and developer news about #llm-pretraining, with the original source and a short summary.

Meta Engineers Detail TLX: A Hardware-Native GPU Compiler Extension for Production ML

The Triton Low-Level Extensions (TLX) framework enables explicit warp-group orchestration, delivering up to 2x forward speedup in GDPA kernels and reducing RecSys inference latency by up to two-thirds.

·2026.11.11 01:00

Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings

Mixing high-resource auxiliary data boosts performance equivalent to 2–3× more unique target data on validation loss.

·2·2026.10.13 09:00

Study reveals 'mode-hopping' in LM pre-training where models switch between memorization and generalization

OLMo3-32B scores dropped from 81% to 0% on an arithmetic eval between 2.17T and 2.19T tokens before recovering.

·1·2026.10.01 09:00

Magic AI Releases 'Magic V5 e24', Achieving Performance Similar to DeepSeek V4 Pro with 48x Fewer Computations

Magic AI has released the Magic V5 e24 model, achieving performance similar to DeepSeek V4 Pro with approximately 48x fewer FLOPs.

·2026.09.08 09:00

Cosmopedia, a 25 Billion Token Dataset, Released

HuggingFace has released Cosmopedia, a large-scale open synthetic dataset for LLM pretraining.

·2024.03.20 09:00