AI Briefing#pretraining
The latest AI and developer news about #pretraining, with the original source and a short summary.
The latest AI and developer news about #pretraining, with the original source and a short summary.
The self-supervised encoder achieves 49.3% tactile-to-video recall at 4K hours, significantly outperforming reconstruction-based baselines.
The method stores precomputed n-gram counts on host memory to improve validation loss and enable domain adaptation without training.
Experiments with models from 135M to 1B parameters show LIFT outperforms standard Transformers on language modeling and reasoning tasks under token-matched budgets.
Steven Byrnes argues that LLMs excel in mathematics due to the high accuracy (over 99%) of mathematical literature and the resulting quality of pre-training data, rather than ease of verification.
In pre-training progress from 2019 to 2025, data improvements provided 3.24 times greater compute efficiency gains than model improvements.
Magic announced that a new pretraining recipe reached performance similar to DeepSeek V4 Pro Base with approximately 50x fewer FLOPs, and that training scaled up by approximately 10x subsequently outperformed public base models in perplexity evaluations.
LAION has released BVD, an open-source multimodal video dataset containing 10 million hours of content.
The IDEA Prune pipeline, which integrates pre-training of expanded models with pruning, improved LLM compression performance.
Sparse data is optimally reused 15–20 times, leading to the proposal of a scaling law that reflects this.
Apple proposed LINK, a data intervention technique leveraging bilingual lexicons, to significantly improve multilingual LLM training efficiency.
AntLing has open-sourced six Ling-3.0-based models applying WSM technology.
We confirmed that the optimal number of domain data repetitions increases with model size during LLM scaling.
LittleLearner demonstrated that pre-training data determines the knowledge acquisition limits of models by training on specific educational curriculum data.
Sophisticated probing of models is used to estimate knowledge cutoffs and pre-training dates.
Mirendil will conduct pre- and post-training of AI models using Google Cloud's TPU and NVIDIA GPU.
A parameter selection technique has been proposed that reduces forgetting of existing knowledge while training only on new data.
Google officially announced that it has begun its most ambitious pre-training run to date.
It improves Nemotron model performance by generating high-quality synthetic data using existing tasks as seeds.
New research shows that with sufficient compute power, training large models on low-quality data without filtering is possible.
**Andrej Karpathy**, an OpenAI co-founding member, has joined Anthropic's **Claude pre-training** team.
OpenAI co-founder Andrej Karpathy has joined Anthropic's pretraining team.
OpenAI co-founder Andrej Karpathy is joining Anthropic's pre-training unit.
During pretraining, LMs repeatedly oscillated between pattern imitation and generalization.