AI Briefing#scaling-laws
The latest AI and developer news about #scaling-laws, with the original source and a short summary.
The latest AI and developer news about #scaling-laws, with the original source and a short summary.
Baron et al. propose a scaling law that recommends ~7,000x more parameters than Chinchilla at 10^29 FLOPs, using hyperparameter transfer and ensembling to maintain loss reduction on fixed datasets.
The largest 1B-parameter model achieves 44.71% average zero-shot QA accuracy, surpassing the 39.79% of the corresponding GPT baseline.
The hierarchical network architecture demonstrates 1.3x compute efficiency on validation loss and up to 5.37x gains on out-of-distribution code compared to standard BPE baselines.
The study presents the first tri-modal Masked Diffusion Models pretrained on text, image-text, and audio-text data, achieving strong results at 3B scale.
Sparse data is optimally reused 15–20 times, leading to the proposal of a scaling law that reflects this.
Reexamines the premise that algorithmic progress, rather than data volume, is the primary driver of AI model performance improvements.
Anthropic CEO Dario Amodei acknowledged the AI trust crisis and shared his views on regulation and power concentration.
Dario Amodei emphasized that AI regulation should serve as a means of risk management and competition promotion through fair institutions, rather than concentrating power.
Dyna-2 demonstrated scaling laws for both humans and robots using 1 million hours of human video.
Xiaomi validated VLA scaling using 100,000 hours of manipulation data collected without robots.
Meta developed a sequence learning platform that separates offline user modeling from online ranking.
Moonshot AI has released the weights and technical documentation for Kimi K3, an MoE model with 2.78 trillion parameters.
Xiaomi has unveiled a robot foundation model trained on 100,000 hours of embodiment-free data.
Xiaomi has unveiled Xiaomi-Robotics-1, a robot policy model built on 100,000 hours of embodiment-agnostic pretraining.
The paper proposes a new methodology that separates memorization and generalization in language models to measure model capacity.
A Continuous Diffusion-based speech model was shown to follow scaling laws while demonstrating potential for efficient inference.
This analyzes scaling laws, the predictable relationship between model size, data volume, and compute on training loss.
A study has found that 'Rosetta Neurons,' which are commonly found across different neural networks, become more specialized as model scale increases.
A strategic response is needed to close the gap between AI's rapid pace of advancement and the slow pace of policy legislation.
Cost pressure is expected to push the majority of AI workloads from high-performance models toward efficient small models.
Text optimization is an efficient learning mechanism that complements weight updates and a new scaling axis.
Anthropic emphasized that multifaceted AI safety research is urgently needed to prepare for rapid AI advancement and its accompanying risks.
A paper titled "Unified Neural Scaling Laws" explaining the performance scaling of neural networks has been published.
An interview with Greg Brockman covering OpenAI's technical turning points and the structural changes needed to achieve AGI.