AI Briefing#pretraining
The latest AI and developer news about #pretraining, with the original source and a short summary.
The latest AI and developer news about #pretraining, with the original source and a short summary.
HRM-Text has released a method for pretraining a 1B model for about $1000.
In a 6.9B LLM experiment, AI discourse increased misalignment while aligned discourse reduced it.
It identified broken causality and numerical bias as the core causes of pretraining failures.
More than 50 research and engineering staff have left SpaceXAI since the merger.
EMO has been released, maintaining performance even with only some experts used, thanks to document-level routing.
A 33-year, 103.1B-token Usenet corpus and its cleaning methodology have been released.
Talkie-1930 has been released as a 13B open-weight LLM trained only on text from before 1931.
Talkie, a 13B LLM trained solely on text from before 1931, has been released.
Talkie, a 13B model trained exclusively on pre-1931 text, has been introduced.
A 13B vintage model called Talkie, trained on 260B tokens of pre-1931 English, has been released.
Released Talkie, a 13B vintage LM trained on documents from before 1930.
Trained a 235M-parameter LLM from scratch using a single RTX 5080.
mii-llm released the 0.4B multilingual models Zagreus/Nesso, trained on 64×A100.
Frontier models are harder to prevent **distillation** of than expected, and large-scale pretraining can easily collapse from small **bias** and **causality** violations.
LaCy improves factuality by distinguishing what SLMs should learn from what they should delegate.
When the token-to-parameter ratio is fixed, downstream accuracy can also be predicted by a power law.
Kakao's Kanana Essence, trained on 3T tokens, achieved higher benchmark averages than llama-3.1-8b, while Kanana Nano maximized efficiency through pruning and distillation techniques.
Chinese CLIP, a multimodal model optimized for Chinese culture and domains, has been released.
Unveiled **OFA**, a multimodal pretrained model that unifies tasks, modalities, and architecture into one.
Introducing TAPEX, which efficiently trains table tasks using synthetic data derived from SQL execution results.
This introduces a Warm-starting technique that leverages pretrained BERT or GPT2 to reduce the training cost of Encoder-Decoder models.