AI Briefing#language-modeling
The latest AI and developer news about #language-modeling, with the original source and a short summary.
The latest AI and developer news about #language-modeling, with the original source and a short summary.
The method learns query-dependent weights to compose cached state encodings, improving performance on document QA, needle-in-a-haystack retrieval, and tool use.
The Causal Grassmann layer achieves validation perplexities within 10-15% of size-matched Transformers on Wikitext-2 and slightly outperforms DistilBERT on SNLI.
Experiments with models from 135M to 1B parameters show LIFT outperforms standard Transformers on language modeling and reasoning tasks under token-matched budgets.
The survey covers algorithms, multilinguality, theory, and alternatives like latent or visual tokenization.
Removing the FFN nearly eliminated the performance gap with standard Transformers when the budget was reallocated.
Diffusion language models enable parallel generation but showed limitations in long contexts and batch processing.
Apple researchers implemented 4-step text generation using a 1.7B CFM model.
Gigatoken is an open-source tokenizer that delivers speeds up to 1353x faster than the HuggingFace tokenizer.
Stanford University has released its CS336 course, covering the entire process from designing Transformer models to training and evaluation.
Parcae scales up recurrence to reach 1.3B Transformer-level performance with 770M parameters.
This guide walks through the end-to-end process of training a RoBERTa model from scratch using TensorFlow and TPU.