AI Briefing#transformer
The latest AI and developer news about #transformer, with the original source and a short summary.
The latest AI and developer news about #transformer, with the original source and a short summary.
The open-source model uses a MOMENT-1-large backbone and DARR mode to blend direct predictions with k-nearest-neighbor retrieval from historical data.
The new FactorEngram architecture improves language modeling on 340M and 1B parameter backbones by using basis-level contextual gating for n-gram memory.
A 300M-parameter transformer trained on synthetic recurrent sequences achieves 0.9–2.4 bits per byte on real languages after reading 1 million bytes.
The single-model approach achieved +4.53% Active Users and +6.30% Total Listening Time in a 7-day A/B test on smart speakers.
While an LLM context window represents the total token capacity for a single request, the 'Lost in the Middle' phenomenon causes retrieval rates for middle information to drop with longer inputs, making selective management crucial.
New research demonstrates that combining diverse mechanisms (especially including SSMs) has a greater impact on performance than the placement order of attention mechanisms in Transformer stacks.
The RLT structure, where the recurrent path lengthens and computational load increases during token generation, along with preliminary experimental results, has been released.
StepAudio 3 Gen achieves state-of-the-art performance in TTS and multimodal audio generation using an RVQ token-based discrete generation architecture.
Anthropic researchers released a mathematical framework proving that Induction Heads are the core mechanism for In-context Learning in 2-layer Attention-only Transformers.
Apple has released SimpleDesign, a protein design model that trains directly in the data space without multi-stage training.
Beyond the limitations of TF-IDF, dense vector and transformer-based intent recognition technologies enhance the accuracy of LLM agents.
Extropic released the Z1T design combining Z1 and FPGA, estimating energy efficiency and decoding throughput under conditions excluding final logits operations.
NeoMME, a multilingual multimodal encoder that processes text and images with a single Transformer without a vision tower, has been released.
A small Transformer trained from scratch in 1.5 hours on a 5090 GPU achieved 44% on ARC-AGI, criticizing the low sample efficiency of LLMs.
A small transformer trained for 1.5 hours achieved 44% performance on the ARC-AGI-1 benchmark at a cost of 67 cents.
Google has released TimesFM-3, a foundation model for multivariate time series forecasting based on 330 billion parameters.
Apple released STARFlow2, which combines LLMs and normalizing flows to generate text and images in a unified manner.
This analysis identifies the threshold at which communication costs limit computational performance during data and tensor parallelization of transformer models.
Microsoft has proposed the Full-bandwidth Transformer, which expands vertical feedback channels within models to improve training efficiency and performance.
Removing the FFN nearly eliminated the performance gap with standard Transformers when the budget was reallocated.
Malachyte solved the cold-start problem in retail recommendations using LLM attention-based models and Google Cloud infrastructure.
Analyzed the independent development of LLMs using public configurations, tokenizers, and weights.
Lophius has been released as a tool to streamline language model research tasks based on Jupyter and Transformers.
Meta developed a sequence learning platform that separates offline user modeling from online ranking.