AI Briefing#transformer
The latest AI and developer news about #transformer, with the original source and a short summary.
The latest AI and developer news about #transformer, with the original source and a short summary.
Stanford's CS336 course covers the entire process of LLM development, from data curation to Triton-based FlashAttention2 implementation and DPO alignment.
Stanford University has released its CS336 course, covering the entire process from designing Transformer models to training and evaluation.
Research has revealed that AI-generated CUDA kernels can cause loss functions to diverge during actual model training due to precision issues.
Provides a roadmap and model list for Native Multimodal Modeling (NMM), which goes beyond modular combination to jointly process multiple modalities.
To overcome the context-scaling limits of Transformers, a 'sleep' mechanism is proposed that compresses recent information into fast weights.
The latest open-weight LLMs have adopted KV cache compression and attention optimization techniques for long-context efficiency
Proposes CODA, a GPU kernel abstraction that uses the GEMM-Epilogue approach to solve the memory bottleneck in Transformer training.
CODA, a GPU kernel abstraction that solves the memory bottleneck in Transformer training, has been released
HRM-Text has released a method for pretraining a 1B model for about $1000.
PaddleOCR 3.5 has added Transformers backend support.
Proposes SFHformer, which integrates the FFT mechanism into Transformer to enhance efficiency across various image restoration tasks.
Orthrus claims up to a 7.8x speedup on Qwen3-8B inference while keeping the backbone frozen.
Orthrus, which uses a Diffusion module to boost token generation speed by up to 7.8x while maintaining the accuracy of existing Transformers, has been unveiled.
Amazon extended the Chinchilla scaling law to present a framework that speeds up LLMs without losing accuracy.
Hugging Face presented up to a 24% speed improvement by asynchronizing continuous batching.
The PyTorch library for Hessian/GGN analysis has been rewritten as v1.0.0a1.
The **attention drift** phenomenon in speculative decoding drafters was identified and its performance was improved.
SATFormer is proposed, improving efficiency by selectively reusing early-layer representations on a per-token and per-head basis.
Random cross-layer attention enables layer-wise KV cache sharing.
It was shown that transformers express formal languages more succinctly, and that verification is EXPSPACE-complete.
An 826,985-parameter Transformer built in C++17 achieved a validation loss of 1.6371 after training for 76.2 minutes on a single CPU core.
By viewing the residual stream as a register, a deterministic RPN calculator was implemented using a Transformer.
Meta has released the Sapiens2 vision model family, trained on 1 billion human images, on GitHub.
The Recurrent Transformer boosts both effective depth and decoding efficiency together.