AI Briefing#sequence-modeling
The latest AI and developer news about #sequence-modeling, with the original source and a short summary.
The latest AI and developer news about #sequence-modeling, with the original source and a short summary.
The study demonstrates that gating mechanisms in SSMs cause models to learn in-weights memorization solutions, delaying or preventing convergence to correct in-context learning even without architectural memory limits.
The Causal Grassmann layer achieves validation perplexities within 10-15% of size-matched Transformers on Wikitext-2 and slightly outperforms DistilBERT on SNLI.
The open-source library supports non-commutative geometric spaces like hyperbolic and SO(n) rotations to preserve event order in sequence modeling.
OpenEuroLLM has released Complex KDA, which implements 2D rotations by extending the gate and coefficient ranges of Kimi Delta Attention.
Airbnb has built a Transformer-based sequence model that learns guests' long-term behavioral patterns to recommend optimal accommodations.
Decision Transformer converts reinforcement learning problems into sequence modeling problems, achieving stable policy learning through the Transformer architecture.
It was shown that transformers express formal languages more succinctly, and that verification is EXPSPACE-complete.
Apple's ParaRNN parallelizes training of nonlinear RNNs for LLMs, achieving a 665× speedup.
Hugging Face has integrated Decision Transformer, an offline reinforcement learning model, into the transformers library.