transformers releases

Details

Key New Features

Added HYV4 Model Support: Supports the 780B parameter Mixture-of-Experts (MoE) language model Hy4-Preview. It activates 49B parameters per token and provides a 1M token context window.
- Multi-head Latent Attention (MLA): Compresses keys and values into low-rank latent variables to improve efficiency.
- DeepSeek Sparse Attention (DSA): Uses a lightweight indexer to select the top index_topk keys per query. Only layers marked as "full" in indexer_types run the indexer, while "shared" layers reuse selections from the previous full layer.
- Gated MLA with learnable attention sinks: Each head owns a sink logit that participates in the softmax but contributes no value.
- Independent Hyper-Connections (iHC): Uses hc_mult parallel residual streams instead of a standard residual path, merging and redistributing before and after sublayers.
- Note: Multi-token prediction (MTP) layers are not executed, and their weights included in the checkpoint are ignored upon loading (retained for speculative decoding in other runtimes).
Added VibeVoice Framework Support: Supports the new speech synthesis framework VibeVoice.

Details

Added HYV4 Model Support: Supports the 780B parameter Mixture-of-Experts (MoE) language model Hy4-Preview. It activates 49B parameters per token and provides a 1M token context window.
- Multi-head Latent Attention (MLA): Compresses keys and values into low-rank latent variables to improve efficiency.
- DeepSeek Sparse Attention (DSA): Uses a lightweight indexer to select the top index_topk keys per query. Only layers marked as "full" in indexer_types run the indexer, while "shared" layers reuse selections from the previous full layer.
- Gated MLA with learnable attention sinks: Each head owns a sink logit that participates in the softmax but contributes no value.
- Independent Hyper-Connections (iHC): Uses hc_mult parallel residual streams instead of a standard residual path, merging and redistributing before and after sublayers.
- Note: Multi-token prediction (MTP) layers are not executed, and their weights included in the checkpoint are ignored upon loading (retained for speculative decoding in other runtimes).
Added VibeVoice Framework Support: Supports the new speech synthesis framework VibeVoice.