TritonSigmoid Released for GPU

Released TritonSigmoid as open source. It is a padding-aware sigmoid attention GPU kernel used in single-cell foundation models when representing each cell as a gene sequence, handling variable-length inputs of 200 to 16,000+ tokens without computing on empty padding.

While softmax forces competition among tokens, sigmoid allows strong attention to multiple genes and transcription factors simultaneously.

The main results are as follows.

• Recorded up to 515 TFLOPS on H100.
• Showed higher throughput than FlashAttention-2 (361) and FlashSigmoid (440).
• Achieved lower validation loss than softmax attention on 6 held-out datasets.
• Cell-type separation improved by 25%.
• Training remained stable even under conditions where softmax diverged.

The paper is published on arXiv, and the implementation is available on GitHub.