HRM-Text (GitHub repository)

HRM-Text is a 1B text generation model based on the HRM architecture, enhancing task completion and latent space reasoning to improve pretraining efficiency. The repository provides a full pretraining framework including PrefixLM sequence packing, FlashAttention 3, PyTorch FSDP2 training, evaluation, and checkpoint conversion. The architecture supports HRM, Transformer, TRM, RINS, and UT baselines.

By the published benchmarks, the goal is to reduce foundation model pretraining cost to around $1000. It claims 130-600x less compute and 150-900x less data compared to existing approaches, with reference runs of L (0.6B) at 8 H100 / 50 hours / about $800, and XL (1B) at 16 H100 / 46 hours / about $1,472.

On benchmarks, L achieved GSM8k 77.6%, MATH 51.2%, and MMLU 56.6%, while XL reached GSM8k 84.7%, MATH 56.5%, MMLU 60.7%, ARC-C 81.9%, and BoolQ 86.2%. A Hopper-class GPU is recommended, as the attention path depends on FlashAttention 3.

• data_io cleans and tokenizes the corpus, then performs stratified sampling. Since default training is based on 4 epochs, sampling is also matched to epochs=4.
• Single-node setups place samples in /dev/shm/sampled, while multi-node setups place them on shared storage, with each node regenerating the same data.
• Both Docker image and source installation are supported; for multi-node, NCCL communication should be verified first, then experiment tracking is attached via wandb login.
• Training is run with torchrun, saving to checkpoints/ every epoch. evaluation.main automatically loads the latest checkpoint, and conversion.convert_to_hf exports to Transformers format; evaluation and export use EMA weights by default.

The repository currently implements training, checkpointing, and evaluation, and also provides Transformers format export. Native Transformers support has been merged and will be included in the next release, while native vLLM support is in progress.