Linum AI Releases P-JiT, a 2B Pixel-Space Text-to-Image Model Trained for $21,038

Linum AI, a two-person research lab, released P-JiT (Pyramid-JiT), a 2B parameter pixel-space text-to-image model trained from scratch for a total cost of $21,038.
The model eliminates the Variational Autoencoder (VAE), allowing the diffusion model to learn a purpose-built latent space for more aggressive compression than VAEs, which have an empirical compression ceiling.
P-JiT achieves a 32x32 token reduction, representing a 512² image with the same tokens as a previous 256² image, resulting in a 4x reduction in attention context-window compared to Linum v2.
Training required 64 H100 GPUs for 8,415 hours, with pretraining costing $20,348 and post-training costing $690 using the FDFO variant.

Architecture & Mechanism

Uses a DiT structure with 22 identical blocks merging text and image streams; hidden size is 2,944 (23 heads x 128).
Text is processed by a frozen Qwen3.5-4B encoder (layers 7, 15, 27) and 2 single-modality refiner blocks.
Image patches are compressed to 1024-dim (3:1 compression rate from 3,072-dim RGB) and processed by 2 refiner blocks.
Total parameters are 2.18B at sampling and ~2.42B during training, including training-only heads like PixelREPA and linear readout heads.

Trained in two stages: 512² (610.27M samples) and 1024² (49.97M samples), totaling ~660M samples.
Uses Muon optimizer for 2D block matrices and AdamW for the rest, with a peak learning rate of 2e-4.
Converges in 11x fewer samples than Linum v2.
Hardware setup includes 64 H100 GPUs (8 nodes) using bf16 and FSDP2.

The model is not trained on rendered text, so it cannot generate infographics, PDFs, or posters with text.
The 2B parameter size struggles with full body poses, many entities (e.g., 6 cars), and multi-person interactions.
Developed as a proof-of-concept for future generative video models aimed at reducing inference costs and improving control for animation.