OpenArch Released: A Collection of PyTorch Implementations for Modern LLM Architectures

'OpenArch', a collection of hand-crafted PyTorch implementations of modern open-source LLM architectures based on Sebastian Raschka's LLM Architecture Gallery, has been released. To address the readability issues caused by speed and sharding optimizations in production libraries, this project provides 'read-optimized single-file implementations that clearly reveal structural choices.

Key Implemented Architectures

The forward passes for major models such as Llama 2/3/4, DeepSeek R1, Qwen 3, Gemma 3, Kimi K2, GLM 4.5, and GPT-OSS are currently complete. Each implementation is structured to allow explicit comparison of core components, including Attention (MHA, GQA, MLA, etc.), Normalization (RMSNorm, QK-Norm, etc.), Positional Encoding (RoPE, YaRN, etc.), and MoE routing.

Project Goals and Features

• Learning and Educational Purpose: Designed as a reference material to aid in understanding architectures, rather than competing with production libraries like transformers
• Expansion Plan: Architectures included in Sebastian Raschka's LLM Architecture Gallery will be implemented sequentially; currently, Grok-2.5 and some multimodal models are under development
• License: The project itself follows the Apache License 2.0, while individual model implementations must comply with the licenses of the original models