OpenArch Rebuilds Popular LLM Architectures in Python for Learning
A new open-source project called OpenArch is bringing clarity to the often confusing world of large language model architectures. The GitHub repository contains hand-written PyTorch implementations of the models cataloged in Sebastian Raschka's LLM Architecture Gallery, with each architecture contained in a single readable Python file. The creator builds each model from scratch using the original papers, technical reports, and reference configuration files.
The repository already includes working implementations of many major models, including GPT-2 XL, Llama 2 and 3, OLMo 2, DeepSeek R1, Gemma 3, Mistral 3, Llama 4 Maverick, Qwen 3, Kimi K2, GLM 4.5, and GPT-OSS, with Grok-2.5 under construction. Rather than competing with production libraries like Hugging Face's transformers, the project optimizes for readability and learning, making the structural differences between models easy to compare side by side.
The project highlights how modern LLMs share a common skeleton but differ in dozens of small but important design choices. These include attention variants such as multihead, grouped query, multihead latent, and sliding-window attention, along with differences in normalization strategies, positional encodings like RoPE, dense versus sparse mixture-of-experts routing, and training tricks like multi-token prediction. Production codebases often obscure these choices behind optimizations for speed and sharding, so OpenArch makes them explicit for anyone studying how these models actually work.