marin: 500 Billion Parameter MoE Model, Open Source from Data to Inference

Marin is an open-source software platform for foundation model research and development. It integrates the entire LLM building process into a single framework, covering data curation, tokenization, pre-training, post-training, and evaluation. Its core value lies in disclosing all process knowledge, including failed experiments.

Currently, the focus is on pre-training a Mixture-of-Experts (MoE) model with over 500 billion parameters and 5e24 FLOPs, aiming to solve important challenges for scientists and researchers. Additionally, through an open scaling suite called Delphi, it scales LLM recipes from 3e18 to 1e23 FLOPs and provides scaling laws to predict the performance of larger models using smaller ones.

Marin supports experiment definitions similar to Makefiles. Each step, such as data tokenization and model training, has dependency relationships and is executed in topological order. Example code for training a small model with the TinyStories dataset is provided, which can be modified to scale to large GPU or TPU pods.

Previously, an 8B parameter model surpassing Llama 3.1 8B and the Marin 32B model were trained. It is also applicable to audio-text, DNA, and protein model research. Stanford CRFM and Open Athena lead the core development, and it is released under the Apache-2.0 license.