Experience of Participating in the Open LLM Leaderboard

Amid the rapid development of generative AI in 2023, the Hugging Face Open LLM Leaderboard has established itself as a major benchmark for verifying LLM performance. This leaderboard evaluates a model's reasoning, knowledge, truthfulness, and mathematical ability from multiple angles through 6 metrics: ARC, HellaSwag, MMLU, TruthfulQA, Winogrande, GSM8k.

Model Merge technology for creating low-cost, high-efficiency models is also drawing attention. Model merging is a method that combines existing LLMs to boost performance without separate additional training, enabling efficient modeling while reducing carbon emissions.

The major model merge algorithms are as follows.

• SLERP (Spherical Linear Interpolation): A method that smoothly interpolates in spherical space while preserving the magnitude of vectors in high-dimensional space.
• TIES-Merging: Efficiently merges models by removing parameter redundancy (Trim), resolving sign conflicts (Elect Sign), and eliminating inconsistencies.
• Passthrough: A method that directly connects layers from different LLMs to create a new model, allowing the model's parameter scale to be expanded.