LoRA Speedrun – A Public Wall-Clock Leaderboard for Fine-Tuning Techniques

The LoRA Speedrun project has been released to objectively compare the performance of parameter-efficient fine-tuning (PEFT) techniques such as LoRA/QLoRA. It was designed to overcome the limitation of existing papers reporting performance using different models, data, and hardware.

Core Rules and Environment:

• Fixed Environment: Variables are controlled using the Qwen2.5-1.5B model, the GSM8K dataset, and an NVIDIA L40S GPU (Modal sandbox).
• Measurement Metric: Based on the wall-clock time taken to achieve a specific accuracy (GSM8K 57% or higher).
• Verification System: Only data verified by re-running each record 3 times on identical hardware is accepted.
• Freedom of Choice: Participants are free to choose LoRA Rank, quantization method, learning rate schedule, sequence packing, and whether to use custom kernels.

Like the case of nanoGPT, this project aims to uncover practical optimization techniques through fair competition among fine-tuning methods.