Upgrading Only the Experts

BAR (Branch-Adapt-Route) independently trains domain-specific experts, merges them into a single model, and allows specific experts to be swapped or upgraded without retraining the rest.

While last year's FlexOlmo dealt with modular combination at the pretraining stage, BAR extends this to post-training. The goal is to reduce the performance loss and retraining cost that commonly arise when separately improving areas like math, tool use, and code.

• Train experts separately for each skill
• Merge them into a single model
• The model routes to the appropriate expert based on the input
• Updating just a specific expert doesn't require retraining the whole model

According to AllenAI's explanation, at 7B scale BAR outperformed existing alternatives for updating models after pretraining, was superior to the approach of training a separate dense model and attaching it later, and came quite close to the results of full retraining from scratch.