How to Know When the Pandemic Is Over (and What Models Needed to Learn)

Airbnb's Predictive Data Science team operates forecasting models across thousands of markets for demand and bookings, recognizing that accumulated forecast errors directly translate to business risk. Rather than viewing model retraining as a single action, they distinguish between three independent decisions: Refitting, Respecifying, and Holding.

Refitting is the cheapest option, applying the latest data to the same model structure, but it carries risk every time because recent data may be anomalous and degrade model performance. Respecifying involves changing the model structure or features; it is the key method for improving accuracy when a model is fundamentally flawed, but it incurs significant costs for validation and deployment. Holding means refraining from intervention even when errors occur, playing a crucial role in preventing models from misinterpreting shocks as normal values after a crisis has ended.

Most organizations perform unconditional refitting according to regular schedules, which can cause models to continue learning from past shocks or outliers that are no longer relevant. Therefore, when detecting drift in forecasting models, it is necessary to have the judgment to calmly compare the three options—whether to simply update the data, change the model structure, or refrain from intervention—rather than just updating the data.