How to Safely Manage Large-Scale Dynamic Configuration Changes

Dynamic configuration lets you change runtime behavior without redeploying services, but a bad change can immediately cause an outage. To address this, Airbnb built an internal platform called Sitar, unifying the entire flow of defining, reviewing, testing, and deploying changes into a single system.

There are four key capabilities a platform must have.

• Configs as code: Config is version-controlled in Git just like service code, with review, approval, and audit trails applied.
• Safe rollout: Every change must pass schema validation and automated checks, after which it is gradually expanded starting from a limited scope.
• Isolated testing: Validating changes in local or canary environments beforehand to reduce production risk.
• Multi-tenant flexibility: Deployment triggers, guardrails, and rollout strategies can be configured differently per tenant.

Sitar's architecture is divided into a developer-facing layer, control plane, data plane, and an agent sidecar and client library that run alongside the service. Developers manage config through a default GitHub-centric workflow, while exceptional situations or emergency response are handled through sitar-portal. The control plane handles validation, ownership, access control, and rollout policy, while the data plane is responsible for storing config and deploying it at scale.

Rollouts are not pushed everywhere all at once. After being merged into the main branch, a change is first applied to a limited scope, then gradually expanded while monitoring status, and can be rolled back immediately if anomalies are detected. This approach reduces the blast radius of a bad config and speeds up incident response.

On the service side, the local cache is important. The agent sidecar periodically fetches config from the backend and stores it locally, and the client library reads from that cache. So even if the backend becomes temporarily slow or goes down, the service can continue operating with the last known-good config.

Thanks to this structure, product teams can make changes safely in smaller units and choose rollout methods—automatic, manual, or cron—suited to their team's nature. When an incident occurs, observability tools can be used to trace which config affected whom and when, and then quickly fix it through the portal's emergency flow.