Background Coding Agents: Accelerating Downstream Dataset Migration (Honk, Part 4)

Spotify needed to deprecate two core user datasets and move them to new versions, requiring it to handle about 1,800 direct downstream pipelines and thousands of indirect dependencies within 6 months. They estimated this would take 10 engineering weeks if done manually, and to reduce that, they mobilized Backstage, Fleetshift, and Honk.

First, they used Backstage's endpoint lineage and Codesearch to quickly find affected repositories and finalize the migration scope. Then, they ran the large-scale shift with Fleetshift, allowing them to check progress on a single screen and jump directly into auto-generated PRs.

The most difficult part was context engineering. Spotify's data pipelines were split across three frameworks—BigQuery Runner, dbt, and Scio—and among these, Scio had large variance across teams, making it hard to handle with batch prompts. At the time, Honk had no Claude skills or custom capability to directly read external documentation, so they had to write sufficiently comprehensive context files in advance, and Scio was ultimately excluded from this round of work.

For dbt and BigQuery Runner, simply reusing the human-oriented migration guide often led to incorrect field mapping inference. Once they switched to explicitly specifying mappings in table form, and for fields requiring judgment, leaving them unchanged while attaching comments and guide links above them instead, performance improved significantly.

However, these two frameworks had almost no unit tests at build time, so Honk couldn't run a sufficient feedback loop to verify results and self-correct. As a result, verification before final merge had to rely on manual testing by the downstream owning team.

Nevertheless, they ultimately successfully deployed 240 automated migration PRs. From this experience, Spotify concluded that to effectively use large-scale batch-style agents, standardization of the data landscape and a testing/verification system across repositories are essential. Going forward, a feature for Honk to first read JIRA or documentation and gather context on its own is also on the roadmap.