How Karrot Introduced LLM Release Notes for a Second-hand Trading Server, Readable by Anyone

By first accumulating deployment records as an asset and then layering an LLM translation layer on top, deployment history that only developers used to read was turned into release notes that anyone can check. Without setting up any separate new infrastructure, the pipeline was built using GitHub Actions, Notion, a Rails rake task, and Prompt Studio.

• In Stage 1, notion:record_deploy runs on deployment, gathering the PR list and PR descriptions via git log and the gh CLI, saving them to Notion, and linking the deployment DB and PR DB via a relation.

• Instead of inserting the change diff directly, the PR description already refined by CodeRabbit was used as-is. Since the same SHA can be deployed to multiple regions, upserts were done using a region-HEAD_SHA combination, and if there was no previous deployment SHA, it fell back to the most recent 50 commits.

• In Stage 2, release_notes:daily runs every day at UTC 01:00 (KST 10:00), reading the body text from the Notion PR DB, summarizing it via Prompt Studio, and saving it to the LLM release notes DB.

• A dry_run option was added so output could be checked without side effects, and the KST/UTC boundary was aligned based on Time.zone.yesterday to reduce date mismatches.

Prompt quality was tuned using classification rules and a terminology conversion table. Change types were divided into five labels with priorities assigned, and rules were added—such as converting notification-batch to "notification dispatch batch job"—along with region notation rules, so that even non-developers could read them. The data accumulated this way is reused not only for release notes but also for deployment frequency, per-region filtering, and contributor tracking.