Commerce Claim Index CDC Migration Reduces Absolute Latency from 14.3s to 0.8s

Changed the collection method for the commerce claim search index from notification-based to CDC, resolving issues with missing data and timestamp contamination.
The median absolute latency decreased by approximately 18 times, from 14.3 seconds to 0.8 seconds, and zero field mismatches were confirmed in a full reconciliation of approximately 42,000 records over 7 days.
During the full reconciliation process, 166 defects were discovered where the existing index generated inconsistent values for identical inputs.
To reduce the load on the source DB for the entire index batch, the batch size was adjusted to 5,000 records and duplicate queries were removed, reducing the number of queries by approximately 1,100 times.

Verification Method and Results

The migration decision was made by reconciling across three axes: missing data, values, and latency.
The difference in record counts by year is 0, excluding the year spanning the start of the retention period, and there are 0 documents present only in the new index.
A full reconciliation of new changes over 7 consecutive days showed 0 field mismatches, while past periods were verified via per-path sampling.

Removed duplicates in personal information lookups within the entire index batch, reducing the lookup target from 16.43 million to 366,000 records.
Changed the batch size from 200 to 5,000 records and set the rate limit based on the total volume of the entire batch.
After the measures, CPU usage decreased by 45%, and the required time was shortened from 12.9 minutes to 11.6 minutes.

Agents are delegated only for tasks where verification is possible, while humans perform judgment and assume responsibility.
Agents execute reconciliation runs and aggregate summaries, while humans are responsible for result judgment and abandonment decisions.