Start Experiments with One Click! Building a Recommendation Experiment System That Shortens Iteration Cycles

To speed up iteration on its recommendation algorithm, Azar first introduced a low-code experiment system that let experiments be turned on without code changes or deployments. It automatically assigned experiment groups and applied new algorithms using only JSON configuration, but as the organization grew and experiments increased, the configuration files became excessively large and the structure grew complex.

To solve this problem, the system was redesigned as PolicyEngine, separating Policy and Experiment. Default settings that don't change often were placed in Policy, while settings that apply only to experiment targets were placed in Experiment, with Experiment overriding Policy — reducing complex segment branching and configuration conflicts.

For expressing groups, a Python-based DSL was adopted instead of JSON. It uses familiar syntax like and, or, not, and in, while actually operating as a restricted subset language, securing both readability and expressiveness. Parsing and execution were handled with ast and simpleeval.

The effects of the DSL were immediate.

• Common patterns in country-specific detailed settings were bundled into functions, reducing 7,000 lines to 50 lines, about a 99% reduction
• A common function was created to render dozens of configuration combinations, shortening configuration change and review time from 4 days to half a day, an 87% reduction

To prevent conflicts between experiments, static-analysis-based MECE verification was added. After parsing the DSL with ast, it is converted into Z3 expressions to automatically check for overlaps and omissions, blocking conflicts in advance at the experiment configuration PR stage.

To improve operational convenience, a real-time monitoring dashboard was also built. With Lucene query, specific countries or groups can be filtered and checked, reducing the metric dilution problem in small-scale experiments and enabling quick verification of whether the settings were applied correctly.

As a result, the average time spent writing and reviewing experiment configurations dropped from 2 days to 30 minutes, about a 97% reduction. Most importantly, experiment owners can now write configurations directly without help from software engineers, and currently most recommendation experiments proceed through configuration changes alone.