Network Efficiency for ML Workloads (Part 1): Feature Trimmer

Pinterest's online ML serving operates on a root-leaf architecture. The root fetches features from the feature store and preprocesses them, while the leaf performs inference on GPU. The problem was that the root sent too many features to the leaf, causing the network to become a bottleneck before anything else—the leaf's peak network usage exceeded GPU SM activity. The root had to use the network-optimized m6in instance just to meet SLA.

First, they applied lz4 compression to fbthrift RPC, reducing root-leaf network usage by 20%. In exchange, CPU increased by 5% and p90 latency rose by 5ms (about 10%). Compression was effective, but the underlying structure of continuously sending unused data remained unchanged.

The solution was Feature Trimmer. The root treats each leaf model's model signature as the source of truth, allowlisting only the necessary features and removing the rest. The signature lives in module_info.json inside the TorchScript .pt archive, and when the signature changes, a new model is forked to safely maintain versioned lookup and fallback.

Synchronization is handled within the deployment pipeline.

• During the training stage, module_info.json is exported as a standalone artifact.
• During the bundle build stage, per-model signatures are collected to create a bundle-level mapping.
• Root configs are deployed in the order Canary → ACA → Production.
• If a signature is missing, the build doesn't fail—it warns and skips instead.

This approach maintains trimming criteria even when there are multiple model bundles with independent rollout/rollback cycles. Allowlists carry less management overhead than blocklists, and can safely keep pace even when root and leaf have different deployment cadences.