Optimizing Recommendation System with JDK Vector API

Netflix's Ranker is a large-scale service that powers personalization rows on the home screen, and among its tasks, video serendipity scoring accounted for about 7.5% of CPU per node. The core question was "how different is this new title from what's been watched so far," and the existing approach calculated cosine similarity between candidates and viewing history using an M×N nested loop, which did not scale well.

The first step was introducing batching, which turned candidate and history embeddings into a matrix multiplication. About 98% of requests were single, but the remaining 2% large batches accounted for roughly half of total throughput, making batching optimization well worth it.

However, the first implementation actually made performance about 5% worse. There were two reasons.

• Creating a new double[][] on every request increased GC pressure, and the non-contiguous memory layout of 2D arrays also hurt cache efficiency.
• The Java-based matrix multiplication was a simple scalar implementation that failed to leverage SIMD.

In the next step, the data was converted into a flat double[] buffer, and ThreadLocal<BufferHolder> was used to reuse buffers for candidates and history. Buffers were allowed to grow as needed but never shrink, reducing per-request allocations, and the contiguous row-major memory made access patterns predictable.

For the matrix multiplication engine, BLAS was considered first, but it didn't deliver the expected gains on the production path. The netlib-java F2J fallback path, JNI transition overhead, the batching mismatch with column-major layout, and additional copies and allocations tangled with embedding computation all held it back.

The final solution was the JDK Vector API. This feature allows expressing SIMD in pure Java, selecting a lane width suited to the host CPU via DoubleVector.SPECIES_PREFERRED without any native dependencies or JNI, and accumulating dot products with fma(). When the Vector API isn't available, it falls back to a scalar path, but even that path branches through MatMulFactory to use a loop-unrolled optimized implementation.

In the end, this work wasn't just an algorithmic change—it was an optimization that brought together batching + flat buffers + ThreadLocal reuse + a SIMD kernel. It maintained the same serendipity score while lowering per-request CPU cost, ultimately reducing the cluster footprint.