JD Ads Cuts AI Inference Latency 15% by Rethinking the CPU Side

JD’s advertising business says it has cut end-to-end inference latency by 15% by rebuilding the CPU side of its AI serving stack, rather than buying more GPUs. The work was done with Huawei’s Kunpeng server team and tested on 5% of live traffic during the 618 mid-year shopping festival, one of China’s largest e-commerce events.

The problem was arithmetic. Generative recommendation models pushed compute demand up roughly a thousandfold, while ad auctions still have to answer within about 300 milliseconds. JD concluded that the bottleneck lay in general-purpose computing, not graphics chips. Using Huawei’s Lingqu interconnect, the two sides compressed four levels of cache into two and built a 300TB shared memory pool, then rewrote the remote procedure calls that stitch the system together. The company says KV cache and RPC performance improved substantially.

The collaboration is now spreading from advertising into search and recommendation, with both sides working on a SuperCache layer and sandboxed environments for AI agents. For Chinese platforms, squeezing efficiency out of commodity servers is becoming as important as securing scarce accelerators.

Latency is money in advertising: a slower auction means fewer bids and cheaper inventory. If the gains hold at full traffic, the approach offers a template for other platforms facing the same thousandfold demand curve without a matching GPU budget.