Piqama: Pinterest's Quota Management Ecosystem

Piqama is an ecosystem Pinterest built to handle diverse resources like memory, CPU, QPS, and network bandwidth as a common quota management platform. Through UI portal, REST, and Thrift, it connects quota schema management, validation, permission checks, deployment, and enforcement into a single lifecycle.

The core idea is to consolidate the way each service used to handle quota separately into a common control plane, while still allowing service-specific custom logic where needed. Quota definitions can have a hierarchical structure, validation is extended through a pluggable framework, and updates go through owner-based authorization.

After operation, usage feedback, governance, and optimization follow. The Piqama client collects enforcement and usage statistics, and systems that don't use the client leave data through a separate feedback loop. This statistics data is stored in Apache Iceberg on Amazon S3 and goes through pre-aggregation to improve space efficiency.

This data becomes input for automatic right-sizing. A separate service reads Presto, Iceberg, and user-defined data sources to perform quota adjustments that reflect organic growth, traffic bursts, and underutilization, with strategies for capacity-based quota currently applied.

There are two real-world application cases.

• Big Data Processing Platform: In the combination of Moka and Yunikorn, Piqama manages physical resource quotas such as memory, vcore, and max concurrent applications.
• Online storage rate limiting: Rules are deployed via PinConf, rate limiting decisions are made on the local data path, and Piqama handles quota lifecycle and right-sizing as an async control plane.

On the Big Data side, the Yunikorn Config Updater periodically reads Piqama's latest quota values to update configuration, and resource usage after job completion flows into S3 and the resource database. When usage exceeds budget, the maximum resource is lowered to control the burning speed of over-budget projects.

On the online services side, Piqama addresses the limitations of the existing rate limit framework — non-declarative rules, manual modification, and fixed thresholds. Pinterest is extending this into the Service-Protection Framework (SPF) to also handle throttling and concurrency control, and integration has already been completed for several services including TiDB and Key-Value Store.

The future direction is to more strongly connect quota, entitlement, and budget, and to provide more sophisticated auto-right-sizing. The target is set to expand to high-load platforms such as PinCompute, ML Training Platform, and LLM Serving Services.