Toss Reveals AI Platform Architecture to Streamline Service Development

The Toss AI Platform team built a platform centered around the GenAI Portal and GenAI Gateway to reduce repetitive infrastructure tasks such as model integration, permission verification, and usage aggregation, allowing service teams to focus on core development. This architecture integrates the entire process from model usage application to actual invocation, security checks, and cost management, aiming to improve operational efficiency.

Model Usage Application and Governance Management

The GenAI Portal is a web service that manages model usage applications and configurations, controlling permissions and budgets at the project level. It defines usage conditions by inputting data grades (L1~L4) and service impact levels (SLA grades), issuing API keys after internal policy reviews, such as checking the scope of permitted innovative financial services. This enables consistent management of the same models and budgets across multiple services.

Invocation Processing and Security Guardrails

The GenAI Gateway handles actual model invocations, performing API key authentication, usage limits, and input guardrail checks. It detects and blocks personal information and prompt attacks before external model calls, and masks sensitive information when logging to balance security and issue tracking. Internal tool calls are verified separately via the MCP Gateway, clearly separating permissions for model invocation and tool execution.

Cost and Capacity Optimization Strategies

Budget management applies both cumulative cost and tokens per minute (TPM) limits to prevent cost overruns and service interruptions. A routing strategy prioritizes pre-provisioned throughput (PTU) and automatically switches to pay-as-you-go (PAYG) when capacity is insufficient, ensuring both stability and cost efficiency. Additionally, logic for reserving and adjusting token counts for incomplete requests, such as streaming responses, maintains accurate real-time usage tracking.