LangSmith LLM Gateway: Runtime Governance Integrated into the Agent Lifecycle

LangChain has launched LangSmith LLM Gateway, a runtime governance layer within LangSmith, where agent development, observability, and evaluation take place. This gateway operates between agents and LLM providers, enforcing cost limits and redacting sensitive data before requests reach the model.

Previously, observability and enforcement were separate, creating the hassle of having to cross-reference signals across multiple tools whenever an issue occurred. However, LLM Gateway unifies policy enforcement and tracing within LangSmith, providing an environment where blocked requests can be immediately traced to the relevant trace and fixed.

The key features currently available in the beta version are as follows:

• Spend limits: Set cost caps at the organization, workspace, user, or API key level to prevent unexpected overcharges.
• Spend visibility: Provides real-time cost totals by workspace and user.
• PII and secrets detection: Automatically redacts sensitive data from requests and responses before they are passed to the model or logged in traces.
• Trace continuity: All calls that pass through the gateway remain identical to traces in existing LangSmith workspaces, preserving observability.
• Audit logging: Logs all administrative actions to meet the needs of compliance and security teams.

Setup is extremely simple. Just change the base_url in your existing code to the LangSmith Gateway endpoint and set your API key, and you can immediately apply powerful governance features while keeping your existing code intact.