NVIDIA Reveals Formal Method Using Z3 SMT Solver to Verify AI Agent Privilege Escalation

NVIDIA's OpenShell project has released a formal method that uses the Z3 SMT solver to deterministically verify whether privilege changes in AI agents exceed approved scopes. This technology aims to block risks of AI agents bypassing sandboxes or escalating privileges in unexpected ways.

How Formal Verification Works

Existing AI-based reviews are probabilistic, potentially missing details and incurring double computational costs. In contrast, formal methods provide deterministic results in milliseconds. OpenShell models network ports (Int), hosts/paths (String), and Glob patterns (Regular Expression) using SMT logic.

The verification logic checks whether the set difference between "the set of actions allowed by the proposed policy" and "the set of actions allowed by the pre-defined safe policy" is empty.

• Unsat (Unsatisfiable): No violations, within safe scope
• Sat (Satisfiable): Violations exist, excessive privileges found
• Unknown: Modeling limitations, handled with fail-closed behavior

Real-World Application and Security Checks

OpenShell uses Rust to encode agent proposals into a format understandable by Z3. For example, attempts to bypass Layer 7 (HTTP/REST) proxies to use credentials via Layer 4 (wire protocols) are detected by Z3 as capability expansion. The system automatically performs the following four core security checks.

• Link-local reach: Blocks access to internal addresses such as 169.254.0.0/16
• L7 bypass credentialed: Blocks credential access via protocols unchecked by proxies
• Credential reach expansion: Blocks the addition of new credential access paths
• Capability expansion: Blocks the addition of new HTTP methods to existing access paths

This method is a technology already validated in critical systems such as aviation control and internet routing, and is expected to contribute to enhancing the reliability of AI agent governance.