Building AI Agents for AR Glasses and XR Devices with NVIDIA XR AI

Developers of AR glasses and wearable devices have struggled with building complex infrastructure, including integrating real-time camera/microphone streams, connecting multimodal AI models, and linking enterprise data. NVIDIA XR AI addresses these challenges by providing a reusable foundation that connects XR devices to GPU-accelerated AI services based in the cloud, data centers, and edge.

Currently released as a beta version, this open source library allows developers to build intelligent XR agents that understand users' visual information, grasp voice or text intent, and call enterprise tools. It is optimized to provide real-time guidance and information to workers in hands-free environments such as field service, manufacturing, healthcare, and education.

The core architecture of NVIDIA XR AI consists of a modular combination of the following components:

• NVIDIA Cosmos: Environmental understanding through Visual Grounding
• NVIDIA Nemotron: Language understanding, reasoning, and Tool Calling
• Model Context Protocol (MCP): Connecting enterprise tools and data sources
• NVIDIA NeMo Agent Toolkit: Workflow orchestration between models and tools
• NVIDIA CloudXR: Spatial content rendering for rich 3D interaction

This architecture is designed to separate media transmission, model serving, and tool access. Video pixels are kept in shared memory while only lightweight metadata moves through the system, maximizing efficiency by reducing unnecessary model inference and data movement. It also supports multi-user and multi-agent scenarios, allowing flexible scaling across various devices and clients on a single foundation.