Google launches Guided Vision in Gemini Live for real-time visual assistance

Google has launched Guided Vision in Gemini Live, a feature designed to provide real-time, conversational visual assistance for people who are blind or have low vision. Developed in close collaboration with the accessibility community, the tool allows users to share their camera during a Gemini Live session to receive dynamic audio descriptions and natural verbal reframing cues.

How it works

When Guided Vision is active, Gemini acts as a conversational partner that interprets the camera feed. If the camera is positioned incorrectly, the AI provides specific verbal prompts—such as asking the user to pan slowly, tilt downward, or step back—to ensure it captures clear visual context. This enables users to ask detailed follow-up questions about their surroundings.

Key capabilities

Guided Vision supports various everyday tasks where quick visual verification is needed:

• Reading text: Deciphering fine print, nutrition labels, appliance dials, and menus in dim lighting.
• Object localization: Finding dropped items or locating specific objects in cluttered spaces.
• Visual details: Identifying colors, patterns, and shapes to assist with tasks like matching outfits.
• Environment exploration: Providing descriptive overviews of unfamiliar rooms or spaces.

Development and availability

To train the model, Google partnered with Aira, utilizing tens of thousands of hours of visually interpreted data. Over 1,000 members of Aira’s Trusted Tester network helped stress-test the system, while Aira specialists contributed to establishing safety guardrails. The feature supports multiple languages, informed by testing with blind and low-vision communities in India, Brazil, Singapore, Indonesia, and Japan.

Guided Vision is now available on Android 9 and above in regions where Gemini Live is supported. Users can access it via the Gemini app settings, Android Accessibility Shortcuts, or through integration with Google TalkBack. Google notes that while the tool is an assistive utility, it is not a replacement for mobility aids like white canes and should not be used for navigation or obstacle detection.