Reimagining the Mouse Pointer for the AI Era

Google has unveiled an AI-enabled pointer experiment that combines pointer and voice input based on Gemini.

The core idea is to let users invoke AI directly on top of the document, image, or webpage they're currently viewing, without needing to move to a separate AI window.

The four principles presented are as follows.

• Maintain the flow: Handle tasks directly within the current workflow, like summarizing a PDF, converting a table into a pie chart, or doubling recipe ingredients.
• Show and tell: Read the visual and semantic context around the pointer together, accurately understanding what the user is pointing at, whether it's a paragraph, a word, part of an image, or a code block.
• This and that: Naturally combine short instructions with gestures to simplify complex requests.
• Pixels to actionable entities: Turn pixels on screen into actionable entities like places, dates, or objects, directly connecting notes or video scenes to actions.

Google stated that it will apply this feature to Gemini in Chrome, enabling users to point at a specific part of a webpage and immediately request comparisons, summaries, or visualizations. It also plans to extend the same approach to products like Magic Pointer in Googlebook and Disco from Google Labs. Users can currently try demos of image editing and finding places on maps in Google AI Studio.