Agents Now Have Eyes — Click, Describe, Done

The Element Selector feature has been introduced, allowing UI elements in an app to be modified with a single click instead of having to describe each one in text. Now users can directly select a specific element in an app via Agent or Assistant and change it immediately.

Here's how Element Selector works:

• Turn on Element Selector: Enable the feature next to the chat input tool for Agent and Assistant.
• Click to select: Directly click on the UI element you want to modify, such as a button or heading.
• Describe the change: Assistant updates text, colors, and styles, while Agent adds features such as turning a static list into a searchable list.

The core of this feature is that it maps UI components to their exact location in the codebase. Once an element is selected, the Agent immediately grasps information such as:

• Exact file and line number references
• Style properties and current state
• Related functions and event handlers

Through this, the Agent doesn't simply guess but accurately understands and executes the change the user wants. Currently, this feature is available for new JavaScript-based apps, with plans to expand to other tech stacks in the future.