Zero-Shot Vision Library for Robotics Released

Developed by the Pollen Robotics team, pollen-vision is an open-source library that helps robots recognize and manipulate unknown objects.

The library focuses on combining ready-to-use Zero-Shot models, requiring no additional training, to build a 3D object detection pipeline that estimates the 3D spatial coordinates (x, y, z) of objects.

The core models included are as follows:

• OWL-VIT: Text-based 2D object localization and bounding box generation
• Mobile SAM: Object mask generation via a lightweight segmentation model
• RAM (Recognize Anything Model): Object tagging and recognition within images

Users can easily implement object detection and segmentation by simply inputting a text prompt and an image, and it is designed to run in real time (few fps) on consumer GPUs.