Estimating the 6D pose of objects unseen during training is highly desirable yet challenging. Zero-shot object 6D pose estimation methods address this challenge by leveraging additional task-specific supervision provided by large-scale, photo-realistic synthetic datasets. However, their performance heavily depends on the quality and diversity of rendered data and they require extensive training.
In this work, we show how to tackle the same task but without training on specific data. We propose FreeZe, a novel solution that harnesses the capabilities of pre-trained geometric and vision foundation models. FreeZe leverages 3D geometric descriptors learned from unrelated 3D point clouds and 2D visual features learned from web-scale 2D images to generate discriminative 3D point-level descriptors. We then estimate the 6D pose of unseen objects by 3D registration based on RANSAC. We also introduce a novel algorithm to solve ambiguous cases due to geometrically symmetric objects that is based on visual features.
We comprehensively evaluate FreeZe across the seven core datasets of the BOP Benchmark, which include over a hundred 3D objects and 20,000 images captured in various scenarios. FreeZe consistently outperforms all state-of-the-art approaches, including competitors extensively trained on synthetic 6D pose estimation data.
Given a 3D model of a novel query object and an RGBD image, we compute their geometric and visual features using frozen geometric and vision foundation models which do not require any task-speficic training.
We create rendered images from the 3D model to extract visual features of the query object, which we then back project to the object point cloud. Concurrently, we extract geometric features, which we then fuse with the visual features.
Similarly, we compute visual and geometric features of the target object imaged in the input crop, and fuse the two as before. Although the query and target objects are from two different modalities (a 3D model the former, and an RGBD image the latter), we employ the same vision and geometric encoders to compute their features.
Lastly, we input the fused features to a registration algorithm based on feature matching to estimate the object 6D pose.
@article{caraffa2024freeze,
title={FreeZe: Training-free zero-shot 6D pose estimation with geometric and vision foundation models},
author = {Caraffa, Andrea and Boscaini, Davide and Hamza, Amir and Poiesi, Fabio},
journal = {arXiv:2312.00947},
year = {2024}
}