SAM-guided Graph Cut for 3D Instance Segmentation

Haoyu Guo*, He Zhu, Sida Peng, Yuang Wang, Yujun Shen, Ruizhen Hu*, Xiaowei Zhou* ;

"∗ Equal contribution † Corresponding authors This paper addresses the challenge of 3D instance segmentation by simultaneously leveraging 3D geometric and multi-view image information. Many previous works have applied deep learning techniques to 3D point clouds for instance segmentation. However, these methods often failed to generalize to various types of scenes due to the scarcity and low-diversity of labeled 3D point cloud data. Some recent works have attempted to lift 2D instance segmentations to 3D within a bottom-up framework. The inconsistency in 2D instance segmentations among views can substantially degrade the performance of 3D segmentation. In this work, we introduce a novel 3D-to-2D query framework to effectively exploit 2D segmentation models for 3D instance segmentation. Specifically, we pre-segment the scene into several superpoints in 3D, and formulate the task into a graph cut problem. The superpoint graph is constructed based on 2D segmentation models, enabling great segmentation performance on various types of scenes. We employ a GNN to further improve the robustness, which can be trained using pseudo 3D labels generated from 2D segmentation models. Experimental results on the ScanNet200, ScanNet++ and KITTI-360 datasets demonstrate that our method achieves state-of-the-art segmentation performance. Code will be made publicly available for reproducibility."