Relationship Spatialization for Depth Estimation

Xiaoyu Xu, Jiayan Qiu, Xinchao Wang, Zhou Wang ;

"Considering the role played by the relationships between objects in monocular depth estimation (MDE), it can be easily told that relationships, such as ‘in front of’ and ‘behind’, provide explicit spatial priors for depth estimation. However, it is hard to answer the questions that which kinds of relationships embed with the useful spatial cues for MDE? And how much these relationships contribute to the MDE? We term the task of answering these two questions as Relationship Spatialization. To this end, we strive to spatializing the relationships by devising a novel learning-based framework. Specifically, given the monocular image, the image representations and the corresponding scene graph are firstly extracted, and the in-graph relationship representations are learnt to be obtained. Then, the relationship representations from the graph space are spatially aligned with the image representations from the visual space, followed by a redundancy elimination. Finally, we feed the concatenation of the image representations and the modified relationship representations into a depth predictor, which estimates the monocular depth with relationship spatialization. Experiments on KITTI, NYU v2 and ICL-NUIM datasets shows the effectiveness of relationship spatialization on MDE. Moreover, adopting our framework to current state-of-the-art MDE models leads to marginal improvement on most evaluation metrics."