Learning Shape Priors for Single-View 3D Completion and Reconstruction

Jiajun Wu, Chengkai Zhang, Xiuming Zhang, Zhoutong Zhang, William T. Freeman, Joshua B. Tenenbaum; The European Conference on Computer Vision (ECCV), 2018, pp. 646-662

The problem of single-view 3D shape completion or reconstruction is challenging, because among the many possible shapes that explain an observation, most are implausible and do not correspond to natural objects. Recent research in the field has tackled this problem by exploiting the expressiveness of deep convolutional networks. In fact, there is another level of ambiguity that is often overlooked: there are usually multiple plausible shapes that fit the 2D image equally well; i.e., the ground truth shape is non-deterministic given the input. Existing fully supervised approaches fail to address this issue, and often produce blurry mean shapes with smooth surfaces but no fine details. In this paper, we propose ShapeHD, pushing the limit of single-view shape completion and reconstruction by integrating deep generative models with adversarially learned shape priors. The learned priors serve as a regularizer, penalizing the model only if its output is unrealistic, not if it deviates from the ground truth. Our design thus overcomes both types of ambiguities aforementioned. Experiments demonstrate that ShapeHD outperforms state-of-the-arts by a large margin on both shape completion and shape reconstruction.