3D Attention-Driven Depth Acquisition for Object Identification

Kai Xu1,2,3Yifei Shi1,  Lintao Zheng1,  Junyu Zhang3,  Min Liu1,  Hui Huang3,

Hao Su4, Daniel Cohen-Or5, Baoquan Chen2

1National University of Defense Technology,    2Shandong University,    3Shenzhen University    4Stanford University    5Tel-Aviv University

ACM Transactions on Graphics (SIGGRAPH Asia 2016), 35(6)


Figure 1: We present an autonomous system for active object identification in an indoor scene (a), with consecutive depth acquisitions, for online scene modeling. The scene is first roughly scanned, and segmented to generate 3D object proposals. Targeting an object proposal (b), the robot performs multi-view object identification, based on a 3D shape database, driven by a 3D Attention Model. The retrieved 3D models are inserted into the scanned scene (c), replacing the corresponding object scans, thus incrementally constructing a 3D scene model (d).


We address the problem of autonomous exploring unknown objects in a scene by consecutive depth acquisitions. The goal is to model the scene via identifying the objects online, from among a large collection of 3D shapes. Fine-grained shape identification demands a meticulous series of observations attending to varying views and parts of the object of interest. Inspired by the recent success of attention-based models for 2D recognition, we develop a 3D Attention Model that selects the best views to scan from, as well as the most informative regions in each view to focus on, to achieve efficient object recognition. The region-level attention leads to focus-driven features which are quite robust against object occlusion. The attention model, trained with the 3D shape collection, encodes the temporal dependencies among consecutive views with deep recurrent networks. This facilitates order-aware view planning accounting for robot movement cost. In achieving instance identification, the shape collection is organized into a hierarchy, associated with pre-trained hierarchical classifiers. The effectiveness of our method is demonstrated on an autonomous robot (PR) that explores a scene and identifies the objects for scene modeling.

Paper PDF


PDF, 12.5MB

Conference Slides

PDF, 4.5MB



MP4, 31MB


Algorithm Overview

Figure 2: Our core contribution is a recurrent 3D attention model (3D-RAM) (d). At each time step, it takes a depth image as input, updates its internal state, performs shape classification and estimates the next-best-view. To make instance-level classification feasible, we organize the shape collection with a classification hierarchy (b). At each node of the hierarchy, a multi-view 3D shape classifier (c) assigns the associated shapes into K overlapping groups (a). Online object identification is guided by 3D-RAM while traversing down the hierarchy for coarse-to-fine classification (e). The feature extractor and classifier of the corresponding node are employed by the 3D-RAM.


Figure 3: Our recurrent 3D attention model (b) is a two-layer stacked recurrent neural network, with a MV-CNN (a) plugged in. The subnetworks of MV-RNN responsible for feature extraction, feature aggregation and classification are substituted by CNN1, view pooling and CNN2 of MV-CNN, respectively. The corresponding subnetworks are shaded in same color. (b) shows a two-time-step expansion of MV-RNN. Dashed arrows indicate information flow across time steps while solid ones represent that within a time step.


Figure 4: Visualization of identification process and attention (both on acquired depth images and identified shapes) for six real objects.

We thank the anonymous reviewers for their valuable comments and suggestions. This work was supported in part by NSFC(61572507, 61532003, 61522213, 61232011), 973 Program (2015CB352501, 2014CB360503), Guangdong Science and Technology Program (2015A030312015, 2014B050502009, 2014TX01X033, 2016A050503036) and Shenzhen Innovation Program (JCYJ20151015151249564).

We have released the source code of our MV-RNN, which is built on top of Torch.

Github repository

@article {xu_siga16,
    title = {3D Attention-Driven Depth Acquisition for Object Identification},
    author = {
Kai Xu and Yifei Shi and Lintao Zheng and Junyu Zhang and Min Liu and Hui Huang and Hao Su and Daniel Cohen-Or and Baoquan Chen},
    journal = {ACM Transactions on Graphics (Proc. of SIGGRAPH Asia 2016)},
    volume = {35},
    number = {6},
    pages = {Article No. 238},
year = {2016}

    Back to TopHome