Neural Scanning: Rendering and determining geometry of household objects using Neural Radiance Fields

Floris Erich, Baptiste Bourreau, Chun Kwang Tan, Guillaume Caron, Yusuke Yoshiyasu, Noriaki Ando

Accepted for 2023 IEEE/SICE International Symposium on System Integration: Pre-print paper.

In this paper we present a hardware and software framework for Neural Scanning of household objects using Neural Radiance Fields (NeRF). The NeRF technique tries to learn a probabilistic representation of radiance and density, that can be used to render objects and to export objects' geometry. Our framework allows for easy scanning of the objects by rotating the object while using cameras in a static position. The objects we scan are mostly taken from the Yale-CMU-Berkeley (YCB) object set, and we release our scans as part of a public dataset.

This research is subsidized by New Energy and Industrial Technology Development Organization (NEDO) under a project JPNP20016. This paper is one of the achievements of joint research with and is jointly owned copyrighted material of ROBOT Industrial Basic Technology Collaborative Innovation Partnership.

Datasets

In the paper we consider 10 objects from the YCB set for qualitative analysis and 4 objects that we scanned in a previous work for quantitive and qualitive analysis. We include "masked images and transforms", which can be directly loaded using instant-ngp (place them in the data/nerf folder and run ./build/testbed --scene data/nerf/NAME), as well as exported meshes that can be loaded using your favorite 3D viewing or editting software (such as Meshlab). By clicking on the thumbnail in the Preview column, an animation will start to see the objects from multiple angles.

YCB objects

For the YCB objects, in the paper we compare our results with the TSDF textured scans from the original YCB set.

Object Name	Masked images and transforms	Exported mesh
001_chips_can	Zipped	Zipped
002_master_chef_can	Zipped	Zipped
003_cracker_box	Zipped	Zipped
021_bleach_cleanser	Zipped	Zipped
022_windex_bottle	Zipped	Zipped
024_bowl	Zipped	Zipped
035_power_drill	Zipped	Zipped
036_wood_block	Zipped	Zipped
048_hammer	Zipped	Zipped
054_softball	Zipped	Zipped

Household objects

For a quantitive analysis we have additionally scanned four household objects using both our neural scanner and a commercial 3D scanner. We included both the exported mesh using neural scanning and the ground truth mesh using the commercial 3D scanner.

Object Name	Masked images and transforms	Exported mesh	RGBD comparison	Ground truth mesh
cup_noodle	Zipped	ply mesh	ply mesh	Zipped
coffee_mug	Zipped	ply mesh	ply mesh	Zipped
sparkling_lemon	Zipped	ply mesh	ply mesh	Zipped
tissues_box	Zipped	ply mesh	ply mesh	Zipped