A Dataset of 3D-Scanned Frequent Family Gadgets


Many latest advances in pc imaginative and prescient and robotics depend on deep studying, however coaching deep studying fashions requires a large number of information to generalize to new situations. Traditionally, deep studying for pc imaginative and prescient has relied on datasets with tens of millions of things that had been gathered by internet scraping, examples of which embody ImageNet, Open Pictures, YouTube-8M, and COCO. Nonetheless, the method of making these datasets will be labor-intensive, and may nonetheless exhibit labeling errors that may distort the notion of progress. Moreover, this technique doesn’t readily generalize to arbitrary three-dimensional shapes or real-world robotic information.

Actual-world robotic information assortment could be very helpful, however troublesome to scale and difficult to label (determine from BC-Z).

Simulating robots and environments utilizing instruments corresponding to Gazebo, MuJoCo, and Unity can mitigate lots of the inherent limitations in these datasets. Nonetheless, simulation is just an approximation of actuality — handcrafted fashions constructed from polygons and primitives typically correspond poorly to actual objects. Even when a scene is constructed straight from a 3D scan of an actual setting, the movable objects in that scan will act like fastened background surroundings and won’t reply the way in which real-world objects would. Resulting from these challenges, there are few giant libraries with high-quality fashions of 3D objects that may be included into bodily and visible simulations to offer the range wanted for deep studying.

In “Google Scanned Objects: A Excessive-High quality Dataset of 3D Scanned Family Gadgets”, introduced at ICRA 2022, we describe our efforts to handle this want by creating the Scanned Objects dataset, a curated assortment of over 1000 3D-scanned frequent home items. The Scanned Objects dataset is usable in instruments that learn Simulation Description Format (SDF) fashions, together with the Gazebo and PyBullet robotics simulators. Scanned Objects is hosted on Open Robotics, an open-source internet hosting setting for fashions suitable with the Gazebo simulator.

Historical past

Robotics researchers inside Google started scanning objects in 2011, creating high-fidelity 3D fashions of frequent family objects to assist robots acknowledge and grasp issues of their environments. Nonetheless, it turned obvious that 3D fashions have many makes use of past object recognition and robotic greedy, together with scene development for bodily simulations and 3D object visualization for end-user purposes. Subsequently, this Scanned Objects challenge was expanded to deliver 3D experiences to Google at scale, accumulating a lot of 3D scans of family objects by means of a course of that’s extra environment friendly and value efficient than conventional commercial-grade product images.

Scanned Objects was an end-to-end effort, involving improvements at almost each stage of the method, together with curation of objects at scale for 3D scanning, the event of novel 3D scanning {hardware}, environment friendly 3D scanning software program, quick 3D rendering software program for high quality assurance, and specialised frontends for internet and cellular viewers. We additionally executed human-computer interplay research to create efficient experiences for interacting with 3D objects.

Objects that had been acquired for scanning.

These object fashions proved helpful in 3D visualizations for On a regular basis Robots, which used the fashions to bridge the sim-to-real hole for coaching, work later revealed as RetinaGAN and RL-CycleGAN. Constructing on these earlier 3D scanning efforts, in 2019 we started getting ready an exterior model of the Scanned Objects dataset and remodeling the earlier set of 3D photos into graspable 3D fashions.

Object Scanning

To create high-quality fashions, we constructed a scanning rig to seize photos of an object from a number of instructions beneath managed and punctiliously calibrated circumstances. The system consists of two machine imaginative and prescient cameras for form detection, a DSLR digital camera for high-quality HDR coloration body extraction, and a computer-controlled projector for sample recognition. The scanning rig makes use of a structured mild method that infers a 3D form from digital camera photos with patterns of sunshine which might be projected onto an object.

The scanning rig used to seize 3D fashions.
A shoe being scanned (left). Pictures are captured from a number of instructions with totally different patterns of sunshine and coloration. A shadow passing over an object (proper) illustrates how a 3D form will be captured with an off-axis view of a shadow edge.

Simulation Mannequin Conversion

The early inside scanned fashions used protocol buffer metadata, high-resolution visuals, and codecs that weren’t appropriate for simulation. For some objects, bodily properties, corresponding to mass, had been captured by weighing the objects at scanning time, however floor properties, corresponding to friction or deformation, weren’t represented.

So, following information assortment, we constructed an automatic pipeline to resolve these points and allow using scanned fashions in simulation methods. The automated pipeline filters out invalid or duplicate objects, routinely assigns object names utilizing textual content descriptions of the objects, and eliminates object mesh scans that don’t meet simulation necessities. Subsequent, the pipeline estimates simulation properties (e.g., mass and second of inertia) from form and quantity, constructs collision volumes, and downscales the mannequin to a usable measurement. Lastly, the pipeline converts every mannequin to SDF format, creates thumbnail photos, and packages the mannequin to be used in simulation methods.

The pipeline filters fashions that aren’t appropriate for simulation, generates collision volumes, computes bodily properties, downsamples meshes, generates thumbnails, and packages all of them to be used in simulation methods.
A group of Scanned Object fashions rendered in Blender.

The output of this pipeline is a simulation mannequin in an acceptable format with a reputation, mass, friction, inertia, and collision data, together with searchable metadata in a public interface suitable with our open-source internet hosting on Open Robotics’ Gazebo.

The output objects are represented as SDF fashions that check with Wavefront OBJ meshes averaging 1.4 Mb per mannequin. Textures for these fashions are in PNG format and common 11.2 Mb. Collectively, these present excessive decision form and texture.


The Scanned Objects dataset accommodates 1030 scanned objects and their related metadata, totaling 13 Gb, licensed beneath the CC-BY 4.0 License. As a result of these fashions are scanned fairly than modeled by hand, they realistically mirror actual object properties, not idealized recreations, lowering the issue of transferring studying from simulation to the true world.

Enter views (left) and reconstructed form and texture from two novel views on the proper (determine from Differentiable Stereopsis).
Visualized motion scoring predictions over three real-world 3D scans from the Reproduction dataset and Scanned Objects (determine from Where2Act).

The Scanned Objects dataset has already been utilized in over 25 papers throughout as many tasks, spanning pc imaginative and prescient, pc graphics, robotic manipulation, robotic navigation, and 3D form processing. Most tasks used the dataset to offer artificial coaching information for studying algorithms. For instance, the Scanned Objects dataset was utilized in Kubric, an open-sourced generator of scalable datasets to be used in over a dozen imaginative and prescient duties, and in LAX-RAY, a system for looking cabinets with lateral entry X-rays to automate the mechanical seek for occluded objects on cabinets.

We hope that the Scanned Objects dataset will probably be utilized by extra robotics and simulation researchers sooner or later, and that the instance set by this dataset will encourage different house owners of 3D mannequin repositories to make them out there for researchers in every single place. If you want to strive it your self, head to Gazebo and begin searching!


The authors thank the Scanned Objects workforce, together with Peter Anderson-Sprecher, J.J. Blumenkranz, James Bruce, Ken Conley, Katie Dektar, Charles DuHadway, Anthony Francis, Chaitanya Gharpure, Topraj Gurung, Kristy Headley, Ryan Hickman, John Isidoro, Sumit Jain, Brandon Kinman, Greg Kline, Mach Kobayashi, Nate Koenig, Kai Kohlhoff, James Kuffner, Thor Lewis, Mike Licitra, Lexi Martin, Julian (Mac) Mason, Rus Maxham, Pascal Muetschard, Kannan Pashupathy, Barbara Petit, Arshan Poursohi, Jared Russell, Matt Seegmiller, John Sheu, Joe Taylor, Vincent Vanhoucke, Josh Weaver, and Tommy McHugh.

Particular thanks go to Krista Reymann for organizing this challenge, serving to write the paper, and enhancing this blogpost, James Bruce for the scanning pipeline design and Pascal Muetschard for sustaining the database of object fashions.


Please enter your comment!
Please enter your name here