Portuguese 
Spanish 
Research shows how a cheap grabber with iPhone helped scientists transform real homes into a data source for domestic robots. Project Dobb·E uses human demonstrations to bring machines closer to everyday tasks, like opening drawers, manipulating objects, and interacting with furniture.
Researchers affiliated with New York University and Meta developed Dobb·E, an open domestic robotics system that uses human demonstrations in real homes to train robots to open drawers, close cabinets, manipulate objects, and perform common everyday tasks.
The research was presented in the technical paper “On Bringing Robots Home,” published on arXiv on November 27, 2023, and proposes a cheaper way to collect physical data for machines that need to operate outside controlled laboratories.
At the center of the project is The Stick, a tool created with a manual reacher-grabber type grabber costing $25, 3D printed parts, and used iPhones to record images, depth, and movements during household tasks.
-
Woman Invented the First Dishwasher Using Hot Water and Pressure in 1886 and Founded a Company to Sell It
-
Self-Taught Innovator Develops Colorful Maritime Rockets for Nighttime Ship Communication, Adopted by the U.S. Navy for Safer Visual Messaging
-
IBM Breaks 1-Nanometer Barrier with 0.7 nm Chip Featuring Nearly 100 Billion Transistors and 3D Architecture, Promising AI Revolution
-
The Secret U.S. Underwater Network That Tracked Soviet Submarines for Decades Now Used for Science
Domestic robots learn from real demonstrations
Instead of training only in simulations or organized tables, Dobb·E uses interactions made by people inside homes, where furniture, objects, lighting, and obstacles change from one room to another with little predictability.
During collection, a person holds The Stick, performs a simple action, and lets the iPhone record the visual and spatial data that later helps the system guide the robotic claw in similar situations.
This choice reduces the gap between the training environment and the place where the robot needs to function, as real kitchens, living rooms, and cabinets present variations difficult to fully reproduce in a laboratory.
The Stick transforms gestures into data for AI
The tool was designed to make the collection of demonstrations more accessible, comfortable, and scalable, without requiring each volunteer to directly control a complete robot to teach a household task.
According to the technical paper, models compatible with iPhone Pro 12 or later versions allow capturing RGB videos, depth data, and spatial information at 30 frames per second during the demonstrations.
As The Stick has a visual appearance similar to the robot claw used in tests, researchers sought to reduce the difference between what appears in the human demonstration and what the machine sees when performing the task.
New York Homes Became Training Base
Named Homes of New York, or HoNY, the database created by the project gathers 13 hours of interactions collected in 22 New York homes, with 5,620 trajectories recorded in 216 domestic environments.
The set totals about 1.5 million frames and includes RGB-D videos, as well as annotations on position, orientation, and claw opening, used to train visual representations aimed at the domestic environment.
These representations, called Home Pretrained Representations, or HPR, serve as a basis for robotic policies that need to learn new tasks with few demonstrations in environments different from those used in the initial collection.
Stretch Was Used in Dobb·E Tests
In physical experiments, Dobb·E was applied to Stretch, a mobile robot from Hello Robot used by the team to perform tasks learned from demonstrations recorded with the grabber and the iPhone.
According to the official project page, the system attempted 109 tasks in 10 homes in and around New York, achieving an 81% success rate in the tests presented by the researchers.
The team also reports that Dobb·E can learn a new task with five minutes of demonstration and about 15 minutes of model adaptation before execution by the robot.
Examples shown by the project include opening blinds, pulling kitchen drawers, closing cabinet doors, moving dishes, handling bottles, and interacting with appliances in different homes.
Imitation Learning Leaves the Laboratory
The basis of Dobb·E is imitation learning, a method in which the robot learns by observing human demonstrations, rather than relying solely on trial and error to figure out how to act.
Inside a house, this approach helps deal with tasks that seem simple to people but require the machine to recognize the scenario, estimate distance, position correctly, and physically control the claw.
To expand the collection, researchers prioritized efficiency, safety, and user comfort, as a domestic system needs to record demonstrations in real environments without relying on complex infrastructure or specialized operators.
Limits Still Keep Generalist Robots Away
Despite the results, the technical article points out obstacles such as strong shadows, variable lighting, uneven quality of demonstrations, sensor limitations, and physical restrictions of the hardware used in the tests.
Long tasks, with multiple linked steps, also appear as a challenge for Dobb·E, because they require memory, adaptation during execution, and the ability to handle changes in the environment.
The project was made available with code, models, data, and hardware designs, a measure that allows other researchers to test the system and expand studies on robots capable of operating in real homes.
By transforming everyday movements into structured records, Dobb·E shows how the next stage of artificial intelligence may depend not only on texts, images, and videos online, but also on physical data captured in the real world.
