Robots learn from dogs to understand human gestures and can now locate objects with 89% success.

Written by Fabio Lucas Carvalho

Published on 01/06/2026 at 23:58

1 person reacted to this.

Research from Brown University combines language, human gestures, and computer vision to improve object search by robots, with 89% average success in simulations and inspiration from how dogs interpret pointing, looks, and intentions in interactions with people.

Robots capable of locating objects through language, gestures, and vision achieved 89% average success in simulations at Brown University, in a study accepted for HRI 2026, scheduled for March in Edinburgh.

Robots learn from dogs to interpret human commands

The advancement addresses a common difficulty in the domestic and professional use of machines: understanding incomplete requests. For a person, asking for a key, a cup, or a tool seems simple. For a robotic system, the task involves ambiguity, movement, similar objects, and imperfect clues.

The team at Brown University developed the LEGS-POMDP, a system that combines language, human pointing, and visual observation. The inspiration came from research at the Brown Dog Lab on how dogs interpret gestures and looks, especially when humans point to something.

ARTICLE CONTINUES BELOW

How the system decides where to search

The name LEGS-POMDP refers to a probabilistic structure based on a partially observable Markov decision process. In practice, it helps the machine act when it does not have all the necessary information about the environment, the object, or human intention.

Instead of deciding too quickly, the system maintains hypotheses about the identity and location of the sought item. These hypotheses are updated as new clues appear, including verbal description, gesture direction, and visual reading of the scene.

The combination allows the robot to better explore the space before concluding the search. It can adjust the viewpoint, review a possibility, and delay the final choice until gathering stronger evidence about where the correct object is.

In the experiments, multimodal integration outperformed approaches based solely on language or gestures. The result reinforces the idea that human communication depends on the sum of signals, not a single isolated instruction.

Tests indicate progress, but still with limits

An average rate of 89% was recorded in simulations described as demanding. The team also conducted tests with a real quadruped robot, used as qualitative validation of the approach. The research will be presented at HRI 2026, from March 16 to 19, 2026.

The use of a vision-language model expands the system’s ability to interpret scenes. Thus, the machine can relate verbal descriptions, spatial constraints, and visible objects, even when there is disorganization, similarity between items, or obstacles in the way.

The suggested applications involve everyday and industrial environments. In a home, robots could search for medications on a cluttered counter or find glasses among scattered items. In a workshop, they could retrieve parts and tools without excessively precise commands.

Even so, the results do not mean that fully intuitive mechanical assistants are already available. The 89% figure comes from simulations, while physical tests indicate robustness but do not eliminate the challenges of real, varied, and unpredictable environments.

The progress helps bring laboratories closer to everyday situations, where simple requests always carry noise, pauses, and inaccuracies.

The main advancement is in the way of dealing with uncertainty. By observing dogs, human gestures, and natural language, robotics gains a path to create machines less dependent on rigid commands and more capable of interpreting intentions in context.

Click here to check the study.

0 Comments

most recent

older Most voted

Robots learn from dogs to understand human gestures and can now locate objects with 89% success.

Research from Brown University combines language, human gestures, and computer vision to improve object search by robots, with 89% average success in simulations and inspiration from how dogs interpret pointing, looks, and intentions in interactions with people.

Robots learn from dogs to interpret human commands

Great Pyramid has withstood for more than 4,500 years in Egypt, and a study with 37 sensors reveals how its vibrations may have reduced damage in earthquakes.

How the system decides where to search

Tests indicate progress, but still with limits

A giant printer called the “Ferrari” of concrete has landed in Latin America and can already build a 120-square-meter house in just 48 hours, making the traditional construction site look like something from the last century.

Great Atlantic Sargassum Belt has ceased to be a climatic phenomenon and has become a biological system that feeds on itself, with 8,000 kilometers in length, 37 million tons of algae, and scientists confirm that it is now permanent.

Brazil will harvest a record crop of up to 357 million tons of grains, but it does not have the capacity to store a large part of it. The storage deficit has reached the highest level in history and is equivalent to almost the entire production of Argentina.

Under nearly 2 kilometers of ice in Antarctica, scientists have discovered an ancient landscape sculpted by rivers up to 60 million years ago, the size of Wales, preserved as a time capsule that could help predict the advance of ice towards the ocean.

Philippines begin construction of the country’s largest desalination plant in a city without enough fresh water to grow, and the 66,500 m³ per day plant will transform the sea into drinking water for 50,000 homes in 24 months with reverse osmosis technology from SUEZ and JEMCO.

Turkey is building a billion-dollar shortcut to channel billions of dollars by sea with a 45 km long canal, 275 meters wide and 20.75 meters deep, which promises to relieve the Bosphorus amid the crisis in the Strait of Hormuz.

While generating revenue of R$ 43.5 billion and recording a profit of R$ 2 billion, Grupo Mateus laid off 6,673 employees in six states in the North and Northeast.

Tutankhamun’s jewel made with rare glass found in the desert may reveal traces of a gigantic cosmic collision that hit Earth around 29 million years ago and continues to intrigue scientists, archaeologists, and geology experts to this day.

French tourist accidentally enters a volcanic crater in Arkansas and comes out with a 7.46-carat diamond: rare brown stone became the eighth largest find since 1972 in a park where visitors can keep the treasure they find.

Supermarket giant has just closed 28 stores and left 6,673 people unemployed in the North and Northeast of Brazil.

Now Brazil has moved up a level: flying robots are starting to be tested to harvest oranges on farms, selecting ripe fruits with cameras and sensors, sucking each fruit without dropping it, and promising to help producers in the face of labor shortages in orchards.

Robots learn from dogs to understand human gestures and can now locate objects with 89% success.

Research from Brown University combines language, human gestures, and computer vision to improve object search by robots, with 89% average success in simulations and inspiration from how dogs interpret pointing, looks, and intentions in interactions with people.

Robots learn from dogs to interpret human commands

Great Pyramid has withstood for more than 4,500 years in Egypt, and a study with 37 sensors reveals how its vibrations may have reduced damage in earthquakes.

How the system decides where to search

Tests indicate progress, but still with limits

A giant printer called the “Ferrari” of concrete has landed in Latin America and can already build a 120-square-meter house in just 48 hours, making the traditional construction site look like something from the last century.

Great Atlantic Sargassum Belt has ceased to be a climatic phenomenon and has become a biological system that feeds on itself, with 8,000 kilometers in length, 37 million tons of algae, and scientists confirm that it is now permanent.

Brazil will harvest a record crop of up to 357 million tons of grains, but it does not have the capacity to store a large part of it. The storage deficit has reached the highest level in history and is equivalent to almost the entire production of Argentina.

Under nearly 2 kilometers of ice in Antarctica, scientists have discovered an ancient landscape sculpted by rivers up to 60 million years ago, the size of Wales, preserved as a time capsule that could help predict the advance of ice towards the ocean.

Philippines begin construction of the country’s largest desalination plant in a city without enough fresh water to grow, and the 66,500 m³ per day plant will transform the sea into drinking water for 50,000 homes in 24 months with reverse osmosis technology from SUEZ and JEMCO.

Turkey is building a billion-dollar shortcut to channel billions of dollars by sea with a 45 km long canal, 275 meters wide and 20.75 meters deep, which promises to relieve the Bosphorus amid the crisis in the Strait of Hormuz.

Winter 2026 already has a start date in Brazil, but the advance of El Niño could completely change the expected climate pattern with more rain, less intense cold, and a reduced risk of snow in the South of the country.

In Germany, engineers drill kilometers of rock to set up a giant underground radiator that draws the hottest heat ever achieved in a geothermal well.

Pancreatic cancer pill surprises oncologists by doubling survival in phase 3 study and turning scientific data into a rare scene of emotion

New Fiat EV, priced at R$ 77,000, will bring a reinterpretation of the 147 and a consumption equivalent to 70 km/l.

Can a driver install LED headlights in any car? The replacement, which promises greater light range, energy savings, and durability, has become the target of technical restrictions due to the risk of glare in vehicles that did not come from the factory with the technology.

While generating revenue of R$ 43.5 billion and recording a profit of R$ 2 billion, Grupo Mateus laid off 6,673 employees in six states in the North and Northeast.

Tutankhamun’s jewel made with rare glass found in the desert may reveal traces of a gigantic cosmic collision that hit Earth around 29 million years ago and continues to intrigue scientists, archaeologists, and geology experts to this day.

French tourist accidentally enters a volcanic crater in Arkansas and comes out with a 7.46-carat diamond: rare brown stone became the eighth largest find since 1972 in a park where visitors can keep the treasure they find.

Supermarket giant has just closed 28 stores and left 6,673 people unemployed in the North and Northeast of Brazil.

Now Brazil has moved up a level: flying robots are starting to be tested to harvest oranges on farms, selecting ripe fruits with cameras and sensors, sucking each fruit without dropping it, and promising to help producers in the face of labor shortages in orchards.