Rearranging unknown objects in everyday environments (e.g., kitchens, stores, warehouses, etc.) has been recently identified as a major challenging problem in robotics. In contrast to human’s fast adaptation capabilities in novel scenarios, most robot agents are limited to hand-designed environments because they rarely fully understand the scene, which includes multiple heterogeneous components (e.g., objects in start states, objects in goal states, fixed environmental structures, etc). A versatile robot should attend to this semantic information and coordinate its skills (e.g., pick-place, push) in order to adapt to novel scenarios.

PhD student Xibai Lou (Electrical Engineering), in a project called “Object rearrangement with hetero-graph neural networks,” is developing a hierarchical robotic system that exploits the underlying relationships and maximizes the collaborative power of pick-place and push for rearranging unknown objects. The high-level coordinator employs a Heterogeneous Graph Neural Network (HetGNN) to reason about what action to perform on which object, and the low-level planners execute the action in a continuous space. The typical pick-place action primitive, which expensively combines pick, move, and place, is limited to graspable objects and bound to a 90% success rate for novel objects, motivating the collaboration with an expedited and safe non-prehensile action primitive, push. Prior work primarily focused on model-based single action planning in a simplified open tabletop scenario, whereas we are interested in investigating how the heterogeneous scene components affect rearranging unknown objects with both pick-place and push in challenging real-world environments.

Some funding for this project was provided by the 2023 Research Computing-MnDRIVE PhD Graduate Assistantship Program. This program supports U of M PhD candidates pursuing research at the intersection of informatics and any of the five MnDRIVE areas:

  • Robotics
  • Global Food
  • Environment
  • Conditions
  • Cancer Clinical Trials

This project is part of the Robotics MnDRIVE area. See the complete list of the Research Computing-MnDRIVE PhD Graduate Assistantships for 2023.

photo of robot arm with objects, one photo showing "before" and the other "after" sorting