Developing 3D Images

A key challenge of learning a visual representation for the 3D high fidelity geometry of dressed humans lies in the limited availability of the ground truth data (e.g., 3D scanned models), which results in the performance degradation of 3D human reconstruction when applying to real-world imagery. PhD student Yasamin Jafarian (Computer Science), in a project called “Learning High Fidelity Depths of Dressed Humans by Watching Social Media Dance Videos,” is addressing this challenge by leveraging a new data resource: a number of social media dance videos that span diverse appearances, clothing styles, performances, and identities. Each video depicts dynamic movements of the body and clothes of a single person while lacking the 3D ground truth geometry. To learn a visual representation from these videos, this project uses a new self-supervised learning method to use the local transformation that warps the predicted local geometry of the person from an image to that of another image at a different time instant. This allows self-supervision by enforcing a temporal coherence over the predictions. The project also learns the depths along with the surface normals that are highly responsive to local texture, wrinkle, and shade by maximizing their geometric consistency. This method is end-to-end trainable, resulting in high fidelity depth estimation that predicts fine geometry faithful to the input real image. This method outperforms the state-of-the-art human depth estimation and human shape recovery approaches on both real and rendered images.

Some funding for this project was provided by a 2022 University of Minnesota Informatics Institute MnDRIVE PhD Graduate Assistantship. The UMII MnDRIVE Graduate Assistantship program supports U of M PhD candidates pursuing research at the intersection of informatics and any of the five MnDRIVE areas:

  • Robotics, Sensors and Advanced Manufacturing
  • Global Food Ventures
  • Advancing Industry, Conserving Our Environment
  • Discoveries and Treatments for Brain Conditions
  • Cancer Clinical Trials

This project is part of the Robotics, Sensors, and Advanced Manufacturing MnDRIVE area.

Research Computing partners:

  • University of Minnesota Informatics Institute

Complete list of 2022 UMII MnDRIVE PhD Graduate Assistantships.

 

photos of dancing woman with computer-generated images