A team of researchers from the University of California and the two departments of Facebook Reality Labs and AI Research has developed a neural network capable of creating 3D images and 3D figures from 2D images.
Pifu HD, which stands for Pixel-Aligned Implicit Function HD, has been trained to convert 3D images and videos that reproduce dressed people to 2D.
In addition to facial features such as mouth and nose, the system is able to accurately convert wrinkles and the texture of clothing into a corresponding 3D model. Female photographic poses are eloquent, and male poses are not joking either.
Pifu HD is an improved version of the previous AI Pifu, introduced last year. With more pixel information, smaller details can be transferred to a 3D model, such as folds and finger positions. The scientist Shunsuke saito presented the new work in two videos. There is also a scientific document with detailed information.
Pifu HD training
Artificial intelligence is trained to produce 3D images using 2D images, which act as landmarks. In the specific case, a set of fashion images was used, which contains male and female photographic models in various poses and with different articles of clothing.
Much like how it works for medical MRI, each depth level scanned results in a three-dimensional figure. An image encoder calculates the respective 3D coordinate for each 2D pixel and estimates whether it is inside or outside the cross section of the image under consideration. And from image to 3D at that point it's a short step.
Pifu HD encodes parallel images in reduced resolution, and uses the information to create its own 3D model.
The video demo works quite well. Therefore, Pifu can generate a 3D human figure from model images. Plus, a much more detailed 3D model from different images of the same target person with different viewing angles.
We still need some time
Pifu is not fully developed yet and is not efficient enough to be used productively with its 3D models. The speed of creation and learning of this software, however, is astounding.
