A team of researchers from the University of California and the two departments of Facebook Reality Labs and AI Research has developed a neural network capable of creating 3D figures from 2D images.
Pifu HD, which stands for Pixel-Aligned Implicit Function HD, has been trained to convert 3D images and videos that reproduce dressed people to 2D.
In addition to facial features such as mouth and nose, the system is able to accurately convert wrinkles and the texture of clothing into a corresponding 3D model. Female photographic poses are eloquent, and male poses are not joking either.
Pifu HD is an improved version of the previous AI Pifu, introduced last year. With more pixel information, smaller details can be transferred to a 3D model, such as folds and finger positions. The scientist Shunsuke saito presented the new work in two videos. There is also a scientific document with detailed information.
The training of Pifu HD
Artificial intelligence is trained using 2D images, which serve as reference points. In the specific case, a set of fashion images was used, which contains male and female photographic models in various poses and with different clothing items.
Much like it works for medical MRI, each depth level scanned results in a three-dimensional figure. An image encoder calculates the respective 3D coordinate for each 2D pixel and estimates whether it is inside or outside the cross section of the image under examination. And from image to 3D at that point the step is short.
Pifu HD encodes parallel images in reduced resolution, and uses the information to create its own 3D model.
The video demo works quite well. Therefore, Pifu can generate a 3D human figure. Furthermore, a much more detailed 3D model from different images of the same reference person with different viewing angles.
We still need some time
Pifu is not yet fully developed and is not efficient enough to be used productively. Her speed of learning, however, leaves you flabbergasted.
