2D photo to 3D -- PIFuHD

PIFuHD

I used the PIFuHD demo software (on Google's Colaboratory) to convert my 2D drawing into a full 3D OBJ file:

remove.BG

PIFuHD

Blender

EZGif

On Instagram

The resulting 3D OBJ  file

spinning in front

of my 2D source drawing

Still image

from the above

Note:  I first cropped out all of the background with remove.BG, and downloaded the PNG file with a transparent background.  This is what I uploaded to PIFuHD.

I followed the YouTube PIFuHD tutorial video to get through the process.  Although the interface is in Chinese, the narrative was in English, so it was not too impossible to navigate.

PIFuHD tutorial

This is an AI (artificial intelligence) paper that was submitted by researchers from both Facebook and USC, and forwarded to me by Holly Grimm.

The whole project was presented as a larger paper, with examples and 3 other YouTube explanations:

PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization

YouTube presentation of 3D results

WORKS BEST with

 PHOTOS of REAL PEOPLE 

The software demo did work with my more realistic drawing above (and not with my less realistic drawing), but the result wasn't great.  PIFuHD works best with realistic frontal photos of humans, which is what it was designed for.

Therefore I downloaded a free PNG file of a woman (with transparent background), taken from the front -- and the resulting 3D image was a lot better.

pngimg

PIFuHD

Blender

EZGif

Better 3D results,

from a more realistic frontal photo

TWITTER RESULTS

People are posting their results on Twitter:

PIFuHD results on Twitter

Interesting results:

• From a classical painting
• 3D printing

• Example after animating the 3D result in Mixamo.
• Mixamo and Blender
• Augmented Reality
• Using Artbreeder first (the AI program)  -- imagine using "this person does not exist" AI, when it gives full body images

Perhaps someone could slightly alter the input 2D form in Photoshop, or even Deep Dream Generator, and get interesting results using PIFuHD.

BOCCIONI

I tried PIFuHD with a photo of a Umberto Boccioni sculpture-  Spiral Expansion of Muscles in Action.  Amazingly it went through the whole process, but the result was not pretty.

remove.BG

PIFuHD

Rendered in Blender

The Boccioni 3D result

was in two incoherent pieces

I was able to download this

at the end of the PIFuHD process

***

PIFuHD reminds me of around 2012, when we were 3D scanning with the now discontinued 123D Catch software from Autodesk (using Photogrammetry) -- to 3D scan this big float, for instance -- giving us the same kind of artifact debris around and inside the figure.  

We made a 3D file from a large 

Day-of-the-Dead float,

using the old 123D Catch software

 from Autodesk

Likewise we used Scanect and ItSeez3D (like this example) during the Tucson Sculpture Festival 2015.

***

There are some other interesting online experiments in this direction:

• Body Pix (live demo)
• Pose Net (live demo)