Introduction to 3D Painting in VR

Eric Giessmann talked about 3D painting in VR they do together with Piers Goffart at LAVAMACHINE and discussed VR painting tools, software, animation, the difference between “classic” digital art and VR, its peculiarities and essential things to start with if you want to embark on this field.

SONY DSC

Introduction

My Name is Eric Giessmann. I am a freelance director and animator for animated short films from Dormagen near Cologne, Germany. I have been interested in art for as long as I can remember and it started with Bob Ross, Comics and Mangas. I also always liked to create jump&run or roleplay games by myself, with colored pencils on paper first and a few years later – on my first PC. I remember how I started with drawing on paper, stop motion and all that up to the computer animation.

In Holland, I studied animation and specialized in CGI. My graduation film TYPEWRITERHEAD received attention, I won a film prize and went on to become a director.

My new movie Sand Wanderer will soon be premiered online.

Recently, I’ve also been dealing with Virtual Reality quite a lot, and that feels like unpacking a Super Nintendo on Christmas 1992. At LAVAMACHINE, together with another VR artist, we are currently giving workshops to help out people who have not yet recognized the potential of VR.

Digital Art vs VR

Digital art is flexible during the making process. If you draw by hand then that’s basically it. The stroke is complete. With CGI you have the possibility to manipulate the elements during the production until the result is perfect. Plus, it is reproductive in some parts and that means less work. Back in the days, that seemed to be very appealing. At the same time, any digital artist confronts a typical problem of perfectionism: you can’t stop tweaking, and tweaking, and… eventually, it doesn’t mean less work.

But VR is NOT like that. It’s a back twist, and like in the old days, tiny little mistakes are still (or again) allowed to happen there. When I saw “Quillustrations” in the VR Facebook groups, I was inspired and felt like an inventor again. No compulsion, everything was unclean, rough and had exactly the charm I was looking for. I realized:

Limitations = creativity
Randomness is included without extra work
In addition, the incredible field of view
Maximally intuitive
What you see is what you get = no waiting for post-processing
No confusing tons of menus anymore, or at least no restrictions in the working space
And the most important: you again can be an artist and immerse yourself in your artwork

So, I bought a VR Headset almost a year ago, and I do not regret a cent ever since.

VR Software

Every VR software solution has advantages and disadvantages. I once started an Excel sheet where that was listed. The only problem was that the Software solutions are currently so quickly overloaded with updates that you can’t follow. The beautiful thing is that they cost almost nothing and the developers ease the work for artists with monthly or annual subscriptions.

In addition to Tilt Brush, I can recommend Quill, Medium, AnimVR, Blocks, MasterpieceVR, Gravity Sketch, and Tvori. Also, check out the Leap Motion painting software, which is definitely unique but still in the development.

The demand is still there, and we together with many other VR artists investigate it at the moment. Some concepts already exist. I remember, the school where we offered workshops wrote that our concept was too simple. I laughed: that’s exactly the goal to make it simple. You no longer have to be a scientist to create artwork in a three-dimensional space. I had no one yet who would put on the glasses and not say something like “WOOOOW!”.

The features are still in the development. There are some very cool features that do not exist in the “conventional” screen programs or at least make less sense there. For example, the direction-depending brush which looks different depending on the view angle. I also love the animation brush (there are no proper names for the features yet) which automatically keyframes the stroke while drawing. That makes sense in VR.

I think the artist is always his own “tool” somehow and VR is definitely a brand new toolset in this regards. During the work, you notice well how it affects the unused brain area in the spatial imagination.

VR Art Peculiarities

The first thing that stands out is that VR painting tools may not be appropriate for the clean 3D content production. First of all, you have to distinguish between VR software that works with 3D strokes (e.g. Quill, Tiltbrush), mostly without shaders, and software that looks more like made out of construction foam or clay with shading/lighting (Medium, MasterpieceVR).

In the first case the mesh is actually a little bit crappy, and sometimes it takes away a gigantic amount of memory. Especially if you want to do post-processing. For the 3D strokes, you must approach it differently. In this example of Fire Golem, you can see very well that it can stay within 50 MB.

Above all, you have to accept the style as it is.

You can already see some artist dealing with exactly this matter. The goal is to create a beautiful shader which, while shading the body as a whole, blends the individual strokes together. It’s a matter of taste. Sometimes I prefer shaded artworks, sometimes I like to paint the lighting by myself.

In the second case, like Hiero’s Journey. the mesh is clean and you can decide with which polygon number it will be exported. This is similar to programs such as ZBrush or Mudbox with slimmed-down functionality but maximum intuitive.

Still, with VR software there is a tendency: “back to vertex colors”. You can easily paint in VR, but then it is first saved as vertex color. Vertex colors are bound to the polygon number and make the further processing problematic (I can only judge that as an artist). Fortunately, Medium now offers the export option “UV export”, though with automatic mapping.

Hiero’s Journey Production

In case of Hiero’s Journey, I was inspired by the fantasy book with the same name that I still love to read. I first modeled everything in VR and did the basic coloring. Then the reduced the mesh (16%) and went to Substance Painter (including color & normal map).

There, I baked some additional passes and the texture magic happened. Unfortunately, the automatic baking made it really difficult to properly handle the seams of the UV patches.

Anyway, I also created a version for Maya to test the pipeline and shading (albedo, metallic, AO, roughness, normal).

What was nice is that I managed to display the scene correctly in the viewport in real-time, so no rendering was necessary. I finally painted the background and added a polymesh with transparency channel as light rays for Sketchfab.

Animation in VR

Animation in VR differs from classic CG animation. First of all, it’s less or no keyframe and much more frame-by-frame. So, paint, paint again, move or copy paste. It’s like the difference between intuitive hunting and sports hunting. If you like it, well, then you like it.

Some software solutions already give you a RIG and the animation totally feels like puppet animation or playing with action figures as a kid.

Some people know my Cochlear Implant VR animation which basically shows me wearing a Cochlear Implant to restore hearing. It was one of my first entries and only took me 5 days. No post-processing. With“classic animation”, all that stuff going on in the scene would have taken me a lot more time.

It also taught me that you never know if something will be successful or not and you usually don’t know why precisely.

With Timemachine, for example, we (LAVAMACHINE) started with reference pictures and a color scheme based on Piers‘ Comics.

The character was rather intuitive depending on the VR style. Most parts were done with AnimVR. For the mandala, I used Quill because it’s better and faster for doing this kind of stuff. I added a green background, then pressed the Render button and waited 2 hours with the headset on my fan so that the render didn’t fall into the standby mode.

Everything came together in Premiere altogether with the keyed mandala scene. We had to re-cut the timing a little bit because in VR it’s difficult to do fine-tuning. It’s also possible to work on a VR scene in the multiplayer mode, but that’s still a bit buggy.

Advice for the Newcomers

Don’t start with the cheap equipment, otherwise, you will get a cheap impression. The mobile VR devices (like Oculus Go or mobile phone devices) can’t get you what you will get with good VR equipment. They are more for presentation, and most painting/modeling tools on these devices don’t work for us. For example, room scaling (tracking of your head position) and two controllers are essential. At the moment, better start with a Rift or Vive, but that may change with the new VR Headsets generation.

Find a good 3D printing site for the stuff you want to print for your VR Headset. There are already a lot of things you can get by yourself, for example, here.

Don’t expect the VR tools to be bug-free right now. Sometimes you will have to invent stuff by yourself, which can be funny!