Blending in moss / dirt texture using vertex color and a heightmap [Unreal Engine 4]

General / 11 January 2018

Hi all! 

I have started doing some pre-work / research for my graduation project which officially starts in February. For my graduation project I will be building a 3D environment, complete with a story bible that (roughly) outlines the overall world in which the environment resides, and a style guide that describes the rules the environment will adhere to. The working title for the project is 'Isles of Yahlin' for now. I will go more indepth into the actual environment and story in the near future, but for now all you have to know it's a misty, tropical island scattered with ruins from a lost civilization. I'll be using this blog to document my progress and maybe help some of you in the process! 

I have been trying out and experimenting with a ton of different things, with the goal of having a decent toolkit of materials, models and techniques I can pull from once I really started working on the environment. Today I wanted to experiment with vertex blending. For those who don't know what vertex blending is, it's a technique used to blend in different colors or textures on top of a mesh in your environment. Ofcourse you already can blend different textures in your actual material, however if you were to apply the same material on two different objects both will have the same texture blend appear. The added benefit of vertex blending is that it uses the actual vertices on each individual mesh. This way, if you have for example 3 copies of the same statue with the same material, all 3 of them can have moss or cracks of whatever texture you put in appear on them on different places. This is a very common technique used in games as it allows you to make repetition and tiling less obvious. 

Do keep in mind that everything you see here is still placeholder / experimental! Lot's of models and textures are still work in progress.

Below you can see a very simple setup in UE4 for blending moss on a rock model using vertex color. In this case the red vertex color channel acts as a mask. I use the red channel as an alpha mask in the lerp that blends together the rock texture and the (tiling) moss texture. So when vertex painting with a red value of 0 you will expose the moss texture.

And the result:

As you can see this can be an effective way to partially hide mesh intersections. This rock mesh was intersecting with the terrain beneath it. I have the landscape material set up to have the same moss textures as the vertex blended moss, making the seam less obvious.

However, this is still very basic. From here there are multiple ways you can add things to make it a bit more interesting and usefull. For now I have tried making the places the moss appears more natural, in the cracks and crevices! For this I extracted a heightmap using Substance Designer which I can utilize for exacly that. As Substance Painter can as of the time of writing not bake height maps (I used Painter to texture the rock, baked from a high poly mesh from Zbrush), I used a handy node in Designer called 'Normal to Height HQ'. This creates a heightmap from the normal map I was already using anyway that is decent enough to use for this purpose. All textures are still placeholder for now so I didn't feel like putting too much time in getting a well baked heightmap. Some level adjustments... and voilà!

This was actually done with a normal map from a slightly different tiling rock texture... but you get the idea. This one is also too high resolution for how it is utilized, for maps that only really function as a mask like this one you could easily use a 512/512 texture, or lower. What this texture basicly does is mask out cracks and lower lying areas where moss would be better protected from the sun and moisture would accumulate.

Here we just add this new mask texture with the red channel from the vertex color, and clamp it so it doesn't exeed a value of 1, which would make our colors look a bit funny. Then we use the result from that clamp as an alpha mask in the lerp we created earlier. Now when we paint on a mesh using this material...

Already looking a lot more interesting! In this material I already set up the normals / roughness maps the same way so they'll also blend in nicely.  Just set the erase color to anything with a red value of 1 and the paint color with a red value of 0 and you're good to go. I'll probably improve this material in the future. With this current set up, a rock can't be fully covered by moss and will always leave parts of the rocks exposed making it less ideal for hiding mesh intersections. This shouldn't be hard to fix however and for a material this simple it already is a nice result.  I'll need lot's of large cliffs later on so this will already help me break the repetition up a bit that comes with using tiling textures.

I've been playing around with materials a lot lately, but this was the first time I dived a little bit deeper into using vertex colors. Any tips & tricks you know of are always welcome, I'm still learning!