Masked Blend Mode Shows Ambient Occlusion from Behind Object

[moosefreak]

Hi I've been trying to create a shader with an opacity mask similar to the standard shader "Cutout" rendering mode, but it appears to be incompatible with screen-space ambient occlusion. Here is a comparison of the Standard Shader and Amplify:

Standard Shader:


Amplify Shader:


Nodes:



The other option I've tried is using a custom blend mode with Opaque and Transparent or AlphaTest but that causes AO to go around the entire object's quad which is almost worse.

[Amplify_Borba]

Hello, thank you for getting in touch!

In order to best help you, we'll need some further information regarding this issue.

What occlusion solution are you currently using in your project, and what are its parameters?

Are you using forward or deferred rendering path?

Looking forward to your reply!

[moosefreak]

Hi, I’m using the Unity Post Processing Stack, the default Ambient Occlusion. i’ve also tried the Cinematic Image Effects Ambient Occlusion with the same issue. I’ve used every parameter with no change in both solutions. I’m using forward rendering, but get the same issue in deffered

[moosefreak]

Here are my Ambient Occlusion settings in the Post Processing Behavior for Unity's Post Processing Stack:


This is what the Normals debug view looks like in the Post Processing Behavior:


This is what the Depth debug view looks like in the Post Processing Behaviour:


What the Ambient Occlusion debug view looks like in the Post Processing Behavior:



So it seems apparent that the cutout (masked blending mode: Transparent-Cutout + AlphaTest) in Amplify does not write to the Normals buffer, causing the problem. Whereas the Unity Standard Shader does.

Is this a bug? It's been like this in the past few versions, (I am fully up to date) not sure if it ever worked before though.

[Amplify_RnD_Rick]

Hey moosefreak,

We've been doing some tests on our end and it seems this is a Unity limitation. But don't worry as we have figured out a way to surpass it.

So what is happening is exactly like you said, the Depth Normals are not being correctly written, as it's ignoring the clipped values.

This issue is only present when rendering in the Forward Rendering path as when on deferred, the G-Buffer is used and it is correctly constructed.

So lets head to how to fix it in Forward. The way you make it write the depth normals correctly is by setting the properties you use on your shader to create the mask, to have the exact name as the ones used by the Unity internal Depth Normal shader and also their usage to also be exactly the same as Unity as well.

It's a bit confusing but here's a screenshot exemplifying what I've said:

AOTest.jpg (178 KiB) Viewed 7991 times

Notice that we've created a Texture Sampler node with the name MainTex and a Color node with the name Color. The Alpha channel of both nodes are multiplied with each other and its result is set into the Opacity Mask port.

Again, for this method to work, both the properties names must be those ones and usage of each nodes alpha channel to create the opacity mask must be that exact way.

Please let us know if you have further questions as we will gladly help you out.

[moosefreak]

Wonderful! Just want to emphasize (for anyone looking for something similar and because it took me a while to figure out) the Color node HAS to be a Property and not a Constant.

Thank you so much for your help.

I do have a question though, if this is a limit of Unity will this never be fixed in Amplify? And does this make it impossible to change the opacity mask with, say an outline and still keep the changed mask from showing ambient occlusion from behind it?

I have a few nodes and a custom expression that produces an outline that I was planning to use for the opacity mask. But it only seems to let the original MainTex write to the normal buffer. Is there any way to fix this, like wrap my nodes up in a "virtual MainTex"? Something like that? Here's an example showing what I'm talking about.

You can still see the AO through the outline, but not the main texture sample.




The nodes may seem confusing, one is custom. But just the idea of manipulating the opacity mask, is that feasible? Thank you SO much for your help so far though.

[Amplify_RnD_Rick]

No problem at all, we're here to help!

Yes, the Color node must be set as a property.

Unfortunately it's not possible to fix this on from ASE due to the nature of the issue. To give a bit more context, Unity when rendering Depth Normals for the Transparent Cutout Render Type executes the following sub-shader.

Code: Select all

SubShader {
    Tags { "RenderType"="TransparentCutout" }
    Pass {
CGPROGRAM
#pragma vertex vert
#pragma fragment frag
#include "UnityCG.cginc"
struct v2f {
    float4 pos : SV_POSITION;
    float2 uv : TEXCOORD0;
    float4 nz : TEXCOORD1;
    UNITY_VERTEX_OUTPUT_STEREO
};
uniform float4 _MainTex_ST;
v2f vert( appdata_base v ) {
    v2f o;
    UNITY_SETUP_INSTANCE_ID(v);
    UNITY_INITIALIZE_VERTEX_OUTPUT_STEREO(o);
    o.pos = UnityObjectToClipPos(v.vertex);
    o.uv = TRANSFORM_TEX(v.texcoord, _MainTex);
    o.nz.xyz = COMPUTE_VIEW_NORMAL;
    o.nz.w = COMPUTE_DEPTH_01;
    return o;
}
uniform sampler2D _MainTex;
uniform fixed _Cutoff;
uniform fixed4 _Color;
fixed4 frag(v2f i) : SV_Target {
    fixed4 texcol = tex2D( _MainTex, i.uv );
    clip( texcol.a*_Color.a - _Cutoff );
    return EncodeDepthNormal (i.nz.w, i.nz.xyz);
}
ENDCG
    }
}

This is rendered independently on what you have on your own shader.
The main line to take into account, which is the one affecting you is:
clip( texcol.a*_Color.a - _Cutoff );

So any type of variation on how the clip is done on your shader that doesn't match that simply wont be taken into account when creating the depth normals.

There's no easy way to turn to overcome this problem, you can either overwrite Unity internal Depth Normals shader, or somehow have the texture set on the MainTex to already have the outline. P.e. having a separate shader which creates the outline and render it into a render texture which is then used on this one.

[moosefreak]

Thanks for the quick reply. They've all been wonderful, thoughtful replies. Very informative and extremely helpful, thank you. I love your product haha.

I'll have to find a workaround, as my texture already is a rendertexture of many different sprites being animated! haha and I don't think I want to use 2 rendertextures.

I'm considering finding a way to manipulate the mask clip value to let me increase the area around the alpha. Might have to add a shadow or something behind it to do that though. Just thinking out loud.

I'm not very experienced modifying shader code (hence your product) so not sure where I'd begin tackling Unity's Internal Depth Normal Shader manually. But you don't need to go in depth into that haha, you've provided me with more than enough help.

Is this something I, you, or we should report to Unity on a bug/issue tracker or something?

[Amplify_RnD_Rick]

That is a really good question on which I really can only give you my personal opinion!
You should report this issue to let Unity know that their implementation is not flexible enough and has flaws, but honestly this is something that is this way for so long that I'm not seeing it fixed on a near future.