Problems with photoshop-like "lighten" fragment shader

Question

I'm tagging this in OpenGL because I think it's relevant to that space (albeit a different naming).

I'm trying to get photoshop-like blending to work in Flash 11 which uses OpenGL (ES2?). I'm having particular issues with lighten, which I believe should be:

outputColor.rgb = max(base.rgb, blend.rgb);

I believe I am having issues because my layers have transparency. However, even when I pre-multiply my layers, I still am having issues (the image is too dark still).

Am I missing a step here? (This is AGAL, Adobe's gl assembly)

[Shader.Fragment]
tex ft0, v0, fs0 <2d,nearest,nomip,clamp>
tex ft1, v0, fs1 <2d,nearest,nomip,clamp>

// premultiply
mul ft2, ft1.xyzw, ft1.wwww

// premultiplied values
max oc, ft0, ft2

score 3 · Accepted Answer · answered Oct 04 '11 at 16:29

3

Pay attention, you are also multiplying alpha by itself.

So instead of having ( r.a, g.a, b.a, a ), you are outputting ( r.a, g.a, b.a, a.a ), which will lead to darker image if blended with a black background

You may try doing this :

// premultiply
mul ft2, ft1.xyzw, ft1.wwww
mov ft2.w, ft1.w

// premultiplied values
max oc, ft0, ft2

answered Oct 04 '11 at 16:29

crazyjul

2,519
19
26

Yeah I forgot to mention that I tried moving that too, or even setting it to 0 -- however, still is incorrect. I think it might have to deal with the original image having pre-multiplied values ... – ansiart Oct 04 '11 at 17:41
You may be right, Flash premultiply its images, try demultiplying them, maxing out the remultiply. What opengl blending state are you using? – crazyjul Oct 05 '11 at 04:36
src_alpha:one_minus_src_alpha. Yes, it's the pre-multiplied values that are screwing it up. Looks like i'm gonna have to either framebuffer all my textures or manually unmultiply them. Thansk for your help! – ansiart Oct 06 '11 at 02:00
2

If you are outputing premultiply color, don't forget to put the renderstate to one:one_minus_source_alpha. If not the blend equation will be `a*Cs+(1-a)*Cd` where `Cs = a*Ct` thus giving `a²*Ct + (1-a)*Cd` ( Ct = texture color, Cs = color output in pixel shader, Cd color in the back buffer ) – crazyjul Oct 06 '11 at 06:15

skozin · Answer 2 · 2013-01-19T10:50:21.840

As crazyjul mentioned, you have to restore original color values from alpha-premultiplied ones. I believe that the following code should work:

tex ft0, v0, fs0 <2d, nearest, nomip, clamp>
tex ft1, v0, fs1 <2d, nearest, nomip, clamp>

max ft0, ft0, fc0
div ft0.xyz, ft0.xyz, ft0.www // obtain original color 1

max ft1, ft1, fc0
div ft0.xyz, ft0.xyz, ft0.www // obtain original color 2

max ft1.xyz, ft0.xyz, ft1.xyz // lighten

mul ft1.xyz, ft1.xyz, ft1.www // pre-multiply output color with the second alpha
mov oc, ft1

Lines max ft0, ft0, fc0 and max ft1, ft1, fc0 are necessary to avoid divizion by zero alpha value. You can set fc0 constant to contain smth like 0,0,0,0.001.

Also note that we use alpha of the second layer (ft1) as the resulting alpha. I don't know whether it is correct.

Problems with photoshop-like "lighten" fragment shader

2 Answers2