Mdapt & gdapt - dithering treatment [Updated 06/06/14]

RetroArch Additions Shaders
#136

been working on some things here and came up with v2.3. so what’s new?

I eliminated most holes in the vertical line patterns and checkerboard with only two or three lines is working correctly. from now on there will be only one version again, the strict one is an option which you can activate in pass0. yeah… to actually terminate redundant calculations I added just another pass. but don’t worry, performance actually benefits from that.

the detection and handling of these two different patterns is vectorized, so pixels aren’t overwriting each other anymore. the (de)activation of them moved to the last shader pass4. overall the number of false detections especially at text should be way lower in contrast to the last version.

mdapt v2.3

1 Like
#137

and another version. while reading about artificial neural networks I noticed that I can easily swap the entire logic of the shader with a fuzzy one. so basicially what I did is to replace all logical ORs and ANDs with max and min calls. additionally I weighted the dot product result in pass0 with the length of the difference vector of the neighbors and I replaced the threshold function in pass2 with a continous sigmoid one. this changes the aprupt nature of the shader to a more smooth one. in other words, many false detections are gone and the ones which are left aren’t so noticeable anymore.

mdapt v2.4

here is a screenshot series which demonstrates what’s going on inside the shader:

768×448
original picture

768×448
pass0: neighbor analysis by dot product of the neighbor difference vectors

768×448
pass1: pattern rules for vertical lines and checkerboard

768×448
pass2: throwing away isolated areas

768×448
pass3: refining the remaing detections

768×448
pass4: blending

ps: I noticed in another forum that there was some confusion with the pattern recognition in a specific game. by default only checkerboard is activated, but for some games both or only vertical lines are better. so for “tiny toons” I’d recommend VL pattern only:

don’t forget to screw around with the settings if you aren’t satisfied with the results. there is the strict option in pass0, the threshold values in pass2 and the pattern selection in the final pass4.

1 Like
#138

latest version is incredible. Thanks man! Unfortunately I am having a problem when converting to glsl. I am on linux using kms/egl so cg is not an option. Normally your shaders converted and worked perfect but the new one gives me an error. The cg version is working fine but I can’t actually use the glx driver for play only testing. Its unfortunate since I have a great idea for this new version. Perhaps I’ll just have to wait til the cg2glsl script is updated. I’ll just leave the error message below for anyone who might know anything about it. Thanks again.

I am on x86_64 arch linux with latest radeon r600 driver and mesa 9.2.3


RetroArch: Setting RGUI shader: /home/eco/RetroArch/shaders/glsl/dithering/mdapt-v2.4/mdapt.glslp.
RetroArch: Checking GLSL shader support ...
RetroArch: Found GLSL vertex shader.
RetroArch: Found GLSL fragment shader.
RetroArch: Linking GLSL program.
RetroArch: Found GLSL vertex shader.
RetroArch: Shader log: 0:27(2): warning: empty declaration
0:32(2): warning: empty declaration

RetroArch: Found GLSL fragment shader.
RetroArch: Shader log: 0:92(2): warning: empty declaration
0:96(2): warning: empty declaration

RetroArch: Linking GLSL program.
RetroArch: Found GLSL vertex shader.
RetroArch: Shader log: 0:26(2): warning: empty declaration
0:32(2): warning: empty declaration
0:36(2): warning: empty declaration

RetroArch: Found GLSL fragment shader.
RetroArch: Shader log: 0:90(2): warning: empty declaration
0:96(2): warning: empty declaration
0:100(2): warning: empty declaration

RetroArch: Linking GLSL program.
RetroArch: Linker log: error: fragment shader varying TEX0 not written by vertex shader
.
RetroArch [ERROR] :: Failed to link program #1.
RetroArch [ERROR] :: Failed to create GL program #1.
RetroArch [WARN] :: [GL]: Failed to set multipass shader. Falling back to stock.
#139

This looks amazing!

Unfortunately, I get a black screen in both Linux (binary blob) and Windows (OpenGL) with my GTX560. Activating pass by pass, it seems the first pass (mdapt-pass0.cg) displays different colors than in your screenshots. As soon as I activate mdapt-pass1.cg, the screen goes all black.

I tried earlier version (2.2, 2.3), same issue.

#140

to be honest, I don’t know what the problem is. in past version there was in incompatibility with the round function, which I don’t even use in the latest one. I could imagine that the problem lies in the handling of the dot product function, but on another thought it seems unlikely that different gpus are handling it so differently.

I suggest we use another game to compare screens with since it’s kinda difficult to get the same exact picture in SFA3. here is the intro screen from ecco the dolphin (genesis) with mdapt-pass0.cg.

on another thought, maybe it’s a problem that from pass1 till pass3 the fragment shader is only returning a float2 value. doesn’t make problems on my system, but could be on others. could you try out this test version and post screenshots? it only contains the first two passes with hopefully some fixing changes.

ps: is retroarch actually using the alpha channel of the shader result in the graphical display? I’m not sure what happens if this channel is not defined, maybe that’s the reason why the screen gets black. just guessing here.

@timde9 I’m very good with algorithms, not so much with the practical problems of implementations :smiley: I asked for you in the cg2gsl thread, hopefully someone can figure this out. sry for the problems.

#141

Hi Sp00ky,

Thanks for looking into it. Here is what I get with pass0 (still different colors from what you get) http://www.denki-den.com/tmp/mdapt-ecco-pass0.png

And with pass0 and pass1 together (green screen, start flashing bright green as demo starts) http://www.denki-den.com/tmp/mdapt-ecco-pass0and1.png

Same results confirmed both on Windows and Linux. Don’t lose too much sleep over it, I understand shaders can be really hair pulling stuff between nVidia, AMD, Cg, d3d9 Glsl etc… I just wanted to know if other people have the same problem. Your earlier mdapt-4p from the repo seems to work alright so I’ll use that instead =)

#142

seems like there is something weird going on with the color channels. the way pass0 is intended is that the red channel marks a horizontal dithering pixel, green one the vertical direction and blue is just the min of the red and green intensity (which together with red and green gives the white areas). ok it seems that for example many pixels which should be green are instead kinda cyan’ish (which is rgb(1,1,0)). I took it for granted that values outside the [0,1] boundaries are automatically clamped in the pass output, but I could be wrong. so my guess would be that in this case for the pixels the red channel is negative and has a very small absolute value which on your system gets interpreted as a high positive value. same argument for the magenta pixels. let me see what it does if we clamp the output. here is another modification for you to test.

it’s weird though that the third channel (minimum of red and green) doesn’t seem to be affected from this. if red or green is a negative value, the minimum should also be and all these cyan and magenta pixel should be white instead. anyways, check out the test version while I’m still thinking about the problem ^^

#143

It seems to be working well here on my AMD GPU, FWIW.

#144

well no wonder here I guess. as the dev of the shader my GPU is also from AMD. well let’s wait tanuki is saying to the new test version. would be nice if you could look into the issue with cg2gsl.

#145

I’ll certainly take a look at it. It’s maister’s script, and I don’t have much experience with python, but maybe I can at least pinpoint what’s going wrong.

#146

I tried the new test file, but the problem seems to remain both in Linux and Windows: http://denki-den.com/tmp/mdapt-test2-ogl-pass0-ecco.png

The problem is with nVidia OpenGL somewhere since the shader works as intended with d3d9 on Windows: http://denki-den.com/tmp/mdapt-24-d3d-pass0-ecco.png

Talking about which, what driver do people recommend on Windows? d3d9 or gl?

#147

yeah, I don’t really get it. even if you don’t understand what I’m doing there, in the end I’m calculating two values r and g and I’m outputting R=r, G=g and B=min(r,g). so magent’ish (rgb(1,0,1)) or cyan’ish (rgb(0,1,1)) pixels shouldn’t be even possible, since it contradicts with the definition of the blue channel. maybe we’ll figure it out in time.

#148

the new fuzzy logic which I implemented in the last version led to some little holes in the checkerboard handling. so I added a few new rules. and another thing, the output pixel is now a linear combination of the original one and the merged sum, weighted by a probability value that this pixel is indeed part of a dithering pattern. this fits to the new smooth non-aprupt behavior of the shader.

mdapt v2.5

if anyone is interested, personally I like this shader combination.

shaders = 7

...

shader5 = 2xBR-Hybrid-v5-gamma.cg
filter_linear5 = false
scale_type5 = source
scale5 = 2.0

shader6 = advanced-aa.cg
filter_linear6 = true
scale_type6 = source
scale6 = 2.0

you can swap out shader6 though with ddt-extended.cg if you like it a little sharper.

#149

awesome man, stil not working in glsl format though. I am dying to use this as i recently figured out how to use 12 pass shaders in RA instead of just 8 and this would be perfect with a little combo I have been wanting to share but I was waiting for a less false positive prone version of this shader. Just have to wait a little longer I suppose. Thanks yet again for this kick ass shader though.

#150

sounds interesting, I’m looking forward to your shader combo. too bad I can’t help with the cg2gsl problem. if someone figure it out I’ll modify the shader accordingly.

#151

It might make sense to maintain a list of games that have been tested with it, and which work without any false natives/positives. I believe it would be impossible for it to work perfectly with every game possible since there’s so many thousands of different possible dithering patterns. But it could work perfectly for a more finite amount of games.

#152

I don’t know if a list makes too much sense since you can easily tweak the shader settings so that in the specific game the number of false positives is minor or they’re hard to notice. it’s also very dependent on the specific scene. maybe the game looks good at the start and then there is one scene in the midgame where mdapt screws up. overall the standard settings were cautiously chosen.

so a list with settings would probably make more sense. on the other hand, such a list could easily be outdated with a new version (not that I have new ideas atm) and do you really wanna compile a list with hundreds or thousands of games? :smiley:

yeah it’s convenient to throw mdapt on every game but it’s probably better to only use it on games where you know that it uses dithering constantly.

#153

I tweaked mdapt a little further. no new features here but overall a better behavior of the algorithm. the most important change is a new rule for the vertical line pattern. the waterfall in sonic looks flawless again like before the introduction of fuzzy logic.

here is the download and a detailed changelog:

mdapt v2.6

Update 12/06/13:

  • added a new vertical line rule in pass1 to fix some gaps
  • replaced euclidean distance with a fast color metric
  • strict option in pass0 automatically changes behavior in pass1
  • replaced sigmoid function in pass2 with the more suitable tanh (approximation)
  • removed a too aggressive checkerboard rule
#154

I’ve gone through every pass to check for more improvements and funny enough I seem to have overlooked something very basic. mdapt actually never cross checks the horizontal and vertical colors in the CB detection. this is of course a source for false detections. for example this pattern

  R
G B G
  R

in v2.6 gets the strongest CB value of 1.0, in v2.6a the value is now significantly lower corresponding to the color metric value of R-G. apart from this there is nothing new. I think this will be it for 2013 :wink:

mdapt v2.6a

#155

Awesome job! Is this newest shader set batch included with the newest version of PS3 retroarch or do I need to download it separately? I ask because I tried your link and it says the archive is corrupted, etc.