ok I guess there was a platform dependent issue which led to different problems on different systems (like the ps3). I think it was a problem with float precision and the constants I used as pattern tags. by eliminating the denominator and rounding the result (practically the thing you did hyllian in the optimization) it should be fixed now. I also reactivated the smoothing of color transitions. the shader only does it now at edges of non pseudo-transparent dither patterns, sfa3 looks fine again on my system. would be nice if someone can confirm it:
I can confirm it is working now here!
it’s probably gonna take a little time but just wanna inform you guys, that I was able to come up with a solution how to detect dithering patterns where the colors are not strictly equal. and I’m gonna reimplement detection of vertical line dithering again. so look forward to see it in action with the waterfalls in sonic, quicksand in tiny toons or the intro of zero tolerance 
1 Like
Sounds awesome, man. I look forward to checking it out 
1 Like
That’s excellent, Sp00kyfox!
1 Like
while I’m screwing with some values here and there to eliminate false detections enjoy these preview shots 
ps: funny enough, this is working without any kind of thresholds. it’s purely based on equality and dot product (see the previous discussion) checks.
1 Like
ok here we go. I had to reconsider every step and implemented a new approach. the new elementary pattern is only 2x2 pixels big but doesn’t compare the colors themselves but the angles of difference vectors (see the discussion on page 3). so it isn’t so strict with the colors anymore. like I already teased in my last post, mdapt now works fine with transparent vertical line patterns with very few errors (you can deactivate it in pass1 though) and behaves except for some details like the previous version, a few false detections are eliminated. I won’t post new screenshots since besides the now finally working vertical line detections there is nothing groundbreaking.
but enough talk, have at you… ahem here is the download ^^ mdapt v2.0
ps: I have a github account now, would be nice if someone of the admins of the repository could gimme access.
1 Like
Looks beautiful here. Hyllian should be able to hook you up with commit access.
1 Like
Tested it on PS3. It worked.
It’s an improvement from the last versions. Though artifacts as those in SMW life “X” are present yet. The quality for checkerboard dithering is superb, as already proved in the last mdapt 4-pass edition.
The pseudo transparency issues seem to be the chalenge to overcome in the next releases.
SpOOky, have you thought about separating mdapt in two independent shaders? One only for dithering and the other just for pseudo transparency?
1 Like
yeah, I just don’t know what to do about it. it consists of classic dithering patterns. the only objective criteria why it shouldnt be merged is the overall size. try increasing the “min” constant in pass2 and see if the overall detection is still good enough.
the thing is that you can’t really decide which one is which. so I don’t like to differentiate it into these categories but more like in strict and nonstrict patterns. where the strict pattern only consists of 2 colors (like a true checkerboard). and the nonstrict variant instead has regular “up and downs” between pixels.
it’s not a good idea to separate those two in different shaders. for example the pseudo transperancy which is used in SFA3 consists of both types, dependent on the current background. not only would you have empty spots this way but it would also influence pass2 (the isolation termination) since the strict ones aren’t counted anymore. the other way around (only strict ones) is not better. even non-transparent dithering pattern often consists of more than 2 colors, so in the end the result would look like a mess of partial dithering blendings.
1 Like
Some systems, like Genesis/Mega Drive, use dithering much more than pseudo transparency, and SNES is the opposite. Other systems, like CPS2, use both. If you can stomach having 3 versions that excel at one, the other or both might be the best/only way to cover all of your bases without too many false positives.
1 Like
the algorithm doesn’t even differentiate between those two types. dithering and pseudo transparency from a post view after the picture is rendered is the same. I wouldn’t even know how to split the shader into these two types. In mdapt there are only strict and nonstrict patterns and to separate them is not a good idea like I explained.
but I could release a strict-only-version though if there is interest, it only requires some minor modifications anyway. so no big deal. the other way around, nonstrict-only just isn’t even worth it.
1 Like
Sp00kyFox,
Do you intend to post an explanation/tutorial about this new approach?yeah I’ll do that after a little break from the topic.
I updated mdapt again. besides some little changes I threw out some parts that didn’t had purpose anymore due to the new approach. also the code should be more readable now (thx for the inspiration hyllian) and faster. by request there is now a strict version available which only accepts two different colors in patterns. don’t use it on games with pseudo transparency effects. I still recommend the standard version.
from now on I’ll post the download to the new version in the first post, where you’ll find a little introduction and new screenshots
1 Like
I restructured the code and finally got rid of all the if-else structures and ?-operators, so no more branches for you gpu. this gives about 25% performance improvement on my machine and the code should be more clear and readable.
mdapt v2.2a mdapt v2.2a strict
edit: due to the new small pattern size, there was an issue with 45° edges which were interpreted as checkerboard dithering. I added a simple edge detection based on already made calculations to prevent it. here is a comparison, on the left v2.2 and on the right v2.2a:
1 Like
been working on some things here and came up with v2.3. so what’s new?
I eliminated most holes in the vertical line patterns and checkerboard with only two or three lines is working correctly. from now on there will be only one version again, the strict one is an option which you can activate in pass0. yeah… to actually terminate redundant calculations I added just another pass. but don’t worry, performance actually benefits from that.
the detection and handling of these two different patterns is vectorized, so pixels aren’t overwriting each other anymore. the (de)activation of them moved to the last shader pass4. overall the number of false detections especially at text should be way lower in contrast to the last version.
1 Like
and another version. while reading about artificial neural networks I noticed that I can easily swap the entire logic of the shader with a fuzzy one. so basicially what I did is to replace all logical ORs and ANDs with max and min calls. additionally I weighted the dot product result in pass0 with the length of the difference vector of the neighbors and I replaced the threshold function in pass2 with a continous sigmoid one. this changes the aprupt nature of the shader to a more smooth one. in other words, many false detections are gone and the ones which are left aren’t so noticeable anymore.
here is a screenshot series which demonstrates what’s going on inside the shader:
original picture pass0: neighbor analysis by dot product of the neighbor difference vectors pass1: pattern rules for vertical lines and checkerboard pass2: throwing away isolated areas pass3: refining the remaing detections pass4: blending
ps: I noticed in another forum that there was some confusion with the pattern recognition in a specific game. by default only checkerboard is activated, but for some games both or only vertical lines are better. so for “tiny toons” I’d recommend VL pattern only:
don’t forget to screw around with the settings if you aren’t satisfied with the results. there is the strict option in pass0, the threshold values in pass2 and the pattern selection in the final pass4.
1 Like
latest version is incredible. Thanks man! Unfortunately I am having a problem when converting to glsl. I am on linux using kms/egl so cg is not an option. Normally your shaders converted and worked perfect but the new one gives me an error. The cg version is working fine but I can’t actually use the glx driver for play only testing. Its unfortunate since I have a great idea for this new version. Perhaps I’ll just have to wait til the cg2glsl script is updated. I’ll just leave the error message below for anyone who might know anything about it. Thanks again.
I am on x86_64 arch linux with latest radeon r600 driver and mesa 9.2.3
RetroArch: Setting RGUI shader: /home/eco/RetroArch/shaders/glsl/dithering/mdapt-v2.4/mdapt.glslp.
RetroArch: Checking GLSL shader support ...
RetroArch: Found GLSL vertex shader.
RetroArch: Found GLSL fragment shader.
RetroArch: Linking GLSL program.
RetroArch: Found GLSL vertex shader.
RetroArch: Shader log: 0:27(2): warning: empty declaration
0:32(2): warning: empty declaration
RetroArch: Found GLSL fragment shader.
RetroArch: Shader log: 0:92(2): warning: empty declaration
0:96(2): warning: empty declaration
RetroArch: Linking GLSL program.
RetroArch: Found GLSL vertex shader.
RetroArch: Shader log: 0:26(2): warning: empty declaration
0:32(2): warning: empty declaration
0:36(2): warning: empty declaration
RetroArch: Found GLSL fragment shader.
RetroArch: Shader log: 0:90(2): warning: empty declaration
0:96(2): warning: empty declaration
0:100(2): warning: empty declaration
RetroArch: Linking GLSL program.
RetroArch: Linker log: error: fragment shader varying TEX0 not written by vertex shader
.
RetroArch [ERROR] :: Failed to link program #1.
RetroArch [ERROR] :: Failed to create GL program #1.
RetroArch [WARN] :: [GL]: Failed to set multipass shader. Falling back to stock.
This looks amazing!
Unfortunately, I get a black screen in both Linux (binary blob) and Windows (OpenGL) with my GTX560. Activating pass by pass, it seems the first pass (mdapt-pass0.cg) displays different colors than in your screenshots. As soon as I activate mdapt-pass1.cg, the screen goes all black.
I tried earlier version (2.2, 2.3), same issue.
to be honest, I don’t know what the problem is. in past version there was in incompatibility with the round function, which I don’t even use in the latest one. I could imagine that the problem lies in the handling of the dot product function, but on another thought it seems unlikely that different gpus are handling it so differently.
I suggest we use another game to compare screens with since it’s kinda difficult to get the same exact picture in SFA3. here is the intro screen from ecco the dolphin (genesis) with mdapt-pass0.cg.
on another thought, maybe it’s a problem that from pass1 till pass3 the fragment shader is only returning a float2 value. doesn’t make problems on my system, but could be on others. could you try out this test version and post screenshots? it only contains the first two passes with hopefully some fixing changes.
ps: is retroarch actually using the alpha channel of the shader result in the graphical display? I’m not sure what happens if this channel is not defined, maybe that’s the reason why the screen gets black. just guessing here.
@timde9
I’m very good with algorithms, not so much with the practical problems of implementations 
I asked for you in the cg2gsl thread, hopefully someone can figure this out. sry for the problems.