By the tone in your wording you look excited, I’m here discussing a thing that the OP started, not on a crusade so I’m not going to discuss this point if you allow me.
I’m not sure if at this point we need to spoon-feed every single argument. Put a name to this:
I understand that it’s not as easy as with hires pseudo-transparency, but as maths and programming illiterate as I am, I’m already testing with masktools to come up with a code that puts my thoughts into paper.
Here I go, one hour quick fiddling. It’s certainly doable:
uv = 3
sstr = 2.7
amnt = 255
uv2 = uv
STR = string(sstr)
AMN = string(amnt)
clp=last
vblur = clp.mt_convolution("50 99 50","1",U=uv,V=uv)
hblur = clp.mt_convolution("1","50 99 50",U=uv,V=uv)
masky = mt_luts (vblur, hblur, mode="avg", pixels=mt_circle(1)+string(0), expr="x y - abs 2 > 255 0 ?" )
vblur = mt_merge(clp,vblur,masky)
# vinverse function sharpening routines
vblurD = mt_makediff(clp,vblur,U=uv,V=uv)
Vshrp = mt_lutxy(vblur,vblur.mt_convolution("1 4 6 4 1","1",U=uv2,V=uv2),expr="x x y - "+STR+" * +",U=uv2,V=uv2)
VshrpD = mt_makediff(Vshrp,vblur,U=uv2,V=uv2)
VlimD = mt_lutxy(VshrpD,VblurD,expr="x 128 - y 128 - * 0 < x 128 - abs y 128 - abs < x y ? 128 - 0.25 * 128 + x 128 - abs y 128 - abs < x y ? ?",U=uv2,V=uv2)
mt_adddiff(Vblur,VlimD,U=uv,V=uv)
(amnt>254) ? last : (amnt==0) ? clp : mt_lutxy(clp,last,expr="x "+AMN+" + y < x "+AMN+" + x "+AMN+" - y > x "+AMN+" - y ? ?",U=uv,V=uv)
It’s hacky, basically get the difference between a vert blur and a horiz blur. You can do it wiser, compare a single 4x1 pixel window against the pixel above, if it matches… blablabla
this is the result:
you’re making mistakes with your statements about the discussed methods and with the usage of terms, so don’t complain if someone corrects and explains it to you.
that’s a correct and well-known example of a pseudo transparency pattern as opposed to the lion king scene. like I already said, mdapt and gdapt can handle both cases:
I can’t reproduce your result, please post a complete script. converting the image to yv24 (so it can be used with masktool functions) and applying your script does nothing:
imagesource("screenshot.png").converttoYV24(matrix="PC.601")
#turnleft() # doesn't help either
#your script
if I understand your approach correctly you’re just ignoring checkerboard patterns. then again they also can be used for pseudo-transparency (like in SFA3).
The way you talk to me is a bit aggressive, just letting you know…
You are lost in a war of terms, pseudo-transparency.… ‘vertical’ dither… in the end of the day it’s the same whatever is the goal it’s used for. They are interleaved columns of the same (vertical) pixels (for dithering reasons) or related (vertical) pixels (for transparency reasons). And the best proof for showing they are the exact same thing is that my code works on both.
Here the image:
Here my code: http://pastebin.com/DuBnH8S9
The reason I exclude other types of dithering patterns (and hence your gdapt and mdapt shaders) is because 1) they don’t expose a problem when used along CRT shaders on top, and 2) either your shaders or vanilla vinverse leave some very ugly and random dots that stand out too much, even with CRT shaders on top.
well that was exactly my point, right?
I wasn’t the one who persisted to strictly separate between these two if you can remember 
can’t say that it works. ratioresize() is a custom method of yours? anyway since I have example pictures in the original resolution (320x224) I guess the line is unnecessary. script loads fine but like before it doesn’t seem to do anything (apart from a little smearing here and there).
CRT shaders already have the ability to blend dithering patterns anyway because of the pixel bleeding they emulate. it doesn’t matter what pattern is used. try for example the crt-royale shader and play around with the beam_horiz parameters. the whole point of an anti-dithering shader is to get the dither blending without the need to emulate a CRT which then allows you to use scalers like xBR which you can’t use on top of CRT shaders.
well that’s the fundamental problem of this task, balancing between detection-rate and error-proneness. that’s why you have a lot of parameters to tweek for yourself. a filter can’t decide locally wheter a specific pattern is supposed to be dither or actual graphical content. you’ll always have blending errors or dithering which wasn’t detected. restricting it to vertical line patterns doesn’t avoid that (think about text, pillars or the hud in your lion king screenshot). I can only repeat what hunterk said, if you want flawless/accurate behavior you’ll need to use CRT/NTSC shaders. the snes-hires case is a special case where you don’t need to put effort in any kind of detection because you can do the same thing on every part of the picture and it works perfectly.
just wanna add, this doesn’t mean I would advise against implementing your own anti-dithering shader. new approaches can always be helpful. only your motivation to do so seems questionable. just forget the thought that there is a perfect and error-free solution for this problem.
What version of avisynth do you use? depending on that you might need to use other masktools versions. The best you can do is test each part, especially the core of my script.
Test both of these, they should be working (blur for each dimension, you can also use blur(1,0) and blur(0,1) too)
clp=last uv=3
vblur = clp.mt_convolution("50 99 50","1",U=uv,V=uv)
hblur = clp.mt_convolution("1","50 99 50",U=uv,V=uv)
It’s not fair you criticize my script, this is not state of the art work, I’m not a programmer and it’s a mere 1 hour job. I simply wanted to picture that with some work it shouldn’t be too hard to get rid of the pseudo-transparency/vertical dither or whatever you want to name it. Bare CRT shaders don’t work on them as nicely as I wished, at least on crt-hyllian which is my default shader (awesome shader btw) as I showed above.
Here I show it again, easily explains my point: http://imgur.com/a/1Vrgh
I know there’s no “perfect” solution, but as a tradeoff I would rather have some forgivable errors shown in the vinverse method than the very dirty drop outs of the anti-dithering gdapt shader for checkerboard dithers. My opinion.
yeah, this works. those lines are pretty similar to the original vinverse by didée which works perfectly. so nope, no problem with masktools here. I uploaded a package with your script and two screenshots here, does this work for you as intended?
you’re the one who wanted to demonstrate your idea and post your work, I’m merely telling you that I can’t recreate your result with your script. if you don’t want critic then delay the release of content until you’re satisfied with it.
as someone who put a lot of hours into discussions and research about the whole topic (see the beginnings of the mdapt thread) and a working solution I say that this is not as trivial as you might think. the process of dithering itself is not something you can reverse without information loss. if you’re going to put more time in testing your approach with different cases you’ll see that simple solutions don’t suffice here.
yeah… can’t have chroma bleeding only in the picture regions you want. it’s all or nothing with CRT shaders.
you can’t avoid these “dirty drop outs” with a vinverse modification either. vinverse handled the jurassic park scenario satisfactorily because of the doubled horizontal resolution. in the normal case (like with genesis games) as soon as you have some alternatiing vertical lines togethers it’ll get smeared (the lion king intro screen is a good test case). especially text is affected, that’s just unavoidable with a method that only acts locally. you could expand the horizontal search range so that in the corresponding picture the incorrect blending disappears, but then again in another scene some dither won’t be detected anymore.
I think you’ll be able to get the kind of pre-crt-shader processing you want by tweaking GTU or else using aliaspider’s tv-out-tweaks shader*. For example, the scanlines from GTU can be disabled by tweaking a value (I forget which one; the code is commented), and you can fine-tune the blur in both vertical and horizontal directions by modifying the respective bandwidth values. Blending Genesis dithering takes some pretty blurry settings, but I think it’s about as good as you’re going to get without setting up expensive and error-prone detection methods like Sp00kyFox used for mdapt/gdapt.
*This shader is a subset of GTU that’s intended to blend dithering and pseudo-transparency–as well as some color stuff–before pushing it out to a CRT monitor/TV, but I think it should work pretty well feeding into a CRT shader, as well.
Yes, your zip content works here. Only the lion king example, the green part of the landscape is not as good as the one I posted above because I was using yv12 (focusing only on luma). Maybe you would want to check at what step masktools returns the same clip without changes, there was some masktool version compatibility mess a year ago with the release of avs 2.6b5 or so maybe the one you use is not right, or not, I don’t know.
I do 3D, graphics and video work, I don’t do programming nor maths so this is as much I will do with my vinverse mod. I wanted to try to do a 4x1 pixel matrix comparison window, I guess it’s doable with mt_luts but didn’t managed to find out how. You compare it with the pixel row above, if it’s the 100% exact (as it happens with no checkerboard dither) then you’ve got a match, for dither to work they should be in a narrow range of pixel values so you don’t get things like text for example smeared, do a few more negative checks and I think it’s pretty much worth a shader. This is what I would try if I knew how.
The point of the thread was to look for something like snes-hires shader but sharper (or rather, localized), so using GTU or some kind of dumb horizontal blur shader is kind of a moot point, I can check if even a dumb blur with low values is somewhat better, for the time being or forever since no one shares my vision lol.
edit: out of curiosity I tested with the intro screen, I’m surprised how well it works for being a sloppy piece of code. It doesn’t affect the outcome (much) but we were doing it a bit wrong, it should be ConvertToYV24(matrix=“pc.601”)
For reference/comparison, this is how blurry GTU needs to be to properly blend horizontal and vertical dithering patterns on Genesis/MD:
This is a signalResolution of 164.0 and a tvVerticalResolution of 125.0
yeah, I guess it’s an incompatibility with all the different avisynth modifications and custom libraries. no offense though if I sounded too harsh, just wanted to give some feedback.
reminds me of my development process, I had similar ideas. but then some test case pops up which ruins the day. at the end of a stage in Sonic if you jump through the big golden ring some score text (thousand or hundred) appears and the hole in the zeros aswell as the gap between the digits is only one pixel wide. so this approach would blend it. what do you do? one keeps extending the horizontal search range and must notice that in some other scenes the dithering won’t be blended anymore. it’s a mess :lol: if you look at the parameters of gdapt in the shader menu there is an “error prevention lvl”. this is actually just the minimum count of alternating pixels which need to be present so that the shader blends them.
I’d suggest you play around with gdapts code. it’s rather basic in most parts and I think can be modified to implement your ideas without too much struggle. you need to know how the C syntax works though. feel free to ask if you need help
I see what I can do myself in the meantime.
I was using the yv24 colorspace all along for your script. I only used YUY2 for vinverse() since I had an old implementation of this filter by tritical which only supports this and YV16. the newest implementation with more supported color formats is here if you need it too:
“the gap between the digits is only one pixel wide. so this approach would blend it. what do you do? one keeps extending the horizontal search range and must notice that in some other scenes the dithering won’t be blended anymore”
Yes I know, I realized that alternating pixels are not always close in pixel values as seen in the intro screen, so as you say it can clearly affects text. You can mask out those using the gdapt algo which doesn’t blend text (but still blends vertical dither right?). For edges in avisynth there’s a function called mt_expand which expands the mask, so that can level things a bit on edges where the algo stops working. One actually never knows until tested, dealing with no pseudo-transparency vertical dither looks to be a rather easy work using the pixel matrix window approach ( “x y x y” color pattern) but things get hairy with pseudo-transparency since as shown in the Sonic gif there’s no correlation in horizontal nor vertical dimensions in pixels. In this case my hacky code above works because it uses a loose approach, maybe that’s the way to go, refining things a bit specially to minimize artifacts on the not affected areas. You can even mix your current gdapt work, acting this as a filter mask that defines where the vinverse blur approach could apply (you save all the vinverse resharpening code block), that would get rid of most artifacts, performance issues aside ATM. Check below the reworked script*.
Or use the window diff approach for dither and the blur approach for transparency (totally different no stackable shaders), since the latter seems to be a rather rare effect in megadrive, correct me if I’m wrong. Just throwing ideas. Maybe I should learn C, I promised myself to never learn programming, I guess I failed since I do some scripting here and there lol
*This is the script a bit more optimized. Without the resharpening code block gdapt detection algo can mask out the left undesired blurring in text and sprites. This handles planes better as I would expect in a shader so the blurring is more effective (ie. blue clouds)
original…filtered
no, who told you that? gdapt has exactly that problem (depending on the settings). in the first pass it tags a pixel if it and its neighbors are alternating. in the second pass it blends them under the condition that a certain amount of those tagged pixels are in a horizontal line (determined by the “error prevention level” parameter). in the case of the sonic example lvl 3 is enough to avoid the blending. and I was mistaken. there is actually a two pixel hole in those zeros. otherwise that would be pretty much the worst case szenario. even a “10” alone then would already have 5 alternating pixels in a row (the 1, gap between 1 and 0, left line of 0, hole, right line of 0).
your idea that only the vertical line patterns are dealt with can be implemented by adding a vertical cohesion. in the case of gdapt I’d add an additional check in the first pass that the pixel above or below must be of the same color. like you said though that would break the pseudo transparency blending. if I use similarity instead the checkerboard pattern in the lion king level 1 sky would be blended again since the color values are too near together. and just to mention it, in level2 there is actually some pseudo transparency going on in the water which is a mix of vertical lines and checkerboard. that’s the problem with developing a filter with only one test scene.
since neither approach fixes the mistakes of the other one using a masked mix wouldn’t give as any improvement. I guess you overestimated gdapt in that regard
gdapt dither detection is miles better than my approach, I tested and it protects the text and sprite of this screenshot at least (check below). With that I do a mask against my horizontal blur approach which respects checkerboard dither.
orig…gdapt…dogway.approach…d.approach+gdapt.mask
…
……
This is the script used: http://pastebin.com/qxgcfHK1 edit:Updated script to also include the vinverse mod approach, all is left is include the gdapt dither detection routine: http://pastebin.com/65w1Am1L
here a zip with the script and screenshots: anti_stripe_dither.zip
I know I’m ignoring whether this is performance friendly or even mappeable to retroarch’s shader language. All I am saying is that it’s not something conceptually impossible.
still don’t know where this ratioresize() is coming from. and this pack works fine. the problem is just that I don’t see how you came up with the “dogwar approach.png” in the first place. I can’t recreate the result with your script:
clp = ImageSource("raw.png",start=0,end=0).ConvertToYV24(matrix="pc.601")
vblur = clp.mt_convolution("50 99 50","1",U=3,V=3)
hblur = clp.mt_convolution("1","50 99 50",U=3,V=3)
masky = mt_luts(vblur, hblur, mode="avg", pixels=mt_circle(1)+string(0), expr="x y - abs 2 > 255 0 ?")
vblur = mt_merge(clp,vblur,masky)
mt_lutxy(clp,vblur,"x y - abs 13 < x y ?")
ConvertToRGB32(matrix="pc.601")
the raw.png I use has the correct resolution of 320x224. first thing I think is weird is that the variable names for the two blurs are swapped. in your script “vblur” is actually an horizontal blur while “hblur” is the vertical one, but that’s irrelevant in the end. the main issue though is the “masky” definition. I guess you try to mark checkerboard pixels with that line. but I don’t see how that should work (and it doesn’t, at least for me) and why you need an horizontal and vertical blur for that. also the argument “pixels” should be a list of 2-tuples, why are you adding a single “0” to the list? this is how “masky” looks like:
anyways, if you really just wanna avoid blending of checkerboard pixels one can add a simple addtional condition in the first pass of gdapt which checks if the orthogonal neighbors of the pixel are all identical. in that case the tag (for the blending in the second pass) is denied. here, replace the fragment shader of pass0 with this:
/* FRAGMENT SHADER */
float4 main_fragment(in out_vertex VAR, uniform sampler2D decal : TEXUNIT0, uniform input IN) : COLOR
{
float3 C = TEX( 0, 0);
float3 L = TEX(-1, 0);
float3 R = TEX( 1, 0);
float3 U = TEX( 0,-1);
float3 D = TEX( 0, 1);
float tag = 0.0;
if(MODE){
tag = all(L == R) && any(C != L);
}
else{
tag = dot(normalize(C-L), normalize(C-R)) * eq(L,R);
}
if(all(L == R) && all(U == D) && all(L == U))
tag = 0.0;
return float4(C, tag);
}
with this modification you’ll get this on lion king:
are you happy now? 
yeah, var names are swapped lol, didn’t bother to change. The rationale was that vertical blur wouldn’t make a difference on vertical striped dither unlike checkerboard dither, so doing a diff mask would protect everything except vertical stripes.
Ratioresize() is a custom function to resize images, as you said before, in this case to original 320px width size. I thought you would ask about the “Dogway approach.png”, so I swapped it with the real code, which is no less the same than this but in any case for visual simplicity I added the whole thing together in the updated script (second pastebin) of my above post. +string(0) is a no-op (or should be), it’s a left over of my testings you can remove. mt_square(1) or circle means a 3x3 matrix. All this just shows how limited my knowledge in maths is. I would say your mask is wrong, this is easy to check, the equation says that if diff between x and y is over a value of 2, then map to white, if you compare both hblur and vblur outputs there are areas with higher values than 2 that are not mapped to luma white in your mask, (or lower than 3 that are not mapped to 0). I asked you before what AVS version you had to rule out possible issues, maybe you are using still 2.5.8, avs+, or maybe vapoursynth, or maybe an outdated version of masktools, these are my outputs. Anyway and unrelated to the issue in avisynth I recommend you to use the YV12 interleaved RGB format instead of YV24 as the process is closer to what would happen in RGB, otherwise you would need to throw a few more “u=3,v=3” to some lines. The most important value in my script is the mt_binarize, for the intro screen I used 25, for the in game I used 13.
Those images are very good, with much simpler code, it has a hard time on the first one, but considering how cheap you managed to make it work with near results to mine I could call it a winner, I tested on several games and it looks good, pseudo-transp included. I’m glad I converted you to a believer 
I couldn’t implement it to my default shader though, tested several combinations:
shader0 = "./gdapt-new/passes/gdapt-pass0.cg"
filter_linear0 = false
scale_type0 = source
srgb_framebuffer0 = true
shader1 = "./gdapt-new/passes/gdapt-pass1.cg"
filter_linear1 = false
scale_type1 = source
srgb_framebuffer1 = true
shader2 = "./crt-hyllian-glow/linearize.cg"
filter_linear2 = false
srgb_framebuffer2 = true
shader3 = "./crt-hyllian-glow/crt-hyllianmd.cg"
filter_linear3 = false
scale_type3 = viewport
scale3 = 1.0
srgb_framebuffer3 = true
shader4 = "./crt-hyllian-glow/threshold.cg"
filter_linear4 = false
srgb_framebuffer4 = true
shader5 = "./crt-hyllian-glow/blur_horiz.cg"
mipmap_input5 = true
filter_linear5 = true
scale_type5 = source
scale5 = 0.25
srgb_framebuffer5 = true
shader6 = "./crt-hyllian-glow/blur_vert.cg"
filter_linear6 = true
srgb_framebuffer6 = true
shader7 = "./crt-hyllian-glow/resolve3.cg"
filter_linear7 = true
I’m using the multithreaded version of avisynth 2.6 here (Release Thread) and a correponding version of masktools (Forum Post). masktools and common scripts (QTGMC or SMDegrain for example) which are using it are working perfectly fine.
YV12 and YV24 are both planar formats, only the chroma resolution is different. YUY2 is an interleaved format but one would need to stretch the image again to avoid information loss at the format conversion from RGB to YUY2, like I did in my previous example screenshots where I was using an old vinverse implementation.
but well, the avisynth issue doesn’t matter anymore anyway.
thanks. I think I increased the “error prevention lvl” parameter for the screenshots, so with the standard value it should get all of the dithering.
use this as a preset for further configuration: https://github.com/libretro/common-shad … /gdapt.cgp
Yes, it should work. In any case I’m using this build which has a few bugfixes and optimizations.
I used the preset but I rarely manage to stack shaders to crt-hyllian-glow, everything goes fine until the last resolve.cg, which simply throws away everything and outputs source.
shader0 = "./gdapt-new/passes/gdapt-pass0.cg"
filter_linear0 = false
scale_type0 = source
scale0 = 1.0
shader1 = "./gdapt-new/passes/gdapt-pass1.cg"
filter_linear1 = false
scale_type1 = source
scale1 = 1.0
shader2 = "./crt-hyllian-glow/linearize.cg"
filter_linear2 = false
srgb_framebuffer2 = true
shader3 = "./crt-hyllian-glow/crt-hyllianmd.cg"
filter_linear3 = false
scale_type3 = viewport
scale3 = 1.0
srgb_framebuffer3 = true
shader4 = "./crt-hyllian-glow/threshold.cg"
filter_linear4 = false
srgb_framebuffer4 = true
shader5 = "./crt-hyllian-glow/blur_horiz.cg"
mipmap_input5 = true
filter_linear5 = true
scale_type5 = source
scale5 = 0.25
srgb_framebuffer5 = true
shader6 = "./crt-hyllian-glow/blur_vert.cg"
filter_linear6 = true
srgb_framebuffer6 = true
shader7 = "./crt-hyllian-glow/resolve.cg"
filter_linear7 = true
edit: for some reason works (partly, glow is a bit bugged) with this (http://pastebin.com/r0xWjeN1), but not with this (http://pastebin.com/5ybYT2w9)
on the server the last pass is called “resolve2.cg”. make sure you are using a recent copy of the shader repository: https://github.com/libretro/common-shad … master.zip
now in resolve2.cg replace this line
#define CRT_PASS PASS2
with this
#define CRT_PASS PASSPREV4
this makes the shader reference relative so you can put as many shaders before the CRT one as you want.
Yes, I compared both and changed that and still, eventually worked. This is utter perfection. I’m going to update all the shaders from master and check if everything is working fine as before. Thanks for the help.