I will continue working on this. At 4K the same issue that was causing the pink horizontal lines at 1080p was showing up as bands of darker, uneven scanlines. I was able to eliminate this by adjusting the scanline parameters. Basically the more defined and separated the horizontal scanlines are the less likely to have those issues. That’s why lowering the resolution, changing the int. scale mode to 1, cropping the black bars at the top and bottom of the screen and lastly adjusting the scanlines parameters all had the effect of reducing or eliminating those anomalies.
It’s also easy to get less pixelation by lowering the Opp Dir Scanline Pre-Scale setting to 75% or 50%. 75% results in uneven scanline alignment which affects text quite a bit. 50% results in even scanline alignment but the text looks much lower resolution. At least it’s readable and not garbled like when you set internal resolution to 320 x 240. All of this is at 640 x 480 native internal resolution. At this resolution the Scale FX doesn’t really do much to the sprites so most of the smoothing is as a result of the scanlines and mask.
Once you use any resolution other than native the smoothing seems to look better. 1280 x 960 seems to be a good option however, the big issue that remains is once you leave native resolution Ryu’s Super Hadoken has vertical gaps in it.
So as it stands, it is possible to improve the look of things on the Flycast in high resolution games with pixel graphics and even in low res. games. I didn’t want to mention it until I got a fix because once you see something you can’t unsee it but I could see the pink lines in @Tromzy 's last set of screenshots. I think I even saw vertical pink lines. It shouldn’t be like that. I just didn’t want to mention it until I came up with a fix.
If I can’t find a better solution I’ll have to start with the solution that looks both sharp and smooth but leaves Ryu’s Super Hadoken messed up. I don’t think you should have to give up the nice high resolution elements just to get those sprites straightened out though.
Too bad scale FX wasn’t designed to work on hi res elements. I guess the best option might be some sort of SSAA effect.
By the way, I updated to RetroArch 1.9.13 and it seems to be slightly slower than 1.9.12. I also updated my installed cores and guess what? My Flycast core options still show the Cable Type. I think I might revert to 1.9.12 due to the performance regression though.