Editing glsl shaders (specifically, hyllian)

RetroArch Additions Shaders
#1

I’m trying to edit the hyllian shader because I’ve noticed that on some systems (SNES, Genesis) the scanlines vary in brightness, as in, every fifth line or so will be darker than the rest, creating a weird moire-like effect that messes with my eyes. This is running at 720p with integer scale on.

Anyway, I opened hyllian.glsl but I couldn’t understand anything. I’m trying to activate “sharpness hack” and adjust “scanline strength” and “bloom strength.”

Actually, will adjusting these even fix the problem I’m experiencing? Or is this just inevitable when not running in 1080p?

#2

I think the bloom is causing the moire you describe.

#3

good to know- any idea on how to go about altering the bloom strength? The glsl file is just a confusing mess to me :slight_smile: the cg shader was all neatly commented so you could easily adjust the parameters but I can’t even begin to understand the glsl file.

Also curious why I don’t notice the moire-like effect when playing NES but I do notice it on SNES and Genesis.

#4

The GLSL shaders are programmatically converted from Cg and are indeed unreadable spaghetti code. If you can run the conversion script, I find it’s easier just to modify the Cg version and re-convert rather than trying to parse the GLSL versions.

For runtime parameters, the #defines below the ‘else’ in the #ifdef PARAMETER_UNIFORM statement are the values that get hardcoded during conversion, so if you find settings you like, put them in there and they will persist.

#5

Ah, thanks - can you point me toward the conversion script? I haven’t been able to find it on my own.

Edit: Found it-

#6

[QUOTE=hunterk;21902]The GLSL shaders are programmatically converted from Cg and are indeed unreadable spaghetti code. If you can run the conversion script, I find it’s easier just to modify the Cg version and re-convert rather than trying to parse the GLSL versions.

For runtime parameters, the #defines below the ‘else’ in the #ifdef PARAMETER_UNIFORM statement are the values that get hardcoded during conversion, so if you find settings you like, put them in there and they will persist.[/QUOTE]

This might be outside of my skill set, unfortunately. I downloaded the python script, went to the terminal and typed “python cg2glsl.py” and it gave me a syntax error: line 22 print(*arg).

I don’t know how to fix it since I don’t know what the error is.

Edit: well, somehow I’ve broken linux by attempting to install the cgc toolkit from Nvidia, lol. Just couldn’t leave well enough alone. Now I’m in the process of reinstalling everything. I really need someone to hold my hand through this.

#7

Here is the problem I’m seeing with CRT Hyllian:

both of these were taken with RA running at 720p.

Any idea how to fix the moire?

#8

That looks like uneven scaling. Do you have integer scale ON (you mentioned you did) and crop overscan OFF? I’m not having any issues at 720p (1366x768, actually), though I do have some letterboxing at integer scale with Genesis games.

#9

I’m using the same settings for NES, Genesis, and SNES: -integer scale ON -crop overscan OFF -video output resolution: 1080p -retroarch render resolution: 960x720

I notice the problem on SNES and Genesis, but the NES looks perfect for some reason. Did you enlarge the images in the links to full size?

#10

Yes, that seems worse that what the bloom can do. It could be a driver/conversion problem, some shaders do strange stuff when trying them on my android phone.

#11

[QUOTE=Tatsuya79;21947]Yes, that seems worse that what the bloom can do. It could be a driver/conversion problem, some shaders do strange stuff when trying them on my android phone.[/QUOTE]

Well, that’s disappointing if true. Why would the shader work fine with NES but not genesis or SNES? That’s confusing to me.

There just needs to be a way to get simple scanlines of continuous thickness whose brightness can be adjusted. None of the other effects are really important to me. The scanline overlay would suit my needs if it actually lined up with the pixels all the time.

Actually, I was wondering about interlacing.cg- would that convert to glsl and provide simple scanlines at 720p?

Also, speaking of the scanline overlay, I notice that I get perfect results on the NES with render resolution set to 960x720 and video output at 1080p. However, I can never get all the scanlines to line up right with the pixels when using the overlay with Genesis or SNES. Could this indicate a problem with how those cores are handling scaling or something?

I’m beginning to think that a talented shader author needs to show some love for the RPi :-). It’s great that RA has all these options for the power user, but the low power options are kind of lacking right now.

#12

You could probably just mess with the scanline overlay(s) to get it to line up properly. Having a separate one for each system is no big deal.

Interlacing.cg would work, except that it only works at even (i.e., 2x, 4x, 6x, etc.) scale factors. Odd scale factors look pretty crazy with it, unfortunately.

#13

[QUOTE=hunterk;21962]You could probably just mess with the scanline overlay(s) to get it to line up properly. Having a separate one for each system is no big deal.

Interlacing.cg would work, except that it only works at even (i.e., 2x, 4x, 6x, etc.) scale factors. Odd scale factors look pretty crazy with it, unfortunately.[/QUOTE]

I’m starting to think this is a rendering bug with the SNES and Genesis cores in RA- I notice the same problem with caligari, it’s just less pronounced because the scanlines are lighter than in hyllian.

I can get the overlay to line up right with the pixels- sort of- if I increase the scale to 1.07. But this results in an even worse moire effect.

What stumps me is that NES looks perfect- look at the screenshot of SMB. I can get the scanline overlay to line up perfectly at scale 1.00 and there is no moire effect when using hyllian or caligari shaders. This suggests to me that this isn’t a hardware problem but is software related, i.e., something to do with how RA is handling integer scaling for the SNES and Genesis cores.

EDIT: The plot thickens - I managed to get both the scanline overlay to align properly at scale 1.00 and get rid of the moire with CRT-Hyllian on the SNES core with the following settings: -video output: 1080p -retroarch resolution 960x720 -integer scale on -crop overscan ON

The key here seems to be crop overscan. Everything worked like a charm (on SNES) when I turned it on, although now I have letterboxing. As a temporary fix, I just zoom the picture using my TV remote.

The problem appears to be with something crop overscan is doing.

Regrettably, the above settings did nothing to address the problem with the Genesis core.

Conclusion: Crop overscan is broken(?) Enabling it fixed the scaling/moire problem on SNES (albeit with letterboxing) but doesn’t do anything with Genesis. Meanwhile, NES works perfectly with or without it.

Here are some shots of SNES with the above settings. The first shows the Konami logo with the crt-hyllian shader enabled, while the second is a shot of the SMW title screen with the scanline overlay, both displayed perfectly.

I’ve been encountering similar issues with the doom core; I can’t get the scanline overlay to align properly and there is moire with scanline shaders. No combination of settings I’ve tried seems to work. I can get the overlay to align if I use the above settings but then the image is pillarboxed and letterboxed into a small window. For now, I’ve been stretching the image so that it fills the normal 4:3 area using my TV remote :P. It’s silly, but I can’t find anything else that works.

The quake core is an example of a core that works fine out of the box (like NES). I didn’t have to do anything but plop the scanline overlay down, or a shader, and everything looks perfect.

#14

Bump; TLDR version of above post:

There appears to be a bug with crop overscan and/or integer scaling with one or more libretro cores. Scanlines/overlays don’t look right with genesis no matter what combination of settings are used. SNES looks right but only if crop overscan is enabled, leaving ugly letterboxing. Doom looks right but only in a ratio which leaves huge letterboxing. The NES and Quake cores work perfectly for some reason.

I’m using the Linux versions that were installed by the Retropie script. This is with video output at 1080, retroarch resolution 960x720.

I love RA and really appreciate all the work put in by the devs, but is it fair to say at this point that there are still some significant issues to be sorted out with video scaling with the Linux versions of many cores? I’ve spent entire days of my life (weeks?) trying to sort this out.

UPDATE: I managed to get the scanline overlay working with Genesis by setting the RA render resolution to 640x480. It now looks perfect. However, the shaders don’t work right at that resolution or at 960x720 . There does seem to be something going on with integer scaling; NES core for example works fine with shaders at 960x720 but Genesis does not. However, this gives me hope that maybe I can resolve my issues with the other cores by just trying a lower resolution.

#15

I’m more suspicious that it’s related to RetroPie (perhaps it’s pulling an old version or something) because I’m not having any of the behavior you described. In fact, it sounds like crop overscan is doing the opposite of what it should be doing for SNES in your case, as the letterboxing is what gets cropped off when it’s enabled.

All of the main contributors use linuxes as their primary OSes, so it if were as simple as “RA’s scaling is broken on linux,” one of us probably would have noticed.

#16

[QUOTE=hunterk;22005]I’m more suspicious that it’s related to RetroPie (perhaps it’s pulling an old version or something) because I’m not having any of the behavior you described. In fact, it sounds like crop overscan is doing the opposite of what it should be doing for SNES in your case, as the letterboxing is what gets cropped off when it’s enabled.

All of the main contributors use linuxes as their primary OSes, so it if were as simple as “RA’s scaling is broken on linux,” one of us probably would have noticed.[/QUOTE]

I think you’re right re: crop overscan on the SNES; it looks like it’s doing the opposite of what it should. On Genesis, I’m not sure it’s doing anything. With the Genesis core the only way I can get proper uniform pixel scaling is if I set RA’s render resolution to 640x480. I can’t get correct scaling at higher resolutions and crop overscan doesn’t appear to do anything. It’s not a huge deal since Genesis doesn’t really look any better at higher resolutions.

Doom is also of concern; that gets letterboxed into a small window, but I think that’s because it has an internal resolution of 320x200 so it can never fill the screen w/integer scale enabled (unless you have a 1200 pixel tall 1080p monitor, which do exist…)

Anyway, I didn’t mean to jump to conclusions - sorting all this out has been pretty confusing as there are so many variables involved, and I really appreciate all the help you’ve given me thus far. I think you’re correct and that there’s a bug in either one or more of the cores used by RetroPie or in the RetroPie script itself maybe.

here are the versions I’m using:

SNES: 1.1 - SNES9X Next v1.52.4

Genesis: 1.1 - PicoDrive 1.91