Dogway's grading shader (slang)

[Syh]

You’re fine it seems the forums have been down for the better part of the day anyway.

Thanks for changing the stepping I’m just being nitpicky, lol.

About temperature I remember you talking about changing it, I just don’t think I had updated since right before then (I may be wrong.) What is proper D75? As I think I have the temperature set at 7501 or something like that.

Mainly I’d be happy with the decoupling, if you could decouple the base deconvergence (not the zoom version, as the zoom deconvergence seems fine being tied to reflection. Might be harder to untie it this way though 🤷) from it, as well as the afterglow being decoupled. I haven’t looked everything over to see how feasible this is though.

I’m fine with everything that’s in glass being in a single shader, I’d just like to pick and choose my effects more liberally. (Again being nitpicky, I love Swiss army knife shaders. I just like being able to micromanage/power-user these effects.)

[Dogway]

I think afterglow is decoupled, the good thing of everything being in one shader is to reuse code, in glass the afterglow is also supposed to be seen in glass inner reflection, so that’s a good thing.

D75 is 7504 as I could check. I will be more intermitent from now on since I’ll doing other things, but from time to time I will try to look into glass as grade is finished (?). Might be tempted to add AdobeRGB space if enough users ask for it.

[Syh]

I’ll look over glass some and see if I can think of a way to reuse as much code as possible while still decoupling some things. (While keeping things coupled while both being active.)

[c9f5fdda06]

I want Adobe RGB, pls

[rafan]

@Dogway

I recalled what I wanted to say, if you don’t have a calibrated display with an instrument one of the best things you can do is what we have been doing already, green to blue, blue to green, and there’s also some blue to red. This kind of thing removes the purity of colors and hence it’s saturation. With the matrix transform now I can set those to zero and the image will look good.

I’ve been experimenting some more with your presets and sugguestions. Would the following values be somewhat accurate?

• green-blue: -0.10
• blue-red: +0.06
• blue-green: -0.09

I don’t understand why this looks better, but it does (Testing with the PAL preset btw). Could you explain a bit why it are these specific changes that do most of the trick?

Also if I would calibrate my monitor to accurate sRGB profile, do I then don’t apply above measures anymore, or are they then still useful?

On the topic of calibration, I’m still a bit confused. Are the steps with DisplayCal as follows?

Option 1: Creating a new profile and 3D LUT

1. Select the “Video 3D LUT for ReShade (Rec. 709 / 1886)” preset under “Settings”.

2. On the “Profiling” tab, optionally increase the number of patches with the slider. More patches will yield higher accuracy of the resulting profile and 3D LUT.

3. If desired, adjust the lookup table size on the “3D LUT” tab. Sizes 16x16x16, 32x32x32 and 64x64x64 are supported.

4. Click “Calibrate & profile”. Adjust the whitepoint of your display if necessary, then continue on to profiling.

5. Wait for measurements and calculations to finish.

How many patches are recommended for 2? And what size do I choose under 3 (64x64x64)?

If I then save the profile and I’m using your preset, do I then save the profile to the location of LUT2 ("…\reshade\shaders\LUT\64.png")?

Am I then set?

textures = “SamplerLUT1;SamplerLUT2” SamplerLUT1 = “…\reshade\shaders\LUT\16.png” SamplerLUT1_linear = “true” SamplerLUT1_wrap_mode = “clamp_to_border” SamplerLUT1_mipmap = “false” SamplerLUT2 = “…\reshade\shaders\LUT\64.png” SamplerLUT2_linear = “true” SamplerLUT2_wrap_mode = “clamp_to_border” SamplerLUT2_mipmap = “false”

[Dogway]

Yes, more or less, it’s the opposite but you’re getting the same effect than me due to doing also blue to green. My settings before were like:

gb = "0.030000"
bg = "0.100000"

Red usually didn’t touch much because grade was already dealing with it. Now that I color managed retroarch, I can set the above to 0, and it looks same-ish or better. The reason is original palette are pure colors, primary colors which are very saturated, those are the first to go out of gamut on a display not 100% sRGB (most if not all them). As I could find out, compressing the gamut to display’s gamut seems to do similar crosstalk between primary colors as described above manually.

As for DisplayCAL, mostly everything is correct, I use a depth of 64, and the number of patches depends on your patience and your colorimeter. With my current one I can do 115 or 175 just fine.

There’s one caveat though, I haven’t managed to use LUTs made out of DisplayCAL, and I think it’s because I’m messing with gamma ramps (Calibration - Tone Curve -not As Measured-), this is supposed to be ignored if you untick Apply calibration (vcgt), but I haven’t had much luck with it. My alternative is just picking the calibration matrix and doing the transforms to finally add as it was an additional Color Space matrix. If you have the opportunity to calibrate the display and have following issues just let me know.

[Nesguy]

Are you now recommending D75 and no color tinting when using a calibrated color profile?

I get clipped blues if I set the temperature any higher than 6505. Is this normal?

Also, are you still recommending that we set CRT gamma to 2.2 when using an HDTV with a gamma of 2.4? Or has this changed?

[Dogway]

I only use D75 for CPS, PSX, Saturn and the like. I use D93 for SNES, Genesis, N64, and interesting enough I found 8000K enough for 8-bit systems.

For all of them I recommend using the color mangler suggestions if you are not calibrating the screen, which should be also calibrated for gamma depending on your space.

CRTs had an equivalent power gamma of 2.4 because they were also designed to be watched in a dim surround environment, that means a CRT value of 2.222, but actual CRTs were tuned to 2.40. So it depends if you want nostalgic CRT or correct CRT.

[Nesguy]

I’m using an HDTV with 2.4 gamma. Just want to make sure I’m doing this right… does the shader assume an LCD gamma of 2.2?

So, 2.4 CRT gamma with a 2.2 LCD = 2.6 CRT gamma with a 2.4 LCD? Is this correct?

IDK, 2.2 CRT gamma with SMPTE-C mode just looks better to me. The darkest values are too bright otherwise.

[cobhc2016]

What do you mean by color mangler suggestions? Are those the settings you posted with screenshots? As I could calibrate my display to different color temps but can’t at the moment unfortunately and my display is pretty accurate to D65.

[Dogway]

@Nesguy, HDTVs are rarely 2.4 gamma, they are about 2.2, and even if you tweak the TV gamma option to -2 what I was getting was a downward slope not really a gamma curve.

As you said if you really have a calibrated 2.4 gamma HDTV you want your CRT gamma to be 2.4 which is the default. If your LCD gamma is 2.2 lower CRT gamma to 2.2, but as I said CRT’s were tuned with a range between 2.35 and 2.55 gamma that means it can range between 2.2 and 2.40 for an LCD calibrated to 2.2.

@cobhc2016, the easiest is to tweak the settings explained in this post. Basically removing some purity from primaries of the color palette.

[Nesguy]

Thanks for the reply, everything makes sense now!

I double checked and you’re right, this TV is actually 2.2. All my stuff is in storage right now, so I was just eyeballing it Haven’t had a chance to do a proper calibration on this yet, just did basic adjustments using test patterns.

[rafan]

@Dogway

I’ve been testing a bit more the grade shader, thanks also to Nesguy’s slotmask preset from the CRT shader thread. I’m noticing that green gets clipped in the SNES 240p color bars test (column D, E and F are the same for green). Just curious as to why this happens? It’s probably by design, just trying to understand.

Would going the wide gamut route alleviate the above problem, i.e. give green “wide range” in your shader with gradients in colum D, E, F being different?

Earlier you mentioned getting a 99+% sRGB match monitor would be a better route than crappier wide range gamut match, is that still the case? In that sense would it only be worthwhile to get something if it is certified 99%+ DCI gamut (for example)?

With that regards I noticed that in your grade shader code the sRGB and Rec 2020 tables are the same as stock guest-venom, but the DCI one differs as below.

grade:

// to DCI-P3
const mat3x3 DCIP3 = mat3x3(
 1.7842102050781250, -0.83157867193222050,  0.04736841097474098,
-0.6664471626281738,  1.76710486412048340, -0.10065787285566330,
-0.2881578207015991,  0.023684212937951088, 1.26447343826293950);

guest-venom

const mat3 ToDCI = 
mat3(
 2.725394,  -0.795168,   0.041242,
-1.018003,   1.689732,  -0.087639,
-0.440163,   0.022647,   1.100929
);

[Dogway]

From D IIRC it’s WTW (Whiter than White), it’s not supposed to be visible, you can read about it in the test suite instructions, so I don’t think going wide gamut could solve that and if it “solves” it would be a side effect. In any case those typically represent out-of-gamut values, those are the first to get clipped (that’s the whole point of the shader) through the composite pass block.

As I could observe CRT greens are in-gamut for sRGB, but blues and a bit of reds are not so those need gamut compression. In that regard a calibrated wide gamut can help (you will notice more saturated reds and blues). A non calibrated wide gamut in my opinion is the same as a non calibrated sRGB, worthless, but I’m a color geek so take my opinion with a grain of salt : )

About the DCI matrix, you need to update. I was using the DCI white point, but I found out (someone correct me if I’m wrong) that most DCI compliant devices use D65 white point, so I fixed that.

And thanks for having a look, the shader has been without changes for a while, just some formatting changes but if you find something else iffy let me know.

[Dogway]

I see some gr and gb in your preset, but in any case rather than hard clipping there is some soft clipping that soft rolls the highs making them look somewhat flat I also use my display space as space out so the gamut mapping might be flattening out the saturated primaries a bit more.

I’m not sure how old CRTs did work, games render out whites as 255, so is this supposed to be clipped (WTW) on a CRT or not? Might be worth to pass the greyscale ramp of the test suite over an old random CRT (not calibrated). The consensus is that the TV range is 16-235 but probably this was standarized after the 16-bit systems era.

About the blues, CRTs gamut is wider for blues than sRGB, so unless you gamut map to your display it’s going to clip at D93. Or try the bg technique.

[Nesguy]

@Dogway

I’m not sure how old CRTs did work, games render out whites as 255, so is this supposed to be clipped (WTW) on a CRT or not? Might be worth to pass the greyscale ramp of the test suite over an old random CRT (not calibrated). The consensus is that the TV range is 16-235 but probably this was standarized after the 16-bit systems era.

The way I’m reading the instructions in Test Suite, everything after C should be clipped on a CRT(!)

@hunterk Do you have an old uncalibrated CRT and feel like testing this? For the sake of science!

[hunterk]

I don’t have any random TVs, unfortunately, just a bunch of broadcast monitors, but I don’t know if or how well they’ve been calibrated, so I can give it a shot.

[Nesguy]

Ok, I think I solved the clipping thing with guest-dr-venom. Still no idea if CRTs clipped or how much, but shouldn’t we be maximizing the dynamic range on our sRGB LCD displays, regardless of what CRTs did?

There’s still a bit of soft clipping due to the bright pixel boost, but it’s not as bad as it was and I think it makes the image look better, overall. Lowering dark pixel bright boost helps smooth out the low end/reduce posterization a bit when using the SMPTE-C mode. Also, adding a mask will help reduce the clipping. Slot mask strength 0.50 looks good w/these settings, I think.

@rafan

You might try beam shape low/high 15.00/5.00 to see if that eliminates the clipping for you. I still think there’s probably something going on with your display and/or graphics card settings, though. This makes the highlights pretty bloomy but there’s no avoiding it if you want to optimize the dynamic range.

shaders = "10"
shader0 = "shaders_slang/stock.slang"
filter_linear0 = "false"
wrap_mode0 = "clamp_to_border"
mipmap_input0 = "false"
alias0 = ""
float_framebuffer0 = "false"
srgb_framebuffer0 = "false"
scale_type_x0 = "source"
scale_x0 = "1.000000"
scale_type_y0 = "source"
scale_y0 = "1.000000"
shader1 = "shaders_slang/stock.slang"
filter_linear1 = "true"
wrap_mode1 = "clamp_to_border"
mipmap_input1 = "false"
alias1 = ""
float_framebuffer1 = "false"
srgb_framebuffer1 = "false"
scale_type_x1 = "source"
scale_x1 = "1.000000"
scale_type_y1 = "source"
scale_y1 = "1.000000"
shader2 = "shaders_slang/misc/grade.slang"
filter_linear2 = "true"
wrap_mode2 = "clamp_to_border"
mipmap_input2 = "false"
alias2 = "WhitePointPass"
float_framebuffer2 = "false"
srgb_framebuffer2 = "false"
scale_type_x2 = "source"
scale_x2 = "1.000000"
scale_type_y2 = "source"
scale_y2 = "1.000000"
shader3 = "shaders_slang/crt/shaders/guest/afterglow.slang"
filter_linear3 = "true"
wrap_mode3 = "clamp_to_border"
mipmap_input3 = "false"
alias3 = "AfterglowPass"
float_framebuffer3 = "false"
srgb_framebuffer3 = "false"
scale_type_x3 = "source"
scale_x3 = "1.000000"
scale_type_y3 = "source"
scale_y3 = "1.000000"
shader4 = "shaders_slang/crt/shaders/guest/avg-lum.slang"
filter_linear4 = "true"
wrap_mode4 = "clamp_to_border"
mipmap_input4 = "true"
alias4 = "AvgLumPass"
float_framebuffer4 = "true"
srgb_framebuffer4 = "false"
scale_type_x4 = "source"
scale_x4 = "1.000000"
scale_type_y4 = "source"
scale_y4 = "1.000000"
shader5 = "shaders_slang/crt/shaders/guest/linearize.slang"
filter_linear5 = "true"
wrap_mode5 = "clamp_to_border"
mipmap_input5 = "false"
alias5 = "LinearizePass"
float_framebuffer5 = "true"
srgb_framebuffer5 = "false"
scale_type_x5 = "source"
scale_x5 = "1.000000"
scale_type_y5 = "source"
scale_y5 = "1.000000"
shader6 = "shaders_slang/crt/shaders/guest/blur_horiz.slang"
filter_linear6 = "true"
wrap_mode6 = "clamp_to_border"
mipmap_input6 = "false"
alias6 = ""
float_framebuffer6 = "true"
srgb_framebuffer6 = "false"
scale_type_x6 = "source"
scale_x6 = "1.000000"
scale_type_y6 = "source"
scale_y6 = "1.000000"
shader7 = "shaders_slang/crt/shaders/guest/blur_vert.slang"
filter_linear7 = "true"
wrap_mode7 = "clamp_to_border"
mipmap_input7 = "false"
alias7 = "GlowPass"
float_framebuffer7 = "true"
srgb_framebuffer7 = "false"
scale_type_x7 = "source"
scale_x7 = "1.000000"
scale_type_y7 = "source"
scale_y7 = "1.000000"
shader8 = "shaders_slang/crt/shaders/guest/linearize_scanlines.slang"
filter_linear8 = "true"
wrap_mode8 = "clamp_to_border"
mipmap_input8 = "false"
alias8 = ""
float_framebuffer8 = "true"
srgb_framebuffer8 = "false"
scale_type_x8 = "source"
scale_x8 = "1.000000"
scale_type_y8 = "source"
scale_y8 = "1.000000"
shader9 = "shaders_slang/crt/shaders/guest/crt-guest-dr-venom.slang"
filter_linear9 = "true"
wrap_mode9 = "clamp_to_border"
mipmap_input9 = "false"
alias9 = ""
float_framebuffer9 = "false"
srgb_framebuffer9 = "false"
scale_type_x9 = "viewport"
scale_x9 = "1.000000"
scale_type_y9 = "viewport"
scale_y9 = "1.000000"
parameters = "g_gamma_in;g_signal_type;g_gamma_type;g_crtgamut;g_space_out;g_hue_degrees;g_I_SHIFT;g_Q_SHIFT;g_I_MUL;g_Q_MUL;g_lum_fix;g_vignette;g_vstr;g_vpower;g_lum;g_cntrst;g_mid;wp_temperature;g_sat;g_satr;g_satg;g_satb;g_vibr;g_lift;blr;blg;blb;wlr;wlg;wlb;rg;rb;gr;gb;br;bg;LUT_Size1;LUT1_toggle;LUT_Size2;LUT2_toggle;SW;AR;PR;AG;PG;AB;PB;sat;lsmooth;GAMMA_INPUT;TAPSH;GLOW_FALLOFF_H;TAPSV;GLOW_FALLOFF_V;TATE;IOS;OS;BLOOM;brightboost;brightboost1;gsl;scanline1;scanline2;beam_min;beam_max;beam_size;h_sharp;s_sharp;csize;bsize;warpX;warpY;glow;shadowMask;masksize;vertmask;slotmask;slotwidth;double_slot;slotms;mcut;maskDark;maskLight;CGWG;gamma_out;spike;inter;interm;bloom;scans"
g_gamma_in = "2.400000"
g_signal_type = "1.000000"
g_gamma_type = "1.000000"
g_crtgamut = "2.000000"
g_space_out = "0.000000"
g_hue_degrees = "0.000000"
g_I_SHIFT = "0.000000"
g_Q_SHIFT = "0.000000"
g_I_MUL = "1.000000"
g_Q_MUL = "1.000000"
g_lum_fix = "0.000000"
g_vignette = "0.000000"
g_vstr = "40.000000"
g_vpower = "0.200000"
g_lum = "0.000000"
g_cntrst = "0.000000"
g_mid = "0.500000"
wp_temperature = "6505.000000"
g_sat = "0.000000"
g_satr = "0.000000"
g_satg = "0.000000"
g_satb = "0.000000"
g_vibr = "0.000000"
g_lift = "0.000000"
blr = "0.000000"
blg = "0.000000"
blb = "0.000000"
wlr = "1.000000"
wlg = "1.000000"
wlb = "1.000000"
rg = "0.000000"
rb = "0.000000"
gr = "0.000000"
gb = "0.000000"
br = "0.000000"
bg = "0.000000"
LUT_Size1 = "16.000000"
LUT1_toggle = "0.000000"
LUT_Size2 = "64.000000"
LUT2_toggle = "0.000000"
SW = "1.000000"
AR = "0.070000"
PR = "0.050000"
AG = "0.070000"
PG = "0.050000"
AB = "0.070000"
PB = "0.050000"
sat = "0.100000"
lsmooth = "0.900000"
GAMMA_INPUT = "2.400000"
TAPSH = "4.000000"
GLOW_FALLOFF_H = "0.000000"
TAPSV = "4.000000"
GLOW_FALLOFF_V = "0.000000"
TATE = "0.000000"
IOS = "0.000000"
OS = "1.000000"
BLOOM = "0.000000"
brightboost = "0.500000"
brightboost1 = "1.500000"
gsl = "2.000000"
scanline1 = "15.000000"
scanline2 = "5.000000"
beam_min = "1.000000"
beam_max = "1.000000"
beam_size = "0.000000"
h_sharp = "3.000000"
s_sharp = "1.000000"
csize = "0.000000"
bsize = "600.000000"
warpX = "0.000000"
warpY = "0.000000"
glow = "0.000000"
shadowMask = "-1.000000"
masksize = "1.000000"
vertmask = "0.000000"
slotmask = "0.000000"
slotwidth = "2.000000"
double_slot = "1.000000"
slotms = "1.000000"
mcut = "0.250000"
maskDark = "0.500000"
maskLight = "1.500000"
CGWG = "0.300000"
gamma_out = "2.400000"
spike = "1.000000"
inter = "400.000000"
interm = "1.000000"
bloom = "0.000000"
scans = "1.000000"
textures = "SamplerLUT1;SamplerLUT2;SamplerLUT3"
SamplerLUT1 = "shaders_slang/crt/shaders/guest/lut/sony_trinitron1.png"
SamplerLUT1_linear = "true"
SamplerLUT1_wrap_mode = "clamp_to_border"
SamplerLUT1_mipmap = "false"
SamplerLUT2 = "shaders_slang/crt/shaders/guest/lut/sony_trinitron2.png"
SamplerLUT2_linear = "true"
SamplerLUT2_wrap_mode = "clamp_to_border"
SamplerLUT2_mipmap = "false"
SamplerLUT3 = "shaders_slang/crt/shaders/guest/lut/other1.png"
SamplerLUT3_linear = "true"
SamplerLUT3_wrap_mode = "clamp_to_border"
SamplerLUT3_mipmap = "false"

[Dogway]

An image with WTW (235-255) values doesn’t directly mean it’s in PC levels. That area is reserved for padding or overshoots.

The process to convert TV to PC range is by luminance expansion 0 <- 16 and 235 -> 255. So if you leave 255 as it is you are going to have a dull image, that is in the case old CRTs did luma expansion which I don’t really know. Most literature says that for SMPTE-C white corresponds to IRE 100 on a IRE 120 scale, so there’s a hint there, but that’s not the case for NTSC-J systems.

[Nesguy]

Thanks for indulging us in this shader madness, lol. Anything you can do to help us get to the bottom of this would be appreciated.