I see your point.
I’m still not sure whether the artifacts I’m seeing regarding the mask output is to do with the curves alone. There’s an itch with this gamut mapping, that I’d like to at least understand a bit better.
I created a fictitious “Limited 709” space that is using desaturated 709 primaries for red and green. Mapping this one into 709 is the equivalent of the 709 to wide gamut mapping (DCI-P3/2020) but has the benefit that we keep 709 as output space, such that we can possibly rule out issues related to wide gamut modes.
The interesting finding is that the mask output gets messed up when color mapping of “limited 709 space” to regular 709 space is done, i.e. when a color gamut transform is active, while with regular 709 the mask output is a clean RBG.
Below is the patched gamma-correct.h. It’s a quick hack, but it gives the idea. Hopefully this is useful in tracking down why the mask output is so different (wrong?) when a color gamut transform is active.
Note that when you replace gamma-correct.h with this version, then the colour space parameter 709 and sRGB actually switches between “regular 709” space and “limited 709” in 709 space. Also the transform for regular 709 is done pro forma, as it’s a transform from 709toXYZ and XYZ to 709.
For me when using “limited 709” I can see the mask output being messe up when doing macro zoom on the subpixel output in the white screen of the 240p test suite. Contrary when setting regular 709 the mask output is clean RBG.
// SDR Colour output spaces
const mat3 k709_to_XYZ = mat3(
0.412391f, 0.357584f, 0.180481f,
0.212639f, 0.715169f, 0.072192f,
0.019331f, 0.119195f, 0.950532f);
const mat3 kXYZ_to_DCIP3 = mat3 (
2.4934969119f, -0.9313836179f, -0.4027107845f,
-0.8294889696f, 1.7626640603f, 0.0236246858f,
0.0358458302f, -0.0761723893f, 0.9568845240f);
// "709limited" - fictitious 709 space with less saturated red and green primaries
// Wx=0.3127, Wy=0.329 -- D65 white
// Rx=0.50, Ry=0.33 -- less red saturation than 709/sRGB primaries
// Gx=0.30, Gy=0.50 -- less green saturation than 709/sRGB primaries
// Bx=0.15, By=0.06 -- Blue same as 709/sRGB
const mat3 k709limited_to_XYZ = mat3(
0.4182745739f, 0.4034733473f, 0.1287080058f,
0.2760612188f, 0.6724555789f, 0.0514832023f,
0.1422133551f, 0.2689822316f, 0.6778621641f);
// XYZ to 709
const mat3 kXYZ_to_k709 = mat3 (
3.2409699419f, -1.5373831776f, -0.4986107603,
-0.9692436363f, 1.8759675015f, 0.0415550574,
0.0556300797f, -0.2039769589f, 1.0569715142);
float LinearTosRGB_1(const float channel)
{
return (channel > 0.0031308f) ? (1.055f * pow(channel, 1.0f / HCRT_GAMMA_OUT)) - 0.055f : channel * 12.92f;
}
vec3 LinearTosRGB(const vec3 colour)
{
return vec3(LinearTosRGB_1(colour.r), LinearTosRGB_1(colour.g), LinearTosRGB_1(colour.b));
}
float LinearTo709_1(const float channel)
{
return (channel >= 0.018f) ? pow(channel * 1.099f, 1.0f / (HCRT_GAMMA_OUT - 0.18f)) - 0.099f : channel * 4.5f; // Gamma: 2.4 - 0.18 = 2.22
}
vec3 LinearTo709(const vec3 colour)
{
return vec3(LinearTo709_1(colour.r), LinearTo709_1(colour.g), LinearTo709_1(colour.b));
}
vec3 LinearToDCIP3(const vec3 colour)
{
return clamp(pow(colour, vec3(1.0f / (HCRT_GAMMA_OUT + 0.2f))), 0.0f, 1.0f); // Gamma: 2.4 + 0.2 = 2.6
}
void GammaCorrect(const vec3 scanline_colour, inout vec3 gamma_out)
{
if(HCRT_HDR < 1.0f)
{
if(HCRT_OUTPUT_COLOUR_SPACE == 0.0f)
{
const vec3 Rec709_colour = (scanline_colour * k709_to_XYZ) * kXYZ_to_k709;
gamma_out = LinearTo709(Rec709_colour);
}
else if(HCRT_OUTPUT_COLOUR_SPACE == 1.0f)
{
const vec3 Rec709_colour = (scanline_colour * k709limited_to_XYZ) * kXYZ_to_k709;
gamma_out = LinearTo709(Rec709_colour);
}
else
{
const vec3 Rec709_colour = (scanline_colour * k709limited_to_XYZ) * kXYZ_to_k709;
gamma_out = LinearToDCIP3(Rec709_colour);
}
}
else
{
gamma_out = Hdr10(scanline_colour, HCRT_PAPER_WHITE_NITS, HCRT_EXPAND_GAMUT);
}
}
Output when “Limited 709” is selected:
Output with regular 709:
