Would it be possible to create an adaptive brightness shader?

[Squalo]

The brightness compensation/black level methods that crt_yah uses have reminded me of an old idea I had long ago.

I’m sure many of you know how a compressor/limiter works in the world of audio. Well, the idea is to apply the same principles to moving images. Thresholds would be set for shadows, midtones and bright parts of the signal (it could also be used for colour hues in a more advanced version) and the shader would automatically raise or lower the pixel luminance values according to a specified ratio. Attack/release times would need to be finely tuned to make the shifts in brightness smooth and pleasing to the eye but, if implemented correctly, I think it could work very well, and increase dynamic range quite dramatically. It would also be pretty cool when combined with CRT shaders as long as the process takes places “behind the scanlines”.

Of course the idea is not new. Nature figured it out aeons ago and we have been using it in videogames for many years now in the form of HDRR/tonemapping, but as far as I know there’s nothing of the sort that can be applied to vintage games.

@hunterk @guest.r @Hyllian @Jezze

[Squalo]

@anikom15 @DariusG @beans, any other devs around.

I’m sure you are busy with your own projects and life in general, but what do you think? It’s certainly not beyond your powers.

[kokoko3k]

What’s the problem this should aim to solve?

I think it would completely ruin the image for big changes and just a bit for small ones.

eg: black screen with gray text -> black screen with white text.

Reminds me of dynamic contrast seen on cheap TV panels…

(also, compression/limiting reduces the dynamic range, as opposed to expanders seen in audio processing, which I think you were referring to)

There was a “feature/defect” on consumer or old CRTs that would increase the contrast (crushed blacks) as the picture went bright; It is present in koko-aio and named adaptive black, but it wasn’t meant to improve the perceived quality.

[Jobima]

btw, I remember back in CRT days that with crt-breathing (raster-bloom in guest shader) if the frame is kinda bright the black level will be less than if the frame is dark, especially if the contrast in the TV settings is high

[kokoko3k]

Yeah, indeed you were forced to push brightness a bit to avoid the bright images to be too much contrasted (but it was a loop lol)…or at least, that was what i did

[Jobima]

@kokoko3k

indeed you were forced to push brightness a bit to avoid the bright images to be too much contrasted

that what I was doing too and kinda set contrast to middle or lower

[Squalo]

@kokoko3k it would not necessarily solve any actual problems, but it would allow to increase contrast, saturation and perceived brightness without ever crushing blacks/clipping highlights. It would also give us the opportunity to create a few “universal” presets and apply them to almost any content, without having to manage multiple brightness settings for individual games, which would be quite convenient in a plug’n play sense. Brightness and gamma were all over the place back in the day, as you more than likely know. And I’m 100% sure more cool uses for it would be discovered down the road!

@kokoko3k

(also, compression/limiting reduces the dynamic range, as opposed to expanders seen in audio processing, which I think you were referring to)

If used correctly, it can actually increase it in high contrast scenarios, since it provides perceivable access to the darkest shadows and the brightest highlights of a scene, all at the same time. But yeah, it would have to be used with care in order to not look horrible like those corny HDR photographs that got so trendy a few years ago.

@kokoko3k

There was a “feature/defect” on consumer or old CRTs that would increase the contrast (crushed blacks) as the picture went bright; It is present in koko-aio and named adaptive black, but it wasn’t meant to improve the perceived quality.

This idea that I suggest should in theory deal with issues like precisely that one, by maximizing the capabilities of any given display while ensuring that no pixel ever gets clipped or crushed.

I hope you understand my point a bit better now

[anikom15]

@Squalo

without having to manage multiple brightness settings for individual games

I don’t know why you feel like there is a need for this. I have never had issues with a lack of dynamic range in games, even for notoriously dark titles like Doom 64. If you need to manage brightness on an individual basis, either something is wrong with your viewing environment or your monitor.

That might sound harsh, but something like this can only be developed by defining a fixed viewing environment and display conditions. Your viewing environment is the number one contributor to loss of perceptual dynamic range as it goes from the TV to your eyes, followed by the TV itself. Those controls, especially the first one, also has the benefit of not degrading any of the digital information in the video signal itself. Therefore those two factors need to be controlled first, and addressing anything else before hand is just going to create a moving target.

Now, if you can limit the scope of the problem into something more specific, like ‘I want to play dark games on a laptop in an uncontrolled environment (like an airport)’ that may give people more info to work with and build something.

[Squalo]

@anikom15

That might sound harsh

I don’t think you sounded harsh, but even if you did, I don’t really mind harsh. All good

Like I just told kokoko, it’s not because there is a problem to solve or because I need it. Please read my answer to him if you want to know why I think it’s an interesting idea.

On the other topic, my viewing environment is actually pretty well set up with dim, controlled lightning, calibrated screens and whatnot. Managing settings on the basis of specific content is not, again, a matter of need, but one of enjoyment. Especially considering that we are dealing (mostly) with old games here, and like I just told kokoko, brightness, gamma, colour, etc back then were anything but normalized. It was a mess. And basically the only way to deal with that mess, if you actually play those games and care about them, is to manage them almost individually. Well maybe you have a different way, but to each their own right?

See, I enjoy playing Street Fighter. And I enjoy it much more if I can play it like this

than this

It’s not like I need that crt filter, but it makes the game look way better and also much more similar to my childhood memories of it. Now I can send you that preset so you can slap it over Doom64 or Wipeout 2097. Have fun!

[anikom15]

Well that’s the thing. Those CRT shaders shouldn’t be causing any significant loss of dynamic range. You should be able to pull up ramp patterns and not see any crush or banding. If you do, those shaders need to be fixed.

In SDR we only have 8-bits to work with, so I always suggest people to use their display controls first for gamma, white point, and brightness (because scanlines and masks reduce it). Because those are the things that will lead to crush (or dullness for the latter) if they are changed too much. Color correction can affect it as well, but it usually doesn’t have a very visible effect so I don’t really count that as a distortion we need to worry about.

So if you are using CRT shaders and finding you are getting black crush or something, you need to find out why and address it. It should be fixable.

Other than that, what you’re talking about is a bias in three bands. Your raw screenshot of SFII actually has high dynamic range. Look at the meat shop. What’s actually happening with this screen is that there is more bright information than dark, so you want to darken some of that bright information to equalize the ratio of bright to midtone to dark. An expander wouldn’t work because the darks are actually dark enough. What you are looking for is a technique called histogram equalization with a filter across the histogram to smooth it out (biased in the direction of either lighter or darker depending on tonal balance).

[kokoko3k]

I second @anikom15, I can’t see any method that would automagically fix the mess of different brightness by different hardware; that’s because, probably, those output levels, and art itself, was intended for particular screens.

I’ve seen a lot of cps2 games being pale, requiring an higher output gamma, maybe because the coupled screens provided a darker image.

Being core used not provided to the shader by Retroarch, and let alone the emulated subsystem in multiple emulator like MAME and derivates, one could “guess” something by the input resolution, but probably it would be prone to errors, so ineffective in the long run.

The best one can do is to prepend (append would be better, but heavier), a color correction shader that would adapt a formally correct output (maybe calibrated via gamma ramps) to the output screen/environment characteristics and shortcomings.

A calibration helper shader maybe?

Also, from a pure technical point of view, analyzing the picture for the purpose you were asking would be a very, very heavy operation; as it would require sampling every input “pixel”. It is something the core itself could do in a much more efficient way; but again, I’m not sure it would help anyway

[Squalo]

@anikom15 the point was that old games need to be micromanaged individually in order to get the best possible experience out of them. I used those examples (SF2HF, WO2097, or your very own Doom64) because of their evidently different image characteristics in raw form. You can’t use the same brightness settings for those games and expect good results. I configured the crt image I posted with crushed shadows on purpose to show that what I’m suggesting would provide way more headroom for contrast.

@anikom15

the darks are actually dark enough

Exactly, they are already dark enough, so increasing contrast will crush them. The idea is to lift them dynamically. And on the other side of the histogram, controlling the signal up there would yield better looking highlights and also help shaders apply masks and scanlines. And speaking of those, if developed/implemented in tandem with crt simulators, it might actually solve a lot of our traditional lost brightness issues.

@anikom15

What you are looking for is a technique called histogram equalization with a filter across the histogram to smooth it out (biased in the direction of either lighter or darker depending on tonal balance).

This sounds like a good starting point. If you ever find the time and the willingness to advance/develop such a system, I will love to hear what you save to say!

@kokoko3k

I can’t see any method that would automagically fix the mess of different brightness by different hardware

In the framework of emulation, any given system’s inherent brightness properties are irrelevant. It’s a digital signal. The way a game looks depends on how it was developed and designed, its code and assets, it has nothing to do with the machine it’s running on. 255,255,255 is the same white on a SuperNintendo as it is on an Xbox360, or your computer.

@kokoko3k

A calibration helper shader maybe?

Completely different thing from what I have in mind, but it would be really nice to have something like that too

@kokoko3k

Also, from a pure technical point of view, analyzing the picture for the purpose you were asking would be a very, very heavy operation; as it would require sampling every input “pixel”

Your thresholds are just numerical values for luminance. HDDR and tonemapping have been a thing since at least the Playstation 2, a 6gflop chip. And a modern toaster can run a ps2 emulator I’m not a genius as some people around here are, but I’m pretty sure it would be an fairly cheap process.

[kokoko3k]

“old games need to be micromanaged individually”

“The idea is to lift them dynamically”

“histogram equalization” (“sounds like a good starting point.”)

Your thresholds are just numerical values for luminance. HDDR and tonemapping have been a thing since at least the Playstation 2, a 6gflop chip. And a modern toaster can run a ps2 emulator I’m not a genius as some people around here are, but I’m pretty sure it would be an fairly cheap process.

Tonemapping is an useful concept that allows me to express my statements better.
With it, you “map” a color in a wide numerical range to another in a narrower one; and not only it is a relatively cheap and dumb process, but often there are static or dynamic metadatas that provide useful hints to the mapping algorithm.

so it is something like:

[ 1 color+metadata ] --> [1 color]

Strip out the metadata, and you have a single stupid transfer function that may or may not work well, depending on the content.

But Retroarch does NOT provide any metadata:

@kokoko3k

Being core used not provided to the shader by Retroarch, and let alone the emulated subsystem in multiple emulator like MAME and derivates,

So an efficient way to overcom that limitation would be:

@kokoko3k

“guess” something by the input resolution

…but that would assume that:

@kokoko3k

those output levels, and art itself, was intended for particular screens.

…which you seem to to disagree with (?)

@Squalo

The way a game looks depends on how it was developed and designed, it has nothing to do with the machine it’s running on.

What remains on the table? Pretty nothing, so we would be forced to analyze the whole picture to figure out an effective way to map the input to the output that would not ruin the whole art.

And, even assuming such a solution exists, histogram equalization requires data to build the Histogram, right?

@"kokoko3k

analyzing the picture for the purpose you were asking would be a very, very heavy operation; as it would require sampling every input “pixel”. It is something the core itself could do in a much more efficient way; but again, I’m not sure it would help anyway

@Squalo

@kokoko3k

A calibration helper shader maybe?

Completely different thing from what I have in mind, but it would be really nice to have something like that too

Yes, I know. I wrote that, because I think that’s the best you can aim for.

[anikom15]

@Squalo

the point was that old games need to be micromanaged individually in order to get the best possible experience out of them. I used those examples (SF2HF, WO2097, or your very own Doom64) because of their evidently different image characteristics in raw form.

They don’t and I don’t see how arbitrarily altering colors is going to be the best possible experience.

[guest.r]

From my experience the answer is per-game shader settings. Even if an existing preset covers a range of games, like with core presets or console to TV connection presets, per-game presets are currently the last level of micro-management.

OK, average accumulated average screen brightness is sortoff doable, but can still not guarantee a better experience.

Good example of per-game presets are arcade games like Hero Tonma, Contra, Golden Axe etc., where applying negative black levels helps to a more authentic experience.

Another good example are also some Atari ST ports to Amiga, where games looked too dark. Like, not a general rule for a core, but some examples exist.

[Hari-82]

@guest.r

Another good example are also some Atari ST ports to Amiga, where games looked too dark. Like, not a general rule for a core, but some examples exist.

I recently tried (using PUAE) Midnight Resistance for amiga (I think it was a port from ST) and the image is truly dark!

I had an amiga 500 back in the 90’s and I clearly remember playing Midnight Resistance but I have no memories of brightness problems! memory tricks? or real hardware was somehow compensating for that?

[guest.r]

@Hari-82

I had an amiga 500 back in the 90’s and I clearly remember playing Midnight Resistance but I have no memories of brightness problems! memory tricks? or real hardware was somehow compensating for that?

I guess the game looked darker indeed. It’s nothing like the Sega max. Brightness, where this was relevant for all Genesis content. Maybe it’s more pronounced with emulators and modern displays.

[Squalo]

@guest.r

From my experience the answer is per-game shader settings.

That’s exactly what I’m talking about, per-game shader settings.

This is one of my favourite racing games ever. I remember having to crank brightness up and contrast down (a whole lot) on my first crt (a 14" little Sharp :_) back in the teen days, just to see where I was going and not crash the ship into those pitch black walls. Nowadays I’m doing the same thing with a keyboard instead of a remote, and very conveniently I have granular control over most parameters and can save the changes for later use. Why wouldn’t I take advantage of these possibilities?

If someone can’t see why for me the image on the left is a better experience than the one on the right, we just see the world differently and I’m not going to try to convince them, but WO2097 is just too dark and no “viewing conditions” are ever going to change that fact (and don’t get me wrong, I understand and respect the importance of proper viewing conditions, and do what I can to make the best of my own. I also understand their limitations, however).

The hypotethical shader that I propose would fix these extremely dark games by lifting shadows organically while keeping midtones in line with artistic intention and highlights in check.

[anikom15]

@Squalo

The hypotethical shader that I propose would fix these extremely dark games by lifting shadows organically while keeping midtones in line with artistic intention and highlights in check.

You are not proposing a hypothetical shader. You are dreaming and then asking, out of the blue, three busy people to build your dreams for you. How is a real-time shader supposed to know those shadows are too dark? What is the reference to compare against? How quickly does it need to react? How do you balance that given this is a game that changes rapidly frame to frame? This game is supposed to have shadows that look nearly black. That’s the whole aesthetic of that stage. There are other stages in the game that aren’t even dark like that. Cherrypicking a screenshot to make the game look wrong without the context of gameplay really rubs me the wrong way, like how you picked the brightest stage of SFII to suggest it’s too bright as a whole. The fact that you mock ‘viewing conditions’ like it’s a hallucination shows you don’t understand the theory.

If you actually want someone to write a shader for you, you should at least ground the idea in some kind of theory. Even just saying ‘I did [entire process] in Photoshop/GIMP and it gives me this look, how would I do that with shaders?’

Even just documenting games that you feel are too dark or bright and offering an argument as to why they are, either historically or technologically, would be helpful.

[kokoko3k]

This is something a crt shader can’t do

(btw, great idea that thread, lol)