Whoa, those are very nice photos. Much better than the reference shots I took with my cell phone, ha. Will definitely be able to make use of those.
There’s some issues with doing a totally accurate handheld LCD shader. First, if you simply directly scale the LCD cells (a cell being one pixel on the real handheld screen made up of red, green, and blue subpixels) up to a larger size to match the scale of the image being displayed on your monitor, you get very dramatic color separation because the subpixel representations are now very far apart in terms of screen pixels. If you were playing from very far back or on a small, high DPI screen this would be probably be the optimal approach. At a normal viewing distance on a normal monitor it becomes pretty painful to look at, though.
Second, the image tends to darken significantly even at small cell scales. This is because even if you did the minimum representation where 1 pixel on your monitor correlated to 1 subpixel from the handheld screen, if you had a primary color like red being displayed you’d end up with 2 black pixels between every red pixel. Pixels on a typical monitor are pretty large so this gap is much more noticeable than the gap between subpixels on your handheld LCD (although it is still noticeable on the handheld and this is an important factor in replicating the feel of the original screen).
In order to brighten the image up, a lot of LCD shaders, including mine, use magenta/yellow/cyan instead of red/green/blue. This means that if you had a red pixel on screen, instead of just lighting up 1 pixel in the simulated cell (the pixel corresponding to the red subpixel) and having 2 black pixels, it now lights up two pixels in the simulated cell - the magenta (red/blue) and yellow (red/green). This is probably why you said it feels more red than the real thing. This concept could actually be expanded on to stretch out LCD cells into arbitrarily wide color gradients which can be one approach to solving the cell scaling problem mentioned above, but in my photoshop mockups scaling cells in this way still resulted in unpleasant levels of color separation.
The image will probably still be a little too dark after this, so a lot of shaders will actually overlay the cell representation of the screen on top of the “raw” image. Some even just tint the raw image with r/g/b or m/y/c bands to approximate the effect. I use color darkening in my approach to obtain the correct colors and intensities of the subpixels before blending them back with the raw image (well, not quite raw, I add horizontal lines which then do a sort of color averaging blur to mimic the borders between vertically adjacent cells on the actual screen).
What this all comes down to are that some compromises have to be made on current displays. The cell representation I use has a static width of 3 screen pixels (each representing a m/y/c subpixel) so you’re going to get wider color bands compared to a r/g/b representation. You’ll also see multiple cells for every “pixel”, which should be mapped to a single cell on the real screen, if the scale isn’t exactly 3 (you can even get partial cells if the scale isn’t a multiple of 3, but the effect is subtle unless you’re looking really close). Additionally, I partially blend the raw image back in to boost the image brightness, which isn’t realistic at all because now a subpixel that should be black will have some color blended into it. What you get is something like a pixel on the Gameboy that should be 255 red, which would normally be represented by a fully lit red subpixel and “off” (black) green and blue subpixels, would be represented in my shader as two 255 red pixels (those representing hypothetical magenta and yellow subpixels) and a somewhat darker (say, 127) red pixel (corresponding to the hypothetical cyan subpixel). Not physically accurate, but the overall color is much closer to what you’d expect.
I guess you could think of it kind of like scaling the image by fusing multiple Gameboy screens together and stretching the image across the new larger screen rather than just blowing one screen way up. The m/y/c cells and raw image blending tricks just compensate for the fact the simulated cells are 3 times wider than the original cells because each subpixel has to be mapped to a single monitor pixel.
And wow, that turned into a longer post than I expected. I’m actually working on a simple sprite game right now to get some practice with OpenGL development and I want to implement a shader kind of like this in it. It might be a couple weeks, but I hope to refine my LCD shader approach in the process so I can do a meaningful update on the RetroArch shader as well.