I don’t think you can get a shadow mask or aperture grille mask etc to be bright enough on a standard dynamic range monitor. Which will be what someone with a MiSTer is using as HDR is flat out not supported.
If you think about it this makes sense as a mask is only going to darken the screen. At best it will keep it the same and as SDR monitors aren’t as bright as a CRT (my Sony PVM 2730 at least) then you aren’t going to get the brightness (or colours).
(Shameless plug) See my new shader which does everything in HDR for an example of getting the natural brightness of a CRT - you’ll need an LCD capable of 700 nits and more in my experience.
Also 4k is an absolute minimum I’d say for a 600TVL 4:3. As that’s number of lines across the screen to the height of the screen so a 600TVL will actually have 800 (600*(4/3)) lines across the whole width. Minimum pattern is 3 pixels really so that’s 3*800=2400 and the 3840 pixels of a 4k LCD is across a 16:9 screen i.e you have to take off a quarter of the pixels to get into 4:3 ratio i.e you have 2880 pixels to play with.
I squeeze a four pixel pattern into it and it seems to produce a close resolution.
Shaders: slang_shaders/HDR/CRT-sony-pvm-2730-4k-hdr.slangp
TVL description: https://andynumbers.wordpress.com/2016/07/21/lets-talk-about-tvl-count/
More info: