I think slot masks can work at 1080p, but you have to tamper down certain expectations. For one, curvature is probably a bit too hard to get looking good, as there’s not enough resolution to avoid moire. Overscaling is practically a must, too, as you want as much resolution as humanly possible, but of course this is a no-go with hi-res content, as that cuts off too much space. For masks, I personally find mask 9 is the best one to use, as it’s three-pixels wide and plays well with the slot mask parameters. And for scanlines, you really want them to be quite light. Make them too strong and pronounced, and you see a pattern where every other line looks different, which appears to be a side effect of the slot mask itself, and it’s just ugly.
Here’s about where I find the scanline strength to look quite good at 1080p. As you can see, they’re quite light and are almost invisible in places:
The most surefire way to get it looking alright is to use No Scanline mode. This replicates what many smaller, older sets look like, where there are no visible lines even with 240p, and as a bonus it’s a little brighter, but of course it ends up looking somewhat blurrier. The biggest upside to No Scanline Mode, though, is that it does not require integer scaling at all, so it can be freely used with hi-res 480p content without letterboxing. Also, I just found out not too long ago that the very earliest VGA monitors actually used slot masks like their CGA/EGA predecessors and showed no scanlines due to line-doubling on games, so a slot mask No Scanline Mode setup could work quite well for VGA DOS games as well.