Speaking from experience (or my own preferences), for more cartoonish graphics (objects/backgrounds containing low details, usually just flat color surrounded by black outlines), sharp scanlines with little to no masks usually look best. Some exceptions would be those 8-bit Atari/NES/MSX/SG-1000 games for which I sometimes prefer a vintage look (stronger masks, deconvergence, bleeding, etc) to add more flavor to the otherwise very simple/primitive graphics.
For highly-detailed graphics (photo-like in some cases), they almost always look good with or without masks. However applying masks to the detailed graphics can give you really stunning results. Some games with seemingly cartoonish graphics (those CPS3 titles for example) are actually incredibly detailed. These games too will look stunning with mask on.
EDIT:
I also find it difficult to make those pseudo-3d games look good. Outrunners still looks great because of those really detailed graphics I guess. On other hand, games like Rail Chase and Power Drift look like crap now. CRT shaders can’t save them. I still remember how I was wowed by these games back in the arcade. Donkey Kong Country and Vampire Saviour and lots of 2D games still wow me to this day, but I can’t find any way to make these pseudo-3d games look half as good as I remembered 
