This is going to sound like a repeat of an earlier query I made with regard to how the vecx core handles graphics. This time, MAME. Having sort of the same issue: Native MAME defaults to displaying vector games (and its own Vectrex emulation) at whatever resolution my display happens to be, whereas RetroArch seems to want to render the graphics internally at a lower resolution and scale that up.
The chosen internal resolution is hard to pin down but seems to be perhaps 640x480. Although there is a bevy of options in the Retroarch.cfg for specifying resolution, none of them have an impact. I have tried: video_fullscreen_x/y, video_window_x/y, custom_viewport_width/height, setting each to 1920x1080. The only thing any of that accomplishes is to stretch the image to the full dimensions of the display, along with the artifacts of the low-resolution render.
It is unknown whether the fact that the “resolution” setting in RetroArch’s F1 menu caps at 960x720 has any bearing on this phenomenon.
Outside of generating thousands of .bat files, I don’t have the luxury of choosing to use native MAME.EXE for vector games and RetroArch for everything else, so I need to find a way to sort this out.