Disable both Threaded and Synchronous Rendering, enable Fullspeed framerate, set Broadcast to NTSC (might not matter depending on the region of the game you’re playing), and set the cable type to Composite (you might be able to use RGB as well, but stay away from VGA). I did a lot of testing with the microstuttering issues, and these were the settings that got rid of them for me.
Threaded rendering has always caused stuttering at least for me, since it seems to make the framerate setting invalid (possibly by setting it to “normal” instead?). I still get performance dips sometimes with certain transparency effects and when flycast streams/loads files from the disc image. But this kind of performance issue is completely unrelated to the microstuttering issue I’m talking about. My hardware is relatively old at this point, still sporting an overclocked 2500k and a 680 gtx.
Of course, if you have a gsync/freesync monitor that will help a ton too. Just make sure gsync/freesync is actually active. If you have NVidia CP, open it up and go to Display>GSYNC indicator. Next time you load up Retroarch, you should see an indicator on the top right for Vulkan that says “GSYNC” if it’s active. Note, I have seen Retroarch lose Gsync with Vulkan sometimes after quitting games and going back to the main menu. When in doubt, close Retroarch and reopen it again.
I highly recommend making those setting changes I mentioned above, and you should be fine.