I’m glad you found a solution that works well for your needs.
Just sharing my recording settings which may or may not be of assistance to you. I’ll also share @TheNameC’s recommendations since he’s uploaded a number of YouTube videos using my shader preset settings. You really need a high enough bitrate if you want to preserve all of the fine scanline and mask details.
Here are Google’s recommended settings for YouTube:
In general you want as high a bitrate as possible/practical while recording/streaming/trancoding at a consistent resolution across the board with little to no filtering or post processing effects.
https://forums.libretro.com/t/mega-bezel-group-thread/32832/1063?u=cyber