So firstly you have edited the wrong section. if you read the notes,this is for windows. I guess the reason it works for you at the moment is that the resolution you have installed match the ones asked for by RA.
The section you want to edit is below. Starting from line 76 in the same file you have been editing. this sends the height, width and refresh to the algorithm to generate the modeline.
static void switch_res_crt(unsigned width, unsigned height)
{
if (height > 100)
{
video_display_server_switch_resolution(width, height,
ra_set_core_hz, ra_core_hz);
video_driver_apply_state_changes();
}
}
The variable ra_core_hz is what is used for Linux so if you want to hard set that to 138.0f it will then generate all the modeling at 138hz
static void switch_res_crt(unsigned width, unsigned height)
{
if (height > 100)
{
video_display_server_switch_resolution(width, height,
ra_set_core_hz, 138.0f);
video_driver_apply_state_changes();
}
}
I am not sure about interlaced resolution, but the modeline generator will create one and attempt to switch to it. Also this will switch the game to 138hz which mean it will run over twice as fast as it should.
Modelines are tricky to get right, through my testing I found similar issue you have but with 240p as well. The pixelclock, hz, max height and max width all need to calculated together. Also limitations on you GPU can also cause this issue.
Lastly if the limitation is the pixel clock needing to be higher for a 31khz monitor use 3840 as the super resolution. This will keep the pixel clock higher and allow you to reduce the hz. This can be changed in the retroarch.cfg located at ~/.config/retroarch/