It’s possible that the aspect is reacting weirdly because the whole viewport has been rotated, and now to the shader x and y values have been swapped.
This is why in the Mega Bezel that I get the user to turn that stuff off so that the viewport can never get swapped like that.
It’s possible that for this shader and fbneo that having a rotate parameter for the game image may be a good way to go.
I should also mention that this problem will not happen on any recent Mame versions because it does not use the viewport rotation feature and rotates the image in the core before it is sent to retroarch (this is actually “incorrect” based on the retroarch spec, but is ok for a lot of people)