There are several problems why the MAME HLSL vector shader cannot be ported (easily) and the shader itself is none of them.
Actually MAME’s vector.fx shader is not the typical pixel shader. This shader is not applied to the full-screen canvas. It is applied to each vector line, which is also just a rectangle. And the only thing this shader does is rendering round corners to those rectangles, to get smooth line ends, that connect nicely with each other.
RA has no access to those rectangles, it just gets the already rendered vectors as a texture. This is done by the MAME core using DirectDraw - if I’m not mistaken - instead of DirectX. Unfortunately the line rendering of DirectDraw in MAME is flawed. Not only does it not render round corners, it also doesn’t render the lines as rectangles. The result looks more like a parallelogram, where one side is always aligned either to the x or y-axis, depending on the line angle. This becomes apparent when increasing the line thickness.
What you currently can achieve with RA is something like this. (I only invested some minutes, so it’s far from perfect)
What you need to do is:
- Enable the Alternative Renderer in the MAME core settings (otherwise the rendered vector image will only have a size of 640x480)
- In the Slider Controls of MAME increase the Maximum Beam Width to e.g. 5.0
- Use a shader that supports some kind of blurry glow effect (controlling the brightness would also be beneficial)
I used my own shader chain CRT Yah! in the example above. (deactivated scanlines/mask, maximum brightness and halation)