I’ve been thinking of picking up a modded console to use as a old school box for my TV but not sure which to get, I know the Xbox theoretically has the faster GPU but not sure if the PS3 port of Retroarch is more mature or not. Which would give me better shader performance that can handle multiple passes well? Obviously which of the two runs the emulators better is a good thing to know too 
PS3 by far.
On Xbox 360, every render target (ie. frame buffer object) has to fit inside the 10MB of EDRAM that the 360 GPU has. This is an insane limitation that unfortunately totally cripples the shader system that we use for both RetroArch PS3 and RetroArch PC. It is also the reason why so many commercial 360 games run at sub-720p resolutions - and why (speed considerations aside) this also spills over into PS3 ports.
This is currently the big albatross around the Xbox 360 version’s neck since it means we cannot fully implement the shader system and the different ‘scaling factors’ that people can set each pass to. Instead, we have to (maybe) in the future resort to a hack called ‘predicated tiling’ which lets you get away with render targets that are bigger than the total size of the EDRAM by slicing it up into separate batch draw calls that get put then on the command buffer list and processed by the GPU. This of course does have the disadvantage of adding latency and possibly screen anomalies.
But yeah - long story short - PS3 is definitely superior when it comes to shader capabilities. If it were not for the enforced arbitrary limitation of forcing render targets on EDRAM, the 360 would be superior. Unfortunately, that design decision on MS’ part ruins the entire thing for us.
RetroArch PS3 will always likely be the ‘more complete feature-rich port’ from a shader perspective for that reason with 360 limping behind (and limping rather severely behind by now).