For playing with Retroarch on the go I use either my phone or my 3DS.
I gotta say that the 3DS with its lower screen would have a lot of potential if it was possible to have the onscreen overlay in there to use as keyboard or additional controls (like mouse pointer). But this is currently not the case and its current Retroarch port is also very limited in terms of which cores are available for it, it also doesn’t support achievements (and you’d need a wifi connection for the retroachievements anyway, since I don’t think they work offline and you won’t have mobile network connection).
This might change in the future, but I’d say if you really want the best suited portable device currently available to run Retroarch on, go with an Android-based device.
The 3DS / PSP / PS Vita ports might not support all the features and cores, they are not as well supported as Android and they might be more troublesome to get homebrew to work. Unless you are actually keen on playing 3DS / PSP / PS Vita games, you are better off with an Android device that has gamepad controls like the GPD XD with the custom firmware (I’ve read the official GPD XD firmware doesn’t run Retroarch well)…