I just wanted to create this topic in case someone else has this issue. It was something I figured out by myself. It may help on your device also, it did massively on my.
It is important to turn off any overlay from Android. My phone has a program in the app drawer called Games. Retroarch got put there automaticly. That means when I run Retroarch I can use a overlay to show fps etc. Its also suppose to make the game run faster. However it caused massive slowdowns for Retroarch and when I removed Retroarch from it all stuttering was gone. So DONT use these Android overlays and game modes. They dont seem to help at all. At least not in Retroarch.