As you may know from my previous post, it has bothered me that the SNES Mini hasn’t been very well researched in terms of input lag. Especially not how RetroArch on the Mini performs. I’ve also wanted to see good comparisons to RetroPie and RetroArch on PC using the same display, to get a more complete picture. This post is my attempt at improving the knowledge on these things.
What I will do is test input lag using Super Mario World and the following setups:
- SNES Mini using both the built-in emulator (Canoe) and RetroArch
- RetroPie on Raspberry Pi 3 using default settings as well as various input lag reducing settings
- RetroArch on PC (Windows) on a high-end desktop PC with all known input lag reducing settings enabled/maximized
Importantly, I will be using the same gamepad for all tests: The SNES Mini wired controller. For the Raspberry Pi and PC tests, I’ll be using Raphnet’s awesome low-latency adapter to connect the controller to a regular USB port.
Let’s begin with a detailed list of specifications before we proceed to the results.
Test method & hardware/software setup
Super Mario World (NTSC) was used for the tests. The test scene was the very beginning of the level “Yoshi’s Island 2”:
I used my iPhone 8 to record videos of the monitor and controller at 240 FPS. I then counted the frames from the button appearing pressed down until the character on screen reacted (jumped), using the excellent iPhone app “Is It Snappy?” by Chad Austin. The results presented further down are based on 25 samples for each test case and the result is presented as number of frames of input lag at 60 FPS (i.e. the framerate the game runs at, not 240 FPS camera frames). Below is a screenshot of one of the recorded videos.
Note: It would have been better to have an LED connected to the jump button. However, I’m not about to take the soldering iron to my SNES Mini controller. Besides, a previous comparison I did showed a minimal difference (0.05 frames) in the average measured input lag between using an LED and not using an LED.
- Gamepad: Original SNES Mini wired controller
- Gamepad USB adapter: Raphnet Technologies Classic Controller to USB Adapter V2 (model number ADAP-1XWUSBMOTE_V2) - Firmware version 2.1.0. This adapter has a hard coded 1000 Hz USB polling rate (fastest rate the USB standard allows). The rate at which the adapter polls the controller was also set to 1000 Hz (again, the fastest setting available).
- Monitor: Samsung UE22H5005 (22" 1080p LCD TV). 1280x720 resolution was used for all tests, so that the results are comparable (720p is the resolution used by the SNES Mini). The same TV settings and the same HDMI input was used for all tests.
- 4:3 aspect ratio, no border
- Hakchi 2.21f
- retroarch-clover 1.0c (RetroArch 1.4.1)
- Raspberry Pi 3
- Original Raspberry Pi PSU
- RetroPie 4.3 (default image, with no updates applied)
- snes9x-2010 (this is the default SNES emulator)
- Core i7-6700K @ 4.4 GHz
- 16 GB DDR4-2667
- GeForce GTX 1080
- Windows 10 Version 1709 (OS version 16299.192)
- Nvidia GPU driver 388.13
- RetroArch nightly from November 12 2017
- SNES Mini: Default settings. There’s not really anything to modify that will improve the situation. The only possible change would be video_frame_delay, but that’s a very demanding setting so not really suitable for the SNES Mini’s weak hardware.
- RetroPie: I tested both default settings as well as the known settings that affect input lag. The results chart below indicates which settings that were modified for each test case.
RetroArch PC: The setup was optimized for the minimum input lag possible, using every bit of computational power afforded by the overclocked i7:
- video_threaded = false
- video_fullscreen = “true”
- video_windowed_fullscreen = “false”
Finally, just to be clear, vsync was enabled for all tests on all platforms.
Photo of the hardware
Here’s a photo showing most of the hardware used (but not the desktop PC):
Regarding input lag of the Samsung TV
I didn’t use my trusty, low-latency HP Z24i for these tests, since it doesn’t have HDMI (which is required by the SNES Mini). So, to make all measurements comparable, I instead used the Samsung UE22H5005 LCD TV for all tests. From both my own previous tests as well as testing done by Prad.de, we have strong evidence that the HP Z24i has negligible input lag (less than 1 ms). This may come as a shock to you, if you’re one of those who believe that all LCD displays must have a heap of input lag, but this HP monitor is not the only monitor in existence that has virtually no input lag (although the list of such displays isn’t very long).
In order to get a handle on how much input lag the Samsung TV have, I’ve run the RetroArch PC tests on both the Samsung and the HP display so that we can compare the difference. The HP display was tested at native 1920x1200 and the Samsung was tested at 1280x720. The results (average measured input lag for the Super Mario World test case):
- Samsung UE22H5005: 4.6 frames
- HP Z24i: 3.54 frames
Difference: 1.06 frames (17.67 ms)
So, given these results, we can assume the Samsung TV adds ~1 frame of total input lag to the figures presented in the chart below. In other words, to get how each system performs without taking the display into account, subtract 1 frame from the result.
As a side note, the result measured on the HP screen (3.54 frames) is the lowest input lag I’ve ever seen measured for emulated Super Mario World. Given the test scene used and given the fact that Super Mario World is designed to respond to input on the third frame after receiving said input, a real SNES on a CRT will, at best, achieve an average input lag of 3.3 frames. That means we’re some 0.2-0.3 frames or 3-5 ms behind the real thing.
The test results
All results in the chart below are reported as number of frames at 60 FPS, since that is the frame rate at which Super Mario World runs. So, to convert the figures to milliseconds, simply multiply them by 16.67.
First of all, remember that the monitor I’ve tested on has ~1 frame of input lag. So, to get the result of each system without taking the monitor into account, simply subtract 1 from all of the results.
We can see that the SNES Mini with it’s default emulator (Canoe) is pretty fast. A real SNES on a CRT would achieve ~3.3 frames in our test case and the SNES Mini achieves ~4.6 frames if we remove the Samsung TV’s input lag. That’s just ~1.3 frames (~22 ms) behind the real thing. That’s pretty awesome and a job well done by Nintendo, especially given the low computational performance of the Mini’s hardware. The real problem for most people will be that their TV’s add quite a lot of input lag on top of this.
We can also see that the default RetroPie is painfully slow at 8 frames (7 if we remove the Samsung TV’s input lag). Remeber that 8 frames is what we achieve with this comparably fast TV (1 frame of input lag is pretty much as fast as TVs go currently) and a very fast input method. Most people will use standard USB gamepads with standard USB polling rates (125 Hz) and TV’s that add 2 or more frames of lag. The average RetroPie user running a stock setup on his TV might therefore have a total input lag of ~10 frames (167 ms). That’s definitely very noticeable and quite distracting. Please note that a game with less built-in lag than Super Mario World might reduce that figure by 1-2 frames, but it’s still not looking very good.
It’s interesting to see how the RetroPie setup reacts when we, one by one, apply the known input lag reducing settings. Combining them all, we can actually match the SNES Mini. However, this is slightly misleading, as there are a few drawbacks to using these settings. Using the Dispmanx video driver means you lose the ability to use shaders as well as the on screen text (for example when saving). The video_max_swapchain_images=2 setting is also very demanding and many SNES games will not run fullspeed with it enabled. You probably can use it together with the other input lag reducing settings for select 8-bit and 16-bit games, but it would be a bit cumbersome to setup and in that case I’d recommend switching to a more powerful platform (such as x86) instead. Choosing the middle ground of using the Dispmanx driver and disabling threaded video is certainly possible. This works perfectly for NES/SNES and will put you within a frame of the SNES Mini, given a fast enough input device.
We also finally get some hard numbers for how RetroArch performs on the SNES Mini and it’s not pretty. It’s around 2.7 frames (45 ms) slower than Canoe and the difference is definitely noticeable. Exactly why RetroArch is this much slower is something I’ll leave to others to figure out, but my guess is that the difference doesn’t have to be this big. “Someone” should probably look into the video backend and possibly the input handling.
Last but not least, RetroArch on PC manages to edge out all other systems/setups. The difference is mainly thanks to the high performance allowing us to use frame delay to shave off an additional 14 ms (0.84 frames) and arrive at near-console performance.
I’ll end with a caveat: The Mini was tested with a single game. Some games inherently have less input lag than Super Mario World (such as Super Metroid, which responds on the second frame) and such differences affect all tested platforms. However, Nintendo could also be using per-game settings for Canoe that affect input lag. For example, it’s possible that they use additional buffering for games that are harder (more computationally) to emulate, to keep framerate high, which might in turn add additional lag. This is speculation at this point and will have to be the subject of a possible future test, but I’ll leave it here as something to keep in mind.
Thanks for reading another lengthy post!