Asynchronous audio/video for libretro

General Development
#1

This weekend I’ve been working on asynchronous audio for libretro. The idea is to make it far easier to run/port non-emulator games to libretro/RetroArch.

The “emulation” model is highly synchronous audio/video. We have pretty much solved this with dynamic rate control, but it’s not a good match for “real” games. Real games (especially 3D ones) have these properties:

  • Audio is mixed on a different thread, and is not synchronized to video in any way.
  • FPS can be highly variable, and the main loop steps in terms of a time delta.
  • Assets have to be loaded from disk at some point (for anything reasonably complex), stalling the main loop (absolutely requiring threaded audio).

I’ve added two new interfaces to libretro in my audiothread branch:

RETRO_ENVIRONMENT_SET_AUDIO_CALLBACK RETRO_ENVIRONMENT_SET_FRAME_TIME_CALLBACK

If audio callback is used, the normal audio_batch_cb() isn’t called every frame. Instead, RetroArch spawns an audio thread driver which continously calls the audio callback. In that callback, the core will call audio_batch_cb() with new mixed data. Because this runs completely separate from the rest of the core, this ensures no audio dropouts even when FPS crawls (assuming the drivers are not crap, ofc).

Cores using this system should also use frame time callback. Because audio is totally async, disabling vsync would be disastrous. Every frame, you get the frame time delta, and you can go from there. Potentially, you could use this for frameskip as well I guess. Because RetroArch assumes complete control over time, it can “fake” the deltas to support fast-forward, slow-motion, frame-stepping etc.

This feature kinda conflicts with synchronous things like FFmpeg recording, netplay, rewind etc, so async mode is disabled when those features are used. I don’t expect that to become a big issue.

I am still experimenting with these features, but they are quite promising from my tests.

Dynamic frameskip?
#2

I am intrigued by you mentioning we can use this for frameskip as well. Do you mean auto frameskip by this?

What if we would want to use this for usecases like emulators instead which still demand highly synchronous video/audio? Are the two new callbacks independent of it running in either sync or async mode?

We still require “something” for Android - I tried looking at Meancoots old JNI/Java frontend for libretro - hoping that would be better for performance - but instead performance was much, much worse - so it appears we were on the right track avoiding Java/JNI as much as possible. I think by adding “some” kind of decent autoframeskip mechanism we could silence most of the complaints - problem up to now has been though how to do it without the solution being bad/shaky.

#3

You could use frame time callback separately, yes. These two callback are intended for use with games/demos though, but frame time can stand alone and still be useful. Audio thread callback makes no sense for emulators whatsoever.

I think cores would have to implement their own strategies, I don’t think there is one single heuristic that will work well across every core ever regarding frameskip.

#4

Can these patches be cherry picked or are they dependent on the stuff that still has to be merged (ie. the libretro GL improvements)?

#5

Needs more testing. I’ll push up a pull request when it’s ready.