Howto use the audio callback in a core

General Development
#1

Hello,

I’m currently porting a old game engine to libretro and I’m having some problems to understand how a core is supposed to use the audio callback. The main issue is how to cleanly shut down the core’s audio without knowing when the audio callback thread has stopped.

This is what is going on on a high level:

main thread:

  • initialize game
  • spawn audio mixer thread
  • register audio callback via environment_cb(RETRO_ENVIRONMENT_SET_AUDIO_CALLBACK)
  • render frames repatedly in retro_run

audio mixer thread:

  • mix audio into a back buffer
  • wait for a front audio buffer currently being played to be finished playing
  • swap front and back buffers and repeat

(frontent)audio callback thread:

  • wait for a buffer swap from the audio mixer thread
  • upload the new front buffer to the frontend using audio_batch_cb
  • signal the audio mixer thread that the buffer has been uploaded
  • repeat

This runs quite well until I want to unload the game. I have no idea inside of the core what the state of the audio callback thread is. In an older version of retroarch (v1.7.3) it seemed like this thread was torn down before unloading the core. But the most recent version from github seems to have changed. The audio callback thread keeps running until the retro_unload_game completed.

What should a core do in this case? I tried deregistering the audio callback at the beginning of retro_unload_game but then again the core has no idead when the audio callback thread is actually stopped. I tried signalling my code running in the audio callback to stop waiting for a buffer swap and stop accessing any datastructures from the audio mixer thread. But the audio callback thread might actually be blocked inside audio_batch_cb where I have no control over when this will return. Actually I don’t even know if the audio callback thread is actually still runnning as other frontends might treat this differently.

The only thing that could work in my mind is keeping my audio thread around until after retro_unload_game returns and clean it up later in retro_deinit. But even this is making assumptions when the audio callback thread is being stopped. I didn’t find anything mentioned regarding this in the docs or elsewhere. Also in my case the game code will create the audio thread after loading content which defines properties like sample rate and the mixer buffer size use by the mixer thread. So keeping it around until after retro_unload_game is cumbersome as it actually means I cannot unload the game really as this would tear down the audio subsystem inside of it hosting

I hope I’m missing something really abvious. Is there a way to handle this situatiuon gracefully? Should I simply not use the audio callback?

Thanks in advance!

#2

spawn audio mixer thread

You should not need an audio mixer thread as the frontend will handle that for you.

I’ve put together a small framework called pntr_app that leverages the audio callback. You could use it as an example:

#3

Hi RobLoach, Thank you for the quick reply. I looked into your code to see how it avoids any race conditions when unloading the game/core. Let me see if I understand this correctly:

  • Your code doesn’t use an extra thread to do the mixing but calls audio_mixer_mix from libretro-common directly inside of the audio callback
  • audio_mixer itself is thread safe so the main thread can trigger playing some voices while the audio callback runs the mixing in a thread in the frontend
  • retro_unload_game calls pntr_app_close which calls audio_mixer_done
  • audio_mixer_done cleans up allocated data for all the voices by
    • locking the voice mutex
    • deallocating voice data
    • unlocking the voice mutex
    • destroying the voice mutex
    • nulling the voice mutex
    • src is here
  • while retro_unload_game cleans up the mixer the audio callback still runs because audio_driver_stop is called after retro_unload_game (src here)

I hope I’m wrong but your code still looks like it has a race condition to me. It’s not very likely but what if:

  • audio_mixer_done unlocks a voice
  • audio_mixer_mix gets the lock for the voice right after
  • audio_mixer_done destroys the voice’s mutex
  • audio_mixer_mix tries to unlock the voice’s mutex that is already destroyed which is probably not good

Is there something I’m missing there?

Thanks for your help!

#4

one of our core authors on discord said this:

He shouldn’t create his own audio thread, that’s what the frontend’s supposed to do. He should just give the audio_batch_cb the generated samples within the callback he set with RETRO_ENVIRONMENT_SET_AUDIO_CALLBACK (assuming it’s available – if not, just do it on the main thread)

#5

@hunterk Thanks for forwarding this to discord. Really appreciate getting some help.

Ok. Let’s take the custom audio thread out of the equation. It is mainly there so I had to change less in the codebase I was porting to libretro. But let’s assume that I change the code so that the audio callback directly calls the mixer code without any extra threads involved.

This code then needs to be thread safe of course so the main thread can play/stop voices while the mixing code also accesses them from the audio callback. But the problem when tearing down remains. There will be dynamically created synchronization primitives like mutexes, semaphores and the like created by the core for the sake of thread synchronization. Those need to be freed up at some point when it is safe to assume that the audio callback will not fire anymore. From looking at the retroarch code it seems like this is after retro_unload_game when audio_driver_stop is called in retroarch afterwards. So retro_deinit seems to be the first place in the core where one can assume that the audio callback will not fire anymore. But since this isn’t specified anywhere in the documentation other frontends could behave differently. In fact an older versions of retroarch do this differently by calling audio_driver_stop before retro_unload_game.

My main question is: How should I tear down the thread synchronization primitives needed in the audio callback when there is no reliable way of knowing that the callback ended and is disabled?

#6

same guy’s reply:

retro_audio_set_state_callback_t is that reliable way of knowing when you’re done. You’re registering a pair of callbacks with RETRO_ENVIRONMENT_SET_AUDIO_CALLBACK , right? The second being set_state ; use it

and from me: if you want to keep it hanging around until retro_deinit, I don’t think that’s a problem, really.

#7

Aha! So the set_state part of the audio callback fires on the main thread. I though it would also come from the audio driver thread and wondered what it’s purpose was.

While I could clean up things when the state is being set to false I would also have to recreate it again when it is set back to true again as nothing in the docs says that this cannot happen. I think simply doing the cleanup in retro_deinit is simpler then indeed. Thanks for clarification.

Thanks again @hunterk for forwarding. I’ll ask any further questions directly in discord to save you some work.

1 Like