Turns out that I was doing things in a different order than Retroarch does things.
RetroArch’s Order:
- Init Symbols
- Init Environment Callback
- Init Core
- Load Content
- Init Callbacks (video, sound, input, etc)
My order:
- Init Symbols
- Init Callbacks (video, sound, input, etc) <- mesen crashed here
- Init Environment Callback
- Init Core
- Load Content
The crash happened during while initializing the Audio callback. Changing the order to match Retroarch stopped the crash. Strange how only Mesen seems to care about what order these steps happen in.
edit: Note that Mesen is probably too slow to use with runahead, it would need more optimizations first.