Can I send the output video feed to an external server?

RetroArch Additions Tools & Scripts
#1

Hi!

New here, please let me know if I missed this in another topic or in the docs. I’ve been wondering whether I can send the video output of an emulated game into a python script to analyze what I’m seeing in realtime and build on top of it. (Example: track the ball in pong, identify bananas in Mario Kart, read out the scorelines in FIFA, …)

Really, the goal for me would be to take some of my favorite games and try to monitor scorelines as well as emit events when something important happens. What I’ll do with that, who knows… Feed it into a mqtt broker, adapt my room lighting to the game events, whatever. Lulz, you know :slight_smile:

If this can be achieved any other way (like grabbing such events from the game running on a core itself), I’m happy to read into it as well.
But I assume that this wouldn’t be standardized or easily accessed from a compiled game, so reverse engineering events from the video feed would seem the most reliable to me.
I know I’d have to prepare this game by game, but it seems doable once I have the frames stream into a script where I can run it through the likes of OpenCV.

I’ve seen there’s a similar feature baked in for subtitle translation, where video output is sent through OCR, but if I understand correctly, that is happening only on a single frame and only on demand. If at all possible, I’d love if this can be adapted to constantly stream into a server and let me go wild on it.

Let me know if this sounds possible or completely stupid. Just trying to lay the foundations to a hobby project here… Also, thanks for any help already!

1 Like
#2

I don’t know of any way to send full video frames like that currently. You could potentially use the built-in video recording/streaming to do something like that, dunno.

However, most of the time when people want to manipulate stuff in retro games, they use RAM values from the games (see: the cheat interface) rather than analyze the video stream.

#3

Does RetroArch allow me to hook into these RAM values to trigger custom code? I’ll try to look for an intro into this, but if you have any good beginners guide that would help me, I’d be happy if you could share it.

Will this RAM introspection work for all the main consoles emulated by RetroArch, or only the older ones…? And does it expose anything in “readable” form or will I be dealing with weird pointers and whatnot?

#4

I’m not entirely sure how most people do it, tbh. You might check out this alternative frontend that is used to find achievements: https://github.com/RetroAchievements/RALibretro

You might also check out this guy’s video(s), since he’s working on something sort of similar (training machine learning to play video games): https://www.twitch.tv/videos/1019986031

#5

Thanks for pointing me at the RALibretro project. For older games this would be an absolutely perfect starting point.

However, they aren’t compatible with any"newer" consoles which seem to have more complicated RAM paging. So, for those, monitoring the screen output still seems the best bet.

I tried reading into the code of the screen recording plugin for RA, but that’s just too deep for me. No chance I’ll be able to adapt that.

I guess I’ll find another solution to grab a screen capture for my CV script (similar to this project: https://github.com/kevinhughes27/TensorKart)

1 Like