Live game translation in RetroArch, an idea

General
#1

In my free time I like translating retro games. I’ve been thinking of a good way to share my work with others for many months now, and nothing seems appropriate. I’ve been coming back to Secret of Mana recently for instance, there was quite a few things that I re-discovered by playing the original japanese version, but the whole thing translated is a little over 30 pages, with dialogues varying in importance from major plot point to random NPC chatter. I can’t simply release the document as it is, it would be a chore to read. Especially because most readers would only be interested in finding the interesting parts, those that were actually lost in translation at the time.

I’ve toyed with other ideas of course: publishing an in-depth comparison of the original translation with mine, even as a video, was my first idea, but it’s a lot more work and it cannot be exhaustive. In the end I believe the only way is to let people play the game and re-discover the content at their own pace, with the new translation on top. Naturally, ROM patching was out of the question. Part of the reason why Secret of Mana’s original translation was so poor was that the translator at the time was asked to cut his work in half, because of the storage space limitations. No, I needed something that could get rid of all constraints, hence the idea of live translation.

When I heard about ZTranslate and VGTranslate, I thought could be a solution. But, I don’t believe in what they do, or more to the point, in how they do it. OCR is unreliable, especially for Japanese (in some games, kanji are barely legible, even to a human), and the process is painfully slow. I am looking for something that would work in realtime, as to be completely transparent for the player.

I’m also looking for a way to provide a catered translation of a whole game, not a collaborative refinement of an automated process. I’m not against collaborative efforts, but translation is a peculiar thing. Most of the time there isn’t an “objective translation” so reaching some form of consensus on the translation of a game is impossible. My work isn’t necessarly better than anyone else’s, but it’s consistent. It was planned, and thought as a whole package, if you see what I mean.

So here I am, wondering if there would be a way to slap a translation live on top of a game through RetroArch in the future. The idea would be to use memory checks, similar to what RetroAchievements does actually. But instead of awarding a trophy for triggering specific conditions in memory, it would trigger a text to be displayed on top of the game.

Naturally, such a system would only work with hand-crafted translations, and would require translators to do extra work to map everything out, although with a big enough community mapping and translating could easily be done by different persons. Right now though I don’t mind doing both. I believe this is the only way to provide accurate realtime translation.

The problem I’m seeing is that RetroArch doesn’t support plugins, and it would be tough to include that in the base software unless there was a large following. (Although this project could technically accept translation efforts from pretty much anyone, even multiple translations for a single game, so it has potential). I suppose I’d have to provide a fork of RetroArch at first…

Why am I posting this, you wonder? Well I’d love to hear about anyone’s feeling on the matter. Would you find that interesting, as a player, as a translator, as a developer; could it be done differently, etc. Maybe ZTranslate is a better solution, and I just fail to see its potential? My idea involves a lot of work, so I’d rather find out what people think about it first :smile:

3 Likes
#2

Great, I have a nice project for you to do in your spare time.

It’s almost a travesty that’s as bad as the US launch of the TurboGrafx-16 itself that no one knows about these games. Based on what I’ve seen and heard, they might be in the top 5 games released on the platform, yet with the language barrier, I have heard no one in these parts even mention them.

A translation project had started maybe about 10 years ago now but just seemed to stall.

On your topic, I would really like to see this. I tried installing one open source real time translator available on GitHub and the insulation and setup process was probably harder than learning to read Japanese itself.

And after all of that it just didn’t work for me. I’m almost certain I did somehow wrong although I think I followed every instruction.

#3

Yeah sadly there are tons of great games that never made it out of Japan. I once made an extensive cataloguing of retro games, I have counted 971 games on the SNES alone that were exclusively released in Japan. To put things in perspective, 718 games were released for the console in the US. Of course there’s probably quite a few awful games, but that’s just an insanely high number. Add to that 656 NES games, 494 GameBoy games, 338 GameBoy Color games, 466 GameBoy Advance games… Yeah, enough to last a few lifetimes.

Legend of Xanadu makes me realize something else. A translation tool based on OCR has one major caveat: you can never handle games that have VO but not subtitles, which wasn’t uncommon in the early days of VO. Games like that would benefit the most from a memory-based tool.

2 Likes
#4

Have you already tried Ztranslate’s premade packages outside of Retroarch? This is supposed to be the way for humans to curate whatever the ocr outputs. Outside of Retroarch, I couldn’t get standalone Ztranslate to work correctly yet, haven’t tried the few premade packages that are available at the site yet.

#5

I haven’t been able to, no. Currently I’m having trouble registering on their website. But outside of RA I think it’s an even more cumbersome solution, you have to check both the game screen and a side screen with the translation. You can’t even play by only looking at the ztranslate screen because it’s very delayed and slow. A premade package doesn’t change the fact that the tool uses visual recognition based on screenshots taken at given intervals, so it can never be reactive enough for realtime. Perhaps I misunderstand how it works?

#6

The idea is of course to implement packages in RA since it aready supports Ztranslate itself. In RA the OCR results also appear very quick (less than 1 second afer key press, though I haven’t tried this yet with screens that feature a ton of text) for me. I can’t get the Ztranslate client to work correctly outside of RA to attach to emulators though, so I can’t comment on what it’s capabilities are. The package demo video mentions it does also allow user scripting for instance.

#7

I think live translation is an amazing thing.
I have time with translations in my head, if you think Snes has a lot of exclusive Japanese games, you have no idea about DOS/Windows games, there are thousands of really awesome games.

I was just going to tell you about this, I don’t know it, but I understand that it is used to make adjustments in the texts and corrections in the translations.

#8

Local packages or not, it’s still based on screen capture. You need to play the game, capturing and uploading all those screenshots to their server so they can go through OCR, before being able to correct them. But their bandwidth is limited, so you can only do so much before being forced into setting up your own paid plan with Google, if I understand this correctly.

There’s also the issue of speed, even with a local package you’re getting a lot of delay between the original text and the translated one. Ideally what I have in mind would feel like playing a patched ROM, without the need to patch anything.

Still, RA’s code base is dense and it’s been a good… 15 years since I had to work with pure C. Clearly, this idea isn’t going anywhere for a while unless I can motivate people or hire some help haha.

#9

Well, I can tell you that I configured it in its free version and the response was almost immediate, less than a second translating OCR. Then they implemented text-to-speech translation, I have not tested it. As far as I can see, it is now easier to configure.

I thought the Ztranslate files are something like a movie subtitle, which is something more like what you are looking for, I’m not sure, They seem to be having server problems, as he mentions on his discord account.
There they can give better information about the translation packages, maybe they are already working on something similar.
Like you, I don’t like the Google translation and the OCR reader, it is deficient.

I also didn’t know that they had implemented VO subtitle. I think you are confused about this, in the first video on the web, you can see a VO translation of German in real time, the screen on the right is the standalone emulator as a sample.

#10

This video highlights the problem perfectly though. Here you could not play normally, as you’d need to be constantly looking left and right to switch between the game and the translation. You can set it up through RetroArch to have everything in one place, but it doesn’t work for me. I’ve double check all settings, starting the AI service only pauses the game and nothing happens. (And apparently I’m not the only one having issues with the set-up process).

Even if we assume it works, you need to either be faced with the situation in the video you posted (looking at two different windows), or constantly press a key to pause your game and obtain the translation. You can also see in this video that the translation from a local package appears instantly for the game audio, but not when it comes to actual text. Because the software is waiting for the screen capture to parse. It’s the only way it has of knowing what text is being displayed, which is why it’s slow.

I’m sure that’s a sacrifice most would be ready to make, but this is less than ideal. And yes, to create a local package you first have to go through the game taking screenshots of everything, uploading them, OCR and ping Google’s API, before being able to correct the automated translation. From their own website:

How do I make a translation myself? There are two steps to making a translation: sourcing the images, and fixing the translations. (…) When you press “~”, it will grab a screenshot of the window in focus, send the image to the ztranslate.net server and display an automatic translation (this may take a few seconds).

Plus,

For the sourcing step, the ztranslate.net server does an API call to Google to OCR the text and another to translate the text. This costs about $1.50 per 1,000 calls. So, to source a game that needs 10,000 screenshots would cost about $30.

A memory-based tool would be a) faster b) easier to setup thanks to RetroArch and c) independant from web-based APIs.

#11

Yes, I know how it works.
You don’t need to have two windows open, what you are seeing is a demo, it is a project in development.

It would be wonderful to have such a tool, to load translation files (movie subtitle style). But, that has a problem, it only translates to one language.
The whole point of Ztranslate is to translate from any language to any language.
Now, with the packages maybe it can be done, I have no idea, maybe they are already working on that.

#12

what you are seeing is a demo, it is a project in development.

Ah, is that so? That’s an odd thing to use as a front page video to introduce a software :thinking: And yet if you’re not using ztranslate inside RetroArch, how does it work then? How can the translated text overlay the game without any additional software layer on top of the emulator?

But, that has a problem, it only translates to one language. The whole point of Ztranslate is to translate from any language to any language.

Well you’re only getting as many languages as there are translators involved in the project. If I release a japanese to english translation of a game, anyone else could release a french or german translation. They could even use my work as base for all I know.

But I get what you mean. However, unless a translation is done manually, ztranslate can only give you an automated translation… And believe me I’ve tried many translation tools, even so-called “AI” powered translators, and when you’re doing japanese to english for instance, the results vary from passable to downright nonsensical. So, in a sense, you’re also only getting as many language with ztranslate as you would with manual only.

Although, I must say both tools are not necessarily incompatible and if it works for some people, that’s fine by me. It’s just not doing what I need. For example, with what I had in mind it would become possible to turn on translation notes on the fly for those interested. There are many cases where I could make use of something like that, usually to explain things that cannot survive the translation process (jokes, reference to cultural events specific to the country, double meaning of names etc.)

The major issue right now is game compatibility. I’ve taken a look at the memory of Secret of Mana and I haven’t yet found a reliable way to detect which dialogue is being run on screen. I believe there must be something, like a dialogue ID, or the content of the character is slot 1 etc. but so far nothing.

#13

They are not selling hamburgers. You really don’t realize it’s the project’s bulletin board and progress.

The discord is very active, I’m sure they can give you a better answer.
If not, open a reward and wait for someone who wants to work on it.

#14

I don’t know. If I’m working on some software and I need to show what it does, I’m showing the end product, not Visual Studio, no matter who’s the target audience. But it doesn’t matter, that’s a different topic.

If not, open a reward and wait for someone who wants to work on it.

Perhaps in due time, but it’s a lot of work. There’s no way right now I can offer a fair enough reward for that much work, I need to flesh it out and hope I can convince a few people.

#15

I always wanted to ask this question, but never dated because i thought it’s so obvious it would have been done. I also clearly remember an SNES emulator doing this, the game in the middle and the translation on the left in an external window. i don’t remember if it used html or lua. At the time the argument was that translators wanted their project to work on real consoles too, but one doesn’t prevent the other, there is so much games that would benefit from this, and puting away all the hacking hurdles on many titles ! I feel like RetroArch would be perfect for this, i often thought about how the vertical black bars could be used for infos, movelist, notes, maps… It would be the dream :relieved:

#16

That remains to be seen. I’d say it removes some problems, but not all. Having to hack a new font, decompile sometimes encrypted ROMs, being limited by filesize, by text box size and count. All that can become meaningless, but we still need to delve into the memory of the game…

You’re right that such a tool could be used for other things beside translations. You could have personal notes popping up when necessary. If the memory of a game is already mapped out I think it could be rather intuitive even for non-tech savvy users.