HowTo: Generate Pretty, Curated MAME Playlists the Easy(ish) Way

More worthwhile clones suggestions:

sfiii3n.zip - A clone of Street Fighter III Third Strike that doesn’t require CHD (it’s 67.3 MB, cointains everything it needs). Others CPS3 games have NO CD versions too. hcastlek.zip - A clone of Haunted Castle (Castlevania) with rebalanced difficulty - actually playable fairly (the original is pretty much 1 hit KO after a couple of screens) svcplus.zip, svcplusa.zip or svcsplus.zip - Clones of SVC - SNK Vs. Capcom. They have the secret characters available without having to input any select screen code (the most unique characters are secret, like Zero, Mars People, etc). The later one (Super Plus) let you select all of them – the other two have some of them available while keeping others hidden, like the two hidden bosses (Athena and Red Arremer).

Big update I finally plowed through. See changelog in first post. The only major thing I didn’t address was the suggested speed ups to the Friendly Image Renaming AHK. Spent my gumption elsewhere. If anyone wants to mess with that themselves and submit an updated script that’d be grand.

Made 31 or so new icons. Now all 15 major categories and a few others have at least one content icon (sprite). Icon pack download is in Step 5. I’m mostly happy with the selection so I probably won’t make many more. It’s funny, beyond the SF2 fireball I can’t think of a single other sprite that’d work well for Fighting… just out of curiosity, any thoughts?

@chrisq In case you haven’t died of old age I finally got those batch files together. They’re linked in Step 1.

@spinningacorn Re: our convo, the newly expanded ~~~TidyUp.bat (available in the pre-compiled batch file pack or Step 3’s Pastebin link) has rudimentary BIOS-file copying for CPS2 and Neo Geo.

Maybe a punch or kick icon like you see in command lists? Or a ‘KO’ like this one:

Hi Alexandra – I’m interested in one day adapting your work for use on Lakka with MAME 2003, so I thought a first step might be to take you up on this offer and work with your AutoHotKey scripts. Your scripts are derived from the same starting points as the scripts I currently use with MAME 2003, so I ‘speak their language’!

I applied a few speed updates that I worked out for the MAME 2003 playlist generator. One of them I’d mentioned to you before, but here’s the complete list:

[ul] [li]Replaced the regex in the illegal character sanitizing code for both AHK scripts with StrReplace which has a faster implementation in AutoHotKey [/li][li]Changing the playlist generation code to use File.Write and File.Close which have a faster implementation than the AHK File Append feature [/li][li]Load the file lists for ROMs and PNGs into memory and searching for matches in memory rather than consulting the filesystem [/li][/ul]

Other notes:

[ul] [li]The Tetsuya79/reoldemort originals missed a couple of the illegal characters that the RetroArch filename policy forbids. At the same time I changed the RegEx sanitization to use StrReplace I also matched up that code with current RetroArch filename standards. [/li][li]I added a provision for Boxart thumbnails by cloning your work on Snaps and Titles. Kivutar from the dev team has said that they intend for Arcade Flyer art to be used for Boxart with RetroArch. I hope you don’t mind the addition! [/li][/ul]

I think I have been able to successfully blend the relevant code from my ‘forks’ of the roldemort and Tetsuya79 code with yours. I have done some spot testing and the results look right. If you have a chance to try them out I would be interested in hearing your results.

Image renaming AHK: https://github.com/markwkidd/retroarch-arcade-playlist-helpers/blob/master/friendly-image-renaming.ahk Playlist generation and thumbnail copying: https://github.com/markwkidd/retroarch-arcade-playlist-helpers/blob/master/curated-arcade-playlist-generator.ahk

I looked at your code here https://github.com/markwkidd/lakka-arcade-playlist-helpers/blob/master/friendly-image-renaming.ahk . Nice improvements!

Three tips for speeding things up more (adapted from my post upthread ). I’ll suggest line numbers in your script where you’d insert these snippets.

1 Add after line 4 to maximize Autohotkey speed


SetBatchLines -1

2 Trim the dat before the RegExMatch steps. Replace your line 39 with

FileRead, datcontents, %dat%
Loop, Parse, datcontents, `n, `r
 If (InStr(A_LoopField, "<game name=") or InStr(A_LoopField, "<description>"))  ;only keep relevant lines
  slimdat .= A_LoopField "`n"
datcontents := slimdat

3 Skip unwanted dummy images. For example the MAME snaps set pS_snap_fullset_170.zip has about 20K snaps with the MAME mechanical/screenless/device placeholder image. Useless for RetroArch purposes so let us filter out them by filesize. That only takes a few seconds. In the code upthread I moved the dummy images to a separate folder but the below code instead simply skips them. (Note: This step needs testing. The dummy sizes is tested on one MAME snaps set. Will not work well if updates to the snaps set changes the dummy file sizes. But perhaps those files are stable by now? This step will also skip other snaps if they happen to have the same filesize, though there were no such cases when I earlier tested with the pS_snap_fullset_170.zip . A more reliable method is to test file checksums. But that takes much longer time for 40K snaps so should be done by a separate, one-time preprocessing script that moves/deletes the dummy files.)

Insert after your line 43

;Skip known dummy images
;57825 bytes mechanical 9K images
;56475 bytes device  2K
;56467 bytes device  1K 
;53466 bytes screenless system 9K
;That leaves ~14K images in pS_snap_fullset_170.zip
FileGetSize, s, % A_LoopFileFullPath
if (s==57825) or (s==56475) or (s==56467) or (s==53466)
 continue

[QUOTE=roldemort;50629]I looked at your code here https://github.com/markwkidd/retroarch-arcade-playlist-helpers/blob/master/friendly-image-renaming.ahk. Nice improvements!

3 Skip unwanted dummy images. For example the MAME snaps set pS_snap_fullset_170.zip has about 20K snaps with the MAME mechanical/screenless/device placeholder image. Useless for RetroArch purposes so let us filter out them by filesize. That only takes a few seconds. In the code upthread I moved the dummy images to a separate folder but the below code instead simply skips them.

(Note: This step needs testing. The dummy sizes is tested on one MAME snaps set. Will not work well if updates to the snaps set changes the dummy file sizes. But perhaps those files are stable by now? This step will also skip other snaps if they happen to have the same filesize, though there were no such cases when I earlier tested with the pS_snap_fullset_170.zip . A more reliable method is to test file checksums. But that takes much longer time for 40K snaps so should be done by a separate, one-time preprocessing script that moves/deletes the dummy files.)

Insert after your line 43

;Skip known dummy images
;57825 bytes mechanical 9K images
;56475 bytes device  2K
;56467 bytes device  1K 
;53466 bytes screenless system 9K
;That leaves ~14K images in pS_snap_fullset_170.zip
FileGetSize, s, % A_LoopFileFullPath
if (s==57825) or (s==56475) or (s==56467) or (s==53466)
 continue

[/QUOTE]

Firstly, I apologize for missing your speed hacks earlier in the thread. Thanks for helping work them into my version of the script.

I just made the first two alterations verbatim. Regarding the third: It looks like you are only working with a list of dummy file sizes for the snapshots folder. Title screen dummy files have different file sizes to check for, so I have added them to your code. Maybe there is some more testing to do with this dummy removal code, but it seems pretty dang useful. I’d rather start with that in place and remove it if there are unintended consequences.


    ;### Skip known dummy images
    ;57825 bytes mechanical
    ;56475 bytes device
    ;56467 bytes device
    ;53466 bytes screenless system
    ;53472 bytes screenless system
    ;9970 bytes screenless system
    ;55932 bytes device
    ;57281 bytes mechanical
    ;That leaves ~14K images in pS_snap_fullset_170.zip
    FileGetSize, s, % A_LoopFileFullPath
    if (s==57825) or (s==56475) or (s==56467) or (s==53466) or (s==9970) or (s==53472) or (s==55932) or (s==57281)
         continue

Here’s a link to the script, reflecting all these changes in the master branch: https://github.com/markwkidd/retroarch-arcade-playlist-helpers/blob/master/friendly-image-renaming.ahk

Nice! But I now noticed another variable typo error in my snippet number two. Line 42 in your new github code reads

 If (InStr(A_LoopField, "<game name=") or InStr(A_LoopReadLine, "<description>"))  ;only keep relevant lines

but should be changed into

If (InStr(A_LoopField, "<game name=") or InStr(A_LoopField, "<description>"))  ;only keep relevant lines

Sorry about that. I will fix the error in my post above too, in case someone comes and uses only the snippet.

[QUOTE=roldemort;50646]Line 42 in your new github code should be changed into

If (InStr(A_LoopField, "<game name=") or InStr(A_LoopField, "<description>"))  ;only keep relevant lines

Sorry about that. I will fix the error in my post above too, in case someone comes and uses only the snippet.[/QUOTE]

Thanks! I made the correction.

I tested the github code now. Your line 78

 RegExMatch(datcontents, needle, datname)

need a change into

 RegExMatch(datcontents, "U)" needle, datname)

Without that change the script only save one image.

I ran a test on “MAME - ROMs (v0.179_XML).dat” and “pS_snap_fullset_170.zip” The test took 66 seconds to complete . Pretty good optimization from the one hour for the script on the first page of this thread! :slight_smile: Almost all time is spent on the regex step. I think that can be cut down further. Give me a few minutes…

You caught me. I’ve only been dropping in individual image files to rename as I’ve been testing this! :slight_smile: Good catch.

I made the change. I also renamed the github repository, since this script isn’t really a Lakka script (yet).

A new link to the file: https://github.com/markwkidd/retroarch-arcade-playlist-helpers/blob/master/friendly-image-renaming.ahk

I’ll edit my prior posts to point to this location.

New version runs the same test in 13 seconds. Method: the loop first finds the filename position in the slimmed dat and starts a simplified regex from that position.

The new version replaces your lines 66 to 105 with this

posi = 1
needle = <description>(.*)</description>

Loop, Parse, ThumbnailFileList, `n, `r
{
if A_LoopField =
    continue
SplitPath, A_LoopField,,,,filename     ;### trim the file extension from the name
posi := InStr(datcontents, """" filename """",false,posi)  ;filename position in datcontents
RegExMatch(datcontents,"U)" needle, datname, posi) ;start regex from filename position
if !datname1
 continue
newname := char(datname1)
FileCopy, %artsource%\%filename%.png, %destinationfolder%\%newname%.png, 1
}

;change forbidden chars
char(x) {
x := StrReplace(x, "'", "'")
x := StrReplace(x, "&", "_")
x := StrReplace(x, "&", "_")
x := StrReplace(x, "\", "_")
x := StrReplace(x, "/", "_")
x := StrReplace(x, "?", "_")
x := StrReplace(x, ":", "_")
x := StrReplace(x, "<", "_")
x := StrReplace(x, ">", "_")
x := StrReplace(x, "*", "_")
x := StrReplace(x, "|", "_")
return x
}

This new version also saves more images to destinationfolder (13507 images compared to 12629 images). Not sure why there is a difference.

deleted. weird double post

Not to muddy the waters, but there has also been a development in the libretro-thumbnails repo today. A complete set of MAME thumbnails was accepted: https://github.com/libretro/libretro-thumbnails/tree/master/MAME

That set was generated using a prior variation on this same script. The filenames in the RetroArch MAME thumbnail pack are therefore a 1:1 match for what we’ve been working with here, for games up to MAME 0.159. I’m thinking now about how that thumb set can relate to Alexandra’s work here, which I think is ultimately the way I would like my own collection to be displayed. 8)

Back to the script at hand. I merged in your changes, although slightly altered per my OCD naming and formatting preferences.

I can’t really say why there would be more files with your new version, but I’ll tell you why I got interested in speed enhancements to the original script by yourself and Tetsuya79. After I modified your original work to suit MAME 0.78 and Lakka a few months back, I was having erratic results.

Each time I would run the playlist generator it would come up with a different number of playlist entries. After a while I came to realize that if I ran the same script on a faster CPU I stood a better chance of getting the full list of ROMs added to the playlist, but it still wasn’t a guarantee. AutoHotKey was failing in some silent way, apparently ‘giving up’ on matching some playlist entries if it was taking too long.

Once I incorporated my speed increases from upthread, I started getting full MAME 2003 playlists every time. I wouldn’t say whether that is related to your experience now, but it is at least an interesting coincidence.

Good to hear.

Well, if at some later time there is a need to update the MAME snaps in the repo my last code change will save the person doing it approximately 50 seconds. Enjoy that chunk of extra life! :smiley:

I also hope RetroArch will get an option for some type of submenues inside the MAME playlist, something like in Alexandra’s first post.

About the difference in number of output files. I compared the outputs of the newest version (with the posi steps; 13507 images in total) and the previous version (more complex regex; 12629 images). The new output hade 1128 images not in the old output. But the old output also had 250 images not in the new output! Which is weird, not sure why yet.

Anyway, if the github set is complete for the games supported by the latest MAME core then all is good anyway. Though this weird difference could be useful to troubleshoot when moving the core to a newer MAME version.

edit: appears there is 13122 images in https://github.com/libretro/libretro-thumbnails/tree/master/MAME/Named_Snaps

We may be pushing some limit of AutoHotKey, or stumbling on a bug with their code. I don’t really see where the discrepancy would come in from the script.

About the number of files in the github thumbnail repo: When I put together the github set, I started with MAME 0.159, then added in thumbnails with the different name variations used by the Progetto SNAPS datfile in 0.139, 0.78, and 0.37b5. So the current repo is focused on the ‘fixed’ or ‘historical’ libretro MAME cores.

It would only take a few tweaks to this script as well, as some scripted resizing for images wider than 512px, in order to bring the github repo up to date for MAME 0.170.

Love what I’m seeing here; thanks for the all the optimizations. I’ll update the top posts when the dust clears.

I am slightly concerned that thumbnails for some legit games may end up getting filtered out, given the output discrepancies y’all are noticing. I may test later on.

hunterk: Great icon idea, though kind of annoying to sprite-rip from the arcade version. :slight_smile:

It irked me that the newer code weirdly failed to rename some snaps that the old code had no problem with. So I took another look now. This fixes the issue. Replace line 73 in https://github.com/markwkidd/retroarch-arcade-playlist-helpers/blob/master/friendly-image-renaming.ahk with

posi := InStr(datcontents, "game name=""" filename """",false,posi)     ;### filename position in datcontents
if !posi
 posi := InStr(datcontents, "game name=""" filename """")
if !posi
 continue

There were actually two errors. 1 My InStr previously looked only for the filename e.g. “ckong”. But e.g. cloneof=“ckong” comes earlier than game name=“ckong” in the .dat, which tripped things up. Checking for “game name=“ckong”” fixes that. 2 I had assumed the .dat game names were A-Z sorted. But after the game “zzyzzyxx2” the .dat list is in fact unsorted. When a file A was located in that end part the next file B in the loop wasn’t found because the search for it started way down at A’s position near the end of datcontents.

Test results with this script: outputs 13712 renamed files for “MAME - ROMs (v0.179_XML).dat” and “pS_snap_fullset_170.zip”. outputs 11935 renamed files for “MAME 159.dat” (source) and “pS_snap_fullset_170.zip”.

Still differs from the 13122 in the https://github.com/libretro/libretro-thumbnails/tree/master/MAME/Named_Snaps .

That may explain the difference.

(doublepost)

It would be an incredible coincidence if the processed 170 set had the same number of files as the github repo. All new games (and games that have been renamed) from MAME 0.160-0.170 are missing from the repo, and then there are also some additional files in the github repo corresponding to names from pre-0.159 MAME DATs. So the total number is close, but that is unrelated to our question of the script processing all of the thumbs.

Think of the github repo as a ‘rollup’ set designed from 0.37b5-0.159 – it has more than any one of those sets alone, but is missing the very newest stuff entirely. Maybe I’m repeating myself, but with so many MAME thumbnail scripts being discussed I want to try to be clear what the difference is.

At any rate, I’m excited to make your latest update roldemort, on the assumption that you have zapped the source of this discrepancy in thumbnail counts for 0.170. Rock and roll. Next time I have a dedicated few hours I will try to create an update to the github repo that brings in all of the 0.160-0.170 thumbs.