HowTo: Generate Pretty, Curated MAME Playlists the Easy(ish) Way

RetroArch General
#6

Really nice!

1 Like
#7

Thanks for the nice response. Sticky, cool!

markwkidd, good tip. If I tinker with the code again I will try that. I’ve been thinking about how the second script, too, could be a lot more functional, requiring less effort from the user. Not sure I’ll have the gumption to redo it. But maybe!

I’ve been spending way too much time making icons. Just uploaded 75 new ones. They’re mostly headers, most genres still lack content icons. Decided to stick with the convention artwork for headers, sprites for content.

Here’s a glance at everything that exists so far (click for full res): In the case of future updates I’ll just post the new ones. Will hopefully fill in the less covered genres and get at least one sprite-based content icon for each category. For now, though, my wrist hurts.

#8

[QUOTE=Alexandra;45086]markwkidd, good tip. If I tinker with the code again I will try that. I’ve been thinking about how the second script, too, could be a lot more functional, requiring less effort from the user. Not sure I’ll have the gumption to redo it. But maybe! [/QUOTE]

Thanks!

Two thoughts about thumbnails: * If you’re interested in a proposed set of ‘official’ MAME thumbnails you may want to stop by this pull request, where the script has been put to use: https://github.com/libretro/libretro-thumbnails/pull/80 * I don’t know if my version of the thumbnail renamer script would be useful to you either, but in case it is, I have modified the original to automatically rename an entire folder of thumbnails based directly on the MAME datfile rather than generating a RetroArch playlist first and then using the playlist to rename the thumbnails.

;### AUTOHOTKEY SCRIPT TO RENAME ARCADE THUMBNAILS FOR RETROARCH

;---------------------------------------------------------------------------------------------------------
#NoEnv                          ; Recommended for performance and compatibility with future AutoHotkey releases.
#Warn                              ; Enable warnings to assist with detecting common errors.
SendMode Input                  ; Recommended for new scripts due to its superior speed and reliability.
SetWorkingDir %A_ScriptDir%      ; Ensures a consistent starting directory.
;---------------------------------------------------------------------------------------------------------

;### SETUP: ADD YOUR PATHS HERE

artsource = D:\Emulation\MAME RetroArch Thumbnails\mame 037b5	itles-resized
;### Path to the source thumbnails folder on the local machine
;### DO NOT INCLUDE A CLOSING SLASH AT THE END OF THE PATH
;### Example: C:\MAME 0.78 images	itles

processed = D:\Emulation\MAME RetroArch Thumbnails\MAME Onmnibus Thumbnails\Named_Titles
;### NOTE: THIS FOLDER MUST EXIST BEFORE THE SCRIPT IS EXECUTED
;### Path to the destination thumbnail folder on the local machine
;### DO NOT INCLUDE A CLOSING SLASH AT THE END OF THE PATH
;### Example C:\MAME 0.78 images\Named_Titles

dat = D:\Emulation\Original ROM Sets\MAME 0.37b5 Non-Merged\MAME 0.37b5.dat
;### Example: C:\MAME\dats\MAME 078.dat
;### local path to a MAME ROM database file

if !FileExist(dat) or !FileExist(artsource) or !FileExist(processed)
 return     ;### If any of these files and folders desn't exist, exit the script

;### Delete old 'ummatched' log file if it exists 
FileDelete, Unmatched Thumbnails - %dat%.log

FileRead, datcontents, %dat%
FileDelete, %processed%\Unmatched Thumbnails.log    ;### Delete old 'ummatched' log file if it exists
rawcounter = 0
A_Loop_File_Name = 0

Loop, %artsource%\*.png {
 
    SplitPath, A_LoopFileName,,,,filename     ;### trim the file extension from the name

    needle = <game name=.%filename%.(?:| ismechanical=.*)(?:| sourcefile=.*)(?:| cloneof=.*)(?:|  romof=.*)>\R\s*<description>(.*)</description>
    RegExMatch(datcontents, needle, datname)

    fancyname := datname1    ;### extract match #1 from the RegExMatch result
    
    if !fancyname {
        fancyname := filename   ;### the image filename/rom name is not matched in the dat file
         FileAppend, Unmatched Source %artsource%\%filename%.png`n`n, %processed%\Unmatched Thumbnails.log           ;for error checking
        continue                 ;### skip to the next image
    }
    
    ;### Replace characters unsafe for cross-platform filenames with underscore, 
    ;### per RetroArch thumbnail/playlist convention
    fancyname := StrReplace(fancyname, "'", "'")
    fancyname := StrReplace(fancyname, "&", "_")
    fancyname := StrReplace(fancyname, "&", "_")
    fancyname := StrReplace(fancyname, "\", "_")
    fancyname := StrReplace(fancyname, "/", "_")
    fancyname := StrReplace(fancyname, "?", "_")
    fancyname := StrReplace(fancyname, ":", "_")
    fancyname := StrReplace(fancyname, "<", "_")
    fancyname := StrReplace(fancyname, ">", "_")
    fancyname := StrReplace(fancyname, ":", "_")
    fancyname := StrReplace(fancyname, "*", "_")
    fancyname := StrReplace(fancyname, "|", "_")

    destinationfile := processed . "`\" . fancyname . ".png"
;    if FileExist(destinationfile)
;    { 
;    continue    ;### skip onward if this file already exists
;    }    
    FileCopy, %artsource%\%filename%.png, %destinationfile% , 1
}
Organization?
#9

Can someone link to controls.xml? The site appears to be down that usually hosts it.

#10

Here you go, uploaded it to Mediafire. It is version 0.141.1, the latest version at this time.

#11

Thanks Alexandra! Worked great. I didn’t go through all the steps just yet - for now I just used this to cull a massive amount of ROMs :slight_smile: Such an amazing shortcut to get rid of all the silly dupes etc that bloat MAME sets nowadays.

#12

Icon pack updated with 15 new headers. Taking a break for real this time. :stuck_out_tongue:

#13

Cool guide!

I’ll add a few optimization notes on the AutoHotKey processing in step 4. Your “Friendly Image Renaming Script” copies and renames all of the 36K + 35K images. Overkill since earlier steps filtered out a ton of games. A quicker approach is to loop over roms in the created subfolders and only copy/rename their corresponding images. That would take a bigger rewrite. But here are three quicker tweaks

First put

 SetBatchLines -1

at the top of the Friendly Image Renaming Script to increase the script speed, https://autohotkey.com/docs/commands/SetBatchLines.htm

Second slim down the dat file with the below code (takes only a few seconds) and then use “~slim.dat” in Friendly Image Renaming Script instead

 SetBatchLines, -1
x = C:\MAME Roms\~MAME - ROMs (v0.176_XML).dat
y = C:\MAME Roms\~slim.dat
Loop, Read, %x%, %y%
{
If (InStr(A_LoopReadLine, "<game name=") or InStr(A_LoopReadLine, "<description>")) 
 FileAppend %A_LoopReadLine%`n
}

With those two changes to the Friendly Image Renaming Script the time to rename 200 images dropped from 27 to 5-6 seconds. So your one hour might drop to 10-15 minutes or so.

But it gets better. We can filter out some obviously unwanted images before running the Friendly Image Renaming Script. For example the snaps set pS_snap_fullset_170.zip has about 20K snaps with the MAME mechanical/screenless/device placeholder image. We can filter them by filesize and move them to another folder (also only takes a few seconds).

;57825 bytes mechanical  9000 images
;56475 bytes device     2000 images
;56467 bytes device     1000 images
;53466 bytes screenless system 9000 images
;that leaves ~14000 images in pS_snap_fullset_170.zip

FileCreateDir, C:\MAME Roms\~SnapsSkip\
Loop, Files, C:\MAME Roms\~Snaps\*.png
{
FileGetSize, s, % A_LoopFileFullPath
if (s==57825) or (s==56475) or (s==56467) or (s==53466)
 FileMove, % A_LoopFileFullPath , C:\MAME Roms\~SnapsSkip\
}

With all these three tweaks applied the Friendly Image Renaming Script took 2 minutes to process the pS_snap_fullset_170.zip on my system.

edit:

Here is Alexandra’s script with these two tweaks added in

 #NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn  ; Enable warnings to assist with detecting common errors.
SendMode Input  ; Recommended for new scripts due to its superior speed and reliability.
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetBatchLines -1 ;max script speed 

;AUTOHOTKEY SCRIPT TO GENERATE FRIENDLY IMAGE NAMES FOR RETROARCH PLAYLIST USE
;"Friendly Image Renaming Script"

;### SETUP, ADD YOUR PATHS HERE

content = C:\MAME Roms\~Snaps
;### Path to folder containing MAME Snap or Title images

destination = C:\MAME Roms\~SnapsFriendly
;### Path to folder where script should place renamed images

dat = C:\MAME Roms\~MAME - ROMs (v0.176_XML).dat

;### path to a MAME ROM database file
;### example C:\files\MAME - ROMs (v0.164_XML).dat
;### get dat here http://www.emulab.it/rommanager/datfiles.php

;### TIP: If you're renaming both Snaps and Titles, make a second copy of this script, point 
;### the folder paths at the other image type, and run both at once. You'll save time.

;### slim down the dat file to speed up processing
SplitPath, dat, datfolder
slimdat := datfolder "\~slim.dat"

if !FileExist(datfolder "\~slim.dat")
{
Loop, Read, %dat%, %slimdat%
If (InStr(A_LoopReadLine, "<game name=") or InStr(A_LoopReadLine, "<description>")) 
 FileAppend %A_LoopReadLine%`n
}

;### filter out unwanted snap png by size in bytes to speed up processing
;57825 bytes mechanical 9000 images
;56475 bytes device     2000 images
;56467 bytes device     1000 images
;53466 bytes screenless system 9000 images
;that leaves ~14000 images in pS_snap_fullset_170.zip

SplitPath, content, contentroot
FileCreateDir, %contentroot%\~SnapsSkip\
Loop, Files, %content%\*.png
{
FileGetSize, s, % A_LoopFileFullPath
if (s==57825) or (s==56475) or (s==56467) or (s==53466)
 FileMove, % A_LoopFileFullPath , %contentroot%\~SnapsSkip\
}

;### needed: a version of the above filter/move code for the title png files

FileRead, dat, %slimdat%

Loop, %content%\*.png
{

name := SubStr(A_LoopFileName,1,-4) ;trim .png

needle2 = &lt;game name=.%name%. (isbios|isdevice)
if RegExMatch(dat, needle2) 
 continue

needle = &lt;game name=.%name%.(?:| ismechanical=.*)(?:| cloneof=.*)(?:| romof=.*)&gt;\R\s*&lt;description&gt;(.*)&lt;/description&gt;
RegExMatch(dat, needle, datname)
if !datname1
 datname1 := name ;fallback to filename

datname1 := RegExReplace(datname1, "[/\?&lt;&gt;\\:*\|]","") ;replace win forbidden chars  
datname1 := RegExReplace(datname1, "'","'")  ;may need more replace like this
datname1 := RegExReplace(datname1, "&","&") 
;### datname1 := RegExReplace(datname1, " \(.*","")        ;trim stuff like "(World version 2.2)"
datname1 := RegExReplace(datname1, "\.$","") 
;list = %xx%`n%datname1% ;for troubleshooting

FileCopy, %A_LoopFileFullPath%, %destination%\%datname1%.png

}
;msgbox % list  ;for troubleshooting
#14

Great howto @Alexandra!

Some of us don’t use windows though, so RomLister is hard to use. Would you mind sharing the .bat files, playlists or anything else in text format that has the rom names together with the categories?

If you can share that I’ll write some small scripts that makes it useful on raspberry pie/linux.

#15

Thanks! And absolutely. I actually had intended to update with an “easy mode” method that just supplies my batch files and lets 'em rip. I’ll put that together soon.

#16

Great, I’ll try to be patient :wink:

#17

Hey, I tried all this and I’m very gratefull! I had tried RomLister before, but I never figured out how powerful it could be when feeded all the information it asks for. Most of all, thanks for taking the time and explaining everything step by step (instead of just saying “here’s my BATs in case you need them” or something)

I do have a question though. What happens with required BIOS?

Example, you need qsound.zip for (some? most?) CPS2 games. Since I have my games separated by folders (ex: “STGVert” for, say, “19XX”, so “roms/STGVert” will be my rompath whenever I run this game), now they’re separated from their bios file, and RetroArch has no way to find them.

So my question is:

  • Is there a workaround, like having your BIOS files somewhere else RetroArch can read them (does it check “system” for bios?)
  • Or should I simply start again by building a Non-Merged set where every rom has all required files in itself INCLUDING system bios?
#18

[QUOTE=spinningacorn;48389]Hey, I tried all this and I’m very gratefull! I had tried RomLister before, but I never figured out how powerful it could be when feeded all the information it asks for. Most of all, thanks for taking the time and explaining everything step by step (instead of just saying “here’s my BATs in case you need them” or something)

I do have a question though. What happens with required BIOS?

Example, you need qsound.zip for (some? most?) CPS2 games. Since I have my games separated by folders (ex: “STGVert” for, say, “19XX”, so “roms/STGVert” will be my rompath whenever I run this game), now they’re separated from their bios file, and RetroArch has no way to find them.

So my question is:

  • Is there a workaround, like having your BIOS files somewhere else RetroArch can read them (does it check “system” for bios?)
  • Or should I simply start again by building a Non-Merged set where every rom has all required files in itself INCLUDING system bios?[/QUOTE]

Hey… it’s funny that you ask this question, because I was literally dealing with the same issue this past weekend and I had written my own scripts to generate playlists prior to discovering this thread… I came up with a solution that worked for me.

You can check out my script here: https://github.com/singularity098/Generate-MAME-Playlist-for-RetroArch

You could follow the normal instructions for that script, but also change this line from $false to $true, like this:

$writeBiosList=$true

With that done, the script will generate a file “_BIOSList.txt” which should be a list of any file that was tagged as being a device which is not runnable (pretty much a BIOS). So from there you could do something like this to copy all of those files into another dir:

  1. Copy the _BIOSList.txt file that you generated to your main ROMs directory.
  2. Open PowerShell.
  3. Navigate to the ROMs directory with this command: cd ‘<replace with your roms dir>’
  4. Use the BIOSList.txt file to copy all files out of that directory with a command like this: [B]get-content ._BIOSList.txt | foreach {copy-item $ ‘<replace with your destination dir>’}[/B]

It’s a little clunky but it should work. For me I’m dealing with the most recent version of MAME and this produces 400 files (about 9 MB) which I then just drop in with my trimmed down romset, and everything seems to work.

This might depend on which version of MAME you’re trying to work with though. To be honest I am pretty new to this and am still learning about MAME. I see that the older MAME versions have much less metadata in their dat files, and so for older versions of MAME I don’t see the xml node for “isdevice” or “runnable”… and therefore on those files, my script would produce an empty bios list file. :frowning:

As a matter of fact, I am looking at a MAME 078 dat today and I search the entire file for “qsound” and it does not appear in the file… so I am no expert on these files and I don’t even understand how that’s possible. But that method should work for you if using a more recent MAME version.

EDIT: Errr… I may have somewhat misunderstood your problem, but maybe this would be helpful anyway. :stuck_out_tongue:

#19

@spinningacorn Very happy you found this useful!

Regarding your proposed solutions, I don’t know if #1 is possible, and #2 is unnecessary.

I just used my very modest knowledge of common BIOSes to judge which folders needed BIOS .zips dropped in. For example, any folder with a Neo Geo game got neogeo.zip. Any folders with CPS2 roms got qsound.zip. etc. These BIOS files are largely tiny (oops, oxymoron) so the wasted disk space is negligible.

Later, if a game fails to load, a quick check online can tell you if it’s due to a missing BIOS or another reason. I don’t remember if the RomLister database can tell you if a game requires a BIOS… maybe it does? RA MAME cores may also spit out relevant error messages somewhere, but I’m not up on that.

Remember you can get a list of all the BIOS files RomLister knows about with the search B[/B], which can be made into the handy macro @B. I made a BIOS-only folder for easy grabbin’ when needed.

That help?

@chrisq So sorry to keep you waiting. Haven’t had much supernerd bandwidth lately. I haven’t forgotten.

#20

[QUOTE=Alexandra;48604]@spinningacorn I just used my very modest knowledge of common BIOSes to judge which folders needed BIOS .zips dropped in. For example, any folder with a Neo Geo game got neogeo.zip. Any folders with CPS2 roms got qsound.zip. etc. These BIOS files are largely tiny (oops, oxymoron) so the wasted disk space is negligible.[/QUOTE]

Yeah, I figured this would be the most obvious fix, and that’s what I’m doing :slight_smile: I merely was curious what was your personal approach to this, since you didn’t mention it in your guide…

I later realized there was a exclusion list on the AHK file. I pretty much copied every single BIOS files on each category folder and added all of them on the exclusion list of the AHK. Solved.

#21

More worthwhile clones suggestions:

sfiii3n.zip - A clone of Street Fighter III Third Strike that doesn’t require CHD (it’s 67.3 MB, cointains everything it needs). Others CPS3 games have NO CD versions too. hcastlek.zip - A clone of Haunted Castle (Castlevania) with rebalanced difficulty - actually playable fairly (the original is pretty much 1 hit KO after a couple of screens) svcplus.zip, svcplusa.zip or svcsplus.zip - Clones of SVC - SNK Vs. Capcom. They have the secret characters available without having to input any select screen code (the most unique characters are secret, like Zero, Mars People, etc). The later one (Super Plus) let you select all of them – the other two have some of them available while keeping others hidden, like the two hidden bosses (Athena and Red Arremer).

#22

Big update I finally plowed through. See changelog in first post. The only major thing I didn’t address was the suggested speed ups to the Friendly Image Renaming AHK. Spent my gumption elsewhere. If anyone wants to mess with that themselves and submit an updated script that’d be grand.

Made 31 or so new icons. Now all 15 major categories and a few others have at least one content icon (sprite). Icon pack download is in Step 5. I’m mostly happy with the selection so I probably won’t make many more. It’s funny, beyond the SF2 fireball I can’t think of a single other sprite that’d work well for Fighting… just out of curiosity, any thoughts?

@chrisq In case you haven’t died of old age I finally got those batch files together. They’re linked in Step 1.

@spinningacorn Re: our convo, the newly expanded ~~~TidyUp.bat (available in the pre-compiled batch file pack or Step 3’s Pastebin link) has rudimentary BIOS-file copying for CPS2 and Neo Geo.

#23

Maybe a punch or kick icon like you see in command lists? Or a ‘KO’ like this one:

#24

Hi Alexandra – I’m interested in one day adapting your work for use on Lakka with MAME 2003, so I thought a first step might be to take you up on this offer and work with your AutoHotKey scripts. Your scripts are derived from the same starting points as the scripts I currently use with MAME 2003, so I ‘speak their language’!

I applied a few speed updates that I worked out for the MAME 2003 playlist generator. One of them I’d mentioned to you before, but here’s the complete list:

[ul] [li]Replaced the regex in the illegal character sanitizing code for both AHK scripts with StrReplace which has a faster implementation in AutoHotKey [/li][li]Changing the playlist generation code to use File.Write and File.Close which have a faster implementation than the AHK File Append feature [/li][li]Load the file lists for ROMs and PNGs into memory and searching for matches in memory rather than consulting the filesystem [/li][/ul]

Other notes:

[ul] [li]The Tetsuya79/reoldemort originals missed a couple of the illegal characters that the RetroArch filename policy forbids. At the same time I changed the RegEx sanitization to use StrReplace I also matched up that code with current RetroArch filename standards. [/li][li]I added a provision for Boxart thumbnails by cloning your work on Snaps and Titles. Kivutar from the dev team has said that they intend for Arcade Flyer art to be used for Boxart with RetroArch. I hope you don’t mind the addition! [/li][/ul]

I think I have been able to successfully blend the relevant code from my ‘forks’ of the roldemort and Tetsuya79 code with yours. I have done some spot testing and the results look right. If you have a chance to try them out I would be interested in hearing your results.

Image renaming AHK: https://github.com/markwkidd/retroarch-arcade-playlist-helpers/blob/master/friendly-image-renaming.ahk Playlist generation and thumbnail copying: https://github.com/markwkidd/retroarch-arcade-playlist-helpers/blob/master/curated-arcade-playlist-generator.ahk

#25

I looked at your code here https://github.com/markwkidd/lakka-arcade-playlist-helpers/blob/master/friendly-image-renaming.ahk . Nice improvements!

Three tips for speeding things up more (adapted from my post upthread ). I’ll suggest line numbers in your script where you’d insert these snippets.

1 Add after line 4 to maximize Autohotkey speed


SetBatchLines -1

2 Trim the dat before the RegExMatch steps. Replace your line 39 with

FileRead, datcontents, %dat%
Loop, Parse, datcontents, `n, `r
 If (InStr(A_LoopField, "<game name=") or InStr(A_LoopField, "<description>"))  ;only keep relevant lines
  slimdat .= A_LoopField "`n"
datcontents := slimdat

3 Skip unwanted dummy images. For example the MAME snaps set pS_snap_fullset_170.zip has about 20K snaps with the MAME mechanical/screenless/device placeholder image. Useless for RetroArch purposes so let us filter out them by filesize. That only takes a few seconds. In the code upthread I moved the dummy images to a separate folder but the below code instead simply skips them. (Note: This step needs testing. The dummy sizes is tested on one MAME snaps set. Will not work well if updates to the snaps set changes the dummy file sizes. But perhaps those files are stable by now? This step will also skip other snaps if they happen to have the same filesize, though there were no such cases when I earlier tested with the pS_snap_fullset_170.zip . A more reliable method is to test file checksums. But that takes much longer time for 40K snaps so should be done by a separate, one-time preprocessing script that moves/deletes the dummy files.)

Insert after your line 43

;Skip known dummy images
;57825 bytes mechanical 9K images
;56475 bytes device  2K
;56467 bytes device  1K 
;53466 bytes screenless system 9K
;That leaves ~14K images in pS_snap_fullset_170.zip
FileGetSize, s, % A_LoopFileFullPath
if (s==57825) or (s==56475) or (s==56467) or (s==53466)
 continue