HowTo: Generate Pretty, Curated MAME Playlists the Easy(ish) Way

Wow quite the nice effort you put into this guide, great work! :slight_smile:

That’s an interesting idea to cut the game library into several genre playlists. Personally I prefer having everything together for now, easier to search (and because I made it that way already!).

What I’d like to avoid those huge playlist loading lags would be to switch to glui/materialui if it comes to support thumbnails (and get the clock back!). It doesn’t load playlists automatically like xmb and is a lot faster to browse (going from an ingame quickmenu to video settings is such a pain in xmb as an example, and scrolling 8 by 8 when pushing left and right while browsing a playlist is nice too).

Anyway some great script tips I’ll come back to check here, thank you.

(PS: I hardly know how to script a bit by reading other people in that Mame playlist thread! Props to them.)

2 Likes

Great work. Stickied :slight_smile:

1 Like

Thread subscribed!

The version of the AHK scripts I’m using for MAME playlists in Lakka is somewhat mutated from the original at this point, but you might think about adopting a change from my version. Using StrReplace instead of RegExReplace to replace the illegal filename characters resulted in a substantial speedup for me.

For what it’s worth:


;### AUTOHOTKEY SCRIPT TO GENERATE MAME PLAYLIST FOR LAKKA/RETROARCH

;---------------------------------------------------------------------------------------------------------
#NoEnv                          ; Recommended for performance and compatibility with future AutoHotkey releases.
SendMode Input                  ; Recommended for new scripts due to its superior speed and reliability.
SetWorkingDir %A_ScriptDir%      ; Ensures a consistent starting directory.
;---------------------------------------------------------------------------------------------------------

playlistname = MAME.lpl
;### Local path to output the playlist file.
;### Should include the .lpl extension.
;### NOTE: THIS SCRIPT WILL DELETE ANY EXISTING FILE WITH THIS NAME
;### As of 5/2016 using the name "MAME.lpl" results in icons appearing for
;### MAME and its playlist entries in Lakka. Other file names may require 
;### adding icons or copying the MAME set.
;### Default: MAME.lpl

dat = MAME 078.dat
;### Example: C:\MAME\dats\MAME 078.dat
;### local path to a MAME ROM database file

romextension = .zip
;### Enter a . (dot/period) followed by the extension of the ROMs.
;### This should be .zip or .7z
romextensionlength := StrLen(romextension)     ;### automatically calculate the length of the extension

windowsroms = F:\Lakka Staging\roms\MAME 0.78
;### Example: C:\MAME 0.78 Non-Merged\roms
;### DO NOT INCLUDE A CLOSING SLASH AT THE END OF THE PATH
;### This path is a MAME ROMs folder accessible to THIS WINDOWS HOST. These 
;### ROMs should be the same set that will be used in Lakka
;### and should be of the same MAME version as the datfile.

lakkaroms = /storage/roms/MAME 0.78
;### Example: /storage/roms/MAME 0.78
;### DO NOT INCLUDE A CLOSING SLASH AT THE END OF THE PATH
;### MAME ROMs folder on the destination Lakka installation
;### You do not need to have access to these files on the machine which
;### is running this script. The path must accurately point to the location
;### of a copy of the the same ROM set used in the Windows path above.

core = /tmp/cores/mame2003_libretro.so
;### Example: /tmp/cores/mame2003_libretro.so
;### full path to the MAME core which will be executing this playlist

FileDelete, %A_ScriptDir%\%playlistname%  ;clear old MAME.lpl file if present in this directory

FileRead, dat, %dat%
rawcounter = 0

Loop, %windowsroms%\*%romextension% {

    if A_LoopFileName in (neogeo%romextension%,awbios%romextension%,cpzn2%romextension%)
        continue    ;### manually skip these bios files. you can add names of other files to skip

    filename := SubStr(A_LoopFileName,1,(StrLen(A_Loop_File_Name) - romextensionlength)) ;### trim the file extension from the name

    filter1 = <game name=.%filename%. (isbios|isdevice)
    if RegExMatch(dat, filter1)
        continue                    ;### skip if the file listed as a BIOS or device in the dat
    
    needle = <game name=.%filename%.(?:| ismechanical=.*)(?:| sourcefile=.*)(?:| cloneof=.*)(?:|  romof=.*)>\R\s*<description>(.*)</description>
    RegExMatch(dat, needle, datname)

    fancyname := datname1    ;### extract match #1 from the RegExMatch result
    if !fancyname
        fancyname := filename   ;### the file is not matched in the dat file, use the filename instead
        
    ;### Replace characters unsafe for cross-platform filenames with underscore, 
    ;### per RetroArch thumbnail/playlist convention
    fancyname := StrReplace(fancyname, "'", "'")
    fancyname := StrReplace(fancyname, "&", "_")
    fancyname := StrReplace(fancyname, "&", "_")
    fancyname := StrReplace(fancyname, "\", "_")
    fancyname := StrReplace(fancyname, "/", "_")
    fancyname := StrReplace(fancyname, "?", "_")
    fancyname := StrReplace(fancyname, ":", "_")
    fancyname := StrReplace(fancyname, "<", "_")
    fancyname := StrReplace(fancyname, ">", "_")
    fancyname := StrReplace(fancyname, ":", "_")
    fancyname := StrReplace(fancyname, "*", "_")
    fancyname := StrReplace(fancyname, "|", "_")
    
    playlistentry =
(
%lakkaroms%/%filename%%romextension%
%fancyname%
%core%
DETECT
DETECT
%playlistname%
)

    FileAppend, %playlistentry%`n, *%A_ScriptDir%\%playlistname%      ;### the asterisk before the path invokes
                                                                    ;### binary mode. binary mode plus the `n
                                                                    ;### at the end of the line equals unix encoding
}

1 Like

Really nice!

1 Like

Thanks for the nice response. Sticky, cool!

markwkidd, good tip. If I tinker with the code again I will try that. I’ve been thinking about how the second script, too, could be a lot more functional, requiring less effort from the user. Not sure I’ll have the gumption to redo it. But maybe!

I’ve been spending way too much time making icons. Just uploaded 75 new ones. They’re mostly headers, most genres still lack content icons. Decided to stick with the convention artwork for headers, sprites for content.

Here’s a glance at everything that exists so far (click for full res): In the case of future updates I’ll just post the new ones. Will hopefully fill in the less covered genres and get at least one sprite-based content icon for each category. For now, though, my wrist hurts.

[QUOTE=Alexandra;45086]markwkidd, good tip. If I tinker with the code again I will try that. I’ve been thinking about how the second script, too, could be a lot more functional, requiring less effort from the user. Not sure I’ll have the gumption to redo it. But maybe! [/QUOTE]

Thanks!

Two thoughts about thumbnails: * If you’re interested in a proposed set of ‘official’ MAME thumbnails you may want to stop by this pull request, where the script has been put to use: https://github.com/libretro/libretro-thumbnails/pull/80 * I don’t know if my version of the thumbnail renamer script would be useful to you either, but in case it is, I have modified the original to automatically rename an entire folder of thumbnails based directly on the MAME datfile rather than generating a RetroArch playlist first and then using the playlist to rename the thumbnails.

;### AUTOHOTKEY SCRIPT TO RENAME ARCADE THUMBNAILS FOR RETROARCH

;---------------------------------------------------------------------------------------------------------
#NoEnv                          ; Recommended for performance and compatibility with future AutoHotkey releases.
#Warn                              ; Enable warnings to assist with detecting common errors.
SendMode Input                  ; Recommended for new scripts due to its superior speed and reliability.
SetWorkingDir %A_ScriptDir%      ; Ensures a consistent starting directory.
;---------------------------------------------------------------------------------------------------------

;### SETUP: ADD YOUR PATHS HERE

artsource = D:\Emulation\MAME RetroArch Thumbnails\mame 037b5	itles-resized
;### Path to the source thumbnails folder on the local machine
;### DO NOT INCLUDE A CLOSING SLASH AT THE END OF THE PATH
;### Example: C:\MAME 0.78 images	itles

processed = D:\Emulation\MAME RetroArch Thumbnails\MAME Onmnibus Thumbnails\Named_Titles
;### NOTE: THIS FOLDER MUST EXIST BEFORE THE SCRIPT IS EXECUTED
;### Path to the destination thumbnail folder on the local machine
;### DO NOT INCLUDE A CLOSING SLASH AT THE END OF THE PATH
;### Example C:\MAME 0.78 images\Named_Titles

dat = D:\Emulation\Original ROM Sets\MAME 0.37b5 Non-Merged\MAME 0.37b5.dat
;### Example: C:\MAME\dats\MAME 078.dat
;### local path to a MAME ROM database file

if !FileExist(dat) or !FileExist(artsource) or !FileExist(processed)
 return     ;### If any of these files and folders desn't exist, exit the script

;### Delete old 'ummatched' log file if it exists 
FileDelete, Unmatched Thumbnails - %dat%.log

FileRead, datcontents, %dat%
FileDelete, %processed%\Unmatched Thumbnails.log    ;### Delete old 'ummatched' log file if it exists
rawcounter = 0
A_Loop_File_Name = 0

Loop, %artsource%\*.png {
 
    SplitPath, A_LoopFileName,,,,filename     ;### trim the file extension from the name

    needle = <game name=.%filename%.(?:| ismechanical=.*)(?:| sourcefile=.*)(?:| cloneof=.*)(?:|  romof=.*)>\R\s*<description>(.*)</description>
    RegExMatch(datcontents, needle, datname)

    fancyname := datname1    ;### extract match #1 from the RegExMatch result
    
    if !fancyname {
        fancyname := filename   ;### the image filename/rom name is not matched in the dat file
         FileAppend, Unmatched Source %artsource%\%filename%.png`n`n, %processed%\Unmatched Thumbnails.log           ;for error checking
        continue                 ;### skip to the next image
    }
    
    ;### Replace characters unsafe for cross-platform filenames with underscore, 
    ;### per RetroArch thumbnail/playlist convention
    fancyname := StrReplace(fancyname, "'", "'")
    fancyname := StrReplace(fancyname, "&", "_")
    fancyname := StrReplace(fancyname, "&", "_")
    fancyname := StrReplace(fancyname, "\", "_")
    fancyname := StrReplace(fancyname, "/", "_")
    fancyname := StrReplace(fancyname, "?", "_")
    fancyname := StrReplace(fancyname, ":", "_")
    fancyname := StrReplace(fancyname, "<", "_")
    fancyname := StrReplace(fancyname, ">", "_")
    fancyname := StrReplace(fancyname, ":", "_")
    fancyname := StrReplace(fancyname, "*", "_")
    fancyname := StrReplace(fancyname, "|", "_")

    destinationfile := processed . "`\" . fancyname . ".png"
;    if FileExist(destinationfile)
;    { 
;    continue    ;### skip onward if this file already exists
;    }    
    FileCopy, %artsource%\%filename%.png, %destinationfile% , 1
}

Can someone link to controls.xml? The site appears to be down that usually hosts it.

Here you go, uploaded it to Mediafire. It is version 0.141.1, the latest version at this time.

Thanks Alexandra! Worked great. I didn’t go through all the steps just yet - for now I just used this to cull a massive amount of ROMs :slight_smile: Such an amazing shortcut to get rid of all the silly dupes etc that bloat MAME sets nowadays.

Icon pack updated with 15 new headers. Taking a break for real this time. :stuck_out_tongue:

Cool guide!

I’ll add a few optimization notes on the AutoHotKey processing in step 4. Your “Friendly Image Renaming Script” copies and renames all of the 36K + 35K images. Overkill since earlier steps filtered out a ton of games. A quicker approach is to loop over roms in the created subfolders and only copy/rename their corresponding images. That would take a bigger rewrite. But here are three quicker tweaks

First put

 SetBatchLines -1

at the top of the Friendly Image Renaming Script to increase the script speed, https://autohotkey.com/docs/commands/SetBatchLines.htm

Second slim down the dat file with the below code (takes only a few seconds) and then use “~slim.dat” in Friendly Image Renaming Script instead

 SetBatchLines, -1
x = C:\MAME Roms\~MAME - ROMs (v0.176_XML).dat
y = C:\MAME Roms\~slim.dat
Loop, Read, %x%, %y%
{
If (InStr(A_LoopReadLine, "<game name=") or InStr(A_LoopReadLine, "<description>")) 
 FileAppend %A_LoopReadLine%`n
}

With those two changes to the Friendly Image Renaming Script the time to rename 200 images dropped from 27 to 5-6 seconds. So your one hour might drop to 10-15 minutes or so.

But it gets better. We can filter out some obviously unwanted images before running the Friendly Image Renaming Script. For example the snaps set pS_snap_fullset_170.zip has about 20K snaps with the MAME mechanical/screenless/device placeholder image. We can filter them by filesize and move them to another folder (also only takes a few seconds).

;57825 bytes mechanical  9000 images
;56475 bytes device     2000 images
;56467 bytes device     1000 images
;53466 bytes screenless system 9000 images
;that leaves ~14000 images in pS_snap_fullset_170.zip

FileCreateDir, C:\MAME Roms\~SnapsSkip\
Loop, Files, C:\MAME Roms\~Snaps\*.png
{
FileGetSize, s, % A_LoopFileFullPath
if (s==57825) or (s==56475) or (s==56467) or (s==53466)
 FileMove, % A_LoopFileFullPath , C:\MAME Roms\~SnapsSkip\
}

With all these three tweaks applied the Friendly Image Renaming Script took 2 minutes to process the pS_snap_fullset_170.zip on my system.

edit:

Here is Alexandra’s script with these two tweaks added in

 #NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn  ; Enable warnings to assist with detecting common errors.
SendMode Input  ; Recommended for new scripts due to its superior speed and reliability.
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.
SetBatchLines -1 ;max script speed 

;AUTOHOTKEY SCRIPT TO GENERATE FRIENDLY IMAGE NAMES FOR RETROARCH PLAYLIST USE
;"Friendly Image Renaming Script"

;### SETUP, ADD YOUR PATHS HERE

content = C:\MAME Roms\~Snaps
;### Path to folder containing MAME Snap or Title images

destination = C:\MAME Roms\~SnapsFriendly
;### Path to folder where script should place renamed images

dat = C:\MAME Roms\~MAME - ROMs (v0.176_XML).dat

;### path to a MAME ROM database file
;### example C:\files\MAME - ROMs (v0.164_XML).dat
;### get dat here http://www.emulab.it/rommanager/datfiles.php

;### TIP: If you're renaming both Snaps and Titles, make a second copy of this script, point 
;### the folder paths at the other image type, and run both at once. You'll save time.

;### slim down the dat file to speed up processing
SplitPath, dat, datfolder
slimdat := datfolder "\~slim.dat"

if !FileExist(datfolder "\~slim.dat")
{
Loop, Read, %dat%, %slimdat%
If (InStr(A_LoopReadLine, "<game name=") or InStr(A_LoopReadLine, "<description>")) 
 FileAppend %A_LoopReadLine%`n
}

;### filter out unwanted snap png by size in bytes to speed up processing
;57825 bytes mechanical 9000 images
;56475 bytes device     2000 images
;56467 bytes device     1000 images
;53466 bytes screenless system 9000 images
;that leaves ~14000 images in pS_snap_fullset_170.zip

SplitPath, content, contentroot
FileCreateDir, %contentroot%\~SnapsSkip\
Loop, Files, %content%\*.png
{
FileGetSize, s, % A_LoopFileFullPath
if (s==57825) or (s==56475) or (s==56467) or (s==53466)
 FileMove, % A_LoopFileFullPath , %contentroot%\~SnapsSkip\
}

;### needed: a version of the above filter/move code for the title png files

FileRead, dat, %slimdat%

Loop, %content%\*.png
{

name := SubStr(A_LoopFileName,1,-4) ;trim .png

needle2 = &lt;game name=.%name%. (isbios|isdevice)
if RegExMatch(dat, needle2) 
 continue

needle = &lt;game name=.%name%.(?:| ismechanical=.*)(?:| cloneof=.*)(?:| romof=.*)&gt;\R\s*&lt;description&gt;(.*)&lt;/description&gt;
RegExMatch(dat, needle, datname)
if !datname1
 datname1 := name ;fallback to filename

datname1 := RegExReplace(datname1, "[/\?&lt;&gt;\\:*\|]","") ;replace win forbidden chars  
datname1 := RegExReplace(datname1, "'","'")  ;may need more replace like this
datname1 := RegExReplace(datname1, "&","&") 
;### datname1 := RegExReplace(datname1, " \(.*","")        ;trim stuff like "(World version 2.2)"
datname1 := RegExReplace(datname1, "\.$","") 
;list = %xx%`n%datname1% ;for troubleshooting

FileCopy, %A_LoopFileFullPath%, %destination%\%datname1%.png

}
;msgbox % list  ;for troubleshooting

Great howto @Alexandra!

Some of us don’t use windows though, so RomLister is hard to use. Would you mind sharing the .bat files, playlists or anything else in text format that has the rom names together with the categories?

If you can share that I’ll write some small scripts that makes it useful on raspberry pie/linux.

Thanks! And absolutely. I actually had intended to update with an “easy mode” method that just supplies my batch files and lets 'em rip. I’ll put that together soon.

Great, I’ll try to be patient :wink:

Hey, I tried all this and I’m very gratefull! I had tried RomLister before, but I never figured out how powerful it could be when feeded all the information it asks for. Most of all, thanks for taking the time and explaining everything step by step (instead of just saying “here’s my BATs in case you need them” or something)

I do have a question though. What happens with required BIOS?

Example, you need qsound.zip for (some? most?) CPS2 games. Since I have my games separated by folders (ex: “STGVert” for, say, “19XX”, so “roms/STGVert” will be my rompath whenever I run this game), now they’re separated from their bios file, and RetroArch has no way to find them.

So my question is:

  • Is there a workaround, like having your BIOS files somewhere else RetroArch can read them (does it check “system” for bios?)
  • Or should I simply start again by building a Non-Merged set where every rom has all required files in itself INCLUDING system bios?

[QUOTE=spinningacorn;48389]Hey, I tried all this and I’m very gratefull! I had tried RomLister before, but I never figured out how powerful it could be when feeded all the information it asks for. Most of all, thanks for taking the time and explaining everything step by step (instead of just saying “here’s my BATs in case you need them” or something)

I do have a question though. What happens with required BIOS?

Example, you need qsound.zip for (some? most?) CPS2 games. Since I have my games separated by folders (ex: “STGVert” for, say, “19XX”, so “roms/STGVert” will be my rompath whenever I run this game), now they’re separated from their bios file, and RetroArch has no way to find them.

So my question is:

  • Is there a workaround, like having your BIOS files somewhere else RetroArch can read them (does it check “system” for bios?)
  • Or should I simply start again by building a Non-Merged set where every rom has all required files in itself INCLUDING system bios?[/QUOTE]

Hey… it’s funny that you ask this question, because I was literally dealing with the same issue this past weekend and I had written my own scripts to generate playlists prior to discovering this thread… I came up with a solution that worked for me.

You can check out my script here: https://github.com/singularity098/Generate-MAME-Playlist-for-RetroArch

You could follow the normal instructions for that script, but also change this line from $false to $true, like this:

$writeBiosList=$true

With that done, the script will generate a file “_BIOSList.txt” which should be a list of any file that was tagged as being a device which is not runnable (pretty much a BIOS). So from there you could do something like this to copy all of those files into another dir:

  1. Copy the _BIOSList.txt file that you generated to your main ROMs directory.
  2. Open PowerShell.
  3. Navigate to the ROMs directory with this command: cd ‘<replace with your roms dir>’
  4. Use the BIOSList.txt file to copy all files out of that directory with a command like this: [B]get-content ._BIOSList.txt | foreach {copy-item $ ‘<replace with your destination dir>’}[/B]

It’s a little clunky but it should work. For me I’m dealing with the most recent version of MAME and this produces 400 files (about 9 MB) which I then just drop in with my trimmed down romset, and everything seems to work.

This might depend on which version of MAME you’re trying to work with though. To be honest I am pretty new to this and am still learning about MAME. I see that the older MAME versions have much less metadata in their dat files, and so for older versions of MAME I don’t see the xml node for “isdevice” or “runnable”… and therefore on those files, my script would produce an empty bios list file. :frowning:

As a matter of fact, I am looking at a MAME 078 dat today and I search the entire file for “qsound” and it does not appear in the file… so I am no expert on these files and I don’t even understand how that’s possible. But that method should work for you if using a more recent MAME version.

EDIT: Errr… I may have somewhat misunderstood your problem, but maybe this would be helpful anyway. :stuck_out_tongue:

@spinningacorn Very happy you found this useful!

Regarding your proposed solutions, I don’t know if #1 is possible, and #2 is unnecessary.

I just used my very modest knowledge of common BIOSes to judge which folders needed BIOS .zips dropped in. For example, any folder with a Neo Geo game got neogeo.zip. Any folders with CPS2 roms got qsound.zip. etc. These BIOS files are largely tiny (oops, oxymoron) so the wasted disk space is negligible.

Later, if a game fails to load, a quick check online can tell you if it’s due to a missing BIOS or another reason. I don’t remember if the RomLister database can tell you if a game requires a BIOS… maybe it does? RA MAME cores may also spit out relevant error messages somewhere, but I’m not up on that.

Remember you can get a list of all the BIOS files RomLister knows about with the search B[/B], which can be made into the handy macro @B. I made a BIOS-only folder for easy grabbin’ when needed.

That help?

@chrisq So sorry to keep you waiting. Haven’t had much supernerd bandwidth lately. I haven’t forgotten.

[QUOTE=Alexandra;48604]@spinningacorn I just used my very modest knowledge of common BIOSes to judge which folders needed BIOS .zips dropped in. For example, any folder with a Neo Geo game got neogeo.zip. Any folders with CPS2 roms got qsound.zip. etc. These BIOS files are largely tiny (oops, oxymoron) so the wasted disk space is negligible.[/QUOTE]

Yeah, I figured this would be the most obvious fix, and that’s what I’m doing :slight_smile: I merely was curious what was your personal approach to this, since you didn’t mention it in your guide…

I later realized there was a exclusion list on the AHK file. I pretty much copied every single BIOS files on each category folder and added all of them on the exclusion list of the AHK. Solved.

More worthwhile clones suggestions:

sfiii3n.zip - A clone of Street Fighter III Third Strike that doesn’t require CHD (it’s 67.3 MB, cointains everything it needs). Others CPS3 games have NO CD versions too. hcastlek.zip - A clone of Haunted Castle (Castlevania) with rebalanced difficulty - actually playable fairly (the original is pretty much 1 hit KO after a couple of screens) svcplus.zip, svcplusa.zip or svcsplus.zip - Clones of SVC - SNK Vs. Capcom. They have the secret characters available without having to input any select screen code (the most unique characters are secret, like Zero, Mars People, etc). The later one (Super Plus) let you select all of them – the other two have some of them available while keeping others hidden, like the two hidden bosses (Athena and Red Arremer).

Big update I finally plowed through. See changelog in first post. The only major thing I didn’t address was the suggested speed ups to the Friendly Image Renaming AHK. Spent my gumption elsewhere. If anyone wants to mess with that themselves and submit an updated script that’d be grand.

Made 31 or so new icons. Now all 15 major categories and a few others have at least one content icon (sprite). Icon pack download is in Step 5. I’m mostly happy with the selection so I probably won’t make many more. It’s funny, beyond the SF2 fireball I can’t think of a single other sprite that’d work well for Fighting… just out of curiosity, any thoughts?

@chrisq In case you haven’t died of old age I finally got those batch files together. They’re linked in Step 1.

@spinningacorn Re: our convo, the newly expanded ~~~TidyUp.bat (available in the pre-compiled batch file pack or Step 3’s Pastebin link) has rudimentary BIOS-file copying for CPS2 and Neo Geo.