Redump DAT parsing bug? (Libretro-DATs/libretro-db)

Tanuki · 20 March 2019 15:21

Hi all,

I recently checked my ISOs (PS1, SS, SCD, PCE-CD) against the latest Redump DATs. Since it had been a little while, a few of my images were updated to a newer naming convention (versioning changed from (1.2) to (Rev 2) etc…)

I used RetroArch (1.7.6) internal “Scan/Import Content” to update my Playlists (JSON) but despite updating to the latest RDBs available, some games were not detected properly. My OCD got the better of me and I decided to try and compile my own RDBs based on the latest Redump DATs. After some fumbling around, I managed to convert the Redump DATs to libretro-DATs and subsequently to RDBs (with c_converter) following the instructions here (https://github.com/RobLoach/libretro-dats) and there (https://github.com/libretro/RetroArch/tree/master/libretro-db). I am using Ubuntu MATE Server 18.04.

To my disappointment though, the scan is still missing some of the games (or just all of them) or not reflecting the correct names from the original Redump DATs. I am getting different symptoms depending on the platforms.

NEC PCE-CD: the scan detects my ISOs. I get the following messages for all the games and no Playlist gets written. Note that I have the same problem with the RDB from the libretro buildbot. (ClrMAME and manual crc32,mdm5,sha1 checks out with Redump data.)

RetroArch Verbose LOG

[INFO] Parsing CUE file 'Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Seirei Senshi Spriggan (Japan)\Seirei Senshi Spriggan (Japan).cue'...
[INFO] CUE 'Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Seirei Senshi Spriggan (Japan)\Seirei Senshi Spriggan (Japan).cue' primary track: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Seirei Senshi Spriggan (Japan)\Seirei Senshi Spriggan (Japan) (Track 02).bin
 (0, 4969776)
[INFO] Reading first data track...
[INFO] CUE 'Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Seirei Senshi Spriggan (Japan)\Seirei Senshi Spriggan (Japan).cue' crc: 14db03f6
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 01).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 02).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 03).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 04).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 05).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 06).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 07).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 08).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 09).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 10).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 11).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 12).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 13).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 14).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 15).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 16).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 17).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 18).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 19).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 20).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 21).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 22).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 23).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 24).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 25).bin
[INFO] Parsing CUE file 'Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan).cue'...
[INFO] Reading first data track...
[INFO] Comparing with known magic numbers...
[INFO] Could not find compatible system. 
[INFO] CUE 'Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Seirei Senshi Spriggan (Japan)\Seirei Senshi Spriggan (Japan).cue' crc: 14db03f6
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 01).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 02).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 03).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 04).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 05).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 06).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 07).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 08).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 09).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 10).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 11).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 12).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 13).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 14).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 15).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 16).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 17).bined
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 18).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 19).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 20).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 21).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 22).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 23).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 24).bin
[INFO] Pruning file referenced by cue: Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan) (Track 25).bin
[INFO] Parsing CUE file 'Z:\EmuBox\NEC - PC Engine CD - TurboGrafx-CD\Psychic Storm (Japan)\Psychic Storm (Japan).cue'...
[INFO] Reading first data track...

SCD & Saturn: Games with multiple editions are wrongly identified.

Sega CD: “Shining Force CD (USA)” gets identified as “Shining Force CD (USA) (Alt)”

Saturn: “Layer Section (Japan) (3M)” gets identified as "Layer Section (Japan) (6M)

NiGHTS into Dreams… (Japan) is ignored

I somehow get that the “…” in Nights messes around with the parsing. As for the other two, a quick look at the libretro-DATs (my own and the one from buildbot) show that neither “Shining Force CD (USA)” nor “Layer Section (Japan) (3M)” are listed in there (but are of course present in the Redump DAT with different sizes and CRCs from Alt or 6M). If this is by design, then no worries, I’ll try and keep my OCD under control

PS1: This one utterly baffles me! With the RDB from the buildbot, all my games but two are correctly identified. The missing ones are Gunners Heaven (Japan) and Raiden Project (Japan). With my own DAT, it is the other way around! None of the games except those two get identified/written to playlist…

Here is a zip with the Redump DATs, my converted libretro-DATs and compiled RDBs for reference: https://denki-den.dedyn.io/s/MorkoNry8KF4inf

Am I doing something wrong? Anyone else able to replicate?

Thanks in advance!

EDIT: formatting

Tanuki · 3 April 2019 13:57

I have had a little more time to dig into this and tried to pinpoint the source of some of the problems I was having. Looking at the Librero DATs entry, it looks like the parsing process only retains checksums for “Track 01.bin” for identification. The problem is the difference between multiple versions of the same game can be found within the audio tracks rather than the data track.

In the case of “Sonic CD (USA)” for Sega CD, the Redump DAT has three entries:

<game name="Sonic CD (USA)">
		<category>Games</category>
		<description>Sonic CD (USA)</description>
		<rom name="Sonic CD (USA).cue" size="3820" crc="33b99240" md5="638f1e41830319de1914772076096626" sha1="5b085f78a6904d275715dcfeb454ecc262e704e5"/>
		<rom name="Sonic CD (USA) (Track 01).bin" size="139381872" crc="a6184e05" md5="ed96420538f989e94480f810d5f90685" sha1="3c316af52cc9ee8c30976a1cdb66339545dda3bd"/>
		[...]

<game name="Sonic CD (USA) (RE125)">
		<category>Games</category>
		<description>Sonic CD (USA) (RE125)</description>
		<rom name="Sonic CD (USA) (RE125).cue" size="4100" crc="e8a15187" md5="92d0c52b29c25b21e8d0bef3e3086115" sha1="ff275b2a8bf5c301fed311addca55aee8a1a13ab"/>
		<rom name="Sonic CD (USA) (RE125) (Track 01).bin" size="139381872" crc="a6184e05" md5="ed96420538f989e94480f810d5f90685" sha1="3c316af52cc9ee8c30976a1cdb66339545dda3bd"/>
		[...]
		
<game name="Sonic CD (USA) (RE125) (Alt)">
		<category>Games</category>
		<description>Sonic CD (USA) (RE125) (Alt)</description>
		<rom name="Sonic CD (USA) (RE125) (Alt).cue" size="4310" crc="53948c76" md5="ac75d9b8331ba59da608c49e5cc03cb7" sha1="4d190b6a7cc7e2284f7c16f6afaf94ff625dfb8c"/>
		<rom name="Sonic CD (USA) (RE125) (Alt) (Track 01).bin" size="139381872" crc="a6184e05" md5="ed96420538f989e94480f810d5f90685" sha1="3c316af52cc9ee8c30976a1cdb66339545dda3bd"/>
		[...]

All versions share the same “Track 01.bin” checksums. Once parsed with the libretro-dat javascript routine, there is only one entry “Sonic CD (USA) (RE125) (Alt)” as can be seen here in the official libretro metadata from the git repo:

game (
	name "Sonic CD (USA) (RE125) (Alt)"
	description "Sonic CD (USA) (RE125) (Alt)"
	rom ( name "Sonic CD (USA) (RE125) (Alt) (Track 01).bin" size 139381872 crc a6184e05 md5 ed96420538f989e94480f810d5f90685 sha1 3c316af52cc9ee8c30976a1cdb66339545dda3bd )
)

However, a comparison in WinMerge reveals the difference between those 3 version resides in Track 13.

<rom name="Sonic CD (USA) (Track 13).bin" size="14013216" crc="c41a848e" md5="f01a263ce6475c1f43871483e5ab68f2" sha1="21dcc020dae2a0d650eed76540b54897d34ee889"/>
<rom name="Sonic CD (USA) (RE125) (Track 13).bin" size="14013216" crc="282d4f23" md5="88107ee77b536248c0b9149ac1c71ea4" sha1="5797d380e8df06f7d83a37ac55b3072f8a3dff57"/>
<rom name="Sonic CD (USA) (RE125) (Alt) (Track 13).bin" size="14013216" crc="8e9aaeaf" md5="c02b110395c9cb8a6a1066eb5c36835c" sha1="6cfbd4139d60eb7762874aa0308a01f40e0e17fc"/>

While they share the same size, the checksums are different.

The same is true of “Layer Section (Japan) (3M)” and “Layer Section (Japan) (6M)” in the Sega Saturn set. Except the differences appear from Track 02.bin.

        <rom name="Layer Section (Japan) (6M).cue" size="2183" crc="4a838a68" md5="2f21a45ccdf5b439a35d0ad0a4b9f99a" sha1="75932d563fb498e77e0a107efca33c034a75b04c"/>
		<rom name="Layer Section (Japan) (6M) (Track 01).bin" size="16779168" crc="07e9d8da" md5="4cb2552702a5767be5b791efad43762d" sha1="d90ef75e7a8d0ff60fceb58d4e534308b85391ea"/>
		<rom name="Layer Section (Japan) (6M) (Track 02).bin" size="5644800" crc="4385448d" md5="f52bb23aaf351581cea09442d74e0981" sha1="76ee44b608700841f89a3b0c325e7ccf691562c5"/>
		

		<rom name="Layer Section (Japan) (3M).cue" size="2183" crc="6f8ecb12" md5="e6ec1fb1245f5a5ff095bfa29f9d62e1" sha1="7905e5451fbcd372905dae9c4ef2339dff271357"/>
		<rom name="Layer Section (Japan) (3M) (Track 01).bin" size="16779168" crc="07e9d8da" md5="4cb2552702a5767be5b791efad43762d" sha1="d90ef75e7a8d0ff60fceb58d4e534308b85391ea"/>
		<rom name="Layer Section (Japan) (3M) (Track 02).bin" size="5644800" crc="ccffab2f" md5="c7edd32ea27e9596546b6bf8d7148784" sha1="cce1afd9f917210cdf061bdf23982f1597e6639a"/>

Sega - Saturn Libretro MetaDAT

game (
	name "Layer Section (Japan) (6M)"
	description "Layer Section (Japan) (6M)"
	rom ( name "Layer Section (Japan) (6M) (Track 01).bin" size 16779168 crc 07e9d8da md5 4cb2552702a5767be5b791efad43762d sha1 d90ef75e7a8d0ff60fceb58d4e534308b85391ea )
)

I know this will look like nitpicking to some but now that db.libretro.com is also based on the results of those DATs, this may lead to some confusion in the future?

I still don’t understand what is happening with PSX and PC-Engine CD but I’ll keep looking.

EDIT: typos & formatting

metchebe · 3 April 2019 14:43

I posted an issue for PC Engine CD, you might be able to confirm or complement what I reported.

Tanuki · 5 April 2019 11:37

Hi metchebe,

Confirmed. The metadat contains information for Track 01 while RetroArch logs shows it is looking at Track 02. I hope this gets looked at for future releases!

Let’s see if I can pin down the problem with the PSX set this weekend.