NES playlist creation bug: roms playable in nestopia excluded from playlist

RetroArch General
#1

Is there a bug in how Retroarch creates NES playlists? Why does not the playlist generator include all roms whose headerless part (i.e. whole file except first 16 bytes) match a hash in NstDatabase.xml ? Because such roms seem to be playable in the Nestopia core.

Background: I noticed that one NES rom I had was not added to playlist by Retroarch. But when I manually added it by editing the playlist file the rom worked in the Nestopia core.

I got curious and ran some tests. Short story is Retroarch’s playlist creator seems to compare the input file’s hash against (a) a hash list of the headerless part of NES roms and (b) a hash list of whole “headered” NES roms. If either test is a match the file is put on the playlist. (Can someone who knows the code say if I got that right?)

I base this on the observations that a rom like the one I have that matches a file in the database except for the header part (the first 16 bytes) is rejected by the playlist creator. If I remove the header (trim first 16 bytes) the file is included (even though it won’t play in the Nestopia core). And if I add back a “blank” NES header ( 4E 45 53 1A 00 00 00 00 00 00 00 00 00 00 00 00 ) the file is rejected again. Even though all three have all the actual game data.

There is a lot of stuff on google about header mismatches with NES roms. What fuzz about 16 measly bytes! But I read somewhere that Nestopia can handle any header when the rest of the rom file is correct if NstDatabase.xml is present. And sure enough put NstDatabase.xml in the es rom folder and now both the rom I started with and the “blank” NES header file plays.

This takes me back to the question. Is there a bug in how Retroarch creates NES playlists? Why does not the playlist generator include all roms whose headerless part (i.e. whole file except first 16 bytes) match a hash in NstDatabase.xml ? Because such roms seem to be playable in the Nestopia core.

With this suggested change Retroarch’s playlist creator would stop including files that Nestopia can’t play (headerless NES roms) and stop rejecting some files that Nestopia can play when NstDatabase.xml is present.

(As a sidenote it is strange that the nestopia core does not handle headless roms when it already handles blank header roms. If nestopia gets an input file with no NES header it could prefix a blank header and retry. Edit: this has apparently been a complaint for a long time but is not about to be implemented )

#2

The NES databases aren’t based on what Nestopia can play, since we have a number of other cores that have other compatibility. It’s all just based on no-intro dats.

#3

But the Retroarch playlist creator accepts headerless versions of roms that are not in the no-intro dat on Libretro github. For example check Super Mario Bros:

smb.nes (with header) 7D5FAA58 , ab30029efec6ccfc5d65dfda7fbc6e6489a80805 = in dat , included by playlist creator , playable in Nestopia

smb.nes (headerless) 9A2DB086 , 0c4992fc08d2278697339d3b48066e7b5f943598 = not in dat, still included by playlist creator, not playable in Nestopia

If we look at settings > information > database manager > NES .rdb there is both header version hashes and a CRC32 hash of the headerless part there. So it looks like Retroarch also has some additional dat file and uses it to include files during playlist creation. I searched and found the headerless smb.nes CRC hash in the NES files in libretro-database/metadat/ for example here . Note: the metadat files for NES appear to be headerless version hashes while no-intro dat on Libretro github are headered version hashes.

Anyway we can change my talk of a possible bug to feature request talk: why not also allow NES roms into playlist when they have a nes header ( first four bytes 4E 45 53 1A ) and their headerless part ( everything except first 16 bytes) matches the headerless hash Retroarch already has access to in the metadat files? Retroarch already has a hash check function. Why not make that function also check the hash of the headerless part of a NES file during playlist construction?