Scanning vs. Collections

RetroArch General
#1

RetroArch 1.6.7 seems to successfully scan directories and detect the ROMs within them (in this example, uncompressed SNES .smc files), but is only adding a slight fraction to the Collection. If I go back and browse through the target directory RetroArch clearly sees all of the entries, so why isn’t it adding them to the collection? Am I missing something painfully obvious?

Thanks in advance for any and all answers/insight - and if this is a stupid question best answered by RTFM I’d be genuinely grateful for a link to said FM, because I’m having a bear of a time finding clear, quality documentation (though I did find one .pdf with a bloody fantastic Philosophy section, it didn’t really address my issue).

#2

I don’t see any docs specifically talking about the databases, but here are two good related docs:

http://www.lakka.tv/doc/Playlists/

https://www.libretro.com/index.php/getting-started-with-retroarch/

The databases come primarily from the No Intro and Redump databases, so if your ROMs don’t match those (and I can tell you that *.smc files don’t; No-Intro SNES games are all in *.sfc format), the scan won’t pick them up.

1 Like
#3

hunterk, from what I can gather you seem to be one of the main oracles of knowledge here on this forum, so I definitely appreciate the reply. However, I almost sprayed coffee all over my KB when I got through with the links - I know it isn’t the case but damn it almost felt like being trolled; I mean, “Add Content/Scan Directory…and that’s it! We’re done with the basics…” In my best Kyle Broflovski: “Really?”

Why is this mightily important information about “No-Intro” and whatever else you mentioned essentially not documented?

That’s not rhetorical. If people need to hunt down specific romsets in order to get the intended functionality out of the software they should be made aware of this on the front end. Sure, maybe a few casual users will abandon the endeavor but it’s the right thing to do, and just good form. Honest question - are there legal issues or other obstacles which prevent such disclosure?

The Lakka…erm, documentation illustrates the problem as well:

“Lakka includes an internal database that can be used to scan ROMs from many kinds of systems in order to automatically generate playlists. In order for the playlist scanner to recognize the ROMs from these systems, they must be formatted according to a standard which varies from system to system.”

And that’s it - that’s all you get. No explanation. Just a mysterious standard. It seems…sadistic.

So if anyone else out there can further expound on this topic - provided it’s not verboten - it will be really helpful. I’m documenting the trials and tribulations of my nub out-of-the-box-install ordeal, so hopefully one day I can help others forego the same gnashing of teeth.

Again, thanks hunterk for providing a glimmer of insight here, and I hope you and yours aren’t rubbed the wrong way by my commentary. I’m actually having a decent amount of fun fighting with this program, and hope to see it through. But if doing so requires re-downloading of entire ROM libraries (lookin’ at you, PSX) that may be a bridge too far…

3 Likes
Netplay with ROM not included in retroarch's database
#4

+1
You and me and thousands others.
A simple “shortcuts” section to the rom directories will do the job much better.

2 Likes
#5

Yeah, I hear ya. I honestly was surprised that No-Intro/Redump weren’t mentioned anywhere in those docs, which is why I mentioned it here.

Personally, I don’t use the playlists and I think we should just get rid of the whole thing, scanning, thumbnails and all, because it’s not flexible enough and never can/will be, not to mention the tremendous bandwidth cost associated with the thumbnails.

If your PSX games are all in ISO format, you’ll have trouble playing them in beetle-psx, but otherwise, you shouldn’t need to redownload them.

3 Likes
#6

There used to be details about which roms go with which which system in the lakka docs (I wrote it up and added a table). Since then it’s been removed, and I no longer have permissions to change those docs. :horse:

#7

oh yeah, I remember that table! it was great. I’ll try to find out what happened with that…

#8

as a matter of fact I now remember cloning that table here:

I still don’t know why it left the docs though!

2 Likes
#9

[quote=“hunterk, post:5, topic:11811, full:true”] Personally, I don’t use the playlists and I think we should just get rid of the whole thing, scanning, thumbnails and all, because it’s not flexible enough and never can/will be, not to mention the tremendous bandwidth cost associated with the thumbnails.[/quote] In my opinion the playlists per system with their nice controller and cartridge icons (especially with the Retroactive theme) are a great feature. If you just want to get rid of the scanner, ROM databases and thumbnails I can’t tell how many people would miss these features, but removing the playlist feature as a whole would noticeably reduce the comfort of the way you have to interact with RetroArch namely every time you would like to run a game.

Selecting games from a library feels just right while dealing with folder structures plus selecting the appropriate core would feel somewhat old fashioned.

So please don’t kill the playlist feature itself even this would mean we would have to manage them manually.

1 Like
#10

@john.browser and @hunterk I’ve submitted a PR to add a new guide on playlists to the docs: https://github.com/libretro/docs/pull/70

Please feel free to comment on that PR if you have any input on how to improve it.

#11

Looks great to me. Seems I’m not an owner on that repo so I can’t merge it, but I definitely approve :slight_smile:

#12

To everyone: thanks for the great replies, especially the info from @markwkidd and @hunterk.

I’ve just finished pulling some romsets (hoping to Christ/Odin/FSM et al. they’re the “right” ones) so I haven’t tested yet, but from what I understand they’ll not only populate the appropriate Collection tab, but also spit out a thumbnail for each ROM?

@hunterk could you elaborate more on the “tremendous bandwidth cost associated with the thumbnails” - do you mean the initial pulling of the thumbs data from the servers? This is interesting as I agree with @Octopaps, but only to an extent - I believe the ability to sort and select games from a library structure is a significant raison d’être for this platform, but generic cartridge icons are a perfectly acceptable compromise (“oh no, I don’t have the Ice Climber box art, my retro gaming experience is roont!”). As far as scanning goes, I think if the devs commit to a specific “officially unofficial” set - and let people know what they are upfront - there really isn’t a need to spend resources on supporting anything other than those designated sets. Although the @James-F option would be a fine option also - have users map the directory for each console’s ROMS.

Why are PSX games in ISO format problematic? The thick plottens…

HUGE thanks again for the responses, helpful and fascinating stuff!

#13

Re: bandwidth, yeah, we grind through well over 10 TB of bandwidth per month, mostly for thumbnails, and that’s not counting what gets routed through CloudFlare (probably ~10x that).

Re: ISO, the core just doesn’t support loading ISO files directly. Sometimes, you can create a cue sheet that points to the ISO and have it work but it’s not foolproof. The core supports bin/cue and pbp (and perhaps other formats that I’m not remembering at the moment), though I don’t recall whether pbp files will scan properly.

We’ve had nonstop complaints even from people that know about the No-Intro limitation because we can’t cover ROMhacks/translations, we don’t support the exact flavor of MAME ROMs they prefer, we don’t support scanning games compressed with the archive format they prefer, etc. It’s honestly neverending. I don’t think the other core contributors share my extreme views on this topic, but I’m sick of it.

#14

I don’t use thumbnails and scanning myself as I use retroarch as a backend and a frontend to display artwork and playlists. (Like many people). I understand the appeal of just using RA directly and it’s scanning/theming can make it look pretty awesome.

Is there an alternative middle ground for the this feature. To reduce the bandwidth (cost) but still retain the ease and looks of the scanning feature?

Getting rid of the feature totally seems to “extreme” but also the costs saved could go on more “important” things. Being a Patreon member I would like to see money diverted from thumbnail download costs to other things.

Maybe a semi standalone core/program that could build playlists straight off the romfolder and user supplies the game artwork.

#15

Here is a direct link to the new (in a sense) RetroArch guide on ROMs, Playlists, and Thumbnails: https://docs.libretro.com/guides/roms-playlists-thumbnails/

#16

I assumed the database was a combination of both No-Intro and GoodTools for most systems which should catch everything except for recent hacks, no? If not, it should be. Some small additional features would help too, like an option to match games that have the right filename but wrong hash, or pick whichever entry in the database has the shortest Levenshtein distance from the filenames which should work if the name is a little off but still close to the actual title. Those two things should be quite easy to implement. Lastly there really needs to be a graphical way to create/edit a custom database. That will be a lot harder to add I’m guessing but even just those other things should help tremendously.

Please don’t kill the whole thing, it has great potential.

#17

I don’t use thumbnails and scanning myself as I use retroarch as a backend and a frontend to display artwork and playlists. (Like many people).

Keep in mind some platforms (e.g. consoles). don’t have the option to use a separate frontend, and thus are restricted to what Retroarch provides.

Maybe a semi standalone core/program that could build playlists straight off the romfolder and user supplies the game artwork.

Playlist Buddy will do this, and works well in my experience.

#18

Personally, I don’t use the playlists and I think we should just get rid of the whole thing, scanning, thumbnails and all, because it’s not flexible enough and never can/will be, not to mention the tremendous bandwidth cost associated with the thumbnails.

Whoa now, isn’t that a bit of throwing out the baby with the bathwater? I mean Retroarch itself will never be flexible enough for all users either, but that doesn’t mean it should be scrapped :slight_smile:

Bandwidth costs I can sympathize with, there are enough other places to find covers that it wouldn’t be too inconvenient if they needed to be removed for financial reasons.

I strongly believe playlists should remain though; “Load Content”, “Start Folder”, down, down, down, down, down, Console folder, down down down down etc. gets old very fast.

1 Like
#19

I strongly believe playlists should remain though; “Load Content”, “Start Folder”, down, down, down, down, down, Console folder, down down down down etc. gets old very fast.>

Exactly!
As I said many times before, “Shortcuts” or “Favorite Locations” will be very useful indeed.
Moreover, searching for a rom with a keyboard will cut browsing time too, F12 (on-screen keyboard) just doesn’t work in the rom list.

#20

It’s hard to please everyone. Like hunterk, I don’t use playlists or scanning. I adopted the <System>/<Rom> convention in my rom folder a long time ago so “Load Content” would display a nice list of systems. Unfortunately these and other features changed this menu and now I have to “navigate into” my rom folder instead of just seeing the system names. See https://imgur.com/a/6a2hf