Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Readme update #1514

Open
wants to merge 63 commits into
base: master
Choose a base branch
from
Open

Readme update #1514

wants to merge 63 commits into from

Conversation

OctopusButtons
Copy link
Contributor

@OctopusButtons OctopusButtons commented Jan 21, 2025

Thank you to @RobLoach, @zoltanvb, @hunterk, @zach-morris, i30817, for helpful information, answers, and discussions.

Database readme update.

Goals

  • Add clarity on a few key points (below), to address my own questions and questions I've seen confusion about
  • A foundation for some would-be-contributors to become active contributors, and for some Issue Reports to become contributions.
  • To publish clarification about clrmamepro header description header in particular, in the hopes of addressing sourcing mysteries for future or current dats.

List of Additions / Expansions

  • Haven't deleted any information/points, only added, moved, edited/re-phrased.
  • Added intro details about .dat and .rdb files
  • Cheats
    • Description expanded a bit, partly to contrast with 'game information data' which is the main topic of the readme
  • RetroArch's Usage of the Database outline and description.
    • Validation [...]. Previous text about RetroArch verifying against a known good copy has been re-phrased and moved here.
    • Game Naming [...]
    • Thumbnail images based on game names [...]. Explains that thumbnails are only based on game name (with caveats), not on the database, but that the databased assignment of game names allows featured automated retrieval of thumbnails that otherwise wouldn't happen (without user management of filenames).
    • "Explore" Category Search [...]
    • "Information" view per-game [...]
  • Key Field / Matching and index
    • Explanation added: CRC versus Serial, build script code as a reference for which key is in use. (I believe there's a lot of confusion about key matching, with many people imagining it's "hash" not checksum and not serial, and not realizing that in-disc serials are scanned and used for matching, etc. Readme additions intend to address.)
  • Precedence
    • Explanation added. Not only for general info but as background for investigating discrepancies.
  • Game data specified in dats
    • Brief primer and example added, plus an example of collated metadata
  • Header Guidelines for dats
    • Focuses only on the matter of clrmamepro header description field and the current or systematic lack of sourcing/documentation for several dats. Intended to address the situation where, I believe, many people see a typical header description "Manufacturer - Console" and mistakenly think that's required for functionality/by-spec. Specification now given about better documentation of origins, sourcing, including both upstream credit and contributor/scraper/builder
  • Repository Contents expansion / "Guide to these dats" (so to speak)
    • Non-exhaustive guide per-folder for "insightful" cases that illustrate patterns or unique niches (fan translations, other variants) that people might be looking for.
    • Explains upstream bulk data in /dat versus in /metadat, partly covers questions of whether a dat is ad hoc / contribute-able Versus whether it's bulk-overwritten by imports, in order to assist/encourage contributions.
    • Pre-emptive databases small note
  • Sources
    • Expanded intro to remind that the list highlights the bulk coverage source while subsets of the same library may be covered by manual/smaller fragments in other dats. Added note for ">" precedence.
  • Maintenance / Technical Usage
    • Moved the existing admin build/test info here.
    • Changed some header sizes
    • Added brief outline/guideline for Add a New Database, adapted from @RobLoach 's discussions discord, so that it's in a central reference location.
  • Contributions
    • Explains upstream data etc
    • Small-scale corrections
    • Links back to "Adding a New Database" (above)
    • Note/warning about folder structure revision / housekeeping / consolidation. With hypothetical volunteers in mind.
  • Databases & RetroArch Thumbnails
    • Explains that thumbnail repository file names are not updated by database updates, and that thumbnails do not derive from databases. Explains how to contribute fixes to game-name-related thumbnail problems etc.

WIP documentation NOT FINISHED YET
wip notes
Including broad breakdown into all general details and then Maintenance / Technical Usage admin ops.

WIP research into each sub-database.
Key Field CRC vs Serial
WIP per dat descriptions (non-exhaustive).
Contributions draft etc
New explicit note about how thumbnail names (and therefore thumbnail assignment) aren't synced to database name changes.  I'm partly mentioning for people investigating database/thumbnail troubles, but also to maybe get the interest of a volunteer someday.
@zoltanvb
Copy link
Contributor

Wow, that's quite complete :)
Few minor questions added.

README.md Outdated
## Contents
- __Game information database files__.
- __`.dat`__ files in the clrmamepro DAT format, from many [sources](#sources) and across many categories of metadata. The system of dats is multifaceted: alternative or additional sources can be easily added and maintained in a self-contained constituent, some dats may overlap in the games they cover (see [precedence](#precedence)), and some dats cover an exclusive niche of games or attributes.
- __`.rdb`__ files used by RetroArch, compiled and amalgamated from the `.dat` files. [RetroArch Database format](https://github.com/libretro/RetroArch/tree/68b3e5d8e02aff753e01a1f6f8969891910b2e0b/libretro-db#readme) (_no relation to Redis .RDB files_) accommodates RetroArch's [wide range of hardware/OS platforms](https://www.retroarch.com/index.php?page=platforms).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A more generic link would be better (unless there is something in that particular revision):
https://github.com/libretro/RetroArch/blob/master/libretro-db/README.md

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, I didn't realize I copied a history/commit (tree?) link there, meant to have master. Fixing now.

README.md Outdated

The `description " "` and `comment " "` fields within a libretro dat's `clrmamepro ( )` header should be used to clarify the origin and source of the data and file. For example, if a .dat includes 3rd party upstream data processed through a github author's build/scrape script(s), the comment and description (or other appropriate header fields) should contain information about _both_ those aspects of the dat's origin. The description and comment header fields are intended for documentation purposes, are ignored by RetroArch, and can be freely changed without issue.

Only the `name` field of a `.dat` file header must match a uniform system name recognized by RetroArch, not the `description` field.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Uniform" is maybe a bit too strong for what it is. Usually formulated in a "manufacturer - systemname" format, but there are exceptions where e.g. there is no manufacturer, or it is not a typical emulator core, etc... The main thing is that it should match the name that is specified by the core in the .info file: https://github.com/libretro/libretro-super/tree/master/dist/info

Copy link
Contributor Author

@OctopusButtons OctopusButtons Jan 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great point, I forgot to second-guess my statement for exceptions, mind focused on the contrast with [mis-use] of the description field. I'm fixing it like you said, header name should database field in .info files and with link to that repo.

Fixed link to master, was wrongly linked to a specific history/commit
Changed wording about header `name` to fix the wrong statement/generalization pointed out by reviewer.  Name isn't always `manufacturer - system`, and core info files (now linked) detail what it should be.
Extra clear about the `database` field in .info files now
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants