From 233497039ee8a78861194cdd60ce16ecdbc89f42 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Tue, 31 Dec 2024 17:39:13 -0500 Subject: [PATCH 01/62] WIP Update README.md WIP documentation NOT FINISHED YET --- README.md | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 168ec8e2c..f2bbe7b38 100644 --- a/README.md +++ b/README.md @@ -1,16 +1,34 @@ # RetroArch Database -RetroArch incoporates a ROM scanning system to automatically produce playlists. Each ROM that is scanned by the playlist generator is checked against a database of ROMs that are known to be good copies. +RetroArch incoporates a ROM scanning system to automatically produce playlists. Each ROM that is scanned by the playlist generator is checked against a database of ROMs that are known to be good copies. The key field for matching depends on the typical file size of the given system: the crc checksum for systems with smaller file sizes, but instead the serial number for larger files like disc-based systems. [Edit: but the serial must be retrieved/looked up in the first place? Re-check discussion) + +Database entries at minimum contain fields for 1) a game's name and 2) checksum/hash for identifying a particular file. Ideally the entries include further metadata such as a description (to disambiguate variants that may justifiably receive the same game name as another variant), + +The automatic catalogging functions that depend on the database include: + +- Assign a definitive display name for each game file _regardless of filename_ +- Download thumbnail images for each game based on the uniform name assigned by the database, regardless of filename +- A reference search (named "Explore") so that the user can search for games by category, e.g. by Developer, Release Year, Genre, etc. +- Provide an in-app viewable informational screen for each game (Game > Information > Database Entry) + ## Contents - [`cht`](cht) Cheats to various games - [`cursors`](cursors) Provides methods in order to query the playlists -- [`dat`](dat) Customized DAT files, maintained by the libretro team -- [`metadat`](metadat) Different metadata and third-party DATs available to the systems +- [`dat`](dat) Customized DAT files, maintained by the libretro team, including items that do not have contemporary documentation by upstream catalogging groups (e.g. "Virtual Console" variants). +- [`metadat`](metadat) Various metadata and third-party DATs. Examples: + - [`no-intro`](metadat/no-intro) Bulk carry-over data from No-Intro databases. [NOTE EDIT: THIS IS FALSE, research and revise] + - [`redump`](metadat/redump) Bulk carry-over data from Redump databases + - [`hacks`](metadat/hacks) Data for modified versions of commercially released games. These data are set by direct manual commits on the Libretro Github. + - [`homebrew`](metadat/homebrew) Data for non-officially-published games created by independent programmers + - [`libretro-dats`](metadat/libretro-dats) E.g. Fan translations of SNES games, and [NOTE] FDS? Why FDS, check if covered elsewhere + - And more - [`rdb`](rdb) The compiled RetroArch database files - [`scripts`](scripts) Various scripts that are used to maintain the database files +Component databases earlier in the list have precedence over items later in the list. For example, definitions in `dat` will over-ride `metadat` in the final `.rdb` compile if any info conflicts for the same item. + ## Sources Generally, RetroArch's scanner is configured for ROMs that have been validated by [No-Intro](http://datomatic.no-intro.org) or Redump DAT files but many other source databases are also in use. From 0a7a00a7d80ef1be1dfff0aa6403cd4de302a65f Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Sun, 5 Jan 2025 21:29:45 -0500 Subject: [PATCH 02/62] WIP edits README.md --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index f2bbe7b38..02a6c319b 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ -# RetroArch Database +# Libretro Database -RetroArch incoporates a ROM scanning system to automatically produce playlists. Each ROM that is scanned by the playlist generator is checked against a database of ROMs that are known to be good copies. The key field for matching depends on the typical file size of the given system: the crc checksum for systems with smaller file sizes, but instead the serial number for larger files like disc-based systems. [Edit: but the serial must be retrieved/looked up in the first place? Re-check discussion) +RetroArch includes a ROM scanning system to automatically produce playlists. Each ROM that is scanned by the playlist generator is checked against a database of ROMs that are known to be good copies. The key field for matching depends on the typical file size of the given system: the crc checksum for systems with smaller file sizes, but instead the serial number for larger files like disc-based systems. [Edit: but the serial must be retrieved/looked up in the first place? Re-check discussion) Database entries at minimum contain fields for 1) a game's name and 2) checksum/hash for identifying a particular file. Ideally the entries include further metadata such as a description (to disambiguate variants that may justifiably receive the same game name as another variant), @@ -18,7 +18,7 @@ The automatic catalogging functions that depend on the database include: - [`cursors`](cursors) Provides methods in order to query the playlists - [`dat`](dat) Customized DAT files, maintained by the libretro team, including items that do not have contemporary documentation by upstream catalogging groups (e.g. "Virtual Console" variants). - [`metadat`](metadat) Various metadata and third-party DATs. Examples: - - [`no-intro`](metadat/no-intro) Bulk carry-over data from No-Intro databases. [NOTE EDIT: THIS IS FALSE, research and revise] + - [`no-intro`](metadat/no-intro) Bulk carry-over data from No-Intro databases. - [`redump`](metadat/redump) Bulk carry-over data from Redump databases - [`hacks`](metadat/hacks) Data for modified versions of commercially released games. These data are set by direct manual commits on the Libretro Github. - [`homebrew`](metadat/homebrew) Data for non-officially-published games created by independent programmers From fddac63dcd92562df37dea4d2318e74eb81d1bd8 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Mon, 6 Jan 2025 09:53:07 -0500 Subject: [PATCH 03/62] wip edits README.md --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 02a6c319b..b0b1096e9 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,8 @@ The automatic catalogging functions that depend on the database include: - [`cht`](cht) Cheats to various games - [`cursors`](cursors) Provides methods in order to query the playlists -- [`dat`](dat) Customized DAT files, maintained by the libretro team, including items that do not have contemporary documentation by upstream catalogging groups (e.g. "Virtual Console" variants). +- [`dat`](dat) Customized DAT files, maintained by the libretro team, including items that do/did not have contemporary documentation by upstream catalogging groups (e.g. "Virtual Console" variants). + - SNES Virtual Console Variants - [`metadat`](metadat) Various metadata and third-party DATs. Examples: - [`no-intro`](metadat/no-intro) Bulk carry-over data from No-Intro databases. - [`redump`](metadat/redump) Bulk carry-over data from Redump databases From 5a8af9ddcd580f933eec2c3c0a98a8430dff6009 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Tue, 7 Jan 2025 23:09:25 -0500 Subject: [PATCH 04/62] wip notes wip notes --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index b0b1096e9..34f81e05e 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Libretro Database -RetroArch includes a ROM scanning system to automatically produce playlists. Each ROM that is scanned by the playlist generator is checked against a database of ROMs that are known to be good copies. The key field for matching depends on the typical file size of the given system: the crc checksum for systems with smaller file sizes, but instead the serial number for larger files like disc-based systems. [Edit: but the serial must be retrieved/looked up in the first place? Re-check discussion) +RetroArch includes a ROM scanning system to automatically produce playlists. Each ROM that is scanned by the playlist generator is checked against a database of ROMs that are known to be good copies (Change 'good copy' and add to purpose list). The key field for matching depends on the typical file size of the given system: the crc checksum for systems with smaller file sizes, but instead the serial number for larger files like disc-based systems. [Edit: but the serial must be retrieved/looked up in the first place? Re-check discussion. Research Serial being inside the file and retrieved, but not metadata?] Database entries at minimum contain fields for 1) a game's name and 2) checksum/hash for identifying a particular file. Ideally the entries include further metadata such as a description (to disambiguate variants that may justifiably receive the same game name as another variant), From ad2201a23da4897eaa6e119109b972eae2bfaff6 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Sun, 12 Jan 2025 15:49:48 -0500 Subject: [PATCH 05/62] WIP Update README.md Including broad breakdown into all general details and then Maintenance / Technical Usage admin ops. WIP research into each sub-database. --- README.md | 45 +++++++++++++++++++++++++++++---------------- 1 file changed, 29 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 34f81e05e..01a472e39 100644 --- a/README.md +++ b/README.md @@ -1,38 +1,49 @@ # Libretro Database -RetroArch includes a ROM scanning system to automatically produce playlists. Each ROM that is scanned by the playlist generator is checked against a database of ROMs that are known to be good copies (Change 'good copy' and add to purpose list). The key field for matching depends on the typical file size of the given system: the crc checksum for systems with smaller file sizes, but instead the serial number for larger files like disc-based systems. [Edit: but the serial must be retrieved/looked up in the first place? Re-check discussion. Research Serial being inside the file and retrieved, but not metadata?] +The github repository for databases used by RetroArch. RetroArch relies on databases to provide several catalogging functions: -Database entries at minimum contain fields for 1) a game's name and 2) checksum/hash for identifying a particular file. Ideally the entries include further metadata such as a description (to disambiguate variants that may justifiably receive the same game name as another variant), +- __Validation__. Reject or accept files when using the [Import Scanner / Playlist Generator](https://docs.libretro.com/guides/roms-playlists-thumbnails/#working-with-playlists) based on whether the ROM checksum matches the checksum of a known verified completely intact (aka "properly dumped") file. +- __Game Naming__. Assign a definitive and uniform display name for each game file regardless of filename. +- __Thumbnail Images__. Download and display thumbnail images for each game based on the uniform name assigned by the database, regardless of filename +- __Category Search__. A reference search (named "Explore") that allows the user to search for games by category criteria, e.g. by Developer, Release Year, Genre, and other attributes/metadata. +- __Per-Game Information__. Provide an in-app viewable informational screen for each game (Game > Information > Database Entry) -The automatic catalogging functions that depend on the database include: +#### Key Field +The key field for matching varies by file size, i.e. by console media type. -- Assign a definitive display name for each game file _regardless of filename_ -- Download thumbnail images for each game based on the uniform name assigned by the database, regardless of filename -- A reference search (named "Explore") so that the user can search for games by category, e.g. by Developer, Release Year, Genre, etc. -- Provide an in-app viewable informational screen for each game (Game > Information > Database Entry) +- __CRC checksum__ for systems with smaller file sizes, i.e. game media before the advent of disc-based games. +- __Serial Number__ within the ROM file for larger files like disc-based systems. [Edit: How does this work? It searches the binary data? Is it a certain sector to look for? This is NOT metadata?] +Current program code publicly viewable [LINK] shows which type of key field RetroArch uses for each console system. -## Contents +#### Game Data in Repository +Database entries at minimum contain fields for 1) a game's name, i.e. the display name that RetroArch will assign and 2) checksum/hash for identifying a particular file. Ideally the entries include further metadata such as a description (to disambiguate variants that may justifiably receive the same game name as another variant), -- [`cht`](cht) Cheats to various games + +## Repository Contents + +The repository contains many constituent databases that are compiled into `.rdb` files used by RetroArch. _(Note: several items described below are not compiled into RetroArch's game database files, e.g. Cheats, but are described here because they reside within the github database repository.)_ [EDIT: NOTE, separate into "Game Information Databases", ??"Functional/Maintenance", "Other"]?? + +- [`cht`](cht) Cheat codes to various games, collected from any available source on the web including by manual contributions by users who haved used RetroArch's built-in [memory address/value search feature](https://docs.libretro.com/guides/cheat-codes/#retroarch-new-cheat-code-searching) to construct new cheat codes. - [`cursors`](cursors) Provides methods in order to query the playlists - [`dat`](dat) Customized DAT files, maintained by the libretro team, including items that do/did not have contemporary documentation by upstream catalogging groups (e.g. "Virtual Console" variants). - SNES Virtual Console Variants - [`metadat`](metadat) Various metadata and third-party DATs. Examples: - - [`no-intro`](metadat/no-intro) Bulk carry-over data from No-Intro databases. - - [`redump`](metadat/redump) Bulk carry-over data from Redump databases - - [`hacks`](metadat/hacks) Data for modified versions of commercially released games. These data are set by direct manual commits on the Libretro Github. - - [`homebrew`](metadat/homebrew) Data for non-officially-published games created by independent programmers - - [`libretro-dats`](metadat/libretro-dats) E.g. Fan translations of SNES games, and [NOTE] FDS? Why FDS, check if covered elsewhere + - [`no-intro`](metadat/no-intro) Bulk import from upstream No-Intro databases. Generally non-disc-based systems. + - [`redump`](metadat/redump) Bulk import from upstream Redump databases. Generally disc-based systems. + - [`hacks`](metadat/hacks) Data for modified (or "hacked") versions of commercially released games. These data are set by direct manual commits on the Libretro Github. + - [`homebrew`](metadat/homebrew) Data for non-officially-published games created by independent creators/programmers + - [`libretro-dats`](metadat/libretro-dats) E.g. [EDIT: maybe "ad hoc databases for items not covered by upstream bla bla] Fan translations of SNES games, and [NOTE] FDS? Why FDS, check if covered elsewhere - And more - [`rdb`](rdb) The compiled RetroArch database files - [`scripts`](scripts) Various scripts that are used to maintain the database files -Component databases earlier in the list have precedence over items later in the list. For example, definitions in `dat` will over-ride `metadat` in the final `.rdb` compile if any info conflicts for the same item. +#### Precedence +Component databases earlier in the list have precedence over items later in the list. Definitions in `dat` will over-ride `metadat` in the final `.rdb` compile if any info conflicts for the same item. ## Sources -Generally, RetroArch's scanner is configured for ROMs that have been validated by [No-Intro](http://datomatic.no-intro.org) or Redump DAT files but many other source databases are also in use. +Many source databases are in use, as listed below. A large majority of games commonly used in RetroArch are covered by [No-Intro](http://datomatic.no-intro.org) or [Redump](http://redump.org/downloads/) DAT files |System|Source|Repository| |----|---|---| @@ -165,6 +176,8 @@ Generally, RetroArch's scanner is configured for ROMs that have been validated b |Watara - Supervision|[No-Intro](http://datomatic.no-intro.org)|[libretro-dats](https://github.com/robloach/libretro-dats)| |Wolfenstein 3D| | +# Maintenance / Technical Usage + ## Building To build a complete set of RDB files for RetroArch or to generate a single RDB file, see [RetroArch/libretro-db/README.md](https://github.com/libretro/RetroArch/blob/master/libretro-db/README.md). From c45345784f89c660b937f32d6cb43ab97f3fd044 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Sun, 12 Jan 2025 15:50:05 -0500 Subject: [PATCH 06/62] Update README.md --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 01a472e39..5fa3face3 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,8 @@ # Libretro Database -The github repository for databases used by RetroArch. RetroArch relies on databases to provide several catalogging functions: +The github repository for databases used by RetroArch. + +RetroArch relies on databases to provide several catalogging functions: - __Validation__. Reject or accept files when using the [Import Scanner / Playlist Generator](https://docs.libretro.com/guides/roms-playlists-thumbnails/#working-with-playlists) based on whether the ROM checksum matches the checksum of a known verified completely intact (aka "properly dumped") file. - __Game Naming__. Assign a definitive and uniform display name for each game file regardless of filename. From 78a06e7c8df322936955c581ed18915afb4aae4a Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Mon, 13 Jan 2025 10:30:24 -0500 Subject: [PATCH 07/62] WIP Update README.md WIP --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 5fa3face3..a2b644b82 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@ The repository contains many constituent databases that are compiled into `.rdb` - [`cht`](cht) Cheat codes to various games, collected from any available source on the web including by manual contributions by users who haved used RetroArch's built-in [memory address/value search feature](https://docs.libretro.com/guides/cheat-codes/#retroarch-new-cheat-code-searching) to construct new cheat codes. - [`cursors`](cursors) Provides methods in order to query the playlists -- [`dat`](dat) Customized DAT files, maintained by the libretro team, including items that do/did not have contemporary documentation by upstream catalogging groups (e.g. "Virtual Console" variants). +- [`dat`](dat) Customized DAT files maintained by the libretro team, including items that do/did not have contemporary documentation by upstream catalogging groups, e.g. "Virtual Console" variants of SNES games. Also includes some imports from upstream in order to establish precedence in the compilation, e.g. GameCube data from GameTDB. - SNES Virtual Console Variants - [`metadat`](metadat) Various metadata and third-party DATs. Examples: - [`no-intro`](metadat/no-intro) Bulk import from upstream No-Intro databases. Generally non-disc-based systems. @@ -45,7 +45,7 @@ Component databases earlier in the list have precedence over items later in the ## Sources -Many source databases are in use, as listed below. A large majority of games commonly used in RetroArch are covered by [No-Intro](http://datomatic.no-intro.org) or [Redump](http://redump.org/downloads/) DAT files +Many source databases are in use, as listed below. A large majority of games commonly used in RetroArch are covered by [No-Intro](http://datomatic.no-intro.org) or [Redump](http://redump.org/downloads/) DAT files. ">" signs below mean that the multiple listed sources are compiled with the given precedence. |System|Source|Repository| |----|---|---| From 2037602c5b0cf287eeaaeba10fdd886d1ef60da3 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Mon, 13 Jan 2025 10:38:12 -0500 Subject: [PATCH 08/62] WIP Update README.md WIP --- README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index a2b644b82..2de31be17 100644 --- a/README.md +++ b/README.md @@ -41,7 +41,11 @@ The repository contains many constituent databases that are compiled into `.rdb` - [`scripts`](scripts) Various scripts that are used to maintain the database files #### Precedence -Component databases earlier in the list have precedence over items later in the list. Definitions in `dat` will over-ride `metadat` in the final `.rdb` compile if any info conflicts for the same item. +Databases earlier in the list have precedence over items later in the list. Definitions in `dat` will over-ride `metadat` in the final `.rdb` compile if any info conflicts for the same item. + +#### Pre-emptive Databases + +Some databases are maintained even if RetroArch currently has no core for the games/system, e.g. GP32, Vita, Original Xbox, and PS3. ## Sources From fd0890ec87695c23def9dc2de82b8913c69fe8d8 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Mon, 13 Jan 2025 10:43:14 -0500 Subject: [PATCH 09/62] WIP Update README.md WIP --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 2de31be17..d91e09708 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,7 @@ The github repository for databases used by RetroArch. +## Overview RetroArch relies on databases to provide several catalogging functions: - __Validation__. Reject or accept files when using the [Import Scanner / Playlist Generator](https://docs.libretro.com/guides/roms-playlists-thumbnails/#working-with-playlists) based on whether the ROM checksum matches the checksum of a known verified completely intact (aka "properly dumped") file. @@ -49,7 +50,7 @@ Some databases are maintained even if RetroArch currently has no core for the ga ## Sources -Many source databases are in use, as listed below. A large majority of games commonly used in RetroArch are covered by [No-Intro](http://datomatic.no-intro.org) or [Redump](http://redump.org/downloads/) DAT files. ">" signs below mean that the multiple listed sources are compiled with the given precedence. +Many source databases are in use, as listed below. A large majority of games commonly used in RetroArch are covered by [No-Intro](http://datomatic.no-intro.org) or [Redump](http://redump.org/downloads/) DAT files. ">" signs below mean that the multiple listed sources are compiled with the given [precedence](#precedence). |System|Source|Repository| |----|---|---| From 23a7c55df7c6f9b52fb096622faa5b7d84d13e92 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Mon, 13 Jan 2025 10:50:29 -0500 Subject: [PATCH 10/62] WIP Update README.md WIP --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index d91e09708..15ae73db2 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,8 @@ The github repository for databases used by RetroArch. ## Overview -RetroArch relies on databases to provide several catalogging functions: + +Libretro databases allow RetroArch to provide several catalogging functions: - __Validation__. Reject or accept files when using the [Import Scanner / Playlist Generator](https://docs.libretro.com/guides/roms-playlists-thumbnails/#working-with-playlists) based on whether the ROM checksum matches the checksum of a known verified completely intact (aka "properly dumped") file. - __Game Naming__. Assign a definitive and uniform display name for each game file regardless of filename. From 5a2f93181ad7d612a284a36d0e11740f54102929 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Mon, 13 Jan 2025 10:59:23 -0500 Subject: [PATCH 11/62] WIP Update README.md WIP --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 15ae73db2..c38568041 100644 --- a/README.md +++ b/README.md @@ -51,7 +51,7 @@ Some databases are maintained even if RetroArch currently has no core for the ga ## Sources -Many source databases are in use, as listed below. A large majority of games commonly used in RetroArch are covered by [No-Intro](http://datomatic.no-intro.org) or [Redump](http://redump.org/downloads/) DAT files. ">" signs below mean that the multiple listed sources are compiled with the given [precedence](#precedence). +Many source databases are in use, as listed below. A large majority of games commonly used in RetroArch are covered by [No-Intro](http://datomatic.no-intro.org) or [Redump](http://redump.org/downloads/) DAT files. ">" signs below indicate the [precedence](#precedence) order when multiple sources cover the same area of data. |System|Source|Repository| |----|---|---| From 723776bda5e470fe81674418bd7fac70e9755656 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Mon, 13 Jan 2025 23:14:03 -0500 Subject: [PATCH 12/62] WIP Update README.md Key Field CRC vs Serial --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index c38568041..de8b95f9c 100644 --- a/README.md +++ b/README.md @@ -16,9 +16,9 @@ Libretro databases allow RetroArch to provide several catalogging functions: The key field for matching varies by file size, i.e. by console media type. - __CRC checksum__ for systems with smaller file sizes, i.e. game media before the advent of disc-based games. -- __Serial Number__ within the ROM file for larger files like disc-based systems. [Edit: How does this work? It searches the binary data? Is it a certain sector to look for? This is NOT metadata?] +- __Serial Number__ found within the ROM file for larger files like disc-based games. Note the serial is not metadata but encoded within the game's binary data, which is scanned as a byte array by RetroArch. -Current program code publicly viewable [LINK] shows which type of key field RetroArch uses for each console system. +Current [build script code](https://github.com/libretro/libretro-super/blob/master/libretro-build-database.sh#L288) can be used as a reference for which type of key field RetroArch uses for each console system. #### Game Data in Repository Database entries at minimum contain fields for 1) a game's name, i.e. the display name that RetroArch will assign and 2) checksum/hash for identifying a particular file. Ideally the entries include further metadata such as a description (to disambiguate variants that may justifiably receive the same game name as another variant), From 31334f1f1a0add255e88515af11aac98ec550e85 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Mon, 13 Jan 2025 23:45:57 -0500 Subject: [PATCH 13/62] WIP README.md WIP per dat descriptions (non-exhaustive). --- README.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index de8b95f9c..80573c5ac 100644 --- a/README.md +++ b/README.md @@ -30,14 +30,15 @@ The repository contains many constituent databases that are compiled into `.rdb` - [`cht`](cht) Cheat codes to various games, collected from any available source on the web including by manual contributions by users who haved used RetroArch's built-in [memory address/value search feature](https://docs.libretro.com/guides/cheat-codes/#retroarch-new-cheat-code-searching) to construct new cheat codes. - [`cursors`](cursors) Provides methods in order to query the playlists -- [`dat`](dat) Customized DAT files maintained by the libretro team, including items that do/did not have contemporary documentation by upstream catalogging groups, e.g. "Virtual Console" variants of SNES games. Also includes some imports from upstream in order to establish precedence in the compilation, e.g. GameCube data from GameTDB. - - SNES Virtual Console Variants +- [`dat`](dat) Customized DAT files maintained by the libretro team, including items that do/did not have contemporary documentation by upstream catalogging groups, e.g. "Virtual Console" variants of SNES games and non-generalized cores (Cave Story, Doom, etc). Also includes some imports from upstream groups in order to establish precedence in the compilation, e.g. GameCube data from GameTDB. - [`metadat`](metadat) Various metadata and third-party DATs. Examples: - - [`no-intro`](metadat/no-intro) Bulk import from upstream No-Intro databases. Generally non-disc-based systems. - - [`redump`](metadat/redump) Bulk import from upstream Redump databases. Generally disc-based systems. + - [`bbfc`](metadat/bbfc) British Board of Film Classification's ratings for age-appropriateness + - [`elspa`](metadat/elspa) Age-appropriateness/content ratings from the Entertainment and Leisure Software Publishers Association aka the Association for UK Interactive Entertainment ("Ukie") - [`hacks`](metadat/hacks) Data for modified (or "hacked") versions of commercially released games. These data are set by direct manual commits on the Libretro Github. - [`homebrew`](metadat/homebrew) Data for non-officially-published games created by independent creators/programmers - - [`libretro-dats`](metadat/libretro-dats) E.g. [EDIT: maybe "ad hoc databases for items not covered by upstream bla bla] Fan translations of SNES games, and [NOTE] FDS? Why FDS, check if covered elsewhere + - [`libretro-dats`](metadat/libretro-dats) Ad hoc databases for items that were/are not covered by upstream database groups. Currently only fan translations of SNES games, and an FDS dat that includes Virtual Console variants. + - [`no-intro`](metadat/no-intro) Bulk import from upstream No-Intro databases. Generally non-disc-based systems. + - [`redump`](metadat/redump) Bulk import from upstream Redump databases. Generally disc-based systems. - And more - [`rdb`](rdb) The compiled RetroArch database files - [`scripts`](scripts) Various scripts that are used to maintain the database files @@ -51,7 +52,7 @@ Some databases are maintained even if RetroArch currently has no core for the ga ## Sources -Many source databases are in use, as listed below. A large majority of games commonly used in RetroArch are covered by [No-Intro](http://datomatic.no-intro.org) or [Redump](http://redump.org/downloads/) DAT files. ">" signs below indicate the [precedence](#precedence) order when multiple sources cover the same area of data. +A large majority of games commonly used in RetroArch are covered by [No-Intro](http://datomatic.no-intro.org) or [Redump](http://redump.org/downloads/) DAT files, but many other source databases are in use, as listed below. ">" signs below indicate the [precedence](#precedence) order when multiple sources overlap for the same subset of games/data. |System|Source|Repository| |----|---|---| From d369ec9569605a49fd641d4c1652a45bcb620c6c Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Tue, 14 Jan 2025 21:50:54 -0500 Subject: [PATCH 14/62] WIP Update README.md WIP updates --- README.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 80573c5ac..dc830fd09 100644 --- a/README.md +++ b/README.md @@ -26,19 +26,20 @@ Database entries at minimum contain fields for 1) a game's name, i.e. the displa ## Repository Contents -The repository contains many constituent databases that are compiled into `.rdb` files used by RetroArch. _(Note: several items described below are not compiled into RetroArch's game database files, e.g. Cheats, but are described here because they reside within the github database repository.)_ [EDIT: NOTE, separate into "Game Information Databases", ??"Functional/Maintenance", "Other"]?? +The repository contains game information databases that are compiled into `.rdb` files used by RetroArch, cheat code files, and management/admin files and scripts. The non-exhaustive list below serves as a guide to various folders/files in the repository. -- [`cht`](cht) Cheat codes to various games, collected from any available source on the web including by manual contributions by users who haved used RetroArch's built-in [memory address/value search feature](https://docs.libretro.com/guides/cheat-codes/#retroarch-new-cheat-code-searching) to construct new cheat codes. +- [`cht`](cht) Cheat codes to various games, collected from any available source on the web including by manual contributions by users who haved used RetroArch's built-in [memory address/value search feature](https://docs.libretro.com/guides/cheat-codes/#retroarch-new-cheat-code-searching) to construct new cheat codes. Game-specific cheat files remain in plain text format, can be loaded by the user in RetroArch, and are not part of a compilation process like the game information databases. - [`cursors`](cursors) Provides methods in order to query the playlists -- [`dat`](dat) Customized DAT files maintained by the libretro team, including items that do/did not have contemporary documentation by upstream catalogging groups, e.g. "Virtual Console" variants of SNES games and non-generalized cores (Cave Story, Doom, etc). Also includes some imports from upstream groups in order to establish precedence in the compilation, e.g. GameCube data from GameTDB. +- [`dat`](dat) Customized DAT files maintained by the libretro team, including items that do/did not have contemporary documentation by upstream catalogging groups, e.g. "Virtual Console" variants of SNES games and monolithic non-generalized cores (Cave Story, Doom, Quake, etc). Also includes some bulk imports from upstream groups in order to establish precedence in the compilation, e.g. GameCube data from GameTDB. - [`metadat`](metadat) Various metadata and third-party DATs. Examples: - [`bbfc`](metadat/bbfc) British Board of Film Classification's ratings for age-appropriateness - [`elspa`](metadat/elspa) Age-appropriateness/content ratings from the Entertainment and Leisure Software Publishers Association aka the Association for UK Interactive Entertainment ("Ukie") - - [`hacks`](metadat/hacks) Data for modified (or "hacked") versions of commercially released games. These data are set by direct manual commits on the Libretro Github. + - [`hacks`](metadat/hacks) Data for modified (or "hacked") versions of commercially released games. Many of these data are set by direct manual commits on the Libretro Github. - [`homebrew`](metadat/homebrew) Data for non-officially-published games created by independent creators/programmers - [`libretro-dats`](metadat/libretro-dats) Ad hoc databases for items that were/are not covered by upstream database groups. Currently only fan translations of SNES games, and an FDS dat that includes Virtual Console variants. - [`no-intro`](metadat/no-intro) Bulk import from upstream No-Intro databases. Generally non-disc-based systems. - [`redump`](metadat/redump) Bulk import from upstream Redump databases. Generally disc-based systems. + - [`tosec`](metadat/tosec) Bulk import from upstream TOSEC databases. - And more - [`rdb`](rdb) The compiled RetroArch database files - [`scripts`](scripts) Various scripts that are used to maintain the database files @@ -52,7 +53,7 @@ Some databases are maintained even if RetroArch currently has no core for the ga ## Sources -A large majority of games commonly used in RetroArch are covered by [No-Intro](http://datomatic.no-intro.org) or [Redump](http://redump.org/downloads/) DAT files, but many other source databases are in use, as listed below. ">" signs below indicate the [precedence](#precedence) order when multiple sources overlap for the same subset of games/data. +Many source databases are in use as listed below. A large majority of games commonly used in RetroArch are covered by [No-Intro](http://datomatic.no-intro.org) or [Redump](http://redump.org/downloads/) DAT files. ">" signs below indicate the [precedence](#precedence) order when multiple sources overlap for the same subset of games/data. |System|Source|Repository| |----|---|---| @@ -205,7 +206,7 @@ Make sure filenames are Windows file system compatible, and are not too long (eg find -exec basename '{}' ';' | egrep '^.{144,}$' ``` -## Integrations +# Integrations There are a few tools out there that allow integration with libretro's database. From 9b9bf7e384513aec9806ad16644350bc4c4cccd2 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Tue, 14 Jan 2025 21:59:35 -0500 Subject: [PATCH 15/62] WIP Update README.md WIP --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index dc830fd09..a807fd9a1 100644 --- a/README.md +++ b/README.md @@ -21,7 +21,7 @@ The key field for matching varies by file size, i.e. by console media type. Current [build script code](https://github.com/libretro/libretro-super/blob/master/libretro-build-database.sh#L288) can be used as a reference for which type of key field RetroArch uses for each console system. #### Game Data in Repository -Database entries at minimum contain fields for 1) a game's name, i.e. the display name that RetroArch will assign and 2) checksum/hash for identifying a particular file. Ideally the entries include further metadata such as a description (to disambiguate variants that may justifiably receive the same game name as another variant), +Database entries generally at minimum specify 1) a game's name, i.e. the display name that RetroArch will assign in playlists and 2) checksum/hash for identifying a particular file (or sometimes a serial number as a [key field](#key-field). Further metadata for each game is often sourced from multiple databases. ## Repository Contents From fd24c67dc2bca9ae122c80cce95a0304f60c93bf Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Tue, 14 Jan 2025 22:06:48 -0500 Subject: [PATCH 16/62] Update README.md WIP --- README.md | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index a807fd9a1..239191d60 100644 --- a/README.md +++ b/README.md @@ -13,24 +13,21 @@ Libretro databases allow RetroArch to provide several catalogging functions: - __Per-Game Information__. Provide an in-app viewable informational screen for each game (Game > Information > Database Entry) #### Key Field -The key field for matching varies by file size, i.e. by console media type. +The key field for matching varies by file size, i.e. by console media type. Current [build script code](https://github.com/libretro/libretro-super/blob/master/libretro-build-database.sh#L288) can be used as a reference for which type of key field RetroArch uses for each console system. - __CRC checksum__ for systems with smaller file sizes, i.e. game media before the advent of disc-based games. - __Serial Number__ found within the ROM file for larger files like disc-based games. Note the serial is not metadata but encoded within the game's binary data, which is scanned as a byte array by RetroArch. -Current [build script code](https://github.com/libretro/libretro-super/blob/master/libretro-build-database.sh#L288) can be used as a reference for which type of key field RetroArch uses for each console system. - -#### Game Data in Repository +#### Game Data Specified in Databases Database entries generally at minimum specify 1) a game's name, i.e. the display name that RetroArch will assign in playlists and 2) checksum/hash for identifying a particular file (or sometimes a serial number as a [key field](#key-field). Further metadata for each game is often sourced from multiple databases. - ## Repository Contents The repository contains game information databases that are compiled into `.rdb` files used by RetroArch, cheat code files, and management/admin files and scripts. The non-exhaustive list below serves as a guide to various folders/files in the repository. - [`cht`](cht) Cheat codes to various games, collected from any available source on the web including by manual contributions by users who haved used RetroArch's built-in [memory address/value search feature](https://docs.libretro.com/guides/cheat-codes/#retroarch-new-cheat-code-searching) to construct new cheat codes. Game-specific cheat files remain in plain text format, can be loaded by the user in RetroArch, and are not part of a compilation process like the game information databases. - [`cursors`](cursors) Provides methods in order to query the playlists -- [`dat`](dat) Customized DAT files maintained by the libretro team, including items that do/did not have contemporary documentation by upstream catalogging groups, e.g. "Virtual Console" variants of SNES games and monolithic non-generalized cores (Cave Story, Doom, Quake, etc). Also includes some bulk imports from upstream groups in order to establish precedence in the compilation, e.g. GameCube data from GameTDB. +- [`dat`](dat) Customized DAT files maintained by the libretro team, including items that do/did not have contemporary documentation by upstream catalogging groups, e.g. "Virtual Console" variants of SNES games and monolithic non-generalized cores (Cave Story, Doom, Quake, etc). Also includes some bulk imports from upstream groups in order to establish [precedence](#precedence) in the compilation, e.g. GameCube data from GameTDB. - [`metadat`](metadat) Various metadata and third-party DATs. Examples: - [`bbfc`](metadat/bbfc) British Board of Film Classification's ratings for age-appropriateness - [`elspa`](metadat/elspa) Age-appropriateness/content ratings from the Entertainment and Leisure Software Publishers Association aka the Association for UK Interactive Entertainment ("Ukie") @@ -45,7 +42,7 @@ The repository contains game information databases that are compiled into `.rdb` - [`scripts`](scripts) Various scripts that are used to maintain the database files #### Precedence -Databases earlier in the list have precedence over items later in the list. Definitions in `dat` will over-ride `metadat` in the final `.rdb` compile if any info conflicts for the same item. +Databases earlier in the list have precedence over items later in the list. E.g. definitions in `dat` will over-ride `metadat` in the final `.rdb` compile if any info conflicts for the same item. #### Pre-emptive Databases From 704ca79f98b42c35ac0fd97daaefc59eb706a252 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Tue, 14 Jan 2025 22:42:22 -0500 Subject: [PATCH 17/62] WIP Update README.md WIP --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 239191d60..7be8ca0f8 100644 --- a/README.md +++ b/README.md @@ -27,7 +27,7 @@ The repository contains game information databases that are compiled into `.rdb` - [`cht`](cht) Cheat codes to various games, collected from any available source on the web including by manual contributions by users who haved used RetroArch's built-in [memory address/value search feature](https://docs.libretro.com/guides/cheat-codes/#retroarch-new-cheat-code-searching) to construct new cheat codes. Game-specific cheat files remain in plain text format, can be loaded by the user in RetroArch, and are not part of a compilation process like the game information databases. - [`cursors`](cursors) Provides methods in order to query the playlists -- [`dat`](dat) Customized DAT files maintained by the libretro team, including items that do/did not have contemporary documentation by upstream catalogging groups, e.g. "Virtual Console" variants of SNES games and monolithic non-generalized cores (Cave Story, Doom, Quake, etc). Also includes some bulk imports from upstream groups in order to establish [precedence](#precedence) in the compilation, e.g. GameCube data from GameTDB. +- [`dat`](dat) Customized DAT files maintained by the libretro team, including items that do/did not have contemporary documentation by upstream catalogging groups (e.g. "Virtual Console" variants of SNES games), games for monolithic non-generalized cores (Cave Story, Doom, Quake, etc), and the beginning of an interal non-upstream [Playstation Minis database]([https://github.com/OctopusButtons/libretro-database/blob/master/dat/Sony%20-%20PlayStation%20Minis.dat](https://github.com/libretro/libretro-database/blob/master/dat/Sony%20-%20PlayStation%20Minis.dat)). Also includes some bulk imports from upstream groups in order to establish [precedence](#precedence) in the compilation, e.g. GameCube data from GameTDB. - [`metadat`](metadat) Various metadata and third-party DATs. Examples: - [`bbfc`](metadat/bbfc) British Board of Film Classification's ratings for age-appropriateness - [`elspa`](metadat/elspa) Age-appropriateness/content ratings from the Entertainment and Leisure Software Publishers Association aka the Association for UK Interactive Entertainment ("Ukie") From 6c09473c6d17df9b52c5dd547975f6afa3f8807d Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Tue, 14 Jan 2025 22:49:03 -0500 Subject: [PATCH 18/62] Update README.md WIP --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 7be8ca0f8..b50e2cfbe 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ Libretro databases allow RetroArch to provide several catalogging functions: - __Per-Game Information__. Provide an in-app viewable informational screen for each game (Game > Information > Database Entry) #### Key Field -The key field for matching varies by file size, i.e. by console media type. Current [build script code](https://github.com/libretro/libretro-super/blob/master/libretro-build-database.sh#L288) can be used as a reference for which type of key field RetroArch uses for each console system. +The key field for matching generally varies by console typical file size (i.e. media type). Current [build script code](https://github.com/libretro/libretro-super/blob/master/libretro-build-database.sh#L288) can be used as a reference for which type of key field RetroArch uses for each console system. - __CRC checksum__ for systems with smaller file sizes, i.e. game media before the advent of disc-based games. - __Serial Number__ found within the ROM file for larger files like disc-based games. Note the serial is not metadata but encoded within the game's binary data, which is scanned as a byte array by RetroArch. From 624b2797ab9f01a887af2f17af7ddff08ff5df6e Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Tue, 14 Jan 2025 22:50:08 -0500 Subject: [PATCH 19/62] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index b50e2cfbe..f991dedb9 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ The key field for matching generally varies by console typical file size (i.e. m - __Serial Number__ found within the ROM file for larger files like disc-based games. Note the serial is not metadata but encoded within the game's binary data, which is scanned as a byte array by RetroArch. #### Game Data Specified in Databases -Database entries generally at minimum specify 1) a game's name, i.e. the display name that RetroArch will assign in playlists and 2) checksum/hash for identifying a particular file (or sometimes a serial number as a [key field](#key-field). Further metadata for each game is often sourced from multiple databases. +Database entries generally at minimum specify 1) a game's name, i.e. the display name that RetroArch will assign in playlists and 2) checksum/hash for identifying a particular file (or sometimes a serial number as a [key field](#key-field)). Further metadata for each game is often sourced from multiple databases. ## Repository Contents From ee71cbe5d6f8a113a2af54e0391df1d7fc9d7778 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Tue, 14 Jan 2025 22:52:50 -0500 Subject: [PATCH 20/62] WIP Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f991dedb9..5820c433a 100644 --- a/README.md +++ b/README.md @@ -27,7 +27,7 @@ The repository contains game information databases that are compiled into `.rdb` - [`cht`](cht) Cheat codes to various games, collected from any available source on the web including by manual contributions by users who haved used RetroArch's built-in [memory address/value search feature](https://docs.libretro.com/guides/cheat-codes/#retroarch-new-cheat-code-searching) to construct new cheat codes. Game-specific cheat files remain in plain text format, can be loaded by the user in RetroArch, and are not part of a compilation process like the game information databases. - [`cursors`](cursors) Provides methods in order to query the playlists -- [`dat`](dat) Customized DAT files maintained by the libretro team, including items that do/did not have contemporary documentation by upstream catalogging groups (e.g. "Virtual Console" variants of SNES games), games for monolithic non-generalized cores (Cave Story, Doom, Quake, etc), and the beginning of an interal non-upstream [Playstation Minis database]([https://github.com/OctopusButtons/libretro-database/blob/master/dat/Sony%20-%20PlayStation%20Minis.dat](https://github.com/libretro/libretro-database/blob/master/dat/Sony%20-%20PlayStation%20Minis.dat)). Also includes some bulk imports from upstream groups in order to establish [precedence](#precedence) in the compilation, e.g. GameCube data from GameTDB. +- [`dat`](dat) Customized DAT files maintained by the libretro team, including items that do/did not have contemporary documentation by upstream database groups (e.g. "Virtual Console" variants of SNES games), games for monolithic non-generalized cores (Cave Story, Doom, Quake, etc). Also includes some bulk imports from upstream groups in order to establish [precedence](#precedence) in the compilation, e.g. GameCube data from GameTDB. - [`metadat`](metadat) Various metadata and third-party DATs. Examples: - [`bbfc`](metadat/bbfc) British Board of Film Classification's ratings for age-appropriateness - [`elspa`](metadat/elspa) Age-appropriateness/content ratings from the Entertainment and Leisure Software Publishers Association aka the Association for UK Interactive Entertainment ("Ukie") From cb3c051980fc69871a3b6d9a90b4310092368ee5 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Wed, 15 Jan 2025 11:14:16 -0500 Subject: [PATCH 21/62] WIP Update README.md --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 5820c433a..268f1c1a3 100644 --- a/README.md +++ b/README.md @@ -23,11 +23,11 @@ Database entries generally at minimum specify 1) a game's name, i.e. the display ## Repository Contents -The repository contains game information databases that are compiled into `.rdb` files used by RetroArch, cheat code files, and management/admin files and scripts. The non-exhaustive list below serves as a guide to various folders/files in the repository. +The repository contains several different kinds of files: game information databases that are compiled into `.rdb` files used by RetroArch, cheat code files, and management/admin files and scripts. The non-exhaustive list below serves as a guide to various folders/files in the repository. - [`cht`](cht) Cheat codes to various games, collected from any available source on the web including by manual contributions by users who haved used RetroArch's built-in [memory address/value search feature](https://docs.libretro.com/guides/cheat-codes/#retroarch-new-cheat-code-searching) to construct new cheat codes. Game-specific cheat files remain in plain text format, can be loaded by the user in RetroArch, and are not part of a compilation process like the game information databases. - [`cursors`](cursors) Provides methods in order to query the playlists -- [`dat`](dat) Customized DAT files maintained by the libretro team, including items that do/did not have contemporary documentation by upstream database groups (e.g. "Virtual Console" variants of SNES games), games for monolithic non-generalized cores (Cave Story, Doom, Quake, etc). Also includes some bulk imports from upstream groups in order to establish [precedence](#precedence) in the compilation, e.g. GameCube data from GameTDB. +- [`dat`](dat) Customized DAT files maintained by the libretro team, including items that do/did not have contemporary documentation by upstream database groups (e.g. "Virtual Console" variants of SNES games), games for monolithic non-generalized cores (Cave Story, Doom, Quake, etc). Also includes some imports from upstream groups in order to establish [precedence](#precedence) in the compilation, e.g. GameCube data from GameTDB, although most dats from upstream groups reside in [`metadat`](metadat). - [`metadat`](metadat) Various metadata and third-party DATs. Examples: - [`bbfc`](metadat/bbfc) British Board of Film Classification's ratings for age-appropriateness - [`elspa`](metadat/elspa) Age-appropriateness/content ratings from the Entertainment and Leisure Software Publishers Association aka the Association for UK Interactive Entertainment ("Ukie") From 59554cd14cd09a600abbb55b6bda69cabc61f5ab Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Wed, 15 Jan 2025 23:28:58 -0500 Subject: [PATCH 22/62] WIP README.md Contributions draft etc --- README.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 268f1c1a3..d61b685c6 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ Libretro databases allow RetroArch to provide several catalogging functions: The key field for matching generally varies by console typical file size (i.e. media type). Current [build script code](https://github.com/libretro/libretro-super/blob/master/libretro-build-database.sh#L288) can be used as a reference for which type of key field RetroArch uses for each console system. - __CRC checksum__ for systems with smaller file sizes, i.e. game media before the advent of disc-based games. -- __Serial Number__ found within the ROM file for larger files like disc-based games. Note the serial is not metadata but encoded within the game's binary data, which is scanned as a byte array by RetroArch. +- __Serial Number__ found within the ROM file for larger files like disc-based games. Note the serial is not metadata but encoded within the game's binary data, which is scanned (in applicable cases) as a byte array by RetroArch. #### Game Data Specified in Databases Database entries generally at minimum specify 1) a game's name, i.e. the display name that RetroArch will assign in playlists and 2) checksum/hash for identifying a particular file (or sometimes a serial number as a [key field](#key-field)). Further metadata for each game is often sourced from multiple databases. @@ -185,6 +185,8 @@ Many source databases are in use as listed below. A large majority of games com # Maintenance / Technical Usage +Note that the [build script](https://github.com/libretro/libretro-super/blob/master/libretro-build-database.sh) specifies exact `.dat` files and folders in the repository, therefore organizational revisions to the file/folder structure should have corresponding planning in the build script. + ## Building To build a complete set of RDB files for RetroArch or to generate a single RDB file, see [RetroArch/libretro-db/README.md](https://github.com/libretro/RetroArch/blob/master/libretro-db/README.md). @@ -203,6 +205,12 @@ Make sure filenames are Windows file system compatible, and are not too long (eg find -exec basename '{}' ';' | egrep '^.{144,}$' ``` +# Contributions + +A vast majority of the database's game information originates from routine imports from upstream data groups (No-Intro, Redump, TOSEC, GameTDB, etc). General best practice for corrections or additions is for a contributor to go through the systems and channels of the relevant groups, because a change in Libretro's copy of their database would be overwritten and lost with the next import from upstream.. Changes that they incorporate in their databases will eventually cycle through to the Libretro databases. + +In cases where the Libretro `.dat` in question is created and maintained by Libretro, github contributions are acceptable. Refer to the [repository contents list](#repository-contents) above and to Commit/Pull Request [histories](EDIT: LINK) for information about which libretro databases are applicable for github contributions. + # Integrations There are a few tools out there that allow integration with libretro's database. From 01d9ca261215384a02bfcbbbb64a907def5d2a44 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Thu, 16 Jan 2025 18:16:48 -0500 Subject: [PATCH 23/62] WIP Update README.md WIP --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index d61b685c6..8f4920e6e 100644 --- a/README.md +++ b/README.md @@ -23,9 +23,9 @@ Database entries generally at minimum specify 1) a game's name, i.e. the display ## Repository Contents -The repository contains several different kinds of files: game information databases that are compiled into `.rdb` files used by RetroArch, cheat code files, and management/admin files and scripts. The non-exhaustive list below serves as a guide to various folders/files in the repository. +The repository contains several different kinds of files: game information databases that are compiled into `.rdb` files used by RetroArch, cheat code files, and admin/management scripts and files. The non-exhaustive list below serves as a guide to various folders/files in the repository. -- [`cht`](cht) Cheat codes to various games, collected from any available source on the web including by manual contributions by users who haved used RetroArch's built-in [memory address/value search feature](https://docs.libretro.com/guides/cheat-codes/#retroarch-new-cheat-code-searching) to construct new cheat codes. Game-specific cheat files remain in plain text format, can be loaded by the user in RetroArch, and are not part of a compilation process like the game information databases. +- [`cht`](cht) Cheat codes to various games, collected from any available source on the web including by manual contributions by users who haved used RetroArch's built-in [memory address/value search feature](https://docs.libretro.com/guides/cheat-codes/#retroarch-new-cheat-code-searching) to construct new cheat codes. Cheat code files are game-specific, remain in plain text format when downloaded to and used RetroArch, and are not part of a compilation process like the game information databases are. - [`cursors`](cursors) Provides methods in order to query the playlists - [`dat`](dat) Customized DAT files maintained by the libretro team, including items that do/did not have contemporary documentation by upstream database groups (e.g. "Virtual Console" variants of SNES games), games for monolithic non-generalized cores (Cave Story, Doom, Quake, etc). Also includes some imports from upstream groups in order to establish [precedence](#precedence) in the compilation, e.g. GameCube data from GameTDB, although most dats from upstream groups reside in [`metadat`](metadat). - [`metadat`](metadat) Various metadata and third-party DATs. Examples: From fb068c56c7e183b8889a16099921b208c4a87bd0 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Thu, 16 Jan 2025 18:18:32 -0500 Subject: [PATCH 24/62] WIP Update README.md WIP --- README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 8f4920e6e..48efb5a99 100644 --- a/README.md +++ b/README.md @@ -185,8 +185,6 @@ Many source databases are in use as listed below. A large majority of games com # Maintenance / Technical Usage -Note that the [build script](https://github.com/libretro/libretro-super/blob/master/libretro-build-database.sh) specifies exact `.dat` files and folders in the repository, therefore organizational revisions to the file/folder structure should have corresponding planning in the build script. - ## Building To build a complete set of RDB files for RetroArch or to generate a single RDB file, see [RetroArch/libretro-db/README.md](https://github.com/libretro/RetroArch/blob/master/libretro-db/README.md). @@ -205,6 +203,9 @@ Make sure filenames are Windows file system compatible, and are not too long (eg find -exec basename '{}' ';' | egrep '^.{144,}$' ``` +Note that the [build script](https://github.com/libretro/libretro-super/blob/master/libretro-build-database.sh) specifies exact `.dat` files and folders in the repository, therefore organizational revisions to the file/folder structure should have corresponding revisions in the build script. + + # Contributions A vast majority of the database's game information originates from routine imports from upstream data groups (No-Intro, Redump, TOSEC, GameTDB, etc). General best practice for corrections or additions is for a contributor to go through the systems and channels of the relevant groups, because a change in Libretro's copy of their database would be overwritten and lost with the next import from upstream.. Changes that they incorporate in their databases will eventually cycle through to the Libretro databases. From 8ffb27bf44765e45d965ac4dc6e646ade0990d3e Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Thu, 16 Jan 2025 18:23:59 -0500 Subject: [PATCH 25/62] WIP Update README.md WIP --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 48efb5a99..39cef4de2 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ The key field for matching generally varies by console typical file size (i.e. m - __Serial Number__ found within the ROM file for larger files like disc-based games. Note the serial is not metadata but encoded within the game's binary data, which is scanned (in applicable cases) as a byte array by RetroArch. #### Game Data Specified in Databases -Database entries generally at minimum specify 1) a game's name, i.e. the display name that RetroArch will assign in playlists and 2) checksum/hash for identifying a particular file (or sometimes a serial number as a [key field](#key-field)). Further metadata for each game is often sourced from multiple databases. +Database entries generally at minimum specify 1) a game's name, i.e. the display name that RetroArch will assign in playlists and 2) [key field](#key-field) data for matching, which is usually a checksum/hash. Further metadata for each game is often sourced from multiple databases. ## Repository Contents From 4bdcb44b2dc29f3fbe74dab3af1ca589b1eddc45 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Thu, 16 Jan 2025 22:08:30 -0500 Subject: [PATCH 26/62] WIP Update README.md WIP --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 39cef4de2..df49927c7 100644 --- a/README.md +++ b/README.md @@ -208,9 +208,9 @@ Note that the [build script](https://github.com/libretro/libretro-super/blob/mas # Contributions -A vast majority of the database's game information originates from routine imports from upstream data groups (No-Intro, Redump, TOSEC, GameTDB, etc). General best practice for corrections or additions is for a contributor to go through the systems and channels of the relevant groups, because a change in Libretro's copy of their database would be overwritten and lost with the next import from upstream.. Changes that they incorporate in their databases will eventually cycle through to the Libretro databases. +A vast majority of the database's game information originates from routine imports from upstream data groups (No-Intro, Redump, TOSEC, GameTDB, etc). General best practice for corrections or additions is for a contributor to go through the channels/process of the relevant upstream group, because a "fix" to Libretro's copy of their database would be overwritten and lost with the next import from upstream. Upstream changes made by the database groups will eventually be imported to the Libretro databases. -In cases where the Libretro `.dat` in question is created and maintained by Libretro, github contributions are acceptable. Refer to the [repository contents list](#repository-contents) above and to Commit/Pull Request [histories](EDIT: LINK) for information about which libretro databases are applicable for github contributions. +In cases where the Libretro `.dat` in question is created and maintained by Libretro, github contributions are acceptable. Refer to the [repository contents list](#repository-contents) above and to github Histories for information about which libretro databases are applicable for github contributions. # Integrations From b5213201610bf7c699e812fe7d4d0dc713afc693 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Thu, 16 Jan 2025 23:19:19 -0500 Subject: [PATCH 27/62] WIP Update README.md New explicit note about how thumbnail names (and therefore thumbnail assignment) aren't synced to database name changes. I'm partly mentioning for people investigating database/thumbnail troubles, but also to maybe get the interest of a volunteer someday. --- README.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index df49927c7..a593649b9 100644 --- a/README.md +++ b/README.md @@ -208,9 +208,13 @@ Note that the [build script](https://github.com/libretro/libretro-super/blob/mas # Contributions -A vast majority of the database's game information originates from routine imports from upstream data groups (No-Intro, Redump, TOSEC, GameTDB, etc). General best practice for corrections or additions is for a contributor to go through the channels/process of the relevant upstream group, because a "fix" to Libretro's copy of their database would be overwritten and lost with the next import from upstream. Upstream changes made by the database groups will eventually be imported to the Libretro databases. +A vast majority of the database's game information originates from routine imports from upstream data groups (No-Intro, Redump, TOSEC, GameTDB, etc). General best practice for corrections or additions is for a contributor to go through the channels/process of the relevant upstream group. Upstream changes made by the database groups will eventually be imported to the Libretro databases. A "fix" to Libretro's copy of the database would be overwritten and lost with the next import from upstream. -In cases where the Libretro `.dat` in question is created and maintained by Libretro, github contributions are acceptable. Refer to the [repository contents list](#repository-contents) above and to github Histories for information about which libretro databases are applicable for github contributions. +In cases where the `.dat` in question is created and maintained by Libretro, github contributions are acceptable. Refer to the [repository contents list](#repository-contents) above and to github Histories for information about which libretro databases are applicable for github contributions. + +# Databases and RetroArch Thumbnails + +Currently there is no automatic process for syncing libretro [thumbnail repository](https://github.com/libretro-thumbnails/libretro-thumbnails#libretro-thumbnails) image filenames to database game name updates. Databases can assign a game name (aka playlist item name or displayed game title) based on game file's checksum, but thumbnails are only assigned based on name-matching (game name or the ROM filename, with some [flexibility](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails). # Integrations From 4efc669096f36fb3f74b8bfb2cc70849b4c095a1 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Thu, 16 Jan 2025 23:26:15 -0500 Subject: [PATCH 28/62] WIP Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index a593649b9..6d7a1406a 100644 --- a/README.md +++ b/README.md @@ -214,7 +214,7 @@ In cases where the `.dat` in question is created and maintained by Libretro, git # Databases and RetroArch Thumbnails -Currently there is no automatic process for syncing libretro [thumbnail repository](https://github.com/libretro-thumbnails/libretro-thumbnails#libretro-thumbnails) image filenames to database game name updates. Databases can assign a game name (aka playlist item name or displayed game title) based on game file's checksum, but thumbnails are only assigned based on name-matching (game name or the ROM filename, with some [flexibility](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails). +Currently there is no automatic process for updating libretro [thumbnail repository](https://github.com/libretro-thumbnails/libretro-thumbnails#libretro-thumbnails) image filenames based on game name updates in databases. Databases assign a game name (aka playlist item name or displayed game title) based on a game file's checksum, but thumbnails are only assigned if the thumbnail server image file matches the game name or the ROM filename (with some [flexibility](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails)). # Integrations From 35374bf2573dae7a575fdc144a37ada9fb5c3c9b Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Thu, 16 Jan 2025 23:37:17 -0500 Subject: [PATCH 29/62] WIP Update README.md WIP --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 6d7a1406a..01cb86bca 100644 --- a/README.md +++ b/README.md @@ -7,8 +7,8 @@ The github repository for databases used by RetroArch. Libretro databases allow RetroArch to provide several catalogging functions: - __Validation__. Reject or accept files when using the [Import Scanner / Playlist Generator](https://docs.libretro.com/guides/roms-playlists-thumbnails/#working-with-playlists) based on whether the ROM checksum matches the checksum of a known verified completely intact (aka "properly dumped") file. -- __Game Naming__. Assign a definitive and uniform display name for each game file regardless of filename. -- __Thumbnail Images__. Download and display thumbnail images for each game based on the uniform name assigned by the database, regardless of filename +- __Game Naming__. Assign a definitive and uniform display name for each game in a playlist regardless of filename. +- __Thumbnail Images__. Download and display thumbnail images for each game based on the uniform name assigned by the database, regardless of filename. (Thumbnail assignment [note edit]] This can be considered a secondary function of Game Naming rather than the database itself. Note that thumbnails are not assigned by databases or by checksum.) - __Category Search__. A reference search (named "Explore") that allows the user to search for games by category criteria, e.g. by Developer, Release Year, Genre, and other attributes/metadata. - __Per-Game Information__. Provide an in-app viewable informational screen for each game (Game > Information > Database Entry) From 7ef2af7bbc8ae0d6fab88a009d2dc532c9a9431d Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Fri, 17 Jan 2025 22:25:00 -0500 Subject: [PATCH 30/62] WIP Update README.md --- README.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 01cb86bca..3fafd05d8 100644 --- a/README.md +++ b/README.md @@ -4,22 +4,24 @@ The github repository for databases used by RetroArch. ## Overview +#### RetroArch's Usage of the Database + Libretro databases allow RetroArch to provide several catalogging functions: - __Validation__. Reject or accept files when using the [Import Scanner / Playlist Generator](https://docs.libretro.com/guides/roms-playlists-thumbnails/#working-with-playlists) based on whether the ROM checksum matches the checksum of a known verified completely intact (aka "properly dumped") file. - __Game Naming__. Assign a definitive and uniform display name for each game in a playlist regardless of filename. -- __Thumbnail Images__. Download and display thumbnail images for each game based on the uniform name assigned by the database, regardless of filename. (Thumbnail assignment [note edit]] This can be considered a secondary function of Game Naming rather than the database itself. Note that thumbnails are not assigned by databases or by checksum.) +- __Thumbnail Images__. Download and display thumbnail images for games based on the uniform name assigned by the database, regardless of filename. (Note: thumbnails are not directly assigned by the database or by checksum association, but as a secondary effect of databased *game name* assignment if a matching thumbnail is available on the server.) - __Category Search__. A reference search (named "Explore") that allows the user to search for games by category criteria, e.g. by Developer, Release Year, Genre, and other attributes/metadata. -- __Per-Game Information__. Provide an in-app viewable informational screen for each game (Game > Information > Database Entry) +- __Per-Game Information View__. Provide an in-app viewable informational screen for each game (Game > Information > Database Entry) #### Key Field The key field for matching generally varies by console typical file size (i.e. media type). Current [build script code](https://github.com/libretro/libretro-super/blob/master/libretro-build-database.sh#L288) can be used as a reference for which type of key field RetroArch uses for each console system. - __CRC checksum__ for systems with smaller file sizes, i.e. game media before the advent of disc-based games. -- __Serial Number__ found within the ROM file for larger files like disc-based games. Note the serial is not metadata but encoded within the game's binary data, which is scanned (in applicable cases) as a byte array by RetroArch. +- __Serial Number__ found within the ROM file, for larger files like disc-based games, to avoid computing checksums on large files. The serial is not metadata but encoded within the game's binary data, which is scanned (in applicable cases) as a byte array by RetroArch. #### Game Data Specified in Databases -Database entries generally at minimum specify 1) a game's name, i.e. the display name that RetroArch will assign in playlists and 2) [key field](#key-field) data for matching, which is usually a checksum/hash. Further metadata for each game is often sourced from multiple databases. +Database entries generally at minimum specify 1) a game's name, i.e. the display name that RetroArch will assign in playlists and 2) [key field](#key-field) data for matching, which is usually a checksum/hash. Further metadata for each game is often sourced from multiple databases. Databases often contain checksum/hashes for complete informational purposes even for cases where the [key field](#key-field) for matching is the game's internal serial number. ## Repository Contents From e150f26c8e11b9de9e130d159dc1afbd84c9b9ea Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Fri, 17 Jan 2025 23:21:57 -0500 Subject: [PATCH 31/62] WIP Update README.md --- README.md | 37 +++++++++++++++++++++++++------------ 1 file changed, 25 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index 3fafd05d8..1a15f3cda 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,10 @@ # Libretro Database -The github repository for databases used by RetroArch. +The github repository for databases used by RetroArch. The repository contains several different kinds of files: game information databases that are compiled into `.rdb` files used by RetroArch, cheat code files, and admin/management scripts and files. ## Overview -#### RetroArch's Usage of the Database +### RetroArch's Usage of the Database Libretro databases allow RetroArch to provide several catalogging functions: @@ -14,27 +14,40 @@ Libretro databases allow RetroArch to provide several catalogging functions: - __Category Search__. A reference search (named "Explore") that allows the user to search for games by category criteria, e.g. by Developer, Release Year, Genre, and other attributes/metadata. - __Per-Game Information View__. Provide an in-app viewable informational screen for each game (Game > Information > Database Entry) -#### Key Field -The key field for matching generally varies by console typical file size (i.e. media type). Current [build script code](https://github.com/libretro/libretro-super/blob/master/libretro-build-database.sh#L288) can be used as a reference for which type of key field RetroArch uses for each console system. +### Key Field +The key field for matching generally varies by console typical file size (i.e. media type). - __CRC checksum__ for systems with smaller file sizes, i.e. game media before the advent of disc-based games. - __Serial Number__ found within the ROM file, for larger files like disc-based games, to avoid computing checksums on large files. The serial is not metadata but encoded within the game's binary data, which is scanned (in applicable cases) as a byte array by RetroArch. -#### Game Data Specified in Databases -Database entries generally at minimum specify 1) a game's name, i.e. the display name that RetroArch will assign in playlists and 2) [key field](#key-field) data for matching, which is usually a checksum/hash. Further metadata for each game is often sourced from multiple databases. Databases often contain checksum/hashes for complete informational purposes even for cases where the [key field](#key-field) for matching is the game's internal serial number. +Current [build script code](https://github.com/libretro/libretro-super/blob/master/libretro-build-database.sh#L288) can be viewed as a reference for which type of key field RetroArch uses for each console system. + +### Fields Specified in Game Information Databases + +Database entries generally at minimum specify 1) a game's name, i.e. the display name that RetroArch will assign in playlists and 2) [key field](#key-field) data for matching. Further metadata for each game is often sourced from multiple databases. Databases often contain checksum/hashes for informational completeness even in cases where the [key field](#key-field) for matching is the game's internal serial number. + +Example of database entry within [`metadat/no-intro/Atari - 2600.dat`](https://github.com/libretro/libretro-database/blob/master/metadat/no-intro/Atari%20-%202600.dat) for the European region version of _Asteroids_: +``` +game ( + name "Asteroids (Europe)" + description "Asteroids (Europe)" + region "Europe" + rom ( name "Asteroids (Europe).a26" size 8192 crc 0A2F8288 md5 8CF0D333BBE85B9549B1E6B1E2390B8D sha1 1CB8F057ACAD6DC65FEF07D3202088FF4AE355CD ) +) +``` ## Repository Contents -The repository contains several different kinds of files: game information databases that are compiled into `.rdb` files used by RetroArch, cheat code files, and admin/management scripts and files. The non-exhaustive list below serves as a guide to various folders/files in the repository. +The non-exhaustive list below serves as a guide to various folders/files in the repository. - [`cht`](cht) Cheat codes to various games, collected from any available source on the web including by manual contributions by users who haved used RetroArch's built-in [memory address/value search feature](https://docs.libretro.com/guides/cheat-codes/#retroarch-new-cheat-code-searching) to construct new cheat codes. Cheat code files are game-specific, remain in plain text format when downloaded to and used RetroArch, and are not part of a compilation process like the game information databases are. -- [`cursors`](cursors) Provides methods in order to query the playlists +- [`cursors`](cursors) Methods to query playlists. - [`dat`](dat) Customized DAT files maintained by the libretro team, including items that do/did not have contemporary documentation by upstream database groups (e.g. "Virtual Console" variants of SNES games), games for monolithic non-generalized cores (Cave Story, Doom, Quake, etc). Also includes some imports from upstream groups in order to establish [precedence](#precedence) in the compilation, e.g. GameCube data from GameTDB, although most dats from upstream groups reside in [`metadat`](metadat). -- [`metadat`](metadat) Various metadata and third-party DATs. Examples: - - [`bbfc`](metadat/bbfc) British Board of Film Classification's ratings for age-appropriateness - - [`elspa`](metadat/elspa) Age-appropriateness/content ratings from the Entertainment and Leisure Software Publishers Association aka the Association for UK Interactive Entertainment ("Ukie") +- [`metadat`](metadat) Several principal third-party DATs (e.g. No-Intro, Redump, and TOSEC), plus various collections of metadata. Examples: + - [`bbfc`](metadat/bbfc) British Board of Film Classification's ratings for age-appropriateness. + - [`elspa`](metadat/elspa) Age-appropriateness/content ratings from the Entertainment and Leisure Software Publishers Association aka the Association for UK Interactive Entertainment ("Ukie"). - [`hacks`](metadat/hacks) Data for modified (or "hacked") versions of commercially released games. Many of these data are set by direct manual commits on the Libretro Github. - - [`homebrew`](metadat/homebrew) Data for non-officially-published games created by independent creators/programmers + - [`homebrew`](metadat/homebrew) Data for non-officially-published games created by independent creators/programmers. - [`libretro-dats`](metadat/libretro-dats) Ad hoc databases for items that were/are not covered by upstream database groups. Currently only fan translations of SNES games, and an FDS dat that includes Virtual Console variants. - [`no-intro`](metadat/no-intro) Bulk import from upstream No-Intro databases. Generally non-disc-based systems. - [`redump`](metadat/redump) Bulk import from upstream Redump databases. Generally disc-based systems. From da8a02d1f10d586005314c39e3d024621a0d585d Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Fri, 17 Jan 2025 23:36:14 -0500 Subject: [PATCH 32/62] WIP Update README.md --- README.md | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 1a15f3cda..58d5bd6bc 100644 --- a/README.md +++ b/README.md @@ -1,16 +1,22 @@ # Libretro Database -The github repository for databases used by RetroArch. The repository contains several different kinds of files: game information databases that are compiled into `.rdb` files used by RetroArch, cheat code files, and admin/management scripts and files. +The github repository for databases used by RetroArch. ## Overview +The repository contains several different kinds of files: + +- Game information databases that are compiled into `.rdb` files used by RetroArch +- Cheat code files that are game-specific, remain in plain text format when downloaded to and used in RetroArch, and are not part of a compilation process like the game information databases are +- Admin/management scripts and files + ### RetroArch's Usage of the Database Libretro databases allow RetroArch to provide several catalogging functions: - __Validation__. Reject or accept files when using the [Import Scanner / Playlist Generator](https://docs.libretro.com/guides/roms-playlists-thumbnails/#working-with-playlists) based on whether the ROM checksum matches the checksum of a known verified completely intact (aka "properly dumped") file. - __Game Naming__. Assign a definitive and uniform display name for each game in a playlist regardless of filename. -- __Thumbnail Images__. Download and display thumbnail images for games based on the uniform name assigned by the database, regardless of filename. (Note: thumbnails are not directly assigned by the database or by checksum association, but as a secondary effect of databased *game name* assignment if a matching thumbnail is available on the server.) +- __Thumbnail Images__. Download and display thumbnail images for games based on the uniform name assigned by the database, regardless of filename. (Note: thumbnails are not directly assigned by the database or by checksum association, but as a secondary effect of databased *game name* assignment if a matching thumbnail is available on the server. Also note some flexibility in the matching algorithm [link edit]]) - __Category Search__. A reference search (named "Explore") that allows the user to search for games by category criteria, e.g. by Developer, Release Year, Genre, and other attributes/metadata. - __Per-Game Information View__. Provide an in-app viewable informational screen for each game (Game > Information > Database Entry) @@ -22,6 +28,9 @@ The key field for matching generally varies by console typical file size (i.e. m Current [build script code](https://github.com/libretro/libretro-super/blob/master/libretro-build-database.sh#L288) can be viewed as a reference for which type of key field RetroArch uses for each console system. +### Precedence +Databases earlier in the list have precedence over items later in the list. E.g. definitions in `dat` will over-ride `metadat` in the final `.rdb` compile if any info conflicts for the same item. + ### Fields Specified in Game Information Databases Database entries generally at minimum specify 1) a game's name, i.e. the display name that RetroArch will assign in playlists and 2) [key field](#key-field) data for matching. Further metadata for each game is often sourced from multiple databases. Databases often contain checksum/hashes for informational completeness even in cases where the [key field](#key-field) for matching is the game's internal serial number. @@ -40,7 +49,7 @@ game ( The non-exhaustive list below serves as a guide to various folders/files in the repository. -- [`cht`](cht) Cheat codes to various games, collected from any available source on the web including by manual contributions by users who haved used RetroArch's built-in [memory address/value search feature](https://docs.libretro.com/guides/cheat-codes/#retroarch-new-cheat-code-searching) to construct new cheat codes. Cheat code files are game-specific, remain in plain text format when downloaded to and used RetroArch, and are not part of a compilation process like the game information databases are. +- [`cht`](cht) Cheat codes to various games, collected from any available source on the web including by manual contributions by users who haved used RetroArch's built-in [memory address/value search feature](https://docs.libretro.com/guides/cheat-codes/#retroarch-new-cheat-code-searching) to construct new cheat codes. - [`cursors`](cursors) Methods to query playlists. - [`dat`](dat) Customized DAT files maintained by the libretro team, including items that do/did not have contemporary documentation by upstream database groups (e.g. "Virtual Console" variants of SNES games), games for monolithic non-generalized cores (Cave Story, Doom, Quake, etc). Also includes some imports from upstream groups in order to establish [precedence](#precedence) in the compilation, e.g. GameCube data from GameTDB, although most dats from upstream groups reside in [`metadat`](metadat). - [`metadat`](metadat) Several principal third-party DATs (e.g. No-Intro, Redump, and TOSEC), plus various collections of metadata. Examples: @@ -56,9 +65,6 @@ The non-exhaustive list below serves as a guide to various folders/files in the - [`rdb`](rdb) The compiled RetroArch database files - [`scripts`](scripts) Various scripts that are used to maintain the database files -#### Precedence -Databases earlier in the list have precedence over items later in the list. E.g. definitions in `dat` will over-ride `metadat` in the final `.rdb` compile if any info conflicts for the same item. - #### Pre-emptive Databases Some databases are maintained even if RetroArch currently has no core for the games/system, e.g. GP32, Vita, Original Xbox, and PS3. From 6f4c3a8fb1feaa6d18cd087b7aa14353e7251c87 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Sat, 18 Jan 2025 15:47:04 -0500 Subject: [PATCH 33/62] WIP Update README.md WIP --- README.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 58d5bd6bc..1d4076744 100644 --- a/README.md +++ b/README.md @@ -6,17 +6,17 @@ The github repository for databases used by RetroArch. The repository contains several different kinds of files: -- Game information databases that are compiled into `.rdb` files used by RetroArch -- Cheat code files that are game-specific, remain in plain text format when downloaded to and used in RetroArch, and are not part of a compilation process like the game information databases are -- Admin/management scripts and files +- __Game information databases__ that are compiled into `.rdb` files used by RetroArch +- __Cheat code files__ that are game-specific, remain in plain text format when downloaded to and used in RetroArch, and are not part of a compilation process like the game information databases are +- __Admin/management__ scripts and files ### RetroArch's Usage of the Database -Libretro databases allow RetroArch to provide several catalogging functions: +Libretro databases allow RetroArch to provide several automated catalogging functions: - __Validation__. Reject or accept files when using the [Import Scanner / Playlist Generator](https://docs.libretro.com/guides/roms-playlists-thumbnails/#working-with-playlists) based on whether the ROM checksum matches the checksum of a known verified completely intact (aka "properly dumped") file. - __Game Naming__. Assign a definitive and uniform display name for each game in a playlist regardless of filename. -- __Thumbnail Images__. Download and display thumbnail images for games based on the uniform name assigned by the database, regardless of filename. (Note: thumbnails are not directly assigned by the database or by checksum association, but as a secondary effect of databased *game name* assignment if a matching thumbnail is available on the server. Also note some flexibility in the matching algorithm [link edit]]) +- __Thumbnail Images__. Download and display thumbnail images for games based on the uniform name assigned by the database, regardless of filename. (Note: thumbnails are not directly assigned by the database or by checksum association, but as a secondary effect of databased *game name* assignment if a matching thumbnail is available on the server. Also see: [Flexible Name Matching Algorithm](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails)) - __Category Search__. A reference search (named "Explore") that allows the user to search for games by category criteria, e.g. by Developer, Release Year, Genre, and other attributes/metadata. - __Per-Game Information View__. Provide an in-app viewable informational screen for each game (Game > Information > Database Entry) @@ -33,7 +33,7 @@ Databases earlier in the list have precedence over items later in the list. E.g ### Fields Specified in Game Information Databases -Database entries generally at minimum specify 1) a game's name, i.e. the display name that RetroArch will assign in playlists and 2) [key field](#key-field) data for matching. Further metadata for each game is often sourced from multiple databases. Databases often contain checksum/hashes for informational completeness even in cases where the [key field](#key-field) for matching is the game's internal serial number. +Database entries generally at minimum specify 1) a game's name, i.e. the display name that RetroArch will assign in playlists and 2) [key field](#key-field) data for matching. Further metadata for each game is often sourced from multiple databases. Databases often contain checksum/hashes for informational completeness even in cases where the [key field](#key-field) for matching is the game's internal serial number rather than checksum. Example of database entry within [`metadat/no-intro/Atari - 2600.dat`](https://github.com/libretro/libretro-database/blob/master/metadat/no-intro/Atari%20-%202600.dat) for the European region version of _Asteroids_: ``` @@ -49,15 +49,15 @@ game ( The non-exhaustive list below serves as a guide to various folders/files in the repository. -- [`cht`](cht) Cheat codes to various games, collected from any available source on the web including by manual contributions by users who haved used RetroArch's built-in [memory address/value search feature](https://docs.libretro.com/guides/cheat-codes/#retroarch-new-cheat-code-searching) to construct new cheat codes. +- [`cht`](cht) Cheat codes to various games, collected from any available source on the web including by manual contributions from users who haved used RetroArch's built-in [memory address/value search feature](https://docs.libretro.com/guides/cheat-codes/#retroarch-new-cheat-code-searching) to construct new cheat codes. - [`cursors`](cursors) Methods to query playlists. -- [`dat`](dat) Customized DAT files maintained by the libretro team, including items that do/did not have contemporary documentation by upstream database groups (e.g. "Virtual Console" variants of SNES games), games for monolithic non-generalized cores (Cave Story, Doom, Quake, etc). Also includes some imports from upstream groups in order to establish [precedence](#precedence) in the compilation, e.g. GameCube data from GameTDB, although most dats from upstream groups reside in [`metadat`](metadat). -- [`metadat`](metadat) Several principal third-party DATs (e.g. No-Intro, Redump, and TOSEC), plus various collections of metadata. Examples: +- [`dat`](dat) Mostly customized DAT files maintained by the libretro team, including items that do/did not have contemporary documentation by upstream database groups (e.g. "Virtual Console" variants of SNES games), and games for monolithic non-generalized cores (Cave Story, Doom, Quake, etc). Also includes some imports from upstream groups in order to establish [precedence](#precedence) in the compilation, e.g. GameCube data from GameTDB, although most dats from upstream groups reside in [`metadat`](metadat). +- [`metadat`](metadat) Several principal third-party DATs (e.g. No-Intro, Redump, and TOSEC), plus various collections of metadata (Note: some may be deprecated or incomplete). Examples: - [`bbfc`](metadat/bbfc) British Board of Film Classification's ratings for age-appropriateness. - [`elspa`](metadat/elspa) Age-appropriateness/content ratings from the Entertainment and Leisure Software Publishers Association aka the Association for UK Interactive Entertainment ("Ukie"). - [`hacks`](metadat/hacks) Data for modified (or "hacked") versions of commercially released games. Many of these data are set by direct manual commits on the Libretro Github. - [`homebrew`](metadat/homebrew) Data for non-officially-published games created by independent creators/programmers. - - [`libretro-dats`](metadat/libretro-dats) Ad hoc databases for items that were/are not covered by upstream database groups. Currently only fan translations of SNES games, and an FDS dat that includes Virtual Console variants. + - [`libretro-dats`](metadat/libretro-dats) Ad hoc databases for items that were/are not covered by upstream database groups. Currently includes fan translations of SNES games, and an additional FDS dat that mostly overlaps with other sources/dats and may be redundant. - [`no-intro`](metadat/no-intro) Bulk import from upstream No-Intro databases. Generally non-disc-based systems. - [`redump`](metadat/redump) Bulk import from upstream Redump databases. Generally disc-based systems. - [`tosec`](metadat/tosec) Bulk import from upstream TOSEC databases. @@ -71,7 +71,7 @@ Some databases are maintained even if RetroArch currently has no core for the ga ## Sources -Many source databases are in use as listed below. A large majority of games commonly used in RetroArch are covered by [No-Intro](http://datomatic.no-intro.org) or [Redump](http://redump.org/downloads/) DAT files. ">" signs below indicate the [precedence](#precedence) order when multiple sources overlap for the same subset of games/data. +Many source databases are in use as listed below. The table focusses on the 3rd party sources that predominantly cover each specific console library, but other/multiple sources including manual github contributions are maintained and all are compiled together in the final `.rdb` files (see [Repository Contents](#repository-contents) and github History for each dat for details). ">" signs below indicate the [precedence](#precedence) order when multiple sources overlap for the same subset of games/data. |System|Source|Repository| |----|---|---| @@ -229,13 +229,13 @@ Note that the [build script](https://github.com/libretro/libretro-super/blob/mas # Contributions -A vast majority of the database's game information originates from routine imports from upstream data groups (No-Intro, Redump, TOSEC, GameTDB, etc). General best practice for corrections or additions is for a contributor to go through the channels/process of the relevant upstream group. Upstream changes made by the database groups will eventually be imported to the Libretro databases. A "fix" to Libretro's copy of the database would be overwritten and lost with the next import from upstream. +A vast majority of the database's game information originates from routine imports from upstream data groups (No-Intro, Redump, TOSEC, GameTDB, etc). General best practice for corrections or additions is for a contributor to go through the channels/process of the relevant upstream group. Upstream changes made by the database groups will eventually be imported to the Libretro databases. A "fix" to Libretro's copy of the database would be overwritten and lost with the next import from upstream. -In cases where the `.dat` in question is created and maintained by Libretro, github contributions are acceptable. Refer to the [repository contents list](#repository-contents) above and to github Histories for information about which libretro databases are applicable for github contributions. +In cases where the `.dat` in question is created and maintained by Libretro or does not receive bulk over-writes, github contributions are accepted. Refer to the [repository contents list](#repository-contents) above and to github Histories for information about which libretro databases are applicable for github contributions. # Databases and RetroArch Thumbnails -Currently there is no automatic process for updating libretro [thumbnail repository](https://github.com/libretro-thumbnails/libretro-thumbnails#libretro-thumbnails) image filenames based on game name updates in databases. Databases assign a game name (aka playlist item name or displayed game title) based on a game file's checksum, but thumbnails are only assigned if the thumbnail server image file matches the game name or the ROM filename (with some [flexibility](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails)). +Currently there is no automatic process for updating libretro [thumbnail repository](https://github.com/libretro-thumbnails/libretro-thumbnails#libretro-thumbnails) image filenames based on game name updates in databases. Databases assign a game name (aka playlist item name or displayed game title) based on a game file's checksum, but thumbnails are only assigned if the thumbnail server image filename matches the game name or the ROM filename (with some [flexibility](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails)). To help fix a thumbnail, for example in a case where a database game name has been definitively/correctly updated in a way that no longer matches the repository thumbnail name, follow the [Thumbnail Repository readme](https://docs.libretro.com/guides/roms-playlists-thumbnails/#contributing-thumbnails-how-to) and [How-To documentation](https://docs.libretro.com/guides/roms-playlists-thumbnails/#contributing-thumbnails-how-to). # Integrations From 1d6492fa8d8f28748f6376efb65b7292a3c16e83 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Sat, 18 Jan 2025 17:21:40 -0500 Subject: [PATCH 34/62] WIP Update README.md WIP improving breakdown in intro of what the contained files are --- README.md | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 1d4076744..1ad7a981b 100644 --- a/README.md +++ b/README.md @@ -4,11 +4,13 @@ The github repository for databases used by RetroArch. ## Overview -The repository contains several different kinds of files: +The database repository contains several different kinds of files: -- __Game information databases__ that are compiled into `.rdb` files used by RetroArch -- __Cheat code files__ that are game-specific, remain in plain text format when downloaded to and used in RetroArch, and are not part of a compilation process like the game information databases are -- __Admin/management__ scripts and files +- __Game information database files__. + - `.dat` constituent files from many (sometimes-overlapping) sources and across many categories of metadata. + - `.rdb` files compiled and amalgmated from the `.dat` files, one per system, used by RetroArch. [RetroArch Database format](https://github.com/libretro/RetroArch/tree/68b3e5d8e02aff753e01a1f6f8969891910b2e0b/libretro-db#readme) (_no relation to Redis .RDB files_) accomodates RetroArch's [wide range of hardware/OS compatibility](https://www.retroarch.com/index.php?page=platforms). +- __Cheat code `.cht` files__. These are game-specific, remain in plain text, and are used as-is by RetroArch. The repository contains one unitary folder per system, unlike `.dat` files there is no amalgamation process. +- __Admin/management__ scripts and files. ### RetroArch's Usage of the Database @@ -29,7 +31,7 @@ The key field for matching generally varies by console typical file size (i.e. m Current [build script code](https://github.com/libretro/libretro-super/blob/master/libretro-build-database.sh#L288) can be viewed as a reference for which type of key field RetroArch uses for each console system. ### Precedence -Databases earlier in the list have precedence over items later in the list. E.g. definitions in `dat` will over-ride `metadat` in the final `.rdb` compile if any info conflicts for the same item. +Databases earlier in the list have precedence over items later in the list. E.g. definitions in `dat` will over-ride `metadat` in the final `.rdb` compile if any info conflicts for the same item (i.e. for the same key field). ### Fields Specified in Game Information Databases @@ -206,7 +208,7 @@ Many source databases are in use as listed below. The table focusses on the 3rd # Maintenance / Technical Usage -## Building +### Building To build a complete set of RDB files for RetroArch or to generate a single RDB file, see [RetroArch/libretro-db/README.md](https://github.com/libretro/RetroArch/blob/master/libretro-db/README.md). @@ -216,15 +218,22 @@ Alternatively, you can run the following command to rebuild all the RDBs locally make build ``` -## Testing +### Testing Make sure filenames are Windows file system compatible, and are not too long (eg. [ecryptfs limits filenames to 143 characters](https://unix.stackexchange.com/questions/32795/what-is-the-maximum-allowed-filename-and-folder-size-with-ecryptfs/32834#32834))... ``` find -exec basename '{}' ';' | egrep '^.{144,}$' ``` +### Other -Note that the [build script](https://github.com/libretro/libretro-super/blob/master/libretro-build-database.sh) specifies exact `.dat` files and folders in the repository, therefore organizational revisions to the file/folder structure should have corresponding revisions in the build script. +**Folder structure revisions.** The [build script](https://github.com/libretro/libretro-super/blob/master/libretro-build-database.sh) specifies exact `.dat` files and folders in the repository, therefore organizational revisions to the file/folder structure should have corresponding revisions in the build script. + +__Adding New Database: Example__ +- Creat new `metadata/nameofnewdatabasefolder` and appropriately named `.dat` files e.g. `Sony - PlayStation.dat` +- Add the new entry to `libretro-build-database.sh` +- Run ``make build`` to build the RDB files +- New types for RetroArch's `Explore` tab will need some updates to RetroArch. # Contributions From ccf5056b439aa26a3f168d69f956b720cc7885e6 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Sat, 18 Jan 2025 17:29:01 -0500 Subject: [PATCH 35/62] WIP Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 1ad7a981b..3f009cf24 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ The database repository contains several different kinds of files: - __Game information database files__. - `.dat` constituent files from many (sometimes-overlapping) sources and across many categories of metadata. - - `.rdb` files compiled and amalgmated from the `.dat` files, one per system, used by RetroArch. [RetroArch Database format](https://github.com/libretro/RetroArch/tree/68b3e5d8e02aff753e01a1f6f8969891910b2e0b/libretro-db#readme) (_no relation to Redis .RDB files_) accomodates RetroArch's [wide range of hardware/OS compatibility](https://www.retroarch.com/index.php?page=platforms). + - `.rdb` files used by RetroArch, compiled and amalgmated from the `.dat` files. [RetroArch Database format](https://github.com/libretro/RetroArch/tree/68b3e5d8e02aff753e01a1f6f8969891910b2e0b/libretro-db#readme) (_no relation to Redis .RDB files_) accomodates RetroArch's [wide range of hardware/OS compatibility](https://www.retroarch.com/index.php?page=platforms). - __Cheat code `.cht` files__. These are game-specific, remain in plain text, and are used as-is by RetroArch. The repository contains one unitary folder per system, unlike `.dat` files there is no amalgamation process. - __Admin/management__ scripts and files. From d4667d44cfd4a36e191fa4aeb34bf0cb22704e47 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Sat, 18 Jan 2025 20:11:04 -0500 Subject: [PATCH 36/62] WIP Update README.md --- README.md | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 3f009cf24..00e82d8e3 100644 --- a/README.md +++ b/README.md @@ -1,16 +1,12 @@ # Libretro Database -The github repository for databases used by RetroArch. - -## Overview - -The database repository contains several different kinds of files: +The github repository for databases used by RetroArch. The repository contains several different kinds of files: - __Game information database files__. - - `.dat` constituent files from many (sometimes-overlapping) sources and across many categories of metadata. - - `.rdb` files used by RetroArch, compiled and amalgmated from the `.dat` files. [RetroArch Database format](https://github.com/libretro/RetroArch/tree/68b3e5d8e02aff753e01a1f6f8969891910b2e0b/libretro-db#readme) (_no relation to Redis .RDB files_) accomodates RetroArch's [wide range of hardware/OS compatibility](https://www.retroarch.com/index.php?page=platforms). -- __Cheat code `.cht` files__. These are game-specific, remain in plain text, and are used as-is by RetroArch. The repository contains one unitary folder per system, unlike `.dat` files there is no amalgamation process. -- __Admin/management__ scripts and files. + - __`.dat`__ constituent files from many [sources](#sources) and across many categories of metadata. Alternative additional sources can be easily added and maintained in a self-contained constituent. Some `.dat` files may overlap partly or completely in the games they cover (see [precedence](#precedence)), while some `.dat` files cover an exclusive niche of games. + - __`.rdb`__ files used by RetroArch, compiled and amalgmated from the `.dat` files. [RetroArch Database format](https://github.com/libretro/RetroArch/tree/68b3e5d8e02aff753e01a1f6f8969891910b2e0b/libretro-db#readme) (_no relation to Redis .RDB files_) accomodates RetroArch's [wide range of hardware/OS compatibility](https://www.retroarch.com/index.php?page=platforms). +- __Cheat code `.cht` files__. These are game-specific, remain in plain text, and are used as-is by RetroArch. The repository contains one unitary folder for each system, unlike the multifaceted storage of dat files. +- __Admin/management scripts__ and files. ### RetroArch's Usage of the Database @@ -28,6 +24,8 @@ The key field for matching generally varies by console typical file size (i.e. m - __CRC checksum__ for systems with smaller file sizes, i.e. game media before the advent of disc-based games. - __Serial Number__ found within the ROM file, for larger files like disc-based games, to avoid computing checksums on large files. The serial is not metadata but encoded within the game's binary data, which is scanned (in applicable cases) as a byte array by RetroArch. +CRC and serial also serve as RetroArch's primary index. + Current [build script code](https://github.com/libretro/libretro-super/blob/master/libretro-build-database.sh#L288) can be viewed as a reference for which type of key field RetroArch uses for each console system. ### Precedence From c92de506c4f30926e3259bd65aa88852948a5a10 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Sat, 18 Jan 2025 20:12:59 -0500 Subject: [PATCH 37/62] Update README.md From 072cd3debe1e2f5a140173e36aef183e8379bb57 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Sat, 18 Jan 2025 21:19:15 -0500 Subject: [PATCH 38/62] Update README.md --- README.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 00e82d8e3..145708533 100644 --- a/README.md +++ b/README.md @@ -3,9 +3,9 @@ The github repository for databases used by RetroArch. The repository contains several different kinds of files: - __Game information database files__. - - __`.dat`__ constituent files from many [sources](#sources) and across many categories of metadata. Alternative additional sources can be easily added and maintained in a self-contained constituent. Some `.dat` files may overlap partly or completely in the games they cover (see [precedence](#precedence)), while some `.dat` files cover an exclusive niche of games. + - __`.dat`__ constituent files from many [sources](#sources) and across many categories of metadata. The system of dats is multifaceted: alternative or additional sources can be easily added and maintained in a self-contained constituent, and some `.dat` files may overlap partly or completely in the games they cover (see [precedence](#precedence)), while some `.dat` files cover an exclusive niche of games or attributes. - __`.rdb`__ files used by RetroArch, compiled and amalgmated from the `.dat` files. [RetroArch Database format](https://github.com/libretro/RetroArch/tree/68b3e5d8e02aff753e01a1f6f8969891910b2e0b/libretro-db#readme) (_no relation to Redis .RDB files_) accomodates RetroArch's [wide range of hardware/OS compatibility](https://www.retroarch.com/index.php?page=platforms). -- __Cheat code `.cht` files__. These are game-specific, remain in plain text, and are used as-is by RetroArch. The repository contains one unitary folder for each system, unlike the multifaceted storage of dat files. +- __Cheat code `.cht` files__. These are game-specific, remain in plain text, and are used as-is by RetroArch when manually selected and loaded by the user (see [Documentation](https://docs.libretro.com/guides/cheat-codes/)). The repository contains one unitary folder for each system, and multiple different cheat files may exist for the same game. - __Admin/management scripts__ and files. ### RetroArch's Usage of the Database @@ -19,21 +19,21 @@ Libretro databases allow RetroArch to provide several automated catalogging func - __Per-Game Information View__. Provide an in-app viewable informational screen for each game (Game > Information > Database Entry) ### Key Field -The key field for matching generally varies by console typical file size (i.e. media type). +The key field for matching varies by console typical file size (i.e. original media type). -- __CRC checksum__ for systems with smaller file sizes, i.e. game media before the advent of disc-based games. -- __Serial Number__ found within the ROM file, for larger files like disc-based games, to avoid computing checksums on large files. The serial is not metadata but encoded within the game's binary data, which is scanned (in applicable cases) as a byte array by RetroArch. +- __CRC checksum__ for systems with smaller file sizes, e.g. games before the advent of disc-based consoles. +- __Serial Number__ for larger files like disc-based games, to avoid computing checksums on large files. Found within the ROM file. The serial is not metadata but encoded within the game's binary data, which is scanned (in applicable cases) as a byte array by RetroArch. CRC and serial also serve as RetroArch's primary index. Current [build script code](https://github.com/libretro/libretro-super/blob/master/libretro-build-database.sh#L288) can be viewed as a reference for which type of key field RetroArch uses for each console system. ### Precedence -Databases earlier in the list have precedence over items later in the list. E.g. definitions in `dat` will over-ride `metadat` in the final `.rdb` compile if any info conflicts for the same item (i.e. for the same key field). +Databases earlier in the list have precedence over items later in the list. E.g. definitions in `dat` will over-ride `metadat` in the final `.rdb` compile if any info conflicts for the same game (i.e. for the same key field). ### Fields Specified in Game Information Databases -Database entries generally at minimum specify 1) a game's name, i.e. the display name that RetroArch will assign in playlists and 2) [key field](#key-field) data for matching. Further metadata for each game is often sourced from multiple databases. Databases often contain checksum/hashes for informational completeness even in cases where the [key field](#key-field) for matching is the game's internal serial number rather than checksum. +Database entries at minimum specify 1) a game's name, i.e. the display name that RetroArch will assign in playlists and 2) [key field](#key-field) data for matching and indexing. Further metadata for each game is often sourced from multiple databases. For reasons of informationial completeness, future-proofing, and compatibility outside RetroArch, databases contain checksum and cryptographic hashes regardless of the [key](#key-field) used for matching. Example of database entry within [`metadat/no-intro/Atari - 2600.dat`](https://github.com/libretro/libretro-database/blob/master/metadat/no-intro/Atari%20-%202600.dat) for the European region version of _Asteroids_: ``` @@ -44,6 +44,8 @@ game ( rom ( name "Asteroids (Europe).a26" size 8192 crc 0A2F8288 md5 8CF0D333BBE85B9549B1E6B1E2390B8D sha1 1CB8F057ACAD6DC65FEF07D3202088FF4AE355CD ) ) ``` +If other `Atari - 2600.dat` files exist and contain further metadata for the same crc, they would be compiled together with that entry's information in the final `.rdb`. For example, [`metadat/developer/Atari - 2600.dat`](https://github.com/libretro/libretro-database/blob/master/metadat/developer/Atari%20-%202600.dat#L297) would confer the developer attribute "Atari." + ## Repository Contents From ebbbac0dd4c4e6eed77525bc6733a4c0a85680d0 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Sat, 18 Jan 2025 23:09:15 -0500 Subject: [PATCH 39/62] Update README.md --- README.md | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 145708533..890b87d85 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ The github repository for databases used by RetroArch. The repository contains several different kinds of files: - __Game information database files__. - - __`.dat`__ constituent files from many [sources](#sources) and across many categories of metadata. The system of dats is multifaceted: alternative or additional sources can be easily added and maintained in a self-contained constituent, and some `.dat` files may overlap partly or completely in the games they cover (see [precedence](#precedence)), while some `.dat` files cover an exclusive niche of games or attributes. + - __`.dat`__ constituent files from many [sources](#sources) and across many categories of metadata. The system of dats is multifaceted: alternative or additional sources can be easily added and maintained in a self-contained constituent, and some `.dat` files may overlap in the games they cover (see [precedence](#precedence)), while some `.dat` files cover an exclusive niche of games or attributes. - __`.rdb`__ files used by RetroArch, compiled and amalgmated from the `.dat` files. [RetroArch Database format](https://github.com/libretro/RetroArch/tree/68b3e5d8e02aff753e01a1f6f8969891910b2e0b/libretro-db#readme) (_no relation to Redis .RDB files_) accomodates RetroArch's [wide range of hardware/OS compatibility](https://www.retroarch.com/index.php?page=platforms). - __Cheat code `.cht` files__. These are game-specific, remain in plain text, and are used as-is by RetroArch when manually selected and loaded by the user (see [Documentation](https://docs.libretro.com/guides/cheat-codes/)). The repository contains one unitary folder for each system, and multiple different cheat files may exist for the same game. - __Admin/management scripts__ and files. @@ -57,6 +57,8 @@ The non-exhaustive list below serves as a guide to various folders/files in the - [`metadat`](metadat) Several principal third-party DATs (e.g. No-Intro, Redump, and TOSEC), plus various collections of metadata (Note: some may be deprecated or incomplete). Examples: - [`bbfc`](metadat/bbfc) British Board of Film Classification's ratings for age-appropriateness. - [`elspa`](metadat/elspa) Age-appropriateness/content ratings from the Entertainment and Leisure Software Publishers Association aka the Association for UK Interactive Entertainment ("Ukie"). + - [`fb-neosplit`](metadat/fb-neosplit) Includes an XML database (from Logiqx's DTD ROM Management) unique to Arcade ROM scanning: it must be manually selected by the user when doing a Manual Scan, and it defines the component files within each ROM archive. (Also contains a typical `.dat` that treats `.zip` archives as the operative game file as is conventional for arcade emulation.) + - `FinalBurn Neo (ClrMame Pro XML, Arcade only).dat` - [`hacks`](metadat/hacks) Data for modified (or "hacked") versions of commercially released games. Many of these data are set by direct manual commits on the Libretro Github. - [`homebrew`](metadat/homebrew) Data for non-officially-published games created by independent creators/programmers. - [`libretro-dats`](metadat/libretro-dats) Ad hoc databases for items that were/are not covered by upstream database groups. Currently includes fan translations of SNES games, and an additional FDS dat that mostly overlaps with other sources/dats and may be redundant. @@ -73,7 +75,7 @@ Some databases are maintained even if RetroArch currently has no core for the ga ## Sources -Many source databases are in use as listed below. The table focusses on the 3rd party sources that predominantly cover each specific console library, but other/multiple sources including manual github contributions are maintained and all are compiled together in the final `.rdb` files (see [Repository Contents](#repository-contents) and github History for each dat for details). ">" signs below indicate the [precedence](#precedence) order when multiple sources overlap for the same subset of games/data. +Many source databases are in use as listed below. The table focusses on the 3rd party sources that predominantly cover each specific console library, but other/multiple sources including manual github contributions are maintained and all are compiled together in the final `.rdb` files (see [Repository Contents](#repository-contents) and each dat's github History for details). ">" signs below indicate the [precedence](#precedence) order when multiple sources overlap for the same subset of games/data. |System|Source|Repository| |----|---|---| @@ -225,23 +227,24 @@ Make sure filenames are Windows file system compatible, and are not too long (eg ``` find -exec basename '{}' ';' | egrep '^.{144,}$' ``` -### Other -**Folder structure revisions.** The [build script](https://github.com/libretro/libretro-super/blob/master/libretro-build-database.sh) specifies exact `.dat` files and folders in the repository, therefore organizational revisions to the file/folder structure should have corresponding revisions in the build script. +### Adding New Database: Example -__Adding New Database: Example__ -- Creat new `metadata/nameofnewdatabasefolder` and appropriately named `.dat` files e.g. `Sony - PlayStation.dat` +- Create new `metadata/nameofnewdatabasefolder` and appropriately named `.dat` file(s) e.g. `Sony - PlayStation.dat` - Add the new entry to `libretro-build-database.sh` - Run ``make build`` to build the RDB files -- New types for RetroArch's `Explore` tab will need some updates to RetroArch. - +- New types for RetroArch's `Explore` tab would require updates to RetroArch code. # Contributions -A vast majority of the database's game information originates from routine imports from upstream data groups (No-Intro, Redump, TOSEC, GameTDB, etc). General best practice for corrections or additions is for a contributor to go through the channels/process of the relevant upstream group. Upstream changes made by the database groups will eventually be imported to the Libretro databases. A "fix" to Libretro's copy of the database would be overwritten and lost with the next import from upstream. +### Small-Scale Corrections + +A vast majority of the database's game information originates from routine imports from upstream data groups (No-Intro, Redump, TOSEC, GameTDB, etc). In cases where the `.dat` for the entry in question originates from an upstream group, best practice is for a contributor to go through the channels/process of that group. Upstream changes made by the database groups will eventually be imported to the Libretro databases. A seemingly helpful "fix" to Libretro's copy of the database would be overwritten and lost by the next import from upstream. In cases where the `.dat` in question is created and maintained by Libretro or does not receive bulk over-writes, github contributions are accepted. Refer to the [repository contents list](#repository-contents) above and to github Histories for information about which libretro databases are applicable for github contributions. +### Folder structure / Housekeeping revisions.** The [build script](https://github.com/libretro/libretro-super/blob/master/libretro-build-database.sh) specifies exact `.dat` files and folders in the repository, therefore organizational revisions to the file/folder structure (e.g. combining two metadata fragments into one unified folder and file) should have corresponding revisions in the build script. + # Databases and RetroArch Thumbnails Currently there is no automatic process for updating libretro [thumbnail repository](https://github.com/libretro-thumbnails/libretro-thumbnails#libretro-thumbnails) image filenames based on game name updates in databases. Databases assign a game name (aka playlist item name or displayed game title) based on a game file's checksum, but thumbnails are only assigned if the thumbnail server image filename matches the game name or the ROM filename (with some [flexibility](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails)). To help fix a thumbnail, for example in a case where a database game name has been definitively/correctly updated in a way that no longer matches the repository thumbnail name, follow the [Thumbnail Repository readme](https://docs.libretro.com/guides/roms-playlists-thumbnails/#contributing-thumbnails-how-to) and [How-To documentation](https://docs.libretro.com/guides/roms-playlists-thumbnails/#contributing-thumbnails-how-to). From e90b42499e6e10de9fc09aab5fa24206383a563b Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Sat, 18 Jan 2025 23:41:34 -0500 Subject: [PATCH 40/62] Update README.md --- README.md | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 890b87d85..2054e54a2 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ The github repository for databases used by RetroArch. The repository contains s - __Game information database files__. - __`.dat`__ constituent files from many [sources](#sources) and across many categories of metadata. The system of dats is multifaceted: alternative or additional sources can be easily added and maintained in a self-contained constituent, and some `.dat` files may overlap in the games they cover (see [precedence](#precedence)), while some `.dat` files cover an exclusive niche of games or attributes. - __`.rdb`__ files used by RetroArch, compiled and amalgmated from the `.dat` files. [RetroArch Database format](https://github.com/libretro/RetroArch/tree/68b3e5d8e02aff753e01a1f6f8969891910b2e0b/libretro-db#readme) (_no relation to Redis .RDB files_) accomodates RetroArch's [wide range of hardware/OS compatibility](https://www.retroarch.com/index.php?page=platforms). -- __Cheat code `.cht` files__. These are game-specific, remain in plain text, and are used as-is by RetroArch when manually selected and loaded by the user (see [Documentation](https://docs.libretro.com/guides/cheat-codes/)). The repository contains one unitary folder for each system, and multiple different cheat files may exist for the same game. +- __Cheat code `.cht` files__. These are game-specific, remain in plain text, and are used as-is by RetroArch when manually selected and loaded by the user (see [Documentation](https://docs.libretro.com/guides/cheat-codes/)). The repository contains one unitary folder for each system (unlike dats), and multiple different cheat files may exist for the same game. - __Admin/management scripts__ and files. ### RetroArch's Usage of the Database @@ -57,8 +57,8 @@ The non-exhaustive list below serves as a guide to various folders/files in the - [`metadat`](metadat) Several principal third-party DATs (e.g. No-Intro, Redump, and TOSEC), plus various collections of metadata (Note: some may be deprecated or incomplete). Examples: - [`bbfc`](metadat/bbfc) British Board of Film Classification's ratings for age-appropriateness. - [`elspa`](metadat/elspa) Age-appropriateness/content ratings from the Entertainment and Leisure Software Publishers Association aka the Association for UK Interactive Entertainment ("Ukie"). - - [`fb-neosplit`](metadat/fb-neosplit) Includes an XML database (from Logiqx's DTD ROM Management) unique to Arcade ROM scanning: it must be manually selected by the user when doing a Manual Scan, and it defines the component files within each ROM archive. (Also contains a typical `.dat` that treats `.zip` archives as the operative game file as is conventional for arcade emulation.) - - `FinalBurn Neo (ClrMame Pro XML, Arcade only).dat` + - [`fbneo-split`](metadat/fb-neosplit) Includes an XML database (sourced from Logiqx's DTD ROM Management) unique to Arcade ROM scanning: it must be manually selected by the user when doing a Manual Scan, and it defines the component files within each ROM archive. (Also contains a typical `.dat` that treats `.zip` archives as the operative game file as is conventional for arcade emulation.) + - [`mame`](metadat/mame) Similar to `fbneo-split` above. - [`hacks`](metadat/hacks) Data for modified (or "hacked") versions of commercially released games. Many of these data are set by direct manual commits on the Libretro Github. - [`homebrew`](metadat/homebrew) Data for non-officially-published games created by independent creators/programmers. - [`libretro-dats`](metadat/libretro-dats) Ad hoc databases for items that were/are not covered by upstream database groups. Currently includes fan translations of SNES games, and an additional FDS dat that mostly overlaps with other sources/dats and may be redundant. @@ -228,12 +228,12 @@ Make sure filenames are Windows file system compatible, and are not too long (eg find -exec basename '{}' ';' | egrep '^.{144,}$' ``` -### Adding New Database: Example +### Adding A New Database -- Create new `metadata/nameofnewdatabasefolder` and appropriately named `.dat` file(s) e.g. `Sony - PlayStation.dat` +- Create new `metadata/nameofnewdatabasefolder` and appropriately named system `.dat` file(s) e.g. `Sony - PlayStation.dat` with new data - Add the new entry to `libretro-build-database.sh` - Run ``make build`` to build the RDB files -- New types for RetroArch's `Explore` tab would require updates to RetroArch code. +- New types for RetroArch's `Explore` tab require updates to RetroArch code. # Contributions @@ -243,11 +243,17 @@ A vast majority of the database's game information originates from routine impor In cases where the `.dat` in question is created and maintained by Libretro or does not receive bulk over-writes, github contributions are accepted. Refer to the [repository contents list](#repository-contents) above and to github Histories for information about which libretro databases are applicable for github contributions. -### Folder structure / Housekeeping revisions.** The [build script](https://github.com/libretro/libretro-super/blob/master/libretro-build-database.sh) specifies exact `.dat` files and folders in the repository, therefore organizational revisions to the file/folder structure (e.g. combining two metadata fragments into one unified folder and file) should have corresponding revisions in the build script. +### Folder Structure Revisions + +The [build script](https://github.com/libretro/libretro-super/blob/master/libretro-build-database.sh) specifies exact `.dat` files and folders in the repository, therefore organizational housekeeping revisions to the file/folder structure (e.g. combining two metadata fragments into one unified folder and file) require corresponding revisions in the build script. + +### Large-Scale Additions + +See [Adding New Database](#adding-a-new-database). # Databases and RetroArch Thumbnails -Currently there is no automatic process for updating libretro [thumbnail repository](https://github.com/libretro-thumbnails/libretro-thumbnails#libretro-thumbnails) image filenames based on game name updates in databases. Databases assign a game name (aka playlist item name or displayed game title) based on a game file's checksum, but thumbnails are only assigned if the thumbnail server image filename matches the game name or the ROM filename (with some [flexibility](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails)). To help fix a thumbnail, for example in a case where a database game name has been definitively/correctly updated in a way that no longer matches the repository thumbnail name, follow the [Thumbnail Repository readme](https://docs.libretro.com/guides/roms-playlists-thumbnails/#contributing-thumbnails-how-to) and [How-To documentation](https://docs.libretro.com/guides/roms-playlists-thumbnails/#contributing-thumbnails-how-to). +Currently there is no automatic process for updating libretro [thumbnail repository](https://github.com/libretro-thumbnails/libretro-thumbnails#libretro-thumbnails) image filenames based on game name updates in databases. RetroArch uses databases to assign a [game name](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails) based on a game file's checksum (or other [key](#key-field)), but thumbnails are only assigned if the thumbnail server image filename matches the game name or the ROM filename (with some [flexibility](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails)). To help fix a thumbnail, for example in a case where a database game name has been definitively/correctly updated in a way that no longer matches the repository thumbnail name, follow the [Thumbnail Repository readme](https://docs.libretro.com/guides/roms-playlists-thumbnails/#contributing-thumbnails-how-to) and [How To Contribute Thumbnails guide](https://docs.libretro.com/guides/roms-playlists-thumbnails/#contributing-thumbnails-how-to). # Integrations From 26d22ea10aacb30a9b6df43162ebdf946dfe1b0c Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Sat, 18 Jan 2025 23:43:31 -0500 Subject: [PATCH 41/62] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 2054e54a2..c9cf9f268 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ The github repository for databases used by RetroArch. The repository contains s - __Game information database files__. - __`.dat`__ constituent files from many [sources](#sources) and across many categories of metadata. The system of dats is multifaceted: alternative or additional sources can be easily added and maintained in a self-contained constituent, and some `.dat` files may overlap in the games they cover (see [precedence](#precedence)), while some `.dat` files cover an exclusive niche of games or attributes. - __`.rdb`__ files used by RetroArch, compiled and amalgmated from the `.dat` files. [RetroArch Database format](https://github.com/libretro/RetroArch/tree/68b3e5d8e02aff753e01a1f6f8969891910b2e0b/libretro-db#readme) (_no relation to Redis .RDB files_) accomodates RetroArch's [wide range of hardware/OS compatibility](https://www.retroarch.com/index.php?page=platforms). -- __Cheat code `.cht` files__. These are game-specific, remain in plain text, and are used as-is by RetroArch when manually selected and loaded by the user (see [Documentation](https://docs.libretro.com/guides/cheat-codes/)). The repository contains one unitary folder for each system (unlike dats), and multiple different cheat files may exist for the same game. +- __Cheat code `.cht` files__. These are game-specific, remain in plain text, and are used as-is by RetroArch when manually selected and loaded by the user (see [Cheat Code Documentation](https://docs.libretro.com/guides/cheat-codes/)). The repository contains one unitary folder for each system (unlike dats), and multiple different cheat files may exist for the same game. - __Admin/management scripts__ and files. ### RetroArch's Usage of the Database From 78b587ca5a08d19dc49742e67faed2ae5ba3cf68 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Sat, 18 Jan 2025 23:44:15 -0500 Subject: [PATCH 42/62] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index c9cf9f268..e8b3f214a 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ The github repository for databases used by RetroArch. The repository contains s - __Game information database files__. - __`.dat`__ constituent files from many [sources](#sources) and across many categories of metadata. The system of dats is multifaceted: alternative or additional sources can be easily added and maintained in a self-contained constituent, and some `.dat` files may overlap in the games they cover (see [precedence](#precedence)), while some `.dat` files cover an exclusive niche of games or attributes. - __`.rdb`__ files used by RetroArch, compiled and amalgmated from the `.dat` files. [RetroArch Database format](https://github.com/libretro/RetroArch/tree/68b3e5d8e02aff753e01a1f6f8969891910b2e0b/libretro-db#readme) (_no relation to Redis .RDB files_) accomodates RetroArch's [wide range of hardware/OS compatibility](https://www.retroarch.com/index.php?page=platforms). -- __Cheat code `.cht` files__. These are game-specific, remain in plain text, and are used as-is by RetroArch when manually selected and loaded by the user (see [Cheat Code Documentation](https://docs.libretro.com/guides/cheat-codes/)). The repository contains one unitary folder for each system (unlike dats), and multiple different cheat files may exist for the same game. +- __Cheat code `.cht` files__. These are game-specific, remain in plain text, and are used as-is by RetroArch if manually selected and loaded by the user (see [Cheat Code Documentation](https://docs.libretro.com/guides/cheat-codes/)). The repository contains one unitary folder for each system (unlike dats), and multiple different cheat files may exist for the same game. - __Admin/management scripts__ and files. ### RetroArch's Usage of the Database From dcd248db8e59aa3d62eaf2defff0812643b04fa6 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Sat, 18 Jan 2025 23:50:10 -0500 Subject: [PATCH 43/62] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index e8b3f214a..250c3f59f 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,7 @@ game ( rom ( name "Asteroids (Europe).a26" size 8192 crc 0A2F8288 md5 8CF0D333BBE85B9549B1E6B1E2390B8D sha1 1CB8F057ACAD6DC65FEF07D3202088FF4AE355CD ) ) ``` -If other `Atari - 2600.dat` files exist and contain further metadata for the same crc, they would be compiled together with that entry's information in the final `.rdb`. For example, [`metadat/developer/Atari - 2600.dat`](https://github.com/libretro/libretro-database/blob/master/metadat/developer/Atari%20-%202600.dat#L297) would confer the developer attribute "Atari." +If other `Atari - 2600.dat` files exist and contain further metadata for the same crc, they would be compiled together with that entry's information in the final `.rdb`. For example, [`metadat/developer/Atari - 2600.dat`](https://github.com/libretro/libretro-database/blob/master/metadat/developer/Atari%20-%202600.dat#L297) would confer `developer "Atari"`. ## Repository Contents From 98729b09d93e91e29e62381631acbaae5b7cb3d3 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Sun, 19 Jan 2025 12:00:54 -0500 Subject: [PATCH 44/62] WIP Update README.md --- README.md | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 250c3f59f..6c5f688de 100644 --- a/README.md +++ b/README.md @@ -3,9 +3,9 @@ The github repository for databases used by RetroArch. The repository contains several different kinds of files: - __Game information database files__. - - __`.dat`__ constituent files from many [sources](#sources) and across many categories of metadata. The system of dats is multifaceted: alternative or additional sources can be easily added and maintained in a self-contained constituent, and some `.dat` files may overlap in the games they cover (see [precedence](#precedence)), while some `.dat` files cover an exclusive niche of games or attributes. - - __`.rdb`__ files used by RetroArch, compiled and amalgmated from the `.dat` files. [RetroArch Database format](https://github.com/libretro/RetroArch/tree/68b3e5d8e02aff753e01a1f6f8969891910b2e0b/libretro-db#readme) (_no relation to Redis .RDB files_) accomodates RetroArch's [wide range of hardware/OS compatibility](https://www.retroarch.com/index.php?page=platforms). -- __Cheat code `.cht` files__. These are game-specific, remain in plain text, and are used as-is by RetroArch if manually selected and loaded by the user (see [Cheat Code Documentation](https://docs.libretro.com/guides/cheat-codes/)). The repository contains one unitary folder for each system (unlike dats), and multiple different cheat files may exist for the same game. + - __`.dat`__ constituent files from many [sources](#sources) and across many categories of metadata. The system of dats is multifaceted: alternative or additional sources can be easily added and maintained in a self-contained constituent, some `.dat` files may overlap in the games they cover (see [precedence](#precedence)), and some `.dat` files cover an exclusive niche of games or attributes. + - __`.rdb`__ files used by RetroArch, compiled and amalgmated from the `.dat` files. [RetroArch Database format](https://github.com/libretro/RetroArch/tree/68b3e5d8e02aff753e01a1f6f8969891910b2e0b/libretro-db#readme) (_no relation to Redis .RDB files_) accomodates RetroArch's [wide range of hardware/OS platforms](https://www.retroarch.com/index.php?page=platforms). +- __Cheat code `.cht` files__. These are game-specific, remain in plain text, and are used as-is by RetroArch if manually selected by the user (see [Cheat Code Documentation](https://docs.libretro.com/guides/cheat-codes/)). The repository contains one folder for each system (unlike dats), and multiple different cheat files may exist for the same game. - __Admin/management scripts__ and files. ### RetroArch's Usage of the Database @@ -16,7 +16,7 @@ Libretro databases allow RetroArch to provide several automated catalogging func - __Game Naming__. Assign a definitive and uniform display name for each game in a playlist regardless of filename. - __Thumbnail Images__. Download and display thumbnail images for games based on the uniform name assigned by the database, regardless of filename. (Note: thumbnails are not directly assigned by the database or by checksum association, but as a secondary effect of databased *game name* assignment if a matching thumbnail is available on the server. Also see: [Flexible Name Matching Algorithm](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails)) - __Category Search__. A reference search (named "Explore") that allows the user to search for games by category criteria, e.g. by Developer, Release Year, Genre, and other attributes/metadata. -- __Per-Game Information View__. Provide an in-app viewable informational screen for each game (Game > Information > Database Entry) +- __Per-Game Information View__. Provide an in-app viewable informational screen for each game (Game > Information > Database Entry). ### Key Field The key field for matching varies by console typical file size (i.e. original media type). @@ -44,7 +44,7 @@ game ( rom ( name "Asteroids (Europe).a26" size 8192 crc 0A2F8288 md5 8CF0D333BBE85B9549B1E6B1E2390B8D sha1 1CB8F057ACAD6DC65FEF07D3202088FF4AE355CD ) ) ``` -If other `Atari - 2600.dat` files exist and contain further metadata for the same crc, they would be compiled together with that entry's information in the final `.rdb`. For example, [`metadat/developer/Atari - 2600.dat`](https://github.com/libretro/libretro-database/blob/master/metadat/developer/Atari%20-%202600.dat#L297) would confer `developer "Atari"`. +If other `Atari - 2600.dat` files exist and contain further metadata for the same crc, they would be compiled together with that entry's information in the `.rdb`. For example, [`metadat/developer/Atari - 2600.dat`](https://github.com/libretro/libretro-database/blob/master/metadat/developer/Atari%20-%202600.dat#L297) would confer `developer "Atari"`. ## Repository Contents @@ -53,18 +53,22 @@ The non-exhaustive list below serves as a guide to various folders/files in the - [`cht`](cht) Cheat codes to various games, collected from any available source on the web including by manual contributions from users who haved used RetroArch's built-in [memory address/value search feature](https://docs.libretro.com/guides/cheat-codes/#retroarch-new-cheat-code-searching) to construct new cheat codes. - [`cursors`](cursors) Methods to query playlists. -- [`dat`](dat) Mostly customized DAT files maintained by the libretro team, including items that do/did not have contemporary documentation by upstream database groups (e.g. "Virtual Console" variants of SNES games), and games for monolithic non-generalized cores (Cave Story, Doom, Quake, etc). Also includes some imports from upstream groups in order to establish [precedence](#precedence) in the compilation, e.g. GameCube data from GameTDB, although most dats from upstream groups reside in [`metadat`](metadat). -- [`metadat`](metadat) Several principal third-party DATs (e.g. No-Intro, Redump, and TOSEC), plus various collections of metadata (Note: some may be deprecated or incomplete). Examples: +- [`dat`](dat) Mostly customized DAT files maintained by the libretro team, including: + - Items that do/did not have contemporary documentation by upstream database groups, e.g. Virtual Console variants of SNES games + - [DOS](https://github.com/libretro/libretro-database/blob/master/dat/DOS.dat) sourced from TotalDOS via [libretro-database-dos](https://github.com/robloach/libretro-database-dos) + - Games for monolithic non-generalized cores (Cave Story, Doom, Quake, etc) + - Some imports from upstream groups in order to establish [precedence](#precedence) in the compilation, e.g. GameCube data from GameTDB, although most dats from upstream groups reside in [`metadat`](metadat). +- [`metadat`](metadat) Several principal third-party DATs (e.g. No-Intro, Redump, MAME, TOSEC), plus various collections of metadata (some of which may be deprecated). Examples: - [`bbfc`](metadat/bbfc) British Board of Film Classification's ratings for age-appropriateness. - [`elspa`](metadat/elspa) Age-appropriateness/content ratings from the Entertainment and Leisure Software Publishers Association aka the Association for UK Interactive Entertainment ("Ukie"). - - [`fbneo-split`](metadat/fb-neosplit) Includes an XML database (sourced from Logiqx's DTD ROM Management) unique to Arcade ROM scanning: it must be manually selected by the user when doing a Manual Scan, and it defines the component files within each ROM archive. (Also contains a typical `.dat` that treats `.zip` archives as the operative game file as is conventional for arcade emulation.) + - [`fbneo-split`](metadat/fb-neosplit) Includes an XML database (sourced from Logiqx's DTD ROM Management) unique to Arcade ROM scanning: it must be manually selected by the user when running a Manual Scan, it defines the component files within each ROM archive, and is not part of the `.rdb` compile. (Also contains a typical `.dat` that treats `.zip` archives as the operative game file as is conventional for arcade emulation.) - [`mame`](metadat/mame) Similar to `fbneo-split` above. - [`hacks`](metadat/hacks) Data for modified (or "hacked") versions of commercially released games. Many of these data are set by direct manual commits on the Libretro Github. - [`homebrew`](metadat/homebrew) Data for non-officially-published games created by independent creators/programmers. - - [`libretro-dats`](metadat/libretro-dats) Ad hoc databases for items that were/are not covered by upstream database groups. Currently includes fan translations of SNES games, and an additional FDS dat that mostly overlaps with other sources/dats and may be redundant. + - [`libretro-dats`](metadat/libretro-dats) Ad hoc databases for items that were/are not covered by upstream database groups. Currently includes fan translations of SNES games, and an additional FDS dat that may be redundant with other sources. - [`no-intro`](metadat/no-intro) Bulk import from upstream No-Intro databases. Generally non-disc-based systems. - [`redump`](metadat/redump) Bulk import from upstream Redump databases. Generally disc-based systems. - - [`tosec`](metadat/tosec) Bulk import from upstream TOSEC databases. + - [`tosec`](metadat/tosec) Bulk import from upstream TOSEC databases. TOSEC data overlaps with and goes beyond other data sets (No-Intro, Redump), but has lower [precedence](#precedence) in libretro and so generally serves as a secondary stopgap. - And more - [`rdb`](rdb) The compiled RetroArch database files - [`scripts`](scripts) Various scripts that are used to maintain the database files From 8b51d09e2cf489b09e116d92de1343ce2683a6ea Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Sun, 19 Jan 2025 16:58:27 -0500 Subject: [PATCH 45/62] WIP Update README.md --- README.md | 28 +++++++++++++++------------- 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 6c5f688de..83e05b7a4 100644 --- a/README.md +++ b/README.md @@ -3,14 +3,14 @@ The github repository for databases used by RetroArch. The repository contains several different kinds of files: - __Game information database files__. - - __`.dat`__ constituent files from many [sources](#sources) and across many categories of metadata. The system of dats is multifaceted: alternative or additional sources can be easily added and maintained in a self-contained constituent, some `.dat` files may overlap in the games they cover (see [precedence](#precedence)), and some `.dat` files cover an exclusive niche of games or attributes. - - __`.rdb`__ files used by RetroArch, compiled and amalgmated from the `.dat` files. [RetroArch Database format](https://github.com/libretro/RetroArch/tree/68b3e5d8e02aff753e01a1f6f8969891910b2e0b/libretro-db#readme) (_no relation to Redis .RDB files_) accomodates RetroArch's [wide range of hardware/OS platforms](https://www.retroarch.com/index.php?page=platforms). + - __`.dat`__ files in the clrmamepro DAT format, from many [sources](#sources) and across many categories of metadata. The system of dats is multifaceted: alternative or additional sources can be easily added and maintained in a self-contained constituent, some dats may overlap in the games they cover (see [precedence](#precedence)), and some dats cover an exclusive niche of games or attributes. + - __`.rdb`__ files used by RetroArch, compiled and amalgamated from the `.dat` files. [RetroArch Database format](https://github.com/libretro/RetroArch/tree/68b3e5d8e02aff753e01a1f6f8969891910b2e0b/libretro-db#readme) (_no relation to Redis .RDB files_) accommodates RetroArch's [wide range of hardware/OS platforms](https://www.retroarch.com/index.php?page=platforms). - __Cheat code `.cht` files__. These are game-specific, remain in plain text, and are used as-is by RetroArch if manually selected by the user (see [Cheat Code Documentation](https://docs.libretro.com/guides/cheat-codes/)). The repository contains one folder for each system (unlike dats), and multiple different cheat files may exist for the same game. - __Admin/management scripts__ and files. ### RetroArch's Usage of the Database -Libretro databases allow RetroArch to provide several automated catalogging functions: +Libretro databases allow RetroArch to provide several automated cataloging functions: - __Validation__. Reject or accept files when using the [Import Scanner / Playlist Generator](https://docs.libretro.com/guides/roms-playlists-thumbnails/#working-with-playlists) based on whether the ROM checksum matches the checksum of a known verified completely intact (aka "properly dumped") file. - __Game Naming__. Assign a definitive and uniform display name for each game in a playlist regardless of filename. @@ -33,7 +33,7 @@ Databases earlier in the list have precedence over items later in the list. E.g ### Fields Specified in Game Information Databases -Database entries at minimum specify 1) a game's name, i.e. the display name that RetroArch will assign in playlists and 2) [key field](#key-field) data for matching and indexing. Further metadata for each game is often sourced from multiple databases. For reasons of informationial completeness, future-proofing, and compatibility outside RetroArch, databases contain checksum and cryptographic hashes regardless of the [key](#key-field) used for matching. +Database entries for games at minimum specify 1) a game's name, i.e. the display name that RetroArch will assign in playlists and 2) [key field](#key-field) data for matching/indexing and for identifying a file. Further optional metadata may appear. For reasons of informational completeness, future-proofing, and compatibility outside RetroArch, databases contain checksum and cryptographic hashes regardless of the [key](#key-field) used for matching. Example of database entry within [`metadat/no-intro/Atari - 2600.dat`](https://github.com/libretro/libretro-database/blob/master/metadat/no-intro/Atari%20-%202600.dat) for the European region version of _Asteroids_: ``` @@ -44,24 +44,26 @@ game ( rom ( name "Asteroids (Europe).a26" size 8192 crc 0A2F8288 md5 8CF0D333BBE85B9549B1E6B1E2390B8D sha1 1CB8F057ACAD6DC65FEF07D3202088FF4AE355CD ) ) ``` -If other `Atari - 2600.dat` files exist and contain further metadata for the same crc, they would be compiled together with that entry's information in the `.rdb`. For example, [`metadat/developer/Atari - 2600.dat`](https://github.com/libretro/libretro-database/blob/master/metadat/developer/Atari%20-%202600.dat#L297) would confer `developer "Atari"`. +If other `Atari - 2600.dat` files exist in the repository and contain further metadata for the same crc, the data would be compiled together in the `.rdb`. For example, [`metadat/developer/Atari - 2600.dat`](https://github.com/libretro/libretro-database/blob/master/metadat/developer/Atari%20-%202600.dat#L297) would confer `developer "Atari"` to the above data. +#### `.dat` Headers + +The `description " "` and `comment " "` fields within a libretro `clrmamepro ( )` header should explain the origin, source, and/or reason for the dat file's existence. Those header fields are intended for documentation purposes and are ignored by RetroArch, and can be changed in the associated build script (or changed directly in the `.dat` in the case of a manual ad hoc dat file) without issue. ## Repository Contents The non-exhaustive list below serves as a guide to various folders/files in the repository. -- [`cht`](cht) Cheat codes to various games, collected from any available source on the web including by manual contributions from users who haved used RetroArch's built-in [memory address/value search feature](https://docs.libretro.com/guides/cheat-codes/#retroarch-new-cheat-code-searching) to construct new cheat codes. +- [`cht`](cht) Cheat codes to various games, collected from any available source on the web including by manual contributions from users who have used RetroArch's built-in [memory address/value search feature](https://docs.libretro.com/guides/cheat-codes/#retroarch-new-cheat-code-searching) to construct new cheat codes. - [`cursors`](cursors) Methods to query playlists. -- [`dat`](dat) Mostly customized DAT files maintained by the libretro team, including: - - Items that do/did not have contemporary documentation by upstream database groups, e.g. Virtual Console variants of SNES games - - [DOS](https://github.com/libretro/libretro-database/blob/master/dat/DOS.dat) sourced from TotalDOS via [libretro-database-dos](https://github.com/robloach/libretro-database-dos) - - Games for monolithic non-generalized cores (Cave Story, Doom, Quake, etc) - - Some imports from upstream groups in order to establish [precedence](#precedence) in the compilation, e.g. GameCube data from GameTDB, although most dats from upstream groups reside in [`metadat`](metadat). -- [`metadat`](metadat) Several principal third-party DATs (e.g. No-Intro, Redump, MAME, TOSEC), plus various collections of metadata (some of which may be deprecated). Examples: +- [`dat`](dat) Customized DAT files maintained by the libretro team, including: + - Subset data coverage for games or variants that do/did not have contemporary documentation by upstream database groups, e.g. Virtual Console variants of SNES games, fan translations of NEC PC-98 games, and a superceded squib for PSP Minis. + - Monolithic non-generalized cores, e.g. Cave Story, Doom, Quake, etc. + - Data adapted from upstream sources that cover a relatively small number of systems and can therefore can be housed together in a single repository folder without conflict, e.g. DOS, ScummVM, and GameTDB coverage of GameCube and Wii data. (Though many dats from upstream groups reside in [`metadat`](metadat).) +- [`metadat`](metadat) Several principal third-party DATs (e.g. No-Intro, Redump, MAME, TOSEC) that each cover a large number of systems and therefore require their own folders in the repository, plus various collections of metadata (some of which may be deprecated). Examples: - [`bbfc`](metadat/bbfc) British Board of Film Classification's ratings for age-appropriateness. - [`elspa`](metadat/elspa) Age-appropriateness/content ratings from the Entertainment and Leisure Software Publishers Association aka the Association for UK Interactive Entertainment ("Ukie"). - - [`fbneo-split`](metadat/fb-neosplit) Includes an XML database (sourced from Logiqx's DTD ROM Management) unique to Arcade ROM scanning: it must be manually selected by the user when running a Manual Scan, it defines the component files within each ROM archive, and is not part of the `.rdb` compile. (Also contains a typical `.dat` that treats `.zip` archives as the operative game file as is conventional for arcade emulation.) + - [`fbneo-split`](metadat/fbneo-split) Includes an XML database (sourced from Logiqx's DTD ROM Management) unique to Arcade ROM scanning: it must be manually selected by the user when running a Manual Scan, it defines the component files within each ROM archive, and is not part of the `.rdb` compile. (Also contains a typical `.dat` that treats `.zip` archives as the operative game file as is conventional for arcade emulation.) - [`mame`](metadat/mame) Similar to `fbneo-split` above. - [`hacks`](metadat/hacks) Data for modified (or "hacked") versions of commercially released games. Many of these data are set by direct manual commits on the Libretro Github. - [`homebrew`](metadat/homebrew) Data for non-officially-published games created by independent creators/programmers. From 496542c602d26aca5bb64466c7f84031b4075e19 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Sun, 19 Jan 2025 18:29:16 -0500 Subject: [PATCH 46/62] WIP Update README.md --- README.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 83e05b7a4..e31210b72 100644 --- a/README.md +++ b/README.md @@ -14,8 +14,8 @@ Libretro databases allow RetroArch to provide several automated cataloging funct - __Validation__. Reject or accept files when using the [Import Scanner / Playlist Generator](https://docs.libretro.com/guides/roms-playlists-thumbnails/#working-with-playlists) based on whether the ROM checksum matches the checksum of a known verified completely intact (aka "properly dumped") file. - __Game Naming__. Assign a definitive and uniform display name for each game in a playlist regardless of filename. -- __Thumbnail Images__. Download and display thumbnail images for games based on the uniform name assigned by the database, regardless of filename. (Note: thumbnails are not directly assigned by the database or by checksum association, but as a secondary effect of databased *game name* assignment if a matching thumbnail is available on the server. Also see: [Flexible Name Matching Algorithm](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails)) -- __Category Search__. A reference search (named "Explore") that allows the user to search for games by category criteria, e.g. by Developer, Release Year, Genre, and other attributes/metadata. +- __Thumbnail Images__. Download and display thumbnail images for games based on the uniform name assigned by the database, regardless of filename. (Thumbnails are __not__ directly assigned by the database or by checksum association, but as a secondary effect of databased *game name* assignment if a matching thumbnail is available on the server. Also see: [Flexible Name Matching Algorithm](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails).) +- __Category Search ("Explore")__. Allows the user to find games by selecting category criteria, e.g. by Developer, Release Year, Genre, and other attributes/metadata. - __Per-Game Information View__. Provide an in-app viewable informational screen for each game (Game > Information > Database Entry). ### Key Field @@ -46,9 +46,11 @@ game ( ``` If other `Atari - 2600.dat` files exist in the repository and contain further metadata for the same crc, the data would be compiled together in the `.rdb`. For example, [`metadat/developer/Atari - 2600.dat`](https://github.com/libretro/libretro-database/blob/master/metadat/developer/Atari%20-%202600.dat#L297) would confer `developer "Atari"` to the above data. -#### `.dat` Headers +#### Header Guidelines for DATs -The `description " "` and `comment " "` fields within a libretro `clrmamepro ( )` header should explain the origin, source, and/or reason for the dat file's existence. Those header fields are intended for documentation purposes and are ignored by RetroArch, and can be changed in the associated build script (or changed directly in the `.dat` in the case of a manual ad hoc dat file) without issue. +The `description " "` and `comment " "` fields within a libretro dat's `clrmamepro ( )` header should be used to clarify the origin and source of the data and file. For example, if a .dat includes 3rd party upstream data processed through a github author's build/scrape script(s), the comment and description should contain information about both those aspects of the dat's origin. The description and comment header fields are intended for documentation purposes, are ignored by RetroArch, and can be freely changed without issue. + +Only the `name` field of a `.dat` file header must match a uniform system name recognized by RetroArch, not the `description` field. ## Repository Contents @@ -58,12 +60,12 @@ The non-exhaustive list below serves as a guide to various folders/files in the - [`cursors`](cursors) Methods to query playlists. - [`dat`](dat) Customized DAT files maintained by the libretro team, including: - Subset data coverage for games or variants that do/did not have contemporary documentation by upstream database groups, e.g. Virtual Console variants of SNES games, fan translations of NEC PC-98 games, and a superceded squib for PSP Minis. - - Monolithic non-generalized cores, e.g. Cave Story, Doom, Quake, etc. + - Games for monolithic non-generalized cores, e.g. Cave Story, Doom, Quake, etc. - Data adapted from upstream sources that cover a relatively small number of systems and can therefore can be housed together in a single repository folder without conflict, e.g. DOS, ScummVM, and GameTDB coverage of GameCube and Wii data. (Though many dats from upstream groups reside in [`metadat`](metadat).) - [`metadat`](metadat) Several principal third-party DATs (e.g. No-Intro, Redump, MAME, TOSEC) that each cover a large number of systems and therefore require their own folders in the repository, plus various collections of metadata (some of which may be deprecated). Examples: - [`bbfc`](metadat/bbfc) British Board of Film Classification's ratings for age-appropriateness. - [`elspa`](metadat/elspa) Age-appropriateness/content ratings from the Entertainment and Leisure Software Publishers Association aka the Association for UK Interactive Entertainment ("Ukie"). - - [`fbneo-split`](metadat/fbneo-split) Includes an XML database (sourced from Logiqx's DTD ROM Management) unique to Arcade ROM scanning: it must be manually selected by the user when running a Manual Scan, it defines the component files within each ROM archive, and is not part of the `.rdb` compile. (Also contains a typical `.dat` that treats `.zip` archives as the operative game file as is conventional for arcade emulation.) + - [`fbneo-split`](metadat/fbneo-split) Includes an XML database (sourced from Logiqx's DTD ROM Management) unique to Arcade ROM scanning: it must be manually selected by the user when running a Manual Scan, it defines the component files within each ROM archive, and is not part of the `.rdb` compile. (Also contains a typical `.dat` that treats `.zip` archives as the operative game file.) - [`mame`](metadat/mame) Similar to `fbneo-split` above. - [`hacks`](metadat/hacks) Data for modified (or "hacked") versions of commercially released games. Many of these data are set by direct manual commits on the Libretro Github. - [`homebrew`](metadat/homebrew) Data for non-officially-published games created by independent creators/programmers. From f8c30b18f307c188d9d77b2406b1ea13ef859049 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Mon, 20 Jan 2025 14:44:57 -0500 Subject: [PATCH 47/62] WIP Update README.md --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index e31210b72..188dc8e4d 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ Libretro databases allow RetroArch to provide several automated cataloging funct - __Validation__. Reject or accept files when using the [Import Scanner / Playlist Generator](https://docs.libretro.com/guides/roms-playlists-thumbnails/#working-with-playlists) based on whether the ROM checksum matches the checksum of a known verified completely intact (aka "properly dumped") file. - __Game Naming__. Assign a definitive and uniform display name for each game in a playlist regardless of filename. - __Thumbnail Images__. Download and display thumbnail images for games based on the uniform name assigned by the database, regardless of filename. (Thumbnails are __not__ directly assigned by the database or by checksum association, but as a secondary effect of databased *game name* assignment if a matching thumbnail is available on the server. Also see: [Flexible Name Matching Algorithm](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails).) -- __Category Search ("Explore")__. Allows the user to find games by selecting category criteria, e.g. by Developer, Release Year, Genre, and other attributes/metadata. +- __Category Search ("Explore")__. Allows the user to find/view games that match selected criteria, e.g. by Developer, Release Year, Genre, and other attributes/metadata. - __Per-Game Information View__. Provide an in-app viewable informational screen for each game (Game > Information > Database Entry). ### Key Field @@ -56,16 +56,16 @@ Only the `name` field of a `.dat` file header must match a uniform system name r The non-exhaustive list below serves as a guide to various folders/files in the repository. -- [`cht`](cht) Cheat codes to various games, collected from any available source on the web including by manual contributions from users who have used RetroArch's built-in [memory address/value search feature](https://docs.libretro.com/guides/cheat-codes/#retroarch-new-cheat-code-searching) to construct new cheat codes. +- [`cht`](cht) Cheat codes to various games, collected from any available source on the web including by manual [contributions from users](https://github.com/libretro/libretro-database/pulls?q=is%3Apr+is%3Aclosed+cheats) who have used RetroArch's built-in [memory address/value search feature](https://docs.libretro.com/guides/cheat-codes/#retroarch-new-cheat-code-searching) to construct new cheat codes. - [`cursors`](cursors) Methods to query playlists. - [`dat`](dat) Customized DAT files maintained by the libretro team, including: - Subset data coverage for games or variants that do/did not have contemporary documentation by upstream database groups, e.g. Virtual Console variants of SNES games, fan translations of NEC PC-98 games, and a superceded squib for PSP Minis. - - Games for monolithic non-generalized cores, e.g. Cave Story, Doom, Quake, etc. + - Game data for monolithic non-generalized cores, e.g. Cave Story, Doom, Quake, etc. - Data adapted from upstream sources that cover a relatively small number of systems and can therefore can be housed together in a single repository folder without conflict, e.g. DOS, ScummVM, and GameTDB coverage of GameCube and Wii data. (Though many dats from upstream groups reside in [`metadat`](metadat).) - [`metadat`](metadat) Several principal third-party DATs (e.g. No-Intro, Redump, MAME, TOSEC) that each cover a large number of systems and therefore require their own folders in the repository, plus various collections of metadata (some of which may be deprecated). Examples: - [`bbfc`](metadat/bbfc) British Board of Film Classification's ratings for age-appropriateness. - [`elspa`](metadat/elspa) Age-appropriateness/content ratings from the Entertainment and Leisure Software Publishers Association aka the Association for UK Interactive Entertainment ("Ukie"). - - [`fbneo-split`](metadat/fbneo-split) Includes an XML database (sourced from Logiqx's DTD ROM Management) unique to Arcade ROM scanning: it must be manually selected by the user when running a Manual Scan, it defines the component files within each ROM archive, and is not part of the `.rdb` compile. (Also contains a typical `.dat` that treats `.zip` archives as the operative game file.) + - [`fbneo-split`](metadat/fbneo-split) Includes an XML database (sourced from Logiqx's DTD ROM Management) for special use in to arcade ROM scanning: it must be manually selected by the user when running a Manual Scan, it defines the component files within each ROM archive, and is not part of the `.rdb` compile. (Also contains a typical `.dat` that treats `.zip` archives as the operative game file.) - [`mame`](metadat/mame) Similar to `fbneo-split` above. - [`hacks`](metadat/hacks) Data for modified (or "hacked") versions of commercially released games. Many of these data are set by direct manual commits on the Libretro Github. - [`homebrew`](metadat/homebrew) Data for non-officially-published games created by independent creators/programmers. From 14519cdbbe89608c6c0c5fc4d060662c2d47b703 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Mon, 20 Jan 2025 15:20:03 -0500 Subject: [PATCH 48/62] WIP Update README.md Adding brief troubleshooting section --- README.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 188dc8e4d..a8f7b0b84 100644 --- a/README.md +++ b/README.md @@ -65,7 +65,7 @@ The non-exhaustive list below serves as a guide to various folders/files in the - [`metadat`](metadat) Several principal third-party DATs (e.g. No-Intro, Redump, MAME, TOSEC) that each cover a large number of systems and therefore require their own folders in the repository, plus various collections of metadata (some of which may be deprecated). Examples: - [`bbfc`](metadat/bbfc) British Board of Film Classification's ratings for age-appropriateness. - [`elspa`](metadat/elspa) Age-appropriateness/content ratings from the Entertainment and Leisure Software Publishers Association aka the Association for UK Interactive Entertainment ("Ukie"). - - [`fbneo-split`](metadat/fbneo-split) Includes an XML database (sourced from Logiqx's DTD ROM Management) for special use in to arcade ROM scanning: it must be manually selected by the user when running a Manual Scan, it defines the component files within each ROM archive, and is not part of the `.rdb` compile. (Also contains a typical `.dat` that treats `.zip` archives as the operative game file.) + - [`fbneo-split`](metadat/fbneo-split) Includes an XML database (sourced from Logiqx's DTD ROM Management) for special use in arcade ROM scanning: it must be manually selected by the user when running a Manual Scan, it defines the component files within each ROM archive, and is not part of the `.rdb` compile. Also contains a typical `.dat`. - [`mame`](metadat/mame) Similar to `fbneo-split` above. - [`hacks`](metadat/hacks) Data for modified (or "hacked") versions of commercially released games. Many of these data are set by direct manual commits on the Libretro Github. - [`homebrew`](metadat/homebrew) Data for non-officially-published games created by independent creators/programmers. @@ -243,6 +243,12 @@ find -exec basename '{}' ';' | egrep '^.{144,}$' - Run ``make build`` to build the RDB files - New types for RetroArch's `Explore` tab require updates to RetroArch code. +# Troubleshooting Brief + +A helpful set of steps for diagnosing the cause of a database or game/name identification issue, in order to help fix it, is: + + + # Contributions ### Small-Scale Corrections From acd3057bf79850ed57a4f0fdbe44b40f915e951f Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Mon, 20 Jan 2025 16:56:03 -0500 Subject: [PATCH 49/62] Update README.md Moved out tentative "Troubleshooting" to new draft Docs site databases.md --- README.md | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/README.md b/README.md index a8f7b0b84..5e11c2238 100644 --- a/README.md +++ b/README.md @@ -243,17 +243,11 @@ find -exec basename '{}' ';' | egrep '^.{144,}$' - Run ``make build`` to build the RDB files - New types for RetroArch's `Explore` tab require updates to RetroArch code. -# Troubleshooting Brief - -A helpful set of steps for diagnosing the cause of a database or game/name identification issue, in order to help fix it, is: - - - # Contributions ### Small-Scale Corrections -A vast majority of the database's game information originates from routine imports from upstream data groups (No-Intro, Redump, TOSEC, GameTDB, etc). In cases where the `.dat` for the entry in question originates from an upstream group, best practice is for a contributor to go through the channels/process of that group. Upstream changes made by the database groups will eventually be imported to the Libretro databases. A seemingly helpful "fix" to Libretro's copy of the database would be overwritten and lost by the next import from upstream. +A vast majority of the database's game information originates from routine imports from upstream data groups (No-Intro, Redump, TOSEC, GameTDB, etc). In cases where the `.dat` for the entry at issue originates from an upstream group, best practice is for a contributor to go through the channels/process of that group. Upstream changes made by the database groups will eventually be imported to the Libretro databases. A seemingly helpful "fix" to Libretro's copy of the database would be overwritten and lost by the next import from upstream. In cases where the `.dat` in question is created and maintained by Libretro or does not receive bulk over-writes, github contributions are accepted. Refer to the [repository contents list](#repository-contents) above and to github Histories for information about which libretro databases are applicable for github contributions. From 64e3d4aad74cad3052d7b5d765fda8e90528fd06 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Mon, 20 Jan 2025 17:21:48 -0500 Subject: [PATCH 50/62] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 5e11c2238..8516fbb2e 100644 --- a/README.md +++ b/README.md @@ -46,7 +46,7 @@ game ( ``` If other `Atari - 2600.dat` files exist in the repository and contain further metadata for the same crc, the data would be compiled together in the `.rdb`. For example, [`metadat/developer/Atari - 2600.dat`](https://github.com/libretro/libretro-database/blob/master/metadat/developer/Atari%20-%202600.dat#L297) would confer `developer "Atari"` to the above data. -#### Header Guidelines for DATs +### Header Guidelines for DATs The `description " "` and `comment " "` fields within a libretro dat's `clrmamepro ( )` header should be used to clarify the origin and source of the data and file. For example, if a .dat includes 3rd party upstream data processed through a github author's build/scrape script(s), the comment and description should contain information about both those aspects of the dat's origin. The description and comment header fields are intended for documentation purposes, are ignored by RetroArch, and can be freely changed without issue. From 1bd9f08224728eed4f215f083f7151bfac729b0d Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Mon, 20 Jan 2025 17:57:40 -0500 Subject: [PATCH 51/62] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 8516fbb2e..4fda76332 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ The key field for matching varies by console typical file size (i.e. original me CRC and serial also serve as RetroArch's primary index. -Current [build script code](https://github.com/libretro/libretro-super/blob/master/libretro-build-database.sh#L288) can be viewed as a reference for which type of key field RetroArch uses for each console system. +Current [build script code](https://github.com/libretro/libretro-super/blob/master/libretro-build-database.sh#L245) can be viewed as a reference for which type of key field RetroArch uses for each console system. ### Precedence Databases earlier in the list have precedence over items later in the list. E.g. definitions in `dat` will over-ride `metadat` in the final `.rdb` compile if any info conflicts for the same game (i.e. for the same key field). From fe1e00c3a0d7f9aaa27cdef38f0437660b7bfcf0 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Mon, 20 Jan 2025 17:58:13 -0500 Subject: [PATCH 52/62] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 4fda76332..0cb3a8d23 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ Databases earlier in the list have precedence over items later in the list. E.g ### Fields Specified in Game Information Databases -Database entries for games at minimum specify 1) a game's name, i.e. the display name that RetroArch will assign in playlists and 2) [key field](#key-field) data for matching/indexing and for identifying a file. Further optional metadata may appear. For reasons of informational completeness, future-proofing, and compatibility outside RetroArch, databases contain checksum and cryptographic hashes regardless of the [key](#key-field) used for matching. +Database entries for games at minimum specify 1) a game's name, i.e. the display name that RetroArch will assign in playlists and 2) [key field](#key-field) data for matching/indexing and for identifying a file. Further optional metadata may appear. For reasons of informational completeness, future-proofing, and compatibility outside RetroArch, databases contain checksum and cryptographic hashes regardless of the key used for matching. Example of database entry within [`metadat/no-intro/Atari - 2600.dat`](https://github.com/libretro/libretro-database/blob/master/metadat/no-intro/Atari%20-%202600.dat) for the European region version of _Asteroids_: ``` From 0f732fffeebabfc751338ee82b22284cec333857 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Mon, 20 Jan 2025 17:58:59 -0500 Subject: [PATCH 53/62] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 0cb3a8d23..432ad9ffa 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,7 @@ game ( rom ( name "Asteroids (Europe).a26" size 8192 crc 0A2F8288 md5 8CF0D333BBE85B9549B1E6B1E2390B8D sha1 1CB8F057ACAD6DC65FEF07D3202088FF4AE355CD ) ) ``` -If other `Atari - 2600.dat` files exist in the repository and contain further metadata for the same crc, the data would be compiled together in the `.rdb`. For example, [`metadat/developer/Atari - 2600.dat`](https://github.com/libretro/libretro-database/blob/master/metadat/developer/Atari%20-%202600.dat#L297) would confer `developer "Atari"` to the above data. +If other `Atari - 2600.dat` files exist in the repository and contain further metadata for the same crc, the data would be compiled together in the `.rdb`. For example, [`metadat/developer/Atari - 2600.dat`](https://github.com/libretro/libretro-database/blob/master/metadat/developer/Atari%20-%202600.dat#L296) would confer `developer "Atari"` to the above data. ### Header Guidelines for DATs From 65d7423562952186c14e0dce6d8d4987a39ceda6 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Mon, 20 Jan 2025 18:00:10 -0500 Subject: [PATCH 54/62] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 432ad9ffa..45d87e5d5 100644 --- a/README.md +++ b/README.md @@ -48,7 +48,7 @@ If other `Atari - 2600.dat` files exist in the repository and contain further me ### Header Guidelines for DATs -The `description " "` and `comment " "` fields within a libretro dat's `clrmamepro ( )` header should be used to clarify the origin and source of the data and file. For example, if a .dat includes 3rd party upstream data processed through a github author's build/scrape script(s), the comment and description should contain information about both those aspects of the dat's origin. The description and comment header fields are intended for documentation purposes, are ignored by RetroArch, and can be freely changed without issue. +The `description " "` and `comment " "` fields within a libretro dat's `clrmamepro ( )` header should be used to clarify the origin and source of the data and file. For example, if a .dat includes 3rd party upstream data processed through a github author's build/scrape script(s), the comment and description (or other appropriate header fields) should contain information about _both_ those aspects of the dat's origin. The description and comment header fields are intended for documentation purposes, are ignored by RetroArch, and can be freely changed without issue. Only the `name` field of a `.dat` file header must match a uniform system name recognized by RetroArch, not the `description` field. From 2dcbeb76fd232e36b54a331e0bf3b37ec1357c0d Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Mon, 20 Jan 2025 18:03:32 -0500 Subject: [PATCH 55/62] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 45d87e5d5..a0aa686d5 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ CRC and serial also serve as RetroArch's primary index. Current [build script code](https://github.com/libretro/libretro-super/blob/master/libretro-build-database.sh#L245) can be viewed as a reference for which type of key field RetroArch uses for each console system. ### Precedence -Databases earlier in the list have precedence over items later in the list. E.g. definitions in `dat` will over-ride `metadat` in the final `.rdb` compile if any info conflicts for the same game (i.e. for the same key field). +Databases earlier in the list have precedence over items later in the list. E.g. definitions in `/dat` will over-ride `/metadat` in the final `.rdb` compile if any info conflicts for the same game (i.e. for the same key field). ### Fields Specified in Game Information Databases From 56a00165608f29703cb1bfaf6a238b11da03c455 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Mon, 20 Jan 2025 18:06:51 -0500 Subject: [PATCH 56/62] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index a0aa686d5..063c990df 100644 --- a/README.md +++ b/README.md @@ -261,7 +261,7 @@ See [Adding New Database](#adding-a-new-database). # Databases and RetroArch Thumbnails -Currently there is no automatic process for updating libretro [thumbnail repository](https://github.com/libretro-thumbnails/libretro-thumbnails#libretro-thumbnails) image filenames based on game name updates in databases. RetroArch uses databases to assign a [game name](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails) based on a game file's checksum (or other [key](#key-field)), but thumbnails are only assigned if the thumbnail server image filename matches the game name or the ROM filename (with some [flexibility](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails)). To help fix a thumbnail, for example in a case where a database game name has been definitively/correctly updated in a way that no longer matches the repository thumbnail name, follow the [Thumbnail Repository readme](https://docs.libretro.com/guides/roms-playlists-thumbnails/#contributing-thumbnails-how-to) and [How To Contribute Thumbnails guide](https://docs.libretro.com/guides/roms-playlists-thumbnails/#contributing-thumbnails-how-to). +Currently there is no automatic process for updating libretro [thumbnail repository](https://github.com/libretro-thumbnails/libretro-thumbnails#libretro-thumbnails) image filenames based on game name updates in databases. RetroArch uses databases to assign a [game name](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails) based on a game file's checksum (or other [key](#key-field)), but thumbnails are only assigned if the thumbnail server image filename matches the game name or the ROM filename (with some [flexibility](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails)). To help fix a thumbnail, for example in a case where a database game name has been definitively/correctly updated in a way that no longer matches the repository thumbnail name, follow the [Thumbnail Repository readme](https://github.com/libretro-thumbnails/libretro-thumbnails#libretro-thumbnails) and [How To Contribute Thumbnails guide](https://docs.libretro.com/guides/roms-playlists-thumbnails/#contributing-thumbnails-how-to). # Integrations From 50d85e4fedc5843cc4122495857b455a91e14f5d Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Mon, 20 Jan 2025 18:08:21 -0500 Subject: [PATCH 57/62] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 063c990df..da6504268 100644 --- a/README.md +++ b/README.md @@ -261,7 +261,7 @@ See [Adding New Database](#adding-a-new-database). # Databases and RetroArch Thumbnails -Currently there is no automatic process for updating libretro [thumbnail repository](https://github.com/libretro-thumbnails/libretro-thumbnails#libretro-thumbnails) image filenames based on game name updates in databases. RetroArch uses databases to assign a [game name](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails) based on a game file's checksum (or other [key](#key-field)), but thumbnails are only assigned if the thumbnail server image filename matches the game name or the ROM filename (with some [flexibility](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails)). To help fix a thumbnail, for example in a case where a database game name has been definitively/correctly updated in a way that no longer matches the repository thumbnail name, follow the [Thumbnail Repository readme](https://github.com/libretro-thumbnails/libretro-thumbnails#libretro-thumbnails) and [How To Contribute Thumbnails guide](https://docs.libretro.com/guides/roms-playlists-thumbnails/#contributing-thumbnails-how-to). +Currently there is no automatic process for updating libretro [thumbnail repository](https://github.com/libretro-thumbnails/libretro-thumbnails#libretro-thumbnails) image filenames based on game name updates in databases. RetroArch uses databases to assign a [game name](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails) based on a game file's checksum (or other [key](#key-field)), but thumbnails are only assigned if the thumbnail server image filename matches the game name or the ROM filename (with some [flexibility](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails)). To help fix a thumbnail, for example in a case where a database game name has been definitively/correctly updated in a way that no longer matches the repository thumbnail name, follow the [Thumbnail Repository readme](https://github.com/libretro-thumbnails/libretro-thumbnails#libretro-thumbnails) and [How to Contribute to Thumbnails guide](https://docs.libretro.com/guides/roms-playlists-thumbnails/#contributing-thumbnails-how-to). # Integrations From 6a6a76874c18067728c9d805da8fd37c1b7666d0 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Mon, 20 Jan 2025 18:09:21 -0500 Subject: [PATCH 58/62] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index da6504268..5a44647ef 100644 --- a/README.md +++ b/README.md @@ -261,7 +261,7 @@ See [Adding New Database](#adding-a-new-database). # Databases and RetroArch Thumbnails -Currently there is no automatic process for updating libretro [thumbnail repository](https://github.com/libretro-thumbnails/libretro-thumbnails#libretro-thumbnails) image filenames based on game name updates in databases. RetroArch uses databases to assign a [game name](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails) based on a game file's checksum (or other [key](#key-field)), but thumbnails are only assigned if the thumbnail server image filename matches the game name or the ROM filename (with some [flexibility](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails)). To help fix a thumbnail, for example in a case where a database game name has been definitively/correctly updated in a way that no longer matches the repository thumbnail name, follow the [Thumbnail Repository readme](https://github.com/libretro-thumbnails/libretro-thumbnails#libretro-thumbnails) and [How to Contribute to Thumbnails guide](https://docs.libretro.com/guides/roms-playlists-thumbnails/#contributing-thumbnails-how-to). +Currently there is no automatic process for updating libretro [thumbnail repository](https://github.com/libretro-thumbnails/libretro-thumbnails#libretro-thumbnails) image filenames based on game name updates in databases. RetroArch uses databases to assign a [game name](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails) based on a game file's checksum (or other [key](#key-field)), but thumbnails are only assigned if the thumbnail server image filename matches the game name or the ROM filename (with some [flexibility](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails)). To help fix a thumbnail, for example in a case where a database game name has been definitively/correctly updated in a way that no longer matches the repository thumbnail name, follow the [Thumbnail Repository Readme](https://github.com/libretro-thumbnails/libretro-thumbnails#libretro-thumbnails) and [How to Contribute to Thumbnails Guide](https://docs.libretro.com/guides/roms-playlists-thumbnails/#contributing-thumbnails-how-to). # Integrations From 4cdbc0fea17466788f949ea95f3df7e94b1a0ccf Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Mon, 20 Jan 2025 18:14:44 -0500 Subject: [PATCH 59/62] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 5a44647ef..65a29eb9e 100644 --- a/README.md +++ b/README.md @@ -261,7 +261,7 @@ See [Adding New Database](#adding-a-new-database). # Databases and RetroArch Thumbnails -Currently there is no automatic process for updating libretro [thumbnail repository](https://github.com/libretro-thumbnails/libretro-thumbnails#libretro-thumbnails) image filenames based on game name updates in databases. RetroArch uses databases to assign a [game name](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails) based on a game file's checksum (or other [key](#key-field)), but thumbnails are only assigned if the thumbnail server image filename matches the game name or the ROM filename (with some [flexibility](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails)). To help fix a thumbnail, for example in a case where a database game name has been definitively/correctly updated in a way that no longer matches the repository thumbnail name, follow the [Thumbnail Repository Readme](https://github.com/libretro-thumbnails/libretro-thumbnails#libretro-thumbnails) and [How to Contribute to Thumbnails Guide](https://docs.libretro.com/guides/roms-playlists-thumbnails/#contributing-thumbnails-how-to). +Currently there is no automatic process for updating libretro [thumbnail repository](https://github.com/libretro-thumbnails/libretro-thumbnails#libretro-thumbnails) image filenames based on game name updates in databases. RetroArch uses databases to assign a [game name](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails) based on a game file's checksum (or other [key](#key-field)), but thumbnails are only assigned if the thumbnail server image filename matches the game name or the ROM filename (with some [flexibility](https://docs.libretro.com/guides/roms-playlists-thumbnails/#custom-thumbnails)). To help fix a thumbnail, for example in a case where the database game name has been definitively/correctly updated in a way that no longer matches the repository thumbnail name, follow the [Thumbnail Repository Readme](https://github.com/libretro-thumbnails/libretro-thumbnails#libretro-thumbnails) and [How to Contribute to Thumbnails Guide](https://docs.libretro.com/guides/roms-playlists-thumbnails/#contributing-thumbnails-how-to). # Integrations From ae08155e884f93f9b3419b26651c48e9feb8787d Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Tue, 21 Jan 2025 19:24:55 -0500 Subject: [PATCH 60/62] edit (reviewer point): fixed link Fixed link to master, was wrongly linked to a specific history/commit --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 65a29eb9e..245599f15 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ The github repository for databases used by RetroArch. The repository contains s - __Game information database files__. - __`.dat`__ files in the clrmamepro DAT format, from many [sources](#sources) and across many categories of metadata. The system of dats is multifaceted: alternative or additional sources can be easily added and maintained in a self-contained constituent, some dats may overlap in the games they cover (see [precedence](#precedence)), and some dats cover an exclusive niche of games or attributes. - - __`.rdb`__ files used by RetroArch, compiled and amalgamated from the `.dat` files. [RetroArch Database format](https://github.com/libretro/RetroArch/tree/68b3e5d8e02aff753e01a1f6f8969891910b2e0b/libretro-db#readme) (_no relation to Redis .RDB files_) accommodates RetroArch's [wide range of hardware/OS platforms](https://www.retroarch.com/index.php?page=platforms). + - __`.rdb`__ files used by RetroArch, compiled and amalgamated from the `.dat` files. [RetroArch Database format](https://github.com/libretro/RetroArch/blob/master/libretro-db/README.md) (_no relation to Redis .RDB files_) accommodates RetroArch's [wide range of hardware/OS platforms](https://www.retroarch.com/index.php?page=platforms). - __Cheat code `.cht` files__. These are game-specific, remain in plain text, and are used as-is by RetroArch if manually selected by the user (see [Cheat Code Documentation](https://docs.libretro.com/guides/cheat-codes/)). The repository contains one folder for each system (unlike dats), and multiple different cheat files may exist for the same game. - __Admin/management scripts__ and files. From 369eb76f58e7ced49090ea10f1ba429624e09d0a Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Tue, 21 Jan 2025 19:55:53 -0500 Subject: [PATCH 61/62] Fixed (reviewer point) heder `name` generalization Changed wording about header `name` to fix the wrong statement/generalization pointed out by reviewer. Name isn't always `manufacturer - system`, and core info files (now linked) detail what it should be. --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 245599f15..94292e718 100644 --- a/README.md +++ b/README.md @@ -48,9 +48,9 @@ If other `Atari - 2600.dat` files exist in the repository and contain further me ### Header Guidelines for DATs -The `description " "` and `comment " "` fields within a libretro dat's `clrmamepro ( )` header should be used to clarify the origin and source of the data and file. For example, if a .dat includes 3rd party upstream data processed through a github author's build/scrape script(s), the comment and description (or other appropriate header fields) should contain information about _both_ those aspects of the dat's origin. The description and comment header fields are intended for documentation purposes, are ignored by RetroArch, and can be freely changed without issue. +The `description " "` and `comment " "` fields within a libretro dat's `clrmamepro ( )` header should be used to clarify the origin, source, and/or purpose of the data and file. The description and comment header fields are intended for documentation purposes, are ignored by RetroArch, and can be freely changed without issue. For example, if a .dat includes 3rd party upstream data processed through a github author's build/scrape script(s), the comment and description (or other appropriate header fields) should contain information about _both_ those aspects of the dat's origin. If the .dat file is meant to cover a particular niche of data, the description field should explain it. -Only the `name` field of a `.dat` file header must match a uniform system name recognized by RetroArch, not the `description` field. +The `name` field of a `.dat` file header should match the name that is specified by the core in the [.info file for the core](https://github.com/libretro/libretro-super/tree/master/dist/info) (often but not always `manufacturer - systemname` or similar). The `description` field should be descriptive and informative about the `.dat` file's origin and purpose. ## Repository Contents From 5956557698c46bfc0210da22f7f90c2207d0af27 Mon Sep 17 00:00:00 2001 From: OctopusButtons <62315590+OctopusButtons@users.noreply.github.com> Date: Tue, 21 Jan 2025 20:10:53 -0500 Subject: [PATCH 62/62] Even better fixes on (reviewer point) `name` header Extra clear about the `database` field in .info files now --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 94292e718..cc4adacf7 100644 --- a/README.md +++ b/README.md @@ -48,9 +48,9 @@ If other `Atari - 2600.dat` files exist in the repository and contain further me ### Header Guidelines for DATs -The `description " "` and `comment " "` fields within a libretro dat's `clrmamepro ( )` header should be used to clarify the origin, source, and/or purpose of the data and file. The description and comment header fields are intended for documentation purposes, are ignored by RetroArch, and can be freely changed without issue. For example, if a .dat includes 3rd party upstream data processed through a github author's build/scrape script(s), the comment and description (or other appropriate header fields) should contain information about _both_ those aspects of the dat's origin. If the .dat file is meant to cover a particular niche of data, the description field should explain it. +The `description " "` and `comment " "` fields within a libretro dat's `clrmamepro ( )` header should be used to clarify the origin, source, and/or purpose of the data and file. The description and comment header fields are __intended for documentation__ purposes, are ignored by RetroArch, and can be freely changed without issue. For example, if a .dat includes 3rd party upstream data processed through a github author's build/scrape script(s), the comment and description (or other appropriate header fields) should contain information about _both_ those aspects of the dat's origin. If the .dat file is meant to cover a particular niche of data, the description field should explain it. -The `name` field of a `.dat` file header should match the name that is specified by the core in the [.info file for the core](https://github.com/libretro/libretro-super/tree/master/dist/info) (often but not always `manufacturer - systemname` or similar). The `description` field should be descriptive and informative about the `.dat` file's origin and purpose. +The `name` field (and filename) of a `.dat` file header should match the `database` field that is specified in the [.info file for the cores that use it](https://github.com/libretro/libretro-super/tree/master/dist/info) (often but not always `Manufacturer - Systemname` or similar). The `description` field should be descriptive and informative about the `.dat` file's origin and purpose. ## Repository Contents