CHD Disk Image Format (MAME)

Members research, findings and information that can be useful towards the PlayStation 1.
null
Active PSXDEV User
Active PSXDEV User
Posts: 61
Joined: Dec 18, 2021
PlayStation Model: SCPH-1002

Post by null » October 26th, 2022, 3:43 am

Another annoying bug of CHD/CHDMAN.

http://github.com/mamedev/mame/issues/10308

User avatar
nocash
Verified
PSX Aficionado
PSX Aficionado
Posts: 541
Joined: Nov 12, 2012
Contact:

Post by nocash » October 26th, 2022, 9:19 am

It doesn't store CUE Index values?
Yes, that's one of the basic flaws. The CHD format as such could store them in the subchannel data, but CHDMAN is just ignoring most of the CUE Index values.

One could convert CUE/BIN to TOC/BIN (the BIN with 990h-byte sectors) and then to CHD. If there's no tool that can do that: Convert CUE/BIN to CCD/IMG/SUB and then to TOC/BIN (maybe such tools exist) and then to CHD. That kind of toolchains are likely to imply conversion errors, but it should work in theory (except that CHD will then happily erase all Index 0 sectors).

CHD is a rather lossy compression format, and it's a bit silly to use lossless FLAC audio compression without properly preserving the more basic subchannel data. And it's totally overcomplicated with the metadata stuff with different sector sizes. It would have been easier to store all sectors always in 930h+60h byte format (and to store the TOC also in that same format instead of coming up with metadata stuff).

Anyways, I've experimented with a few more sector sizes... and checked how they are stored in the old CHCD metadata format... that should be fully solved now:

Code: Select all

CHCD Metadata (94Ch bytes, plus 10h-byte metadata header)
  000h 4     Number of tracks (N) (1..99)
  004h N*18h Track entries
  ...  ..    Zeropadding to 94Ch-byte size (when less than 99 tracks)
 Track entries:
  000h 4     Track Type       (0..7, CHCD=# in table below) (eg. 6=MODE2_RAW)
  004h 4     Subchannel Type  (0=RW, 1=RW_RAW, 2=None)
  008h 4     Sector Size      (800h, 914h, 920h or 930h)
  00Ch 4     Subchannel Size  (0 or 60h)
  010h 4     Number of Frames (aka number of sectors)
  014h 4     Padding Frames   (0..3) (to make Total Frames a multiple of 4)
Track types:

Code: Select all

 For TYPE and PGTYPE (and CHCD numeric type 0..7):
  "MODE1/2048" or "MODE1"                   CHCD=0  800h-byte ;\Data Mode1
  "MODE1/2352" or "MODE1_RAW"               CHCD=1  930h-byte ;/
  "MODE2/2336" or "MODE2"          ;\dupe?  CHCD=2  920h-byte ;\.
  "MODE2/2336" or "MODE2_FORM_MIX" ;/       CHCD=5  920h-byte ;
  "MODE2/2048" or "MODE2_FORM1"             CHCD=3  800h-byte ; Data Mode2
  "MODE2/2324" or "MODE2_FORM2"             CHCD=4  914h-byte ;
  "MODE2/2352" or "MODE2_RAW" or "CDI/2352" CHCD=6  930h-byte ;/
  "AUDIO" (stored as big-endian samples!!!) CHCD=7  930h-byte ;-Audio CD-DA
I think I do now understand all relevant CDROM-related parts of the CHD format. Well, except two small details:

For tracks, is there a difference between "MODE2" and "MODE2_FORM_MIX"?
For subchannels, what exactly is "RW" versus "RW_RAW"?

There are two common ways for storing subchannels in 60h bytes, with differently arranged/interleaved bits (or "cooked" bits as they are called in CHD, whatever "cooking" means in that context").
In the CLAYS sample that I had uploaded some days ago, I've marked the subchannels as "RW" and stored them as 60h-byte array in this format:

Code: Select all

00 00 00 00 00 00 00 00 00 00 00 00   ;P
01 01 01 00 00 00 00 00 02 00 5A 28   ;Q (for first sector at 00:02:00)
00 00 00 00 00 00 00 00 00 00 00 00   ;R
00 00 00 00 00 00 00 00 00 00 00 00   ;S
00 00 00 00 00 00 00 00 00 00 00 00   ;T
00 00 00 00 00 00 00 00 00 00 00 00   ;U
00 00 00 00 00 00 00 00 00 00 00 00   ;V
00 00 00 00 00 00 00 00 00 00 00 00   ;W
Though I guess there's a 50:50 chance that it's exactly the wrong format, and maybe above should have been called "RW_RAW"? Getting that wrong would be a major mistake. It would be nice to have a test case for that... but that might be difficult...

Quick question: Do you know a way to create or test CHD files with subchannel data?

The problem is that CHDMAN doesn't check if RW or RW_RAW contain the expected data format. If you feed it with wrong data then it will just create a compressed file with that data. One could test a CHD with subchannel data with emulators that do support and use and require subchannel data (eg. for libcrypted games). And then hope that the emulator is actually distinguising RW and RW_RAW correctly as it was originally intended by the CHD developer.

To unravel what was intended:
CHD is just adopting the format/names RW and RW_RAW from CDRDAO's TOC files.
CDRDAO has apparently adopted the format/names RW and RW_RAW from SCSI, ATAPI, or ASPI drivers.

CHD defines RW and RW_RAW as so:

Code: Select all

  "RW"         normal "cooked" 96 bytes per sector
  "RW_RAW"     raw uninterleaved 96 bytes per sector
CDRDAO defines RW and RW_RAW as so:

Code: Select all

  RW: packed R-W sub-channel data (96 bytes, L-EC data will be generated if required),
  RW_RAW: raw R-W sub-channel data (interleaved and L-EC data already calculated, 96 bytes).
That, hmmm, going by that definitions, the terms "uninterleaved" and "interleaved" are both referring to RW_RAW, that is perfectly unclear - help!

null
Active PSXDEV User
Active PSXDEV User
Posts: 61
Joined: Dec 18, 2021
PlayStation Model: SCPH-1002

Post by null » October 26th, 2022, 12:57 pm

nocash wrote: Do you know a way to create or test CHD files with subchannel data?
If you mean like the .SBI, I guess it's not possible. You need some external file, you cannot store it inside the .CHD.

http://forum.fobby.net/index.php?t=msg&goto=6645
wiki.recalbox.com wrote: For PS1 games protected by LibCrypt, normally you have SBI (Subchannel Information) files, you keep them and put them with the CHDs, otherwise your games will not pass.
http://wiki.recalbox.com/en/tutorials/u ... ion/chdman

User avatar
nocash
Verified
PSX Aficionado
PSX Aficionado
Posts: 541
Joined: Nov 12, 2012
Contact:

Post by nocash » October 26th, 2022, 3:30 pm

I meant the whole subchannel data, for example something like CloneCD's .SUB files. Some Libcrypted games might be dumped in that format.

The .SBI format contains only a summary of the special Libcrypt related sectors. But yes, one could also create yet another conversion tool for converting .SBI to .SUB, and then do further conversions to end up with a hopefully intact CHD file.

Ideally, CHDMAN (or any CHDMAN-clone) should do all those things automatically, without needing external tools to convert CUE/BIN/SBI to CCD/IMG/SUB to TOC/BIN to CHD.

The downside is that CHD cannot filter-out standard subchannel values, so the subchannel data won't compress very well (it would become a lot bigger than the SBI file).

I am getting the feeling that CHD is only rhetorically supporting subchannels - without ever having used that feature in practice. As far as I remember, the https://github.com/mamedev/mame/issues/10308 page mentioned that even MAME cannot read subchannel data from CHD files.

null
Active PSXDEV User
Active PSXDEV User
Posts: 61
Joined: Dec 18, 2021
PlayStation Model: SCPH-1002

Post by null » October 27th, 2022, 5:31 pm

nocash wrote: For tracks, is there a difference between "MODE2" and "MODE2_FORM_MIX"?
I guess the difference is the sub-header?

In the CDRDAO.
MODE2:2336 bytes
MODE2_FORM_MIX:2336 bytes including the sub-header

Sub-Headers
http://www.musoware.com/wiki/index.php? ... ub-Headers

So for now we can classify CHD as a lossy compression format until they fix the bug for the audio track indexes beyond 01 in the extracted .CUE? I don't know why they're not alarmed by this bug, looks like the team knew about it and still not fixing it since CHDv3? How come they can tolerate it if the purpose of MAME is about preserving things?

User avatar
nocash
Verified
PSX Aficionado
PSX Aficionado
Posts: 541
Joined: Nov 12, 2012
Contact:

Post by nocash » October 27th, 2022, 9:09 pm

null wrote: October 27th, 2022, 5:31 pm I guess the difference is the sub-header?
In the CDRDAO.
MODE2:2336 bytes
MODE2_FORM_MIX:2336 bytes including the sub-header
The sentence "including the sub-header" is just there for confusion (it's pretty much impossible not to include the sub-header in those sectors). The sub-header is stored twice in MODE2 sectors (as sub-header, and copy of sub-header):
http://problemkaputt.de/psxspx-cdrom-se ... coding.htm
null wrote: October 27th, 2022, 5:31 pm Sub-Headers
http://www.musoware.com/wiki/index.php? ... ub-Headers
Not that! That seems to be for album covers or streaming service or MP3 tags or whatever. CDROM sub-headers are 4-byte tall (most importantly used to distinguish between normal data sectors and XA-ADPCM audio sectors):
http://problemkaputt.de/psxspx-cdrom-xa ... rleave.htm
null wrote: October 27th, 2022, 5:31 pm So for now we can classify CHD as a lossy compression format until they fix the bug for the audio track indexes beyond 01 in the extracted .CUE?
It's not only the Index 0 sectors that are lost, it's also the whole Lead-In area that's missing, and that can contain things like Disc Type, First and Last Track number, Lead-Out location, Session info, CATALOG, ISRC, CD-TEXT, those are rarely needed, but any half-decent CDROM format is able to store that information. Some formats are also storing Error flags (I've never dealt with that, but it might be important for some consoles other than PSX).
null wrote: October 27th, 2022, 5:31 pm I don't know why they're not alarmed by this bug, looks like the team knew about it and still not fixing it since CHDv3? How come they can tolerate it if the purpose of MAME is about preserving things?
That would require a good understanding of what is on CD, how is it stored on a CD, what can be dumped, how is that info stored in CUE/CCD/NRG/TOC/MDS/etc. files. Plus, they cannot know what is currently stored in CHD files (until I've finished writing the documentation for them). And, where they are now is that they don't even know how to compile the CHDMAN source code without linking in unneccessary crap (the EXE should be 250Kbytes max and work on any computer).

null
Active PSXDEV User
Active PSXDEV User
Posts: 61
Joined: Dec 18, 2021
PlayStation Model: SCPH-1002

Post by null » October 31st, 2022, 12:50 am

CHDMAN v0.249
You do not have the required permissions to view the files attached to this post.

null
Active PSXDEV User
Active PSXDEV User
Posts: 61
Joined: Dec 18, 2021
PlayStation Model: SCPH-1002

Post by null » November 2nd, 2022, 12:19 am

Looks like the beta prototypes in hidden palace uses CloneCD with SUB you can test some of those demos. You can download it directly in their site.

http://hiddenpalace.org/Category:PlayStation_prototypes

null
Active PSXDEV User
Active PSXDEV User
Posts: 61
Joined: Dec 18, 2021
PlayStation Model: SCPH-1002

Post by null » November 3rd, 2022, 3:40 am

Some bugs about the CDI.

[CDI/2352]/redump (CD-i) #2784
http://github.com/mamedev/mame/issues/2784
GNU Compact Disc fields wrote: Value/Description
AUDIO - Audio/Music (2352 — 588 samples)
CDG - Karaoke CD+G (2448)
MODE1/2048 - CD-ROM Mode 1 Data (cooked)
MODE1/2352 - CD-ROM Mode 1 Data (raw)
MODE2/2048 - CD-ROM XA Mode 2 Data (form 1) *
MODE2/2324 - CD-ROM XA Mode 2 Data (form 2) *
MODE2/2336 - CD-ROM XA Mode 2 Data (form mix)
MODE2/2352 - CD-ROM XA Mode 2 Data (raw)
CDI/2336 - CDI Mode 2 Data
CDI/2352 - CDI Mode 2 Data
The modes marked with ‘*’ are not defined in the original CUE sheet format specification.
http://www.gnu.org/software/ccd2cue/man ... _0029.html

null
Active PSXDEV User
Active PSXDEV User
Posts: 61
Joined: Dec 18, 2021
PlayStation Model: SCPH-1002

Post by null » November 8th, 2022, 1:35 am

nocash wrote: For tracks, is there a difference between "MODE2" and "MODE2_FORM_MIX"?
nocash wrote: The sentence "including the sub-header" is just there for confusion (it's pretty much impossible not to include the sub-header in those sectors).
According to claunia's source code MODE2 is FORMLESS while the MODE2_FORM_MIX is mixed forms. And there's some info in the Google Patents CDROM Decoder page tells that FORMLESS replaces the subheader with the 2336 bytes user data next to header. So technically there's no subheader for MODE2_FORMLESS, only MODE2_FORM_MIX have it.

http://raw.githubusercontent.com/aaru-d ... nstants.cs
http://patents.google.com/patent/KR100424230B1/en

ps. MODE2_FORMLESS is also called MODE2_FORM0

User avatar
nocash
Verified
PSX Aficionado
PSX Aficionado
Posts: 541
Joined: Nov 12, 2012
Contact:

Post by nocash » November 8th, 2022, 8:32 am

null wrote: November 8th, 2022, 1:35 am the Google Patents CDROM Decoder page tells that FORMLESS ...
Ah, yes, that might be where the two "different" formats came from...

So the formats should be identical - and the only "difference" is whether the recurding tool has made assumptions or verifications on what is stored in the file.

Code: Select all

 MODE2 "formless" --> subheader MAY contain FORM1 or FORM2 or WHATEVER info (the recording tool didn't care about that)
 MODE2_FORM_MIX   --> subheader SHALL contain FORM1 or FORM2 info           (the recording tool did think/know/verify that)
I doubt that there's a direct link from the korean hardware/firmware patent to the cdrdao format. But it might be linked indirectly (if SCSI/ASPI drivers did also use the same notations) (hardware --> driver --> cdrdao).
null wrote: October 31st, 2022, 12:50 amCHDMAN v0.249
No change, just says missing KERNEL32.DLL:AddVectoredExceptionHandler as ever since.
null wrote: November 2nd, 2022, 12:19 am Looks like the beta prototypes in hidden palace uses CloneCD with SUB you can test some of those demos. You can download it directly in their site.
Does that help as test case? I mean, you will need some solid criminal energy to find or create a test case. Like, intentionally creating a CHD file with some kind of broken or nonstandard subchannel data, and then checking if any emulators do respond as expected when reading the corresponding sectors.
The easiest test cases I could think of are:
  • Libcrypted PSX games.
  • Audio Discs with some "visible" glitch (eg. flipping Track number 1-2-1-2-1-2-1-2-1-2-1-2 during playback).
But again, that won't work without criminal energy: It may require finding an emulator that can play CHD files as audio discs and using forged subchannel checksums or investigating when the libcrypt protection kicks (such like: not until reaching level 2 of a game).

Or, try to find people who do still own old computers with physical CDROM drive and CDRDAO software, and try to ask them if they could record a regular CDROM or Audio disc as TOC/BIN image with "--read-subchan" option (albeit risking to get told to go fuck yourself).
null wrote: November 3rd, 2022, 3:40 am Some bugs about the CDI.

[CDI/2352]/redump (CD-i) #2784
http://github.com/mamedev/mame/issues/2784
GNU Compact Disc fields wrote: Value/Description
AUDIO - Audio/Music (2352 — 588 samples)
CDG - Karaoke CD+G (2448)
MODE1/2048 - CD-ROM Mode 1 Data (cooked)
MODE1/2352 - CD-ROM Mode 1 Data (raw)
MODE2/2048 - CD-ROM XA Mode 2 Data (form 1) *
MODE2/2324 - CD-ROM XA Mode 2 Data (form 2) *
MODE2/2336 - CD-ROM XA Mode 2 Data (form mix)
MODE2/2352 - CD-ROM XA Mode 2 Data (raw)
CDI/2336 - CDI Mode 2 Data
CDI/2352 - CDI Mode 2 Data
The modes marked with ‘*’ are not defined in the original CUE sheet format specification.
http://www.gnu.org/software/ccd2cue/man ... _0029.html
I can't read the bug report on the github page... and the post that you've quoted doesn't mention any bugs?

null
Active PSXDEV User
Active PSXDEV User
Posts: 61
Joined: Dec 18, 2021
PlayStation Model: SCPH-1002

Post by null » November 8th, 2022, 1:44 pm

nocash wrote: I can't read the bug report on the github page... and the post that you've quoted doesn't mention any bugs?
The CHDMAN didn't want to accept the CDI/2352 track in a CUE. The error message is "ERROR: Unknown track type [CDI/2352]. Contact MAMEDEV". The workaround is to change the CDI/2352 to MODE2/2352 they say but it looks no one tested it yet so I can't proved if this is a working FIX solution. The issue is still open so there's no hard-coded fix yet in the latest CHDMAN. It's just a minor bug anyway but still a bug.

User avatar
nocash
Verified
PSX Aficionado
PSX Aficionado
Posts: 541
Joined: Nov 12, 2012
Contact:

Post by nocash » December 17th, 2022, 12:40 pm

I`ve released no$psx v2.2 with CHD support.
And the specs for the CHD format can be found here:
http://problemkaputt.de/psxspx-cdrom-di ... d-mame.htm

Post Reply

Who is online

Users browsing this forum: No registered users and 6 guests