CHD Disk Image Format (MAME)

[nocash]

Mono Audio Tracks (with CHD V5 FLAC)
That would be also interesting for testing. The overall CHD FLAC data is always 2 channels (stereo), but it could also contain FLAC Frames with only 1 channel. The simple FLAC decoder doesn't handle that case, and I don't know how it should be handled (only channel 0 used and channel 1 silent, or same output to both channels?), and I don't know if CHDMAN is ever creating such single channel FLAC frames for mono tracks.

[null]

Btw. if you have some conversion tool around, it would be interesting to have the 1-2-3-4 test disc also in formats like CCD, CDI, MDS, NRG, PBP, CUE/BIN and TOC/BIN (other than CHD's cue/toc format) or in any other formats you can think of.

Mono Audio Tracks (with CHD V5 FLAC)?

The shortest seconds of audio cd that can be compressed by CHDMAN is 1.254sec enough for 3 syllables to fit in.

The only format that can be converted to PBP from an audio cd source is CloneCD (CCD/IMG/SUB).

CDemu and CHDMAN are the only app that can convert ROM images to TOC/BIN. CDemu only works in Linux and I don't have that OS.

DiscJuggler is the only app that can output CDI. DiscJuggler only works in old Windows OS like win98, newer OS are not supported anymore and their company stops developing the app for almost 20 years now.

[nocash]

Cool, thanks! That looks like a complete collection of file formats (apart from the missing CDI format).

Testing the various formats in no$psx, there seem to be some issues (missing gaps between tracks, or completely missing track 2-4). I'll look more into that later... it's very well possible that I have some bugs in no$psx... although - what conversion tool did you use? - it might be also possible that the tool did omit some gaps and track numbers.

The mono chd files are working okay with my decoder (=the chd files aren't using any single-channel flac frames).

Good to know that DiscJuggler works on win98. Hmmm, my cdrom drives are buried underneath a stack with my other homeless belongings : / so I can't currently dump a CDI image from real CDRs. But at least I can burn the CUE/BIN on some other PC, which will be useful for testing some audio playback related things on real PSX hardware.

[null]

I used the UltraISO, Alcohol 120%, CDmage and VLC. Other formats that are available are not capable of converting audio CD.

[nocash]

Okay... that are quite a lot of tools, I am not quite sure which tool(s) where used for which files...
(originating from the CUE, it might matter if you had first converted the CUE to a corrupted NRG file, and then converted the corrupted NRG to MDS format, or the like).

I've examined the files via hex editor, and I am 99% sure that the MDS and NRG don't contain pregaps for track 2-4.

The CCD is ascii format, which should be self-explaining, but I guess nobody knows what those ascii strings are meaning in practice... the ALBA entries suggest 150 sector pregap(s), but I am not sure if that means one such gap counted from begin of disc, or one such gap at begin of each track... you could test that by changing the CUE file to use PREGAP 00:04:00 on TRACK 03, then convert to CCD, and check if it's different than the old CCD file.
And same for the NRG and MDS files (just in case I've been wrong when saying that they don't contain pregaps).

CCD has an unknown "PreGapMode" entry with value 0, 1 or 2. One theory is that it does somehow define if/how pregaps are stored in the IMG file... or, another new theory: It might mean that the pregap of 1st track is 0=Audio, 1=Mode1, 2=Mode2. I don't have CloneCD, but that theory could be easily verified by recording some different disc types with CloneCD.

For Alcohol .MDS, I've looked at the Alcohol webpage... and now comes the big triumph: If you stayed out with win9x long enough, then you can download the Alcohol Retro version available for all win9x users for free!
I've downloaded that, it doesn't specifically have "Open .CUE + Save as .MDS" functions, but the following 3 steps are doing it:
- Open .CUE (this does merely add the CUE to a list of "bookmarked" files)
- Mount .CUE as virtual disc (by right-clicking the "bookmarked" file)
- Use "Image Making Wizard" to convert it to MDS/MDF format
The resulting MDF file does contain actual sectors for the pregaps (the binary is a good bit taller than the BIN file).
The only problem is that I don't understand if or how the MDS file is assigning index values to those gaps... it looks as if it's accidentally appending the pregaps as "postgaps" to the preceeding track?

And concerning CHD and normal CUE files:

The CUE files with PREGAPs are now playing fine, without the 2-3-4-hnnn glitch.

And for CHD, I've also implemented the "huff" compression - which is quite simple, but it's different from the map's huffman compression (it's using two huffman trees: the first tree used to create the second tree, and the second tree used to decompress the actual data).

That leaves only "avhu" unsupported... which seems to combine 2-3 different compression methods & versions for some kind of audio and video sectors... I don't know if it's even possible to test all that data format(s) with chd cdrom images.

[null]

Are there any errors in multi track BIN/CUE? Or any corrupted files? I'll try to fix that.

The .CUE results from UltraISO and CDmage are exactly the same, they automatically put pregap values on their own.

About the .SUB, according to some CloneCD format users they can convert the images they made into CHD so it's possible that the CHDMAN supports .SUB files.

I'm still on the process of listing all CHDs with different hunk sizes.

[nocash]

This is all about the 4-track 1-2-3-4 stereo test files:

Intact: The CUE/BIN version
Intact: The V4 CHD version
Intact: The V5 CHD version
Corrupted: The MDS version (missing pregaps)
Corrupted: The NRG version (missing pregaps)
Corrupted: The PBP version (missing pregaps)
Don't know: The CCD version (I don't know that file contains pregaps or not)

To confirm if the pregaps are really missing:
I assume you have used the CUE/BIN as source file, and somehow converted it to MDS,NRG,PBP,CCD.
If you remember how did that... could you repeat that... but this time with the CUE's TRACK 03 PREGAP set to 00:04:00 instead 00:02:00?

If the files contain some different byte(s) then those bytes must somehow encode the pregap info.
If the files are really corrupted (without pregaps) then you should get exactly the same results as from the old CUE file.

---

For CHD, my decoder is now inserting blank sectors in the pregap region, that's working okay & I can play the CHD audio tracks.
But I do rather dislike that CHD is enforcing index 0 and gaps to be replaced by blank sectors.
It would have so much easier to store those sectors as normal compressed data.
And, I think some discs do contain non-blank sectors between tracks, and CHD seems to be just replacing that by blank sectors (like blanking out applause between tracks in live recordings, I am not entirely sure if such discs are using index 0 though).

The four CHD tracks didn't exactly start where they should. It looks as CHD is including dummy sectors at the end of each track (which must be skipped to find the begin of the next track). The 1-2-3-4 disc should have 30 sectors per track, but the CHD file seems to contain 32 sectors per track...

That might be related to this mysterious CHD source code line:

Code: Select all

static constexpr uint32_t TRACK_PADDING    = 4;

But I can't see "TRACK_PADDING" being used anywhere... if somebody can browse github pages: Can you please check if https://github.com/mamedev/ is using "TRACK_PADDING" anywhere?

Another theory would be that the tracks might be padded to the hunksize. To confirm that, just make a new version of the CHD V5 1-2-3-4 audio disc with hunksize 247248 (2448*101), that should/might cause tracks to be padded to 101 sectors instead of 32 sectors (or if it's additionally padded to "TRACK_PADDING" then it would become 104 sectors).

[null]

Okay I don't know if I fixed those files but it looks like the latest Alcohol/CloneCD/UltraISO apps though they can produce their own .CUE file, they can't really make their own .CUE with proper PREGAP values.

Can you please check if https://github.com/mamedev/ is using "TRACK_PADDING" anywhere?

Only these:
https://raw.githubusercontent.com/mamed ... chdman.cpp
https://raw.githubusercontent.com/mamed ... /cdrom.cpp
But what specific question you really wanna ask about TRACK_PADDING?

And the CHD I'm going to upload later.

[null]

To confirm that, just make a new version of the CHD V5 1-2-3-4 audio disc with hunksize 247248 (2448*101)

Audio disc in CHD with 247248 hunksize.

[null]

[WRONG FILE]

[null]

List with compressed sizes for different hunk sizes for 5-10 retail games.

The TXT file contains these games below:
WIPEOUT XL
PANDEMONIUM 2
METAL GEAR SOLID
FINAL FANTASY VIII
SYPHON FILTER 1/2/3
RESIDENT EVIL 1/2/3
JOJO BIZARRE ADVENTURE
MARVEL SUPER HEROES
DRAGON BALL GT FINAL BOUT
METAL SLUG X
HUNTER X HUNTER MABOROSHI NO GREED ISLAND
HUNTER X HUNTER UBAWARETA AURA STONE

[null]

Missing CDI format?

Okay here are the CDI and CHD/TOC/CUE/BIN with POSTGAP values.

[nocash]

I got side-tracked and spent some days on LZMA2 and XZ decompression, whatever that's good for. XZ is just a mess. LZMA2 itself is just a very simple LZMA extension.

Does somebody have a small CUE/BIN image with such smaller sector sizes for testing?

The --sectorsize -ss command is available only for createhd.

No, not the CHD sector size. I've meant the CUE/BIN sector size. If you open a CUE file in text editor, PSX titles usually have "TRACK 01 MODE2/2352", what I am looking for is a test file with values smaller than "2352" (aka 930h).

Discs for PCs or consoles without CD-XA would be likely candidates (if they are recorded without error correction info).

And I've found some old notes saying that "paradox 'pdx-rh.cue' psx demo" is using smaller 920h-byte sectors, but I can't find where to download that demo.

List with compressed sizes for different hunk sizes for 5-10 retail games.

The TXT file contains WIPEOUT XL, PANDEMONIUM 2, METAL GEAR SOLID ...

I can't see the results for "default hunksize" in the TXT, did you forget those?
And less important, if you should have all CHDs in one folder, typing "dir >chd_list.txt" at command prompt would be neat to generate a summary of filenames + filesizes.

Can you please check if https://github.com/mamedev/ is using "TRACK_PADDING" anywhere?

Only these:
https://raw.githubusercontent.com/mamed ... chdman.cpp
https://raw.githubusercontent.com/mamed ... /cdrom.cpp
But what specific question you really wanna ask about TRACK_PADDING?

Thanks! The question was just if TRACK_PADDING is used anywhere (and if so, what it is used for).
Looking at the above source code, it's apparently padding the metadata "FRAMES:value" to a multiple of 4 sectors and then including those padding sectors in the compressed data.
Which is very confusing because it doesn't make any sense to do that, but apparenly CHD is doing so anyways.

To confirm that, just make a new version of the CHD V5 1-2-3-4 audio disc with hunksize 247248 (2448*101)

Audio disc in CHD with 247248 hunksize.

That's very useful! And brings up some more corner cases:

• The "logical size" entry in the CHD header isn't a multiple of the hunksize (causing big problems when not handling that case)
• I'll need to test if that "logical size" implies that the last hunk is smaller than hunksize
• I'll need to allocate a bigger FLAC decoding buffer for such large hunksizes (I've had that on todo list, but now it's time to really do so)

[null]

I got side-tracked and spent some days on LZMA2 and XZ decompression, whatever that's good for.

Nice, it can be use for 7z and for the successor of CHDv5.

What I am looking for is a test file with values smaller than "2352" (aka 930h).

Okay noted, I'll try find that.

I can't see the results for "default hunksize" in the TXT, did you forget those?

Yes it looks like, I'm going upload the TXT file later.

[nocash]

Got the "Audio disc in CHD with 247248 hunksize" working!
It's confusing, but I think I've sorted out how it works:

Code: Select all

CHD Track/Hunk Padding and Missing Index0 sectors
A normal CDROM contains a series of sectors. The CHD format is violating that
in several ways: It's removing Index0/Pregap sectors, and it's instead
inserting dummy/padding sectors between tracks.
  Track        <---- Track1---------> <---- Track2---------> <--End-->
  Section      Index0 IndexN TrackPad Index0 IndexN TrackPad HunkPad
  Real Disc    Yes    Yes    -        Yes    Yes    -        -
  CHD Header   -      Yes    Yes      -      Yes    Yes      -
  CHD Data     -      Yes    Yes      -      Yes    Yes      Yes
That is, the critical parts are:
  Index0/pregap:  Metadata PREGAP:sectors isn't stored in compressed data
  Track padding:  Metadata FRAMES:sectors is rounded up to N*4 sectors
  Hunk padding:   The last hunk is additionally rounded up to hunksize
Missing Index0 might be a problem if a disc contains nonzero data between
tracks (like audio discs with applause in Index0 periods).
Track padding is total nonsense. The final hunk padding makes sense (but 
confusingly that extra padding isn't included in the uncompressed size entry
in CHD header).

I got side-tracked and spent some days on LZMA2 and XZ decompression, whatever that's good for.

Nice, it can be use for 7z and for the successor of CHDv5.

I am already terrified by that thought. The existing CHD versions are such a mess, I am afraid that it could only get worse when adding more corner cases in future versions : /
If they are adding LZMA2 or Zstandard, I hope they will first test if it's really getting smaller or faster with those compression methods...

LZMA2 could make sense for breaking large files into smaller fragments with different compression parameters. But CHD is already breaking the disc into smaller hunks, the compression ratio might get better or worse with the extra LZMA2 overload.

Zstandard source code looks extremely complicated. If I get it right, the raw decompressor is about 5000 lines tall? It would be a pain to implement that. And I wonder if the sheer code size is suitable for small hunks I haven't tried Zstandard, but speed & ratio might be optimized for bigger compression blocks(?) or maybe it's really as good as claimed.

[null]

paradox 'pdx-rh.cue' psx demo

I guess that's the Rest Home by Paradox?

The ZIP contains:
CHD list with default hunksize
Rest Home by Paradox

[null]

Unrelated to .CHD but for .ISZ format. Some missing information about .ISZ format in your website.

PSXSPX CDROM Disk Images Other Formats
.ISZ
This is reportedly a "zipped" .ISO file, using some unknown compression format (unrelated to pkzip .zip file format).

.ISZ is made by UltraISO EZB Systems. They also put the file format specification TXT in their website.

ISZ format
UltraISO uses a proprietary format known as ISZ. The format is advertised as "ISO Zipped", even though it is not a simple zip archive. The format uses zlib or bzip2 to compress the data, and may use AES-128,192 or 256 encryption in the CBC mode (note that this provides no integrity protection and is vulnerable to the padding oracle attack). The file format specification is available publicly on EZB Systems's website. The format is now supported by third party applications such as Daemon Tools, Alcohol 120% and CDemu.

http://www.ezbsystems.com/isz/iszspec.txt
http://en.wikipedia.org/wiki/UltraISO?u ... ISZ_format

[nocash]

.ISZ is made by UltraISO EZB Systems. They also put the file format specification TXT in their website.

Good to know.
Hmmm, UltraISO? I guess that's crap (going by the corrupted audio test files that you had made with UltraISO).

But they have documentation! That's more thoughful than the CHD approach with "design the file format by fiddling around in source code".

Looking through the ISZ TXT file... it doesn't mention filtering out ECC or Error Correction. Oh, and there is this sentence "('sect_size' is always 2048 for ISO CD/DVD images)". Apparently it can't filter out the ECC info because it doesn't even support sectors with ECC info.

CHD list with default hunksize

That will be very useful to compare the filesizes. Ah, what about the DIR listing with just filenames and filesizes?
I could write a parser for extracting that info from your lists, but it might be unneccessary extra work if you can just type DIR on your computer.

Rest Home by Paradox

Thanks! That has those 920h-byte sectors. Oh, I seem to have screwed up support for such 920h-byte CUE/BIN files when "improving" my PREGAP support last week. Okay, fixed that, it's working again.

The Rest Home demo seems to have real error correction (better than the Deathball demo). But there's also one weird bug: Rest Home has the Licence.txt and Logo.tmd sectors stored on A40h-byte (!!!) boundaries (instead of the 920h-byte sector boundaries).

The decompressed CHD sectors are always 990h bytes tall. So the interesting part is where are the 920h bytes stored in that 990h byte region. Which is.. apparently so:

Code: Select all

000h..91Fh   920h-byte Sector Data
920h..98Fh   70h-byte Zerofilled

Uhm, that info isn't helpful, because the 60h-byte Subchannel data could be stored anywhere in the 70h-byte Zerofilled area.

Time to wonder about how to create CHD files with Subchannel data.
My first thought was that CHDMAN would presumably support "File.CUE" bundled with "File.SUB".
No, apparently not.
I am not aware of any official or inoffical ways to specify Subchannel data in CUE files. Or does that exist?

Anyways, that weird TOC/BIN format did support Subchannel data, so we seem to be back at CHDMAN enforcing TOC/BIN format. That's crazy... but there's a gbatemp thread that confirms that one must write a tool that converts CCD/IMG/SUB to TOC/BIN in order to create CHD files with subchannels.

Okay, I don't have CloneCD, and the required test case would be 60h-byte subchannel with less than 930h-byte sector data (and I doubt that CloneCD could create that combination)... so I've written my own tool for creating TOC/BIN test images.

The attached CLAYS.ZIP contains:

Code: Select all

CLAYS.CUE and CLAYS.ISO - old homebrew PSX game
CLAYS_*.TOC - TOC files for different sector sizes
CLAYS.EXE - tool for converting CLAYS.ISO to different .BIN files (for use with above .TOC files)
CLAYS.ASM - source code for CLAYS.EXE
CLAY.BAT - sample batch for converting the TOC/BIN files to CHD with different CHDMAN versions

CHDMAN lets me test v0.145 and older on my PC. The resulting CHD filesizes are:

Code: Select all

       331,470 bytes  20.10.22  17.43 clays_mode1_800h_chdman_v0.145.chd
       331,476 bytes  20.10.22  17.43 clays_mode2_800h_chdman_v0.145.chd
       336,811 bytes  20.10.22  17.43 clays_mode2_860h_chdman_v0.145.chd
       467,281 bytes  20.10.22  17.43 clays_mode2_920h_chdman_v0.145.chd
       469,346 bytes  20.10.22  17.43 clays_mode2_930h_chdman_v0.145.chd
       471,779 bytes  20.10.22  17.43 clays_mode2_980h_chdman_v0.145.chd
       473,474 bytes  20.10.22  17.43 clays_mode2_990h_chdman_v0.145.chd

There, one can neatly see that old CHD versions are reaching worst-case compression ratios for BIN files with sectors bigger than 860h bytes (due to the missing ECC-filter).

And, looking into where the subchannel data is stored in the decompressed 990h-byte area:

Code: Select all

For 980h-byte sectors:
000h..91Fh   920h-byte Sector Data
920h..97Fh   60h-byte Subchannel Data
980h..98Fh   10h-byte Zerofilled

And for 860h-byte sectors:
000h..7FFh   800h-byte Sector Data
800h..85Fh   60h-byte Subchannel Data
860h..98Fh   130h-byte Zerofilled

That isn't directly problematic for old V3/V4 versions. But I suspect that V5 is still using the same data arrangement (whilst simultanously trying to split it into separate 930h + 60h sections and trying to ECC-filter the 930h byte section).

What's needed now...
Please get attached CLAYS.ZIP file, and paste the commands below to windows 64bit command prompt:

Code: Select all

REM below will convert clays.iso to clays_*.bin with different sector sizes
clays.exe

REM below v0.247 does REQUIRE WINDOWS 64bit
chdman_v0.247.exe createcd -i clays_mode1_800h.toc -o clays_mode1_800h_chdman_v0.247.chd
chdman_v0.247.exe createcd -i clays_mode2_800h.toc -o clays_mode2_800h_chdman_v0.247.chd
chdman_v0.247.exe createcd -i clays_mode2_860h.toc -o clays_mode2_860h_chdman_v0.247.chd
chdman_v0.247.exe createcd -i clays_mode2_920h.toc -o clays_mode2_920h_chdman_v0.247.chd
chdman_v0.247.exe createcd -i clays_mode2_930h.toc -o clays_mode2_930h_chdman_v0.247.chd
chdman_v0.247.exe createcd -i clays_mode2_980h.toc -o clays_mode2_980h_chdman_v0.247.chd
chdman_v0.247.exe createcd -i clays_mode2_990h.toc -o clays_mode2_990h_chdman_v0.247.chd

Of course, first rename "chdman.exe" to "chdman_v0.247.exe" (or rename above commands instead).
And upload the resulting seven CHD files, please!

I bet there will a "nasty surprise" for the compression ratio of the 920h and 980h byte files.
And probably some harmless but unintended quirks for the subchannel compression in 860h and 980h byte files.

[null]

Okay these are the CLAYS CHDs including DIR.

[nocash]

That CHDs are looking as expected.

The ECC-Filter doesn't work for 920h/980h byte sectors, resulting in very bad compression ratio.

CHDMAN thinks that it stores the Subchannel-data in a separate Deflated block, but it doesn't actually do so for 860h/980h byte sectors:

Code: Select all

For 980h-byte sectors:
000h..91Fh   920h-byte Sector Data      ;\LZMA/Deflate
920h..92Fh   10h-byte Subchannel Data   ;/
930h..97Fh   50h-byte Subchannel Data   ;\Deflate (intended to contain Subchannel Data)
980h..98Fh   10h-byte Zerofilled        ;/

And for 860h-byte sectors:
000h..7FFh   800h-byte Sector Data      ;800h..85Fh   60h-byte Subchannel Data   ; LZMA/Deflate
860h..92Fh   D0h-byte Zerofilled        ;/
930h..98Fh   60h-byte Zerofilled        ;-Deflate (intended to contain Subchannel Data)

To confirm that, I've zerofilled the bytes at 930h-98Fh right after Inflating the 860h-byte sector, and the decompression is still working with correct CRC16 for 000h-98Fh. So, it's working, but it does probably slightly hurt the compression ratio (and it's somewhere between amusing and confusing when trying to understand the CHD format).