CHD Disk Image Format (MAME)

Members research, findings and information that can be useful towards the PlayStation 1.
User avatar
nocash
Verified
PSX Aficionado
PSX Aficionado
Posts: 541
Joined: Nov 12, 2012
Contact:

CHD Disk Image Format (MAME)

Post by nocash » August 27th, 2022, 11:45 pm

Discussing the CHD format was originally started in another thread (see this post http://www.psxdev.net/forum/viewtopic.p ... 437#p21437 and scroll up to older posts).

Code: Select all

CDROM Disk Images CHD (MAME)
----------------------------

All numbers are stored in Motorola (big-endian) byte ordering.

V1/V2 header:
  000h 08h  ID "MComprHD" (MAME Compressed Hunks of Data)
  008h 4    Length of header (4Ch=V1, 50h=V2)
  00Ch 4    Drive format version (probably 1/2 = V1/V2)
  010h 4    Flags (bit0=DriveHasParent, bit1=AllowWrites)
  014h 4    Compression type (0=None, 1=ZLIB)
  018h 4    Number of sectors per hunk
  01Ch 4    Total number of hunks represented
  020h 4    Number of cylinders on hard disk
  024h 4    Number of heads on hard disk
  028h 4    Number of sectors on hard disk
  02Ch 10h  MD5 checksum on raw data
  03Ch 10h  MD5 checksum on parent file
  N/A  -    V1: Uses fixed 200h-byte Sector size
  04Ch (4)  V2: Number of bytes per sector
  ...  ?    Supposedly followed by map and/or data at whatever locations
 V1/V2 map format (64bit entries with 44bit+20bit):
  44bit     Starting offset within the file
  20bit     Length of data; if == hunksize, data is uncompressed
 Unknown if offset is in Upper 44bit or Lower 44bit.

V3/V4 header:
  000h 08h  ID "MComprHD" (MAME Compressed Hunks of Data)
  008h 4    Length of header (78h=V3, 6Ch=V4)
  00Ch 4    Drive format version (probably 3/4=V3/V4)
  010h 4    Flags (bit0=DriveHasParent, bit1=AllowWrites)
  014h 4    Compression type (0=None, 1=ZLIB, 2=ZLIB_PLUS) (V4: 3=AV)
  018h 4    Total Number of hunks represented
  01Ch 08h  Logical size of the data (in bytes)        ;uh, logical of what?
  024h 08h  Offset to the first blob of metadata
  02Ch 10h  V3: MD5 checksum on raw data            ;\.
  03Ch 10h  V3: MD5 checksum on parent file         ;
  04Ch 4    V3: Number of bytes per hunk            ; V3
  050h 14h  V3: SHA1 checksum on raw data           ;
  064h 14h  V3: SHA1 checksum on parent file        ;/
  02Ch 4    V4: Number of bytes per hunk            ;\.
  030h 14h  V4: SHA1 checksum on raw+meta           ; V4
  044h 14h  V4: SHA1 checksum on raw+meta of parent ;
  058h 14h  V4: SHA1 checksum on raw data           ;/
  ...  ?    Supposedly followed by map, metadata, and/or data ?
 V3/V4 map format:
  000h 8    Starting offset within the file
  008h 4    CRC32 of the uncompressed data
  00Ch 2    Lower 16 bits of length     ;\mixed-endian?
  00Eh 1    Upper 8 bits of length      ;/
  00Fh 1    Flags, indicating compression info (=whut? maybe below V34 stuff?)
V34_MAP_ENTRY_FLAG_TYPE_MASK = 0x0f;     // what type of hunk
V34_MAP_ENTRY_FLAG_NO_CRC    = 0x10;     // no CRC is present
V3-V4 entry types
  V34_MAP_ENTRY_TYPE_INVALID        = 0  invalid type
  V34_MAP_ENTRY_TYPE_COMPRESSED     = 1  standard compression
  V34_MAP_ENTRY_TYPE_UNCOMPRESSED   = 2  uncompressed data
  V34_MAP_ENTRY_TYPE_MINI           = 3  mini: use offset as raw data
  V34_MAP_ENTRY_TYPE_SELF_HUNK      = 4  same as another hunk in this file
  V34_MAP_ENTRY_TYPE_PARENT_HUNK    = 5  same as a hunk in the parent file
  V34_MAP_ENTRY_TYPE_2ND_COMPRESSED = 6  compressed with secondary algorithm
                                         (2nd algorithm is usually FLAC CDDA)

V5 header:
  000h 8    ID "MComprHD" (MAME Compressed Hunks of Data)
  008h 4    Length of header (7Ch=V5)
  00Ch 4    Drive format version (05h=V5)
  010h 4x4  Which custom compressors are used?
  020h 8    Logical size of the data (in bytes)
  028h 8    Offset to the map                    (3D797h)
  030h 8    Offset to the first blob of metadata (7Ch)
  038h 4    Number of bytes per hunk (512k maximum)
  03Ch 4    Number of bytes per unit within each hunk
  040h 14h  SHA1 on raw data
  054h 14h  SHA1 on raw+meta
  068h 14h  SHA1 on raw+meta of parent (0=No parent)
  ...  ..   Metadata Chunk(s)
  ...  ..   Compressed Sectors (aka hunks)
  ...  ..   Map
If compressors[0] == 0, we are uncompressed (including maps)
 V5 uncompressed map format:
  000h 4    32bit offset; // starting offset / hunk size   ;=whut? ;"/"=divide?
 V5 compressed map format header:
  000h 4    Length of compressed map
  004h 6    Offset of first block (48bit)     (E4h, after meta)
  00Ah 2    CRC16 of the map
  00Ch 1    bits used to encode complength
  00Dh 1    bits used to encode self-refs
  00Eh 1    bits used to encode parent unit refs
  00Fh 1    Reserved for future use (probably zero)
  ... probably followed by some kind of bitstream with whatever values
  ...   GUESS: 2bit compression type
  ...   GUESS: Nbit compression length
  ...   GUESS: whatever self-refs and parent unit refs?
  ...   GUESS: 16bit CRC16
  ... (which will result in below info; when expanding Nbit to 24bit, etc.)
 Each compressed map entry, once expanded, looks like:
  000h 1    Compression type
  001h 3    Compressed length (24bit)
  004h 6    Offset (48bit)             (=sum of base+preceeding lengths?)
  00Ah 2    CRC16 of the data          (before/after decompression?)
 V5 compression codecs:
  0,0,0,0 = CHD_CODEC_NONE        ;-uncompressed data
  "zlib" = CHD_CODEC_ZLIB         ;\.
  "lzma" = CHD_CODEC_LZMA         ; general codecs
  "huff" = CHD_CODEC_HUFFMAN      ;
  "flac" = CHD_CODEC_FLAC         ;/
  "cdzl" = CHD_CODEC_CD_ZLIB      ;\.
  "cdlz" = CHD_CODEC_CD_LZMA      ; general codecs with CD frontend
  "cdfl" = CHD_CODEC_CD_FLAC      ;/
  "avhu" = CHD_CODEC_AVHUFF       ;-A/V codecs

V3/V4/V5 Metadata
Overall Metadata chunk format:
  000h 4   Chunk ID (aka Blob Tag) (eg. "CHT2" for each CDROM track)
  004h 1   Unknown (1) maybe some kind of flag/type?
  005h 1   Unknown (0)   ;\maybe MSB(s) of below size?
  006h 1   Unknown (0)   ;/
  007h 1?  Chunk Data Size
  008h 8   Offset to next Chunk (or 0=None, in last chunk)
  010h ..  Chunk Data (eg. "TRACK:1 TYPE:MODE2_RAW ... POSTGAP:0",00h for CHT2)
There can be one or more chunks (a CDROM image has one CHT2 chunk per track).
Summary of Chunk IDs and corresponding Data entries:
  ID_______Data_______________________________________________________
  "GDDD"   "CYLS,HEADS,SECS,BPS"         ;-hard disk standard info
  "IDNT"   ?                             ;-hard disk identify info
  "KEY "   ?                             ;-hard disk key info
  "CIS "   ?                             ;-pcmcia CIS info
  "CHCD"   ?                                                           ;\.
  "CHTR"   "TRACK TYPE SUBTYPE FRAMES"                                 ; CD-ROM
  "CHT2"   "TRACK TYPE SUBTYPE FRAMES PREGAP PGTYPE PGSUB POSTGAP"     ;/
  "CHGT"   ?                                                           ;\Sega
  "CHGD"   "TRACK TYPE SUBTYPE FRAMES PAD PREGAP PGTYPE PGSUB POSTGAP" ;/GD-ROM
  "AVAV"   "FPS.%06d WIDTH HEIGHT INTERLACED CHANNELS SAMPLERATE"      ;\AV
  "AVLD"   ? (A/V Laserdisc frame)                                     ;/
 The ASCII items are separated by spaces as shown above (or commas for GDDD).
 The last item in each chunk is terminated by 00h (at least so for CHT2).
 Most items are followed by a colon and decimal string (eg. TRACK:1), except,
 TYPE,PGTYPE,SUBTYPE,PGSUB are followed by text strings (eg. TYPE:MODE2_RAW).
  CYLS:#          Hard disc number of cylinders
  HEADS:#         Hard disc number of heads
  SECS:#          Hard disc number of sectors
  BPS:#           Hard disc bytes per sector
  TRACK:#         CDROM current track number (1..99)
  TYPE:string     CDROM sector type/size
  SUBTYPE:string  CDROM subchannel info (usually "NONE")
  FRAMES:#        CDROM number of sectors per track (with/without pregap?)
  PAD:#           Sega GDROM only: whatever pad value?
  PREGAP:#        CDROM ... maybe number of pregap sectors? (can be HUGE !!??)
  PGTYPE:string   CDROM ... whatever type?                  (usually "MODE1"??)
  PGSUB:string    CDROM ... whatever subchannel             (usually "RW"??)
  POSTGAP:#       CDROM ... maybe number of pstgap sectors? (usually 0)
  FPS:#.######    AV Video(?)-frames per second? with 6-digit fraction?
  WIDTH:#         AV Width (in pixels?)
  HEIGHT:#        AV Height (in pixels?) (with/without interlace?)
  INTERLACED:#    AV Interlace (maybe a flag that might be maybe 0 or 1?)
  CHANNELS:#      AV Channels (dunno)
  SAMPLERATE:#    AV Samplerate (maybe some audio value in Hertz?)
 For TYPE (and PGTYPE?):
  "MODE1/2048" or "MODE1"                    800h-byte ;\Data Mode1
  "MODE1/2352" or "MODE1_RAW"                930h-byte ;/
  "MODE2/2336" or "MODE2"                    920h-byte ;\.
  "MODE2/2336" or "MODE2_FORM_MIX" (dupe?)   920h-byte ;
  "MODE2/2048" or "MODE2_FORM1"              800h-byte ; Data Mode2
  "MODE2/2324" or "MODE2_FORM2"              914h-byte ;
  "MODE2/2352" or "MODE2_RAW" or "CDI/2352"  930h-byte ;/
  "AUDIO"                                    930h-byte ;-Audio CD-DA
 For SUBTYPE and PGSUB:
  "RW"      60h-byte interleaved   ;normal "cooked" 96 bytes per sector
  "RW_RAW"  60h-byte uninterleaved ;raw uninterleaved 96 bytes per sector
  "NONE"    0-byte                 ;no subcode data stored
EDIT: The findings from this thread are released here: http://problemkaputt.de/psxspx-cdrom-di ... d-mame.htm
Last edited by nocash on December 17th, 2022, 12:38 pm, edited 5 times in total.

User avatar
nocash
Verified
PSX Aficionado
PSX Aficionado
Posts: 541
Joined: Nov 12, 2012
Contact:

Post by nocash » August 28th, 2022, 1:28 am

null wrote: August 27th, 2022, 5:28 pm Update CHDMAN to v0.246 Jul 31, 2022
Sample updated CHD CDROM Psalm 69 JP Demo

Code: Select all

////////////////////////////////////////////////////////////////////////////////////////////
CHDMAN v0.246
              TRACK:1 TYPE:MODE2_RAW SUBTYPE:NONE FRAMES:11519 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.
              TRACK:2 TYPE:AUDIO SUBTYPE:NONE FRAMES:3601 PREGAP:225 PGTYPE:VAUDIO PGSUB:NONE POSTGAP:0.
              TRACK:3 TYPE:AUDIO SUBTYPE:NONE FRAMES:17026 PREGAP:13650 PGTYPE:VAUDIO PGSUB:NONE POSTGAP:0.
////////////////////////////////////////////////////////////////////////////////////////////////////////////
CHDMAN v0.246 doesn't work much better than 0.146 for me (it's less unreliable, but it's now saying that it needs a newer version of windows).
I haven't checked the updated .CHD file yet.

But comparing the CHT2 metadata stuff (from your post above) with the old metadata (from the old .CHD binary)...

Code: Select all

Old CHT2 chunks (with CHD version V5) from CHDMAN v0.146:
TRACK:1 TYPE:MODE2_RAW SUBTYPE:NONE FRAMES:11519 PREGAP:0     PGTYPE:MODE1  PGSUB:RW   POSTGAP:0.
TRACK:2 TYPE:AUDIO     SUBTYPE:NONE FRAMES:3601  PREGAP:225   PGTYPE:MODE1  PGSUB:RW   POSTGAP:0.
TRACK:3 TYPE:AUDIO     SUBTYPE:NONE FRAMES:17026 PREGAP:13650 PGTYPE:MODE1  PGSUB:RW   POSTGAP:0.

New CHT2 chunks (with CHD version V5) from CHDMAN v0.246:
TRACK:1 TYPE:MODE2_RAW SUBTYPE:NONE FRAMES:11519 PREGAP:0     PGTYPE:MODE1  PGSUB:NONE POSTGAP:0.
TRACK:2 TYPE:AUDIO     SUBTYPE:NONE FRAMES:3601  PREGAP:225   PGTYPE:VAUDIO PGSUB:NONE POSTGAP:0.
TRACK:3 TYPE:AUDIO     SUBTYPE:NONE FRAMES:17026 PREGAP:13650 PGTYPE:VAUDIO PGSUB:NONE POSTGAP:0.
That's now quite different... the weird PGTYPE:MODE1 entries are now replaced by PGTYPE:VAUDIO (except on Track 1, which is still "MODE1" despite of being a MODE2 disc)... and PGSUB:RW has changed to PGSUB:NONE.

I've no idea what PGTYPE and PGSUB are meaning (the source code suggests that PG is short for PreGap, whatever that means in that context).
I haven't found the string "VAUDIO" anywhere in the source (only "AUDIO" and "MODE2" etc).

Can you post the original .CUE file for the disc? I am wondering what PREGAP:13650 might mean (13650 leading "silent" sectors in Track 3 ???) Maybe it's becoming more obvious when seeing the .CUE file.

null
Active PSXDEV User
Active PSXDEV User
Posts: 61
Joined: Dec 18, 2021
PlayStation Model: SCPH-1002

Post by null » August 28th, 2022, 2:38 am

nocash wrote: August 28th, 2022, 1:28 am Can you post the original .CUE file for the disc?

Code: Select all

FILE "Psalm 69 (Japan) (Beta) (Track 1).bin" BINARY
  TRACK 01 MODE2/2352
    INDEX 01 00:00:00
FILE "Psalm 69 (Japan) (Beta) (Track 2).bin" BINARY
  TRACK 02 AUDIO
    INDEX 00 00:00:00
    INDEX 01 00:03:00
FILE "Psalm 69 (Japan) (Beta) (Track 3).bin" BINARY
  TRACK 03 AUDIO
    INDEX 00 00:00:00
    INDEX 01 03:02:00
You do not have the required permissions to view the files attached to this post.

User avatar
nocash
Verified
PSX Aficionado
PSX Aficionado
Posts: 541
Joined: Nov 12, 2012
Contact:

Post by nocash » August 31st, 2022, 5:20 am

Okay, that's an odd cue file... and that explains where the huge PREGAP value came from.

I am a bit scared about the LZMA and FLAC compression, I don't know how difficult it will be to write a decompressor for that (or to port an existing decompressor to asm).

For LZMA, there's some SDK (which might be overkill when just needing straight/simple decompression).
The code on this page http://www.nongnu.org/lzip/manual/lzip_manual.html looks easier to deal with... except, it says something about some feature(s) being "optimized away and does not even appear in the code"... so that would work only if those missing features aren't required for CHD files.

Anyways, to get started, it would be easier to have CHD files without LZMA.

Can you make some more DBALL.CHD files for testing? Typing "chdman help createcd" should show how to select different compression methods, which could be either "none" or "type1,type2" (I guess the latter are meant to be placeholders for the actual compression names). The following three formats would be nice (in three separate CHD files):
none -- uncompressed
zlib -- raw zlib (if chdman allows to use that in cdrom images)
cdzl - cd zlib (whatever that is, probably zlib with filtering out header/ecc/edc before compression)

null
Active PSXDEV User
Active PSXDEV User
Posts: 61
Joined: Dec 18, 2021
PlayStation Model: SCPH-1002

Post by null » August 31st, 2022, 7:15 pm

nocash wrote: August 31st, 2022, 5:20 am Can you make some more DBALL.CHD files for testing?
Your LZMA SDK reference is outdated you should use the original one from the author's website. Anyway the LZMA is just an improved and optimized version of LZ77 algorithm according to the author.

Here's the sample file you're looking for.

dball - none (uncompressed)/cdzl (cd zlib)/zlib (raw zlib). The uncompressed sample is bigger than the original BIN/CUE files.
CHDMAN v0.140 CHD v4 (Oct 21 2010) - I don't know if this works on Win98 try this and the commands are different for this version compared to the latest one. CHD v5 is not compatible for CHD v4 and below. Sorry I can't trace the other versions of CHD v3-v1.

Code: Select all

None - Uncompressed
D:\CHDMAN>chdman createcd -c "none" -i "DBALL.cue" -o "DBALL.chd"
chdman - MAME Compressed Hunks of Data (CHD) manager 0.246 (mame0246)
Output CHD:   DBALL.chd
Input file:   DBALL.cue
Input tracks: 1
Input length: 00:07:58
Compression:  none
Logical size: 1,429,632
Compression complete ... final ratio = 100.0%

CD ZLIB
D:\CHDMAN>chdman createcd -c "cdzl" -i "DBALL.cue" -o "DBALL.chd"
chdman - MAME Compressed Hunks of Data (CHD) manager 0.246 (mame0246)
Output CHD:   DBALL.chd
Input file:   DBALL.cue
Input tracks: 1
Input length: 00:07:58
Compression:  cdzl (CD Deflate)
Logical size: 1,429,632
Compression complete ... final ratio = 19.2%

Raw ZLIB
D:\CHDMAN>chdman createcd -c "zlib" -i "DBALL.cue" -o "DBALL.chd"
chdman - MAME Compressed Hunks of Data (CHD) manager 0.246 (mame0246)
Output CHD:   DBALL.chd
Input file:   DBALL.cue
Input tracks: 1
Input length: 00:07:58
Compression:  zlib (Deflate)
Logical size: 1,429,632
Compression complete ... final ratio = 19.4%
You do not have the required permissions to view the files attached to this post.

null
Active PSXDEV User
Active PSXDEV User
Posts: 61
Joined: Dec 18, 2021
PlayStation Model: SCPH-1002

Post by null » August 31st, 2022, 8:07 pm

Code: Select all

Uncompressed
D:\CHDMAN>CHDMAN info -i "DBALL_NONE.chd" -v
chdman - MAME Compressed Hunks of Data (CHD) manager 0.246 (mame0246)
Input file:   DBALL_NONE.chd
File Version: 5
Logical size: 1,429,632 bytes
Hunk Size:    19,584 bytes
Total Hunks:  73
Unit Size:    2,448 bytes
Total Units:  584
Compression:  none
CHD size:     1,449,216 bytes
Metadata:     Tag='CHT2'  Index=0  Length=90 bytes
              TRACK:1 TYPE:MODE2_RAW SUBTYPE:NONE FRAMES:583 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.

     Hunks  Percent  Name
----------  -------  ------------------------------------
        73   100.0%  Uncompressed

Code: Select all

CD Deflate
D:\CHDMAN>CHDMAN info -i "DBALL_CDZL.chd" -v
chdman - MAME Compressed Hunks of Data (CHD) manager 0.246 (mame0246)
Input file:   DBALL_CDZL.chd
File Version: 5
Logical size: 1,429,632 bytes
Hunk Size:    19,584 bytes
Total Hunks:  73
Unit Size:    2,448 bytes
Total Units:  584
Compression:  cdzl (CD Deflate)
CHD size:     275,396 bytes
Ratio:        19.3%
SHA1:         557b44e9efc6a2740b906876614d0b17dd183d4f
Data SHA1:    6753a724b5c9b70be1220a867d2d2d57856b536f
Metadata:     Tag='CHT2'  Index=0  Length=90 bytes
              TRACK:1 TYPE:MODE2_RAW SUBTYPE:NONE FRAMES:583 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.

     Hunks  Percent  Name
----------  -------  ------------------------------------
        73   100.0%  CD Deflate

Code: Select all

Deflate
D:\CHDMAN>CHDMAN info -i "DBALL_ZLIB.chd" -v
chdman - MAME Compressed Hunks of Data (CHD) manager 0.246 (mame0246)
Input file:   DBALL_ZLIB.chd
File Version: 5
Logical size: 1,429,632 bytes
Hunk Size:    19,584 bytes
Total Hunks:  73
Unit Size:    2,448 bytes
Total Units:  584
Compression:  zlib (Deflate)
CHD size:     277,522 bytes
Ratio:        19.4%
SHA1:         557b44e9efc6a2740b906876614d0b17dd183d4f
Data SHA1:    6753a724b5c9b70be1220a867d2d2d57856b536f
Metadata:     Tag='CHT2'  Index=0  Length=90 bytes
              TRACK:1 TYPE:MODE2_RAW SUBTYPE:NONE FRAMES:583 PREGAP:0 PGTYPE:MODE1 PGSUB:NONE POSTGAP:0.

     Hunks  Percent  Name
----------  -------  ------------------------------------
        73   100.0%  Deflate

null
Active PSXDEV User
Active PSXDEV User
Posts: 61
Joined: Dec 18, 2021
PlayStation Model: SCPH-1002

Post by null » September 1st, 2022, 12:54 am

I don't know if these are useful but anyway I'll put it here.

LZMA Decoder in ASM
http://github.com/ilyakurdyukov/micro-lzmadec
http://github.com/upx/upx/tree/devel/src/stub/src/arch

User avatar
nocash
Verified
PSX Aficionado
PSX Aficionado
Posts: 541
Joined: Nov 12, 2012
Contact:

Post by nocash » September 1st, 2022, 10:22 am

Thanks for all those links and uploads - feels a bit like birthday to get some many new files : )

The three new dball files are very useful for testing different file versions, and they are revealing some details that weren't obvious in the chd source code comments... and supporting chd files is looking much more inviting when not needing to care about lzma for now.

chdman 0.140 is - surpisingly - working fine on my computer. It was initially bugging me about missing "unicows.dll" (workaround is to install "Quintessential Media Player", and then copy "unicows.dll" to the chdman folder (which appears to be some obscure/common requirement for running commandline tools from that era)). Anyways, it's working and I can now build my own V4 chd files & examine them in a hex editor.

V3 files would be also interesting, the header is almost same as in V4 (in so far it should be possible to implement working V3-support without ever having seen any V3 files for real). But there appear to be older "metadata" versions (V4 and up are using "CHT2" metadata, I guess V3 must have used "CHCD", and there's also "CHTR" which must have occured in Late-V3 and/or Early-V4 versions).

V1/V2 headers are containing only harddisk related entries - so it should be safe to assume that they were never used for CDROM images.

LZ77 was the oldest (and most sub-optimal) compression method with distance/length pairs, the description "optimized LZ77" does apply to almost all newer compressors no matter how simple or complex they are... I had somehow thought that LZMA would be a bit more complex than things like inflate...

I did see the LZMA SDK on the 7-zip page, but the sample code in the lzip manual looked easier to deal with than digging through megabytes of SDK code. The ASM code that you've found looks even more easy to use... if Ilya's "static.x86.asm" is covering the whole lzma functionality then it seems to be even easier to implement than inflate.

null
Active PSXDEV User
Active PSXDEV User
Posts: 61
Joined: Dec 18, 2021
PlayStation Model: SCPH-1002

Post by null » September 1st, 2022, 6:42 pm

Okay I found the 1st CHDv4 and the last CHDv3/CHDv4. The last CHDv3 and the 1st CHDv4, needs .TOC file in order to createcd .CHD but I don't know what is .TOC file and I don't know how to convert .CUE file to .TOC file? That's why I didn't test to convert a PSX game.

The last CHDv4 is working on me here's the result. I don't know if this works on Win98 you should try it.

ZIP below contains:
1st CHDv4/last CHDv3, CHDv4
DBALL PSX homebrew last CHDv4

Code: Select all

.CUE to .CHD conversion
D:\CHDMAN>chdman -createcd DBALL.cue DBALL.chd
chdman - MAME Compressed Hunks of Data (CHD) manager 0.145 (Feb  5 2012)
trk 1: 583 frames @ offset 0

CD-ROM DBALL.cue has 1 tracks and 583 total frames
Track 01/01 (DBALL.bin:0,583 frames,146 hunks,swap 0,pregap 0,postgap 0)
Compression complete ... final ratio = 80%

Code: Select all

CHD info
D:\CHDMAN>chdman -info DBALL.chd
chdman - MAME Compressed Hunks of Data (CHD) manager 0.145 (Feb  5 2012)
Input file:   DBALL.chd
Header Size:  108 bytes
File Version: 4
Flags:        NO_PARENT, READ_ONLY
Compression:  zlib+
Hunk Size:    9792 bytes
Total Hunks:  146
Logical size: 1,429,632 bytes
SHA1:         38289ad302ac18d37d8c33f5c0a283643d128c59
Raw SHA1:     6753a724b5c9b70be1220a867d2d2d57856b536f
Metadata:     Tag='CHT2'    Length=88
              TRACK:1 TYPE:MODE2_RAW SUBTYPE:NONE FRAMES:583 PREGAP:0 PGTY
You do not have the required permissions to view the files attached to this post.

User avatar
nocash
Verified
PSX Aficionado
PSX Aficionado
Posts: 541
Joined: Nov 12, 2012
Contact:

Post by nocash » September 3rd, 2022, 1:01 pm

Nice! Here's a summary of the different versions...

Code: Select all

  chdman hdr meta  features/requirements/bugs/quirks/failures...
  v?     V1  -     -   ;\supports harddisk format only, not cdrom
  v?     V2  -     -   ;/
  v?     V3? CHCD  ?   ;-?
  v0.130 V3  CHTR  ok  ;\requires cdrdao TOC/BIN as input (CUE/BIN does crash)
  v0.131 V4  CHTR  ok  ;/
  v0.140 V4  CHT2  ok  ;\requires "unicows.dll" (=Quintessential Media Player)
  v0.145 V4  CHT2  ok  ;/
  v0.146 V5  CHT2  bad ;-says output file already exists (crashes on -f force)
  v0.246 V5  CHT2  bad ;-requires "newer version of windows"
The v0.130 and v0.131 are actually working out of the box without crashing or needing dll's. Only the required toc/bin format is a bit on the bizarre side. I think I've heard of 2-3 programs that use different kinds of "toc" files (with tocs in ascii or binary format). In this case, it's apparently needing "cdrdao toc files", which are described here: https://linux.die.net/man/1/cdrdao
For dball.bin, this dball.toc file is working for me:

Code: Select all

  CD_ROM_XA
  TRACK MODE2_RAW
  DATAFILE "dball.bin" 00:00:00 00:07:58
The values 00:00:00 and 00:07:58 are starting offset (within the bin file) and track length, it's also working when omitting those values (chdman will then throw a warning that it's automatically calculating the length of the track from the bin filesize... which is usually exactly what you want (except when storing multiple tracks in a single bin file)).

User avatar
nocash
Verified
PSX Aficionado
PSX Aficionado
Posts: 541
Joined: Nov 12, 2012
Contact:

Post by nocash » September 5th, 2022, 8:56 am

I've been working on parsing the CHD header and metadata - that's more or less complete now. And the sector loading is also already working for some CHD files: Compression "none" is working. And the V3 file with "ZLIB_PLUS" compression is working.

The latter is a bit of a surprise because I have no idea what "ZLIB_PLUS" is supposed to mean. As far as I can tell, it's simply DEFLATED, and there isn't any ZLIB header, least a ZLIB_PLUS (whatever that is).
And the INFLATED data does simply contain the whole sector including header/ECC/EDC, so this also crap without filtering out that values. I hope the later "cdzl" and "cdlz" methods will work better than that.

I've peeked at the "codec.cpp" source code with the decompression functions, but it's hard to make sense of it... it seems to contain a handful of identical-looking "zlib" decompression functions, I don't know what for, and if there are subtle differences between them. I guess it's more realistic to try to decompress the files without looking at the source code... or only look at it for special cases... the "cdzl" compression seems to have an extra 3-byte header in front of the deflated data... the first 8bit seem to be flags for whether to adjust ECC in the 8 sectors... I don't know yet what the other 2 byte are doing.

Understanding the map data will be also difficult: Apart from compressed sectors, there can be also things like SELF and MINI, whatever that is exactly. And V5 seems to use compressed maps... the "decompress_v5_map" function seems to use Huffman and RLE compression for the maps.

For the CHD history... this page https://wiki.mamedev.org/index.php/MAME_0.80u3 says that CDROM support was added in CHDMAN version 0.80u3 - I don't know if such old chdman/mame versions can be found anywhere in internet.

null
Active PSXDEV User
Active PSXDEV User
Posts: 61
Joined: Dec 18, 2021
PlayStation Model: SCPH-1002

Post by null » September 5th, 2022, 5:40 pm

I think the u[No.] are just "intermediate updates" (according to the MAME history page). Those intermediate updates doesn't exist on their page. If you want to download v0.80u3 you should target the v0.81 released because they'll put those info to v0.81 too.

v0.78 doesn't have CDROM support, I didn't test it.
v0.81 the first CHDMAN with CDROM support but can only extract the CHD file to a RAW file, info below.

I'll also put some summary of the CHDMAN history below.

Code: Select all

D:\CHDMAN>chdman -info DBALL.chd
chdman - MAME Compressed Hunks of Data (CHD) manager 0.81 (Apr  1 2004)
Input file:   DBALL.chd
Header Size:  120 bytes
File Version: 3
Flags:        NO_PARENT, WRITEABLE
Compression:  zlib+
Hunk Size:    9792 bytes
Total Hunks:  146
Logical size: 1,429,632 bytes
Metadata:     Tag=43484344  Length=2380
              ...............0.......G....................................

Code: Select all

CHDMAN History

19 Feb 12 v0.145u1
7zip support and change in CHD format (CHDv5).

05 Feb 12 v0.145 (last CHDv4 released)

23 Apr 09 v0.131 (first CHDv4 released)

09 Mar 09 v0.130 (last CHDv3 released)

01 Apr 2004 v0.81
-first CHDv3 released with CDROM support

25 Dec 2003 v0.78
-first CHDv3 released

21 Nov 03 v0.77u1
CHDv3. "hdcomp" tool, got replaced by "chdman". (internal release)

11 Nov 03 v0.77 (last CHDv2 released)

12 Jun 03 v0.70u1
CHDv2.

05 Jun 03 v0.69u3 (last CHDv1 released)
First release to have intermediate 'u' updates.

22 Mar 02 v0.59
Aaron Giles implements CHD (Compressed Hard Disk - later Compressed Hunks of Data).
You do not have the required permissions to view the files attached to this post.

User avatar
nocash
Verified
PSX Aficionado
PSX Aficionado
Posts: 541
Joined: Nov 12, 2012
Contact:

Post by nocash » September 6th, 2022, 3:36 am

Cool, good to know about the older v1/v2 versions, and that chdman was originally called hdcomp back then.

The v0.81 file is nice because it's creating the old "CHCD" metadata... the problem is that it's crashing immediately thereafter (the DBALL.CHD file is only 4868 bytes tall, it's just the header/map/metadata without any data sectors). Does that happen to you, too?

I don't know why it's crashing. It might be an OS issue, or it might dislike something in the .toc file (like linebreak encoding, or missing CD_ROM_XA support or whatever).

Maybe some versions between v0.81 and v0.130 are working better. A working version with the "CHCD" metadata tag would be interesting (you can see that when viewing the .CHD file in a hex editor, "CHCD" in older V3 files, or "CHTR" in newer V3 files).

null
Active PSXDEV User
Active PSXDEV User
Posts: 61
Joined: Dec 18, 2021
PlayStation Model: SCPH-1002

Post by null » September 6th, 2022, 5:19 pm

Yes, that version is also crashing on my PC. But anyway here's the new one.

The last CHDMAN version with [CHCD] is v0.110 and the first CHDMAN version with [CHTR] is v0.111.

ZIP contains:
CHDMAN v0.110/v0.111
CHD produce via CHDMAN v0.110

Code: Select all

D:\CHDMAN>chdman -createcd DBALL.toc DBALL.chd
chdman - MAME Compressed Hunks of Data (CHD) manager 0.110 (Nov  5 2006)

CD-ROM DBALL.toc has 1 tracks and 583 total frames
Compressing track 1 / 1 (file DBALL.bin:0, 583 frames, 146 hunks)
Compression complete ... final ratio = 80%

D:\CHDMAN>chdman -info DBALL.chd
chdman - MAME Compressed Hunks of Data (CHD) manager 0.110 (Nov  5 2006)
Input file:   DBALL.chd
Header Size:  120 bytes
File Version: 3
Flags:        NO_PARENT, READ_ONLY
Compression:  zlib+
Hunk Size:    9792 bytes
Total Hunks:  146
Logical size: 1,429,632 bytes
MD5:          9893d02d8dacfa74264932e8c9cad843
SHA1:         6753a724b5c9b70be1220a867d2d2d57856b536f
Metadata:     Tag=43484344  Length=2380
              ...............0.......G....................................
You do not have the required permissions to view the files attached to this post.

User avatar
nocash
Verified
PSX Aficionado
PSX Aficionado
Posts: 541
Joined: Nov 12, 2012
Contact:

Post by nocash » September 8th, 2022, 4:01 am

That's a nice ZIP, with v0.90, v0.100, v0.110, v0.110, v0.111, v0.112, v0.113, v0.115, v0.118, v0.120, v0.125, v0.141, thanks!.

Any chance that you have a similar collection for versions between v0.146 (file already exists bug) and v0.246 (newer version of windows required)? Maybe there's a V5 version that works on my PC (which would be useful to build some more test images with MODE1 tracks and AUDIO tracks and this or that).

I've converted one of Official Playstation Magazine demo discs to TOC/BIN format for making a CHCD file with multiple tracks. I've got it working, but there was one extra obstacle: It doesn't support BIN filenames with space characters (despite of having the name enclosed in double quotes).

Code: Select all

Original CUE file:
FILE "Official U.S. PlayStation Magazine Demo Disc 34 (Track 1).bin" BINARY
  TRACK 01 MODE2/2352
    INDEX 01 00:00:00
FILE "Official U.S. PlayStation Magazine Demo Disc 34 (Track 2).bin" BINARY
  TRACK 02 AUDIO
    INDEX 00 00:00:00
    INDEX 01 00:02:00
FILE "Official U.S. PlayStation Magazine Demo Disc 34 (Track 3).bin" BINARY
  TRACK 03 AUDIO
    INDEX 00 00:00:00
    INDEX 01 00:02:00

Corresponding TOC file, with renamed space-less BIN filenames:
CD_ROM_XA
TRACK MODE2_RAW
 DATAFILE "opm35_track1.bin"
TRACK AUDIO
 DATAFILE "opm35_track2.bin"
 INDEX 00:02:00
TRACK AUDIO
 DATAFILE "opm35_track3.bin"
 INDEX 00:02:00
I am not sure if I got the INDEX values right. The first 2 seconds on each audio track should have Index 0, and then Index 1 for the main part of each track.

There seems to be little support for Index values or other subchannel data in CHD files.
The CHCD and CHTR metadata doesn't seem to contain any Index info.
The CHT2 metadata has PREGAP info (which isn't exactly the same as Index, but it could be assumed to imply Index 0 within pregap).
Theoretically, the 990h-byte sectors could include fully-fledged 60h-byte subchannel data for each sector, but that seems to be an optional feature (normal CHD files are just having those 60h bytes zerofilled). And that subchannel data could be stored only in sectors for Track 1 and up (not for Lead-in, which would be required for things like CD TEXT or Sessions).
In so far, the CHD format looks a bit incomplete, not really capable to store complete cdrom images. Missing support for Sessions & CD TEXT doesn't really matter for PSX though.

I've cracked the CHD V5 map decompression! It's scattered across hundreds of lines of code in different .cpp and .h files. But when extracting the relevant code, it's actually relative simple:

Code: Select all

  readfile(hdr,10h)                                        ;\read map hdr and
  readfile(src,bigendian32bit[hdr+0])                      ; compressed map
  InitBitstream(src,BigEndianMsbFirst)                     ;/
  i=0                                                      ;\.
  while i<10h                                              ;
    val=GetBits(4), num=1                                  ;
    if val=01h then                                        ; read huffman tree
      val=GetBits(4)                                       ;
      if val<>01h then num=GetBits(4)+3                    ;
    for j=1 to num, codesizes[i]=val, i=i+1                ;
  nonlzh_explode_tree(codetree,codesizes,10h)              ;/
  i=0, typ=0, num=0                                        ;\.
  while i<NumberOfHunks                                    ;
    if num=0                                               ; load huffman coded
      x=GetHuffCode(codetree)                              ; map type values
      if x=07h then      ;COMPRESSION_RLE_SMALL            ;
        num=GetHuffCode(codetree)+03h                      ;
      elseif x=08h then  ;COMPRESSION_RLE_LARGE            ;
        num=GetHuffCode(codetree)*10h                      ;
        num=GetHuffCode(codetree)+num+13h                  ;
      else typ=x, num=1                                    ;
    byte[map+i*0Ch+0]=typ, i=i+1, num=num-1                ;/
  i=0, s=0, p=0             ;index,self,parent             ;\.
  o=bigendian48bit[hdr+4]   ;offset                        ; load other
  while i<NumberOfHunks                                    ; map items
    typ=byte[map+i*0Ch+00h], ofs=o, len=0, crc=0           ;
    if typ<04h then len=GetBits([hdr+0Ch]), crc=GetBits(16);  ;Method 0..3
    elseif typ=04h then len=BytesPerHunk, crc=GetBits(16)  ;  ;Uncompressed
    elseif typ=05h then s=GetBits([hdr+0Dh]), ofs=s        ;  ;New Self
    elseif typ=06h then p=GetBits([hdr+0Eh]), ofs=p        ;  ;New Parent
    elseif typ=09h then typ=05h, ofs=s                     ;  ;Old Self
    elseif typ=0Ah then typ=05h, s=s+1, ofs=s              ;  ;Old Self+1
    elseif typ=0Bh then typ=06h, p=i*SectorsPerHunk, ofs=p ;  ;Direct Parent
    elseif typ=0Ch then typ=06h, ofs=p                     ;  ;Old Parent
    elseif typ=0Dh then typ=06h, p=p+SectorsPerHunk, ofs=p ;  ;Old Parent+1
    else goto error                                        ;
    byte[map+i*0Ch+00h]=typ                                ;
    bigendian24bit[map+i*0Ch+01h]=len                      ;
    bigendian48bit[map+i*0Ch+04h]=ofs                      ;
    bigendian16bit[map+i*0Ch+0Ah]=crc                      ;
    o=o+len, i=i+1                                         ;/
  if bigendian16bit[hdr+0Ah]<>noncrc16(map,i*0Ch) then error ;-final crc check
InitBitstream, GetBits and GetHuffCode should be self-explaining. But there are two uncommon functions:
noncrc16: Uses the same polynomial as for CDROM subchannels, but with initial value FFFFh (instead 0) and with final value left un-inverted (instead of inverting it).
nonlzh_explode_tree: Uses the same concept as for LZH/ARJ huffman trees (it's storing only the number of bits for each code, and the codes are then automatically assigned). But CHD is doing that backwards: It's starting with the biggest codes (instead of smallest codes). For example, if you have three codes with size 1, 2, 2. The traditional standard assignment would be 0, 10, 11. But CHD is instead assigning them as 00, 01, 1.

null
Active PSXDEV User
Active PSXDEV User
Posts: 61
Joined: Dec 18, 2021
PlayStation Model: SCPH-1002

Post by null » September 8th, 2022, 4:09 pm

Nice progress!!

Anyway, CHDMAN v0.217 is the last one to support 32bit OS.
CHDMAN v0.218 and anything up only supports 64bit.

ZIP contains:
CHDMAN v0.147
CHDMAN v0.159
CHDMAN v0.217
You do not have the required permissions to view the files attached to this post.

null
Active PSXDEV User
Active PSXDEV User
Posts: 61
Joined: Dec 18, 2021
PlayStation Model: SCPH-1002

Post by null » September 8th, 2022, 5:31 pm

I hope it helps and you can document this format better than the MAME team.

This source is not from a member of MAME team.
https://docs.rs/chd/latest/chd/codecs/index.html

There's no proper documentation of CHDMAN or CHD from the MAME page as of now and never will because Aaron Giles motto in life is "The code is the documentation". And it looks like he totally abandoned this tool. The MAME team maintains it from now on.

User avatar
nocash
Verified
PSX Aficionado
PSX Aficionado
Posts: 541
Joined: Nov 12, 2012
Contact:

Post by nocash » September 9th, 2022, 3:05 am

No luck with V5, it's only inventing two new errors in v0.159 and v0.217

Code: Select all

  chdman hdr meta  features/requirements/bugs/quirks/failures...
  v0.58  -   -     -   ;-CHD didn't exist in older MAME versions
  v0.59  V1  -     -   ;\.
  v0.71  V2  -     -   ; supports harddisk CHD files only, not cdrom
  v0.78  V3  xxxx  -   ;/
  v0.81  V3  CHCD  bad ;-crashes after creating the CHD file header
  v0.90  V3  CHCD  ok  ;\.
  v0.110 V3  CHCD  ok  ; requires cdrdao TOC/BIN as input (CUE/BIN does crash)
  v0.111 V3  CHTR  ok  ; (warning: BIN filenames may not contain space chars!)
  v0.130 V3  CHTR  ok  ;
  v0.131 V4  CHTR  ok  ;/
  v0.140 V4  CHT2  ok  ;\requires "unicows.dll" (=Quintessential Media Player)
  v0.145 V4  CHT2  ok  ;/
  v0.146 V5  CHT2  bad ;\says output file already exists (crashes on -f force)
  v0.147 V5  CHT2  bad ;/
  v0.159 V5  CHT2  bad ;-crashes instantly
  v0.217 V5  CHT2  bad ;-missing KERNEL32.DLL:AddVectoredExceptionHandler
  v0.246 V5  CHT2  bad ;-requires "newer version of windows"
Thanks for the docs.rs link.
Some pages like https://docs.rs/chd/latest/chd/codecs/t ... Codec.html are looking useful (apparently that's using two separate compression blocks; for data and subchannels).
Some pages like https://docs.rs/chd/latest/chd/metadata ... aRefs.html contain fireworks of seemingly meaningless information (or my browser can't display it in a meaningful way).
And I don't get how to navigate through those hundreds of pages, there are probably pages describing the old V3/V4 "zlib+" compression and the old V3 "CHCD" metadata, but I've no idea how to find those pages.

Using a hex-editor, the CHCD metadata looks roughly like this:

Code: Select all

CHCD Metadata (94Ch bytes, plus 10h-byte metadata header)
  000h 4     Number of tracks (N) (1..99)
  004h N*18h Track entries
  ...  ..    Zeropadding to 94Ch-byte size (when less than 99 tracks)
 Track entries:
  000h 4     Unknown          (6=Data, 7=Audio)
  004h 4     Unknown          (2)
  008h 4     Sector Size      (930h)
  00Ch 4     Unknown          (0)
  010h 4     Number of Frames
  014h 4     Unknown          (2)
The mamedev cdrom.h seems to define the size of that metadata as:

Code: Select all

  static constexpr uint32_t MAX_TRACKS       = 99;
  static constexpr uint32_t METADATA_WORDS   = 1 + MAX_TRACKS * 6;
But that's only the size, without describing the exact contents (and cdrom.cpp doesn't seem to use the above definitions at all) (and github seems to have rolled out an update today and ditched the last bit of old browser support).
Last edited by nocash on September 9th, 2022, 4:46 pm, edited 1 time in total.

null
Active PSXDEV User
Active PSXDEV User
Posts: 61
Joined: Dec 18, 2021
PlayStation Model: SCPH-1002

Post by null » September 9th, 2022, 4:04 am

Oh did you have browser problems with GitHub?

This is the src used on the documentation.
http://github.com/SnowflakePowered/chd- ... chd-rs/src

Maybe later or tomorrow I'll add some more versions of CHDMAN that you can test.

null
Active PSXDEV User
Active PSXDEV User
Posts: 61
Joined: Dec 18, 2021
PlayStation Model: SCPH-1002

Post by null » September 9th, 2022, 3:54 pm

ZIP contains:
CHDMAN v0.148
CHDMAN v0.149
CHDMAN v0.150
CHDMAN v0.174
CHDMAN v0.179
chd-rs GitHub src
You do not have the required permissions to view the files attached to this post.

Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests