Extracting custom container (.DAT) package from disc

[Yagotzirck]

I'd like your feedback on this experiment(especially if you have played the game and know how tracks are supposed to sound like):
https://www.sendspace.com/file/itk8ap

It's not too random to sound implausible, but still a bit too "funny" to make me think it's a loyal reproduction of the in-game track, then again I haven't played the game from start to finish so I don't know how it's supposed to sound like

[DD-Indeed]

This only has sample collection and instrument set, no SEQ data ? Opened it on VGMTrans and there was no sequence data. And I know which track this propably is, heard similar one on the PC version. But I propably need to find the matching .XM or .669 for it, to have that sequence data.

[Yagotzirck]

https://www.sendspace.com/file/k89chh
This time I used the title screen theme since it's easily accessible/recognizable; I also included the .xm, VB and VH files from which I made the PSF, but well - for some reason the piano notes aren't played at all, and the audible channels don't play as they should in my opinion.
Either I'm missing something in XMPlay2's configuration, or I just have to accept that the only way to get this to play right is creating PSFs by hacking the game's exe

Yet another way could be converting the samples inside the .VAB file back to .WAV and importing them inside the .xm files, possibly by automating the process since by doing it manually you'd have to reconstruct the samples' looping points once you import them inside the .xm file (as if manually importing the samples alone isn't tedious enough)... but oh well, this is probably getting to the point where the effort is no longer justified

[DD-Indeed]

I just can't get this to play, the tracker softwares I have do no understand that .XM at all, but at least we have the samples available, and those can be saved as WAV to be used in those tracker softwares. But like I suspect, the music is made with tracker software, which is quite difficult to get head around. Some devs used MIDI-gear to make the music, such as stuff from Roland, Yamaha etc, and those can be easily extracted from the PSF's as SEQ files, and then converted into MIDI again.

It could be also that the files are getting messed up upon extracting them from the data, to me it seems that the notes in files are just simple gibberish when played on tracker software, both XM and 669.


Perhaps the actual extracting of those from the filesystem is out of question, but maybe there could be a way to get those via emulation. The music is not streamed as an actual rendered track as we know by now, instead, the game loads the sequence data + soundfonts into sound memory and plays them from there. And that's the place where to get those. I dunno that is there any tools to seek those from there during emulation

[Squaresoft74]

You guys got me curious with that one so trying my luck with patching the original exe.
Got the the sound system initilalized and the samples bank loaded to SPU RAM so far.
I'll try to figure out how the XM file is loaded / played next and let you know if i can get something working.

[DD-Indeed]

Thanks! Not only this would be useful for this occation, but for other games, where you have no possibility to extract the music from the disc. Most of the games use already rendered audio files like I stated and those can be easily extracted with jPSXdec, but there is a lot of games that require processing of that audio data with PS1 sound processing to get them working. Which is why it would be major thing if someone would be able to play those custom music files thru emulation and dump the tracks into lossless WAV, without the need to actually play the game and record them via audio capture manually.

[Yagotzirck]

It could be also that the files are getting messed up upon extracting them from the data, to me it seems that the notes in files are just simple gibberish when played on tracker software, both XM and 669.

Both XM and VB/VH files I've extracted from the .DAT file and included in that archive are the exact same used by the game, I know because I searched them inside a RAM dump and they matched perfectly; the notes looking like gibberish could be a result of automatic conversion from midi (see the hex editor screen in one of my previous posts, where there's an embedded string which kinda reveals that).
Anyway I agree, unless the samples are somehow converted back to raw pcm while preserving loop points (if any) and placed back inside the .XM files, extracting them from the DAT archive is pretty much useless (unless the playback routine used by the game gets isolated in such a way that we can feed it the extracted files, isolating the driver code from the xm sequences + samples).

@Squaresoft: Great, keep us updated
I'd like to know the technical details of the process as well, if you can be arsed

[DD-Indeed]

Gotta wait then, I'm running out of ideas after two days working on this

And there's so many of those tracker softwares to work with, most of which are DOS-based, so I had to use DOSBox for those too, and some of them are in Amiga-formats. Some of those softwares understood those XM/669 files, some didn't.

[Yagotzirck]

Any specific reason for not simply sticking with OpenMPT? It's regularly updated, works on modern Windows (7 or later) and practically supports every tracker format in existence (or at least the vast majority of them).

[DD-Indeed]

I couldn't get them working with that either. Could be that I didn't do something right. But that PSF worked on foobar2000 with PSF plugin, but it only had three instruments, and there should be 15 of them playing in the opening theme.

[Squaresoft74]

I'd like to know the technical details of the process as well, if you can be arsed

You may want to give a look here or here.
I've already ripped a few custom/compressed format games after getting comfortable enough with regular pQES/pBAV ones, but none using XM yet so really trying my luck here as i'm still learning with any new rip i'm attempting.
Would you happen to have a USA mcd save file for the game where some bgm other than the title one would play ?

[DD-Indeed]

Helped with this, I did that save file, played the game a bit to get into first savepoint, music was playing while I did that.

https://www.sendspace.com/file/87ou1d


It's on MCR format, but can be converted to MCD.

[Yagotzirck]

I'd like to know the technical details of the process as well, if you can be arsed

You may want to give a look here or here.
I've already ripped a few custom/compressed format games after getting comfortable enough with regular pQES/pBAV ones, but none using XM yet so really trying my luck here as i'm still learning with any new rip i'm attempting.

Thanks, I already took a glance at those guides and while they're great I think they're quite incomplete (for example they don't provide any thorough instructions about how to separate the driver code from soundbanks, it's just briefly mentioned but that's about it).

Would you happen to have a USA mcd save file for the game where some bgm other than the title one would play ?

Even better - I found the function that loads and plays .XM files, so by changing the value in a0 as soon as the breakpoint is triggered I can play whatever I want from the title screen (I use the PAL version though, so you might want to get it and I might give you more instructions about it... or if you can't really be arsed I might get the NTSC version myself and see if what I found still applies)

EDIT: this was only partially true, since I can indeed select different .XMs but the soundbank seems to stay the same one associated to the titlescreen theme, leading to lousy results; this definitely needs further analysis.

Anyway, just for reference (again this refers to the PAL version):

• The function that selects the .XM file to play is at offset 0x8007B0A4
• It takes two arguments in registers a0 and a1; no idea about what a1 is, but the first (a0) definitely selects which .xm file to load and play
• a0 can take the following values (decimal): 10, 192, anything inside the range [195, 209], and 1004

Hope that helps somehow, I'll continue examining it a bit more tomorrow.

[Squaresoft74]

The XM loading routine you mentionned for the USA version is sub_8007C198.
Altering $a1 seems to mess with instruments (while not changing the loaded VAB itself), which results in title track to play just like the PSF file you posted (with $a1 set with 0x00 for example).
It might be used to pick some track specific values from a table for playback ?

You're right about the soundbank, it's being loaded first, at least for title screen as far as i could check for now:
Break on sub_8007BCBC
8007BCC0: li $a0, 0x0008 <-- that's the title screen VAB file ID.
Finding all VAB file ID and pairing them with the correct XM files should allow to change the title screen bgm.

I'll try looking for the same in game as it should be easier to isolate.

for example they don't provide any thorough instructions about how to separate the driver code from soundbanks, it's just briefly mentioned but that's about it.

Can you elaborate, not sure i fully understand the question here, not a native English speaker.

@ DD-Indeed
Thanks for the save file, should save some time.
EDIT
Seems the MCR file you posted is empty.

[DD-Indeed]

@ DD-Indeed
Thanks for the save file, should save some time.
EDIT
Seems the MCR file you posted is empty.

Try this one, I converted this from MCR to MCD, and checked on HxD that there is some data

https://www.sendspace.com/file/i5lvkc

[Squaresoft74]

This one is fine, thank you !
I'll check more later tonight.

@Yagotzirck
Ok using the USA version, break on sub_8007C198 and use the following values (most are assumed for now as i just checked against 2 bgms available from DD-Indeed save file):
Bgm parameters
$a0=C3 | $a1=00 ?
$a0=C4 | $a1=01 ?
$a0=C5 | $a1=02 ?
$a0=C6 | $a1=03 ?
$a0=C7 | $a1=04 ?
$a0=C8 | $a1=05 ?
$a0=C9 | $a1=06
$a0=CA | $a1=07
$a0=CB | $a1=08 ?
$a0=CC | $a1=09 ?
$a0=CD | $a1=0A ?
$a0=CE | $a1=0B ?
$a0=CF | $a1=0C ?
$a0=D0 | $a1=0D ?
$a0=D1 | $a1=0E // Title screen BGM

Seems to change the bgm just fine in game.
If doing from title screen, you need to apply twice to get it to work.
I'll check later if/where samples bank get loaded, unless it's a single shared one ?
I didn't extract all VAB files from UNI3.DAT yet to check them.

Title screen bgm samples are at 0x1010 (SPU RAM) by default.
XM files seems to always be at 0x80124108.

[Yagotzirck]

Can you elaborate, not sure i fully understand the question here, not a native English speaker.

There are two ways to extract PSFs (at least that I'm aware of):

1. The naive way: once you've got the game exe stripped of everything unrelated to music playback you embed the tracker/sequence + vab file related to each track and convert the exe to .PSF, and you repeat for each track
2. The best way: in order to avoid redundancy, the driver (or in other words, the exe containing the music player's code) is kept in a separate file, and each PSF/minipsf only contains sequence + VAB data related to the track they're associated to, and all PSF files share the same driver code without including it inside each PSF/minipsf

I'd prefer to use the 2nd approach for obvious reasons, but neither of those guides explain how to do that (or maybe they do and it just went over my head).

Anyway thanks for the info about the XM function, will check it out this evening.

I didn't extract all VAB files from UNI3.DAT yet to check them.

I made a tool that extracts .VAB and .xm files from the .DAT archive, I can host it here if you think it could have any use.
One thing I noticed is that the IDs used for VAB and XM files match the indices inside the DAT archive, for instance:

Break on sub_8007BCBC
8007BCC0: li $a0, 0x0008 <-- that's the title screen VAB file ID.

The same VAB gets extracted by my tool as 0008.VAB, same goes for all the xm files.

---

EDIT: Alright, I've added the sounds' extractor as an attachment; I think the VB/VH combinations are actually unrelated to music and used for in-game sounds instead, but I've included them anyway, just in case.

I also checked this combination out of all the ones you posted:

$a0=C3 | $a1=00 ?

Unfortunately the instruments don't match, so the pattern of decreasing values for the argument in a1 doesn't seem to hold

I'd continue examining it but I'm a bit too burned out at the moment, maybe another time.

martiangothic_extractor.zip

[DD-Indeed]

Better leave it for now, you both have done great job nevertheless.

[Yagotzirck]

Well I think we're more than halfway through, once we understand a few more details about the way VABs and XMs are loaded and the playback routine is found this can be considered done; unfortunately I can't take a full-immersion approach since I've got some other stuff to do at the moment (I mostly tinker with it at evenings if I'm not too burned out), otherwise this probably would've been finished already by now.

[DD-Indeed]

Yea, there's no rush, and you should take breaks too Even I have gotten exhausted after been trying to get those XM's to work for some days, but no luck. At least I managed to rip out the PC version and uploaded that to YT:

https://www.youtube.com/watch?v=kU-fAiT-A1o


It definitely sounds bit different than the PS1 version songs, but the compositions should be the same for most part, just different soundfont/instruments and mixing for PS1