Extracting custom container (.DAT) package from disc

Post a topic about yourself to let others know your skills, hobbies, etc.
User avatar
Squaresoft74
Verified
/// PSXDEV | ELITE ///
/// PSXDEV | ELITE ///
Posts: 303
Joined: Jan 07, 2016
PlayStation Model: SCPH-7502
Location: France
Contact:

Post by Squaresoft74 » August 14th, 2020, 8:26 pm

Yagotzirck wrote: August 12th, 2020, 7:41 pm [*]The best way: in order to avoid redundancy, the driver (or in other words, the exe containing the music player's code) is kept in a separate file, and each PSF/minipsf only contains sequence + VAB data related to the track they're associated to, and all PSF files share the same driver code without including it inside each PSF/minipsf
I'd prefer to use the 2nd approach for obvious reasons, but neither of those guides explain how to do that (or maybe they do and it just went over my head).
This is mostly interesting to go that way when multiple sequences use a same VAB / samples bank set.
To do so you need to make use of tags and basically proceed as follow:

Using PSFLab, save your driver as driver.psf then rename it as driver.psflib

Create a new PSF file and import your VAB file at the address you defined in your driver.
Using Edit -> EXE Settings, set this address as the Text section start
Update the Text section size so it can hold the whole file.
Save it as vab.psf then rename it as vab.psflib

Create a new PSF file and import your sequence file at the address you defined in your driver.
Using Edit -> EXE Settings, set this address as the Text section start
Update the Text section size so it can hold the whole file.
Using Edit -> Tag.. add those two lines:
_lib=driver.psflib
_lib2=vab.psflib
Save it as sequence.psf then rename it as sequence.minipsf

If done correctly you should be able to play the resulting file with your PSF player.

About Martian Gothic:
Some XM files might make use of 0006.VH/0007.VB, 0188.VH/0189.VB, 0190.VH/0191.VB and 0193.VH/0194.VB. instead of 0008.VAB
So i think the Bgm parameters i previously posted might be correct, just the currently loaded VAB that isn't.

Thanks for the extractor, as you mentionned most VAB / VH,VB sets are just for sfx.
Any chance you could update it at some point so it would extract the whole content ?
Maybe the game also makes use of another file to play the XM/VAB stuff, i don't really know atm as i never dealt with this format so far.

I'll keep digging with this game when i have some free time, hopefully we'll get something at some point. :)

@DD-Indeed
Thanks for the OST. :)

Yagotzirck
Verified
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 131
Joined: Jul 17, 2013

Post by Yagotzirck » August 14th, 2020, 9:46 pm

@DD-Indeed: good job, how did you rip it?
If you managed to get those .xm files to play it would be nice to upload them as well somewhere :)

@Squaresoft: Thank you, that's exactly the kind of instructions I was looking for :D
As for the extractor, sure thing - when I manage to get access to my main PC I'll provide the source code as well, it's not particularly complicated so it should be easy to play with/modify at will.
There's also a lot of .TIM files and compressed entries (which, when uncompressed, turned out to be TIM images as well, those I examined anyway) inside that archive, so it's probably wiser to extract the various contents in separate directories to keep things as clean as possible... I'll work on it this evening ;)

User avatar
DD-Indeed
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 165
Joined: Jun 07, 2014
I am a: Average DIY-guy
PlayStation Model: SCPH-5502

Post by DD-Indeed » August 15th, 2020, 12:26 am

Yagotzirck wrote: August 14th, 2020, 9:46 pm @DD-Indeed: good job, how did you rip it?
If you managed to get those .xm files to play it would be nice to upload them as well somewhere :)

There's no XM files in the PC version at all, all of the dialogue, sound effects and music is stored as WAV, but they're packed into this ATF-format, which I was able to open with DragonUnpacker's HyperRipper-tool and extract those WAV's out of them. The PC version music is also heavily compressed, which I find weird, lost of fuzzy sounds in those tracks, especially during the outro's.


Only thing related to the tracker files on the PC version were those 669-files, that the HyperRipper found, but those didn't work at all, and HyperRipper stated that they can be false results --> gibberish :D



@Squaresoft74:


PC version's music files were in separate 10-20 sec pieces, that I had to overlap into row to make them sound as one track (Title track was split into six music14.wav-files)

So maybe the same thing can happen with the PS1 version too, meaning that, like, the theme music consists:

0190.VH/0190.VB
0191.VH/0191.VB
0192.VH/0192.VB
0193.VH/0193.VB
0194.VH/0194.VB
0195.VH/0195.VB



Those are then combined together to create single track, maybe with slight overlapping too.

Yagotzirck
Verified
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 131
Joined: Jul 17, 2013

Post by Yagotzirck » August 15th, 2020, 7:52 am

Well the compression is certainly weird, even more so if we consider that it's in .wav chunks which is supposed to be raw pcm; either it has low sampling rate/bits per sample, or .wav is just a container for a compressed format rather than pcm... or they just messed up when exporting/recording the tracker files as wav :D

User avatar
DD-Indeed
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 165
Joined: Jun 07, 2014
I am a: Average DIY-guy
PlayStation Model: SCPH-5502

Post by DD-Indeed » August 15th, 2020, 8:48 am

Could be anything, we have no clue :D But we know that it's made with tracker software, rather than MIDI-systems (Resident Evil's etc) or CD Audio tracks, and that the PS1 version definitely has sections of the songs in separate files like in PC version (there's fluctuation with the tempo and awkward pauses within the songs, if you listen closely)

Yagotzirck
Verified
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 131
Joined: Jul 17, 2013

Post by Yagotzirck » August 15th, 2020, 10:07 am

I'm not too sure about that; unlike the PC version which has streams of wav files, all the PSX has to do is loading the soundbank to SPU RAM and then send playback commands to the SPU according to the .xm files (which act kinda like a musical score)... unless you mean that some tracks use more than one VAB soundbank which get switched during playback, which wouldn't be too implausible.

Anyway, here's another extractor which extracts all the entries (source code included); the files I couldn't figure out what they were for (not in an automated way at least, lots of files are related to in-game text, but they lack a header so I can't filter them out in an easy/practical way) have .bin extension.
Also, this time the entries are extracted in a folder named "UNI3.DAT entries", in the same path as the UNI3.DAT archive.

To whoever is going to take it: If you're going to compile it yourself, make sure you compile for a 32-bit target; the decompressing function performs some pointer arithmetic assuming pointers are 32 bit, so if you compile it for x86-64 a runtime crash is pretty much guaranteed.
You do not have the required permissions to view the files attached to this post.

User avatar
DD-Indeed
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 165
Joined: Jun 07, 2014
I am a: Average DIY-guy
PlayStation Model: SCPH-5502

Post by DD-Indeed » August 15th, 2020, 10:56 am

Yagotzirck wrote: August 15th, 2020, 10:07 am I'm not too sure about that; unlike the PC version which has streams of wav files, all the PSX has to do is loading the soundbank to SPU RAM and then send playback commands to the SPU according to the .xm files (which act kinda like a musical score)... unless you mean that some tracks use more than one VAB soundbank which get switched during playback, which wouldn't be too implausible.

Anyway, here's another extractor which extracts all the entries (source code included); the files I couldn't figure out what they were for (not in an automated way at least, lots of files are related to in-game text, but they lack a header so I can't filter them out in an easy/practical way) have .bin extension.
Also, this time the entries are extracted in a folder named "UNI3.DAT entries", in the same path as the UNI3.DAT archive.

To whoever is going to take it: If you're going to compile it yourself, make sure you compile for a 32-bit target; the decompressing function performs some pointer arithmetic assuming pointers are 32 bit, so if you compile it for x86-64 a runtime crash is pretty much guaranteed.
I tested this now, extracted the files and opened them in PSound. They're played in really fast speed, but when you force them to 8000 Hz sample rate from the config, they play correctly, althou muddy as hell.


So far it seems that the first 19 sounds are the instrument samples for the songs, rest is just voice and noise effects etc. I need to go through them all tomorrow, so far gone through 40 and there's 263 of them, and I need to go to sleep :D

User avatar
Squaresoft74
Verified
/// PSXDEV | ELITE ///
/// PSXDEV | ELITE ///
Posts: 303
Joined: Jan 07, 2016
PlayStation Model: SCPH-7502
Location: France
Contact:

Post by Squaresoft74 » August 15th, 2020, 11:25 am

Thanks for the updated extractor ! :)
Gave it a quick try but it seems you made it to extract the VAB files so it will skip the extra 4 bytes some of them have.
Ex for 0008.VAB which is indexed @ 0x00000040 with 00480700 14600200
I take it this reads as being located @ offset 0x00074800 and has a 0x00026014 size
Following this, heading over 0x00074800 and selecting a 0x00026014 bloc, it should also include the 240C0000 bytes at the very beginning of the file.
This reads as 0x00000C24 and corresponds to those extra 4 bytes + VH size, so it's the VB pointer.
It's needed and should be preserved in raw files. :)

"VabLoad" (sub_8007BCBC), i named it like that for now, that copies/loads 0008.VAB to/from 0x8017F6E0 expects it with those extra bytes.

That's how i injected the file in my wip driver to get the samples bank loaded to SPU RAM for now (after patching "VabLoad" by removing the CD copy calls).

Also quick suggestion, could you make it output filenames using HEX numbering instead of decimal ?
Just to make it more obvious which file is loaded when tinkering with breakpoints.

Yagotzirck
Verified
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 131
Joined: Jul 17, 2013

Post by Yagotzirck » August 15th, 2020, 12:41 pm

I think that 32-bit field containing the value 0x24 is used for pretty much the same thing I'm using it: to discern a VAB entry containing both header and body from an entry consisting of a VH header followed by another entry which acts as the body (VB), so it makes sense to keep it (for PSF purposes, at least).

Anyway check this out, should comply to new requests (hex names and 0x24 ID field included in VAB files):
https://www.sendspace.com/file/5hmj14

I also fixed a bug where TIM files were saved twice with .TIM and .bin extension, will update source code as well tomorrow (I guess I'll put it on github, uploading a thousand update archives gets tedious real quick), since DD-Indeed isn't the only one who needs some rest :D

User avatar
Squaresoft74
Verified
/// PSXDEV | ELITE ///
/// PSXDEV | ELITE ///
Posts: 303
Joined: Jan 07, 2016
PlayStation Model: SCPH-7502
Location: France
Contact:

Post by Squaresoft74 » August 15th, 2020, 12:49 pm

Many thanks for the update, i'll get back to it later tomorow too after some good rest aswell. ;)

User avatar
DD-Indeed
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 165
Joined: Jun 07, 2014
I am a: Average DIY-guy
PlayStation Model: SCPH-5502

Post by DD-Indeed » August 15th, 2020, 7:10 pm

Okay, I went now through all of the soundfiles and these are the instrument sample packs:

0007.VB
0008.VAB
0189.VB
0191.VB
0194.VB



But I have zero idea than how the game picks those up. If it does it the easy way, it just picks up XM and one of those VB/VAB packs for each track. The hard way is that it picks up XM and then individual sample pieces from different VB/VAB packs.

Yagotzirck
Verified
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 131
Joined: Jul 17, 2013

Post by Yagotzirck » August 15th, 2020, 7:54 pm

Good job at filtering them up; from what I've seen in the disassembly, the game selects lots of files by passing hardcoded values to certain functions (e.g. the function that selects which XM file to play), so I wouldn't be surprised if soundbanks are selected in a similar fashion (especially considering there's only 5 of them).

User avatar
Squaresoft74
Verified
/// PSXDEV | ELITE ///
/// PSXDEV | ELITE ///
Posts: 303
Joined: Jan 07, 2016
PlayStation Model: SCPH-7502
Location: France
Contact:

Post by Squaresoft74 » August 15th, 2020, 8:02 pm

I pointed that out previously ;)
Squaresoft74 wrote: August 14th, 2020, 8:26 pm About Martian Gothic:
Some XM files might make use of 0006.VH/0007.VB, 0188.VH/0189.VB, 0190.VH/0191.VB and 0193.VH/0194.VB. instead of 0008.VAB
So i think the Bgm parameters i previously posted might be correct, just the currently loaded VAB that isn't.

If i can end up getting a working driver that would at least play the title screen XM bgm, it then shouldn't t be a problem to make it use a different VAB or VH/VB set.
This would allow to test and pair corresponding XM files. :)

User avatar
DD-Indeed
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 165
Joined: Jun 07, 2014
I am a: Average DIY-guy
PlayStation Model: SCPH-5502

Post by DD-Indeed » August 15th, 2020, 8:04 pm

Okay, so it picks up them individually then from all packs, this makes things even more harder :D

Only if we could emulate just the music loading process of the game to get those playing and then get them ripped from the datastream.

Or, last chance is just to force each track playing in the Main Menu(in Options, to prevent game going into the intro video again) and then just record them as lossless WAV with good program. But do we have any idea how many songs there actually is ? My bet is that there's 10-12 songs, PC version has 15, I remember that PS1 cut something out. Or then people haven't triggered those songs in the game, as some of them are somewhat hidden into certain trigger points, where you might not get in the game.

Yagotzirck
Verified
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 131
Joined: Jul 17, 2013

Post by Yagotzirck » August 15th, 2020, 8:17 pm

I'm not on my main PC now, but IIRC there are 15 xm files extracted from UNI3.DAT, then again some of them might be unused by the game and left there since that seems to be common practice for PSX games :D

User avatar
Squaresoft74
Verified
/// PSXDEV | ELITE ///
/// PSXDEV | ELITE ///
Posts: 303
Joined: Jan 07, 2016
PlayStation Model: SCPH-7502
Location: France
Contact:

Post by Squaresoft74 » August 15th, 2020, 8:28 pm

DD-Indeed wrote: August 15th, 2020, 8:04 pm Okay, so it picks up them individually then from all packs, this makes things even more harder :D
We can already alter the XM file that gets loaded at title screen or in game (using your save file).
Problem is that the currently loaded samples bank at this point is 0008.VAB.
This is handled by sub_8007BCBC at game boot and it's harcoded to load a single file, 0008.VAB in this case.
So we can't really patch it to make it load a different VH/VB set which are 2 files.

Unless you could provide a game save that would have a bgm using a different samples bank loaded from another VH/VB set so i could start investigating from here ?

User avatar
DD-Indeed
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 165
Joined: Jun 07, 2014
I am a: Average DIY-guy
PlayStation Model: SCPH-5502

Post by DD-Indeed » August 15th, 2020, 8:36 pm

Well I can play it a bit further and hopefully it would load different songs by that, and save the game when those occur.


Btw, you can't use savestates ?

User avatar
Squaresoft74
Verified
/// PSXDEV | ELITE ///
/// PSXDEV | ELITE ///
Posts: 303
Joined: Jan 07, 2016
PlayStation Model: SCPH-7502
Location: France
Contact:

Post by Squaresoft74 » August 15th, 2020, 8:41 pm

I could but i'm using both no$psx (to debug) and BizHawk (to ram search/watch) so a memcard save file is more convenient in this case. :)

User avatar
DD-Indeed
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 165
Joined: Jun 07, 2014
I am a: Average DIY-guy
PlayStation Model: SCPH-5502

Post by DD-Indeed » August 15th, 2020, 8:46 pm

Well, if I'm gonna play it through, it takes couple days, as it's difficult game and contains so many puzzles :D But I can make regular saves and savestates, will be using Beetle PSX HW thru RetroArch.



Edit: This ain't gonna work. I have progressed pretty well in the game (despite it being clunky as hell), but the music tracks plays very randomly, and saverooms tend to have that same track going on everytime, and when you go into the hallways, the music changes to different one and you can't save there. And sometimes the music plays just briefly and then suddenly cuts, and sometimes it has long pauses.

This takes a while... :D

User avatar
Squaresoft74
Verified
/// PSXDEV | ELITE ///
/// PSXDEV | ELITE ///
Posts: 303
Joined: Jan 07, 2016
PlayStation Model: SCPH-7502
Location: France
Contact:

Post by Squaresoft74 » August 16th, 2020, 12:03 pm

No worry, take all the time you need :)
Maybe a save from a late point in game should do ?

Post Reply

Who is online

Users browsing this forum: No registered users and 9 guests