Extracting custom container (.DAT) package from disc

Post a topic about yourself to let others know your skills, hobbies, etc.
Post Reply
User avatar
DD-Indeed
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 165
Joined: Jun 07, 2014
I am a: Average DIY-guy
PlayStation Model: SCPH-5502

Extracting custom container (.DAT) package from disc

Post by DD-Indeed » August 2nd, 2020, 8:51 pm

Haven't been here for a while, but been curious to know that is there any way to extract packed file, that uses .DAT container ?

I've investigated about that format and it seems that it's used mostly as custom created container, meaning that usually those are something not in common formats, and would require basically to examine it manually and figure out how it works.


If there's any tools or people who would help with this, would be pleased. The game in particular is Martian Gothic Unification, which has pretty much all of the game's data crypted into this .DAT file that cannot be accessed. So far I examined the disc with jPSXdec and only found all of the dialogue cuts and couple images + videos. Everything else from the music to the graphics are stored in that .DAT file, that cannot be accessed.

Yagotzirck
Verified
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 131
Joined: Jul 17, 2013

Post by Yagotzirck » August 4th, 2020, 2:53 am

If file format reverse engineering is your sort of thing and have some experience with at least a hex editor I might consider a possible collaboration, even though admittedly I'm a bit rusty for this kind of stuff.

Do you need to extract resources for a specific reason, or is it more for the sake of curiosity?

User avatar
DD-Indeed
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 165
Joined: Jun 07, 2014
I am a: Average DIY-guy
PlayStation Model: SCPH-5502

Post by DD-Indeed » August 4th, 2020, 11:19 pm

Well that's the thing, I can't work on that, not my cup of tea. That's why I came here to ask, even thou I tried with some tools to get it open somehow and briefly read about that container format.

For curiosity, and also, for music files. Some games have their music pieces hidden in those packages, either as PSF sequence files or actual audio files. So far, I managed to see the music playback in one emulator, which showcased that there was multiple channels of audio running and you could mute those out individually, muting down instruments on tracks. Which suggest that the music is stored as PSF sequence (or worse case, as a tracker file, those usually are complete mess when extracted out). But I can't get to it, even thou I have tried.

Yagotzirck
Verified
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 131
Joined: Jul 17, 2013

Post by Yagotzirck » August 5th, 2020, 10:25 pm

If you're not in a rush for getting it done I might give it a look at some point in the future, I don't feel like tackling it on my own right now since I'm a bit busy with other stuff and from my personal experience reversing file formats is something that you know when you start, but don't have any clue/estimate about when you're going to finish (sometimes the file format is trivial/well organized and you're done in a couple of hours, sometimes it turns out to be a nightmare and could take months).

EDIT: I scanned UNI3.DAT with PSound and I could indeed hear some ambient samples, so you were right in saying that it uses tracker/midi-like format for the music.
Reconstructing PSF files (I guess you're only interested in that?) isn't exactly a walk in the park, but not impossible either; I'll see what I can do, but don't expect it to be done anytime soon (if at all).

User avatar
DD-Indeed
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 165
Joined: Jun 07, 2014
I am a: Average DIY-guy
PlayStation Model: SCPH-5502

Post by DD-Indeed » August 6th, 2020, 5:53 am

No I don't have any rush, take your time, thanks in advance!

I'm looking just to extract that package, it should contain graphic files, music files etc. Only things PSound and jPSXdec can see is those ambient/dialogue samples, but it can't open those actual music files, since those are propably saved on that UNI3.DAT package. When I tried the game with PSXeven emulator, there is the sound debug option to show up the music that's playing, and you can see that there's actually separate channels for each instruments, that you can mute, which means that the music is stored in PSF or some sort of tracker format.

Untitled 4.png
You do not have the required permissions to view the files attached to this post.

Yagotzirck
Verified
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 131
Joined: Jul 17, 2013

Post by Yagotzirck » August 6th, 2020, 10:48 pm

Well, the problem with PSF is that it's not a file format extracted as it is from the game's archives - rather, it is a container format which consists of the music samples/soundbank (usually VAB/VB or a custom equivalent format) combined with the assembly routine that plays it (usually called at each VSync interrupt), and the process to find both the playing routine and the soundbank and combining them to make a PSF is done "by hand" and therefore quite time consuming (here is a tutorial that describes the process quite well), so that's most likely the last thing that I'd deal with.

As for the archive itself, its format is quite trivial - if you open it with a hex editor you can see right at the start an array of structures, each containing a pair of little-endian 32-bit values describing the offset and size of each subentry... the "complicated" part would be discerning each sub-entry and extracting them as something which makes sense (e.g. recognizing it's an image file and building a .TIM header to put before the data in the extracted file, etc).

Stay tuned, and hopefully I'll come up with an extractor that deals with most of the subfiles inside that archive 8-)

User avatar
DD-Indeed
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 165
Joined: Jun 07, 2014
I am a: Average DIY-guy
PlayStation Model: SCPH-5502

Post by DD-Indeed » August 7th, 2020, 3:36 am

I have VGMToolbox which I have been using to extract PSF's, and I also have VGMTrans_WTL, but since this archive seems to be crypted, no luck getting those ripped from it.




Edit: PSound actually found those intrument samples within the UNI3.DAT, but they definitely are used in some sort of sequence file, whether as SEQ from PSF's or some sort of tracker files. Those other tools however can't find those sequence files.

Yagotzirck
Verified
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 131
Joined: Jul 17, 2013

Post by Yagotzirck » August 7th, 2020, 7:14 am

The archive isn't encrypted at all - like I said, I could make an extractor that spits out all of the entries within the archive in a couple of minutes if we don't consider "converting" them to formats that most tools can deal with (e.g. TIM for images, VAG for sounds, etc) - the problem is (I guess) that all that VGMToolbox does is scanning files/directories searching for .VAB/VB sound banks and SEQ/SEP sequence(midi-like) files, but if the game uses custom formats then tough luck, you must take the manual route I described earlier.

User avatar
DD-Indeed
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 165
Joined: Jun 07, 2014
I am a: Average DIY-guy
PlayStation Model: SCPH-5502

Post by DD-Indeed » August 8th, 2020, 7:44 pm

Yea, well that's sort of a dead end. But I started to think about that what about having some sort of plugin on emulator or even tool, that would dump SEQ/PSF/Full audio track from the game once the game loads it up ? I mean, the audio is processed separately, so there must be a way to seek that data stream and find the stuff along the way, rather than trying to extract it out from these compressed packs. Sort of using the game to bring those up and then dump them out along the way. Beetle PSX HW has this prototype feature for graphics, that dumps out the graphic data stream as bitmap files,that you can then replace.

Yagotzirck
Verified
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 131
Joined: Jul 17, 2013

Post by Yagotzirck » August 8th, 2020, 10:56 pm

Well I'm not going to say that automating the process is impossible, it's just a bit outside of my current capabilities :D
It would be much more feasible to launch the game with no$psx and setting some breakpoints on functions which upload samples to SPU RAM and send commands to it, in order to get a clearer picture/a starting point and check if they can be extracted manually.

Anyway, are you interested only on ripping PSFs, or the .DAT archive as a whole?
If it's the latter, understand that not all images/sprites will be clear-cut; for example, prerendered backgrounds are made of 32x32(or 64x64, I don't remember right now) tiles which would have to be recombined manually (kinda like a puzzle) once they're extracted, if you want to make some sense out of them.

Yagotzirck
Verified
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 131
Joined: Jul 17, 2013

Post by Yagotzirck » August 9th, 2020, 10:36 pm

Update:
I made a tool that extracts .DAT entries as they are - sound banks appear to be standard .VAB files, but there's also some entries with a "GT20" signature... a quick research reveals that they're compressed entries, so you were half-right in saying the archive was encrypted (it would be more correct to say some entries are compressed, but still).

I tried decompressing a few entries but they were mostly .TIM images; once I finalize the extractor and fully automate the process we'll see if we can retrieve .SEQ files as well :)
Last edited by Yagotzirck on August 10th, 2020, 4:03 am, edited 1 time in total.

User avatar
DD-Indeed
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 165
Joined: Jun 07, 2014
I am a: Average DIY-guy
PlayStation Model: SCPH-5502

Post by DD-Indeed » August 10th, 2020, 2:48 am

Great job! This gives at least insight for the filesystem, but the audio is still mystery. Trouble with PSF's is that they sometimes contain custom tricks and hacks, that the universal PSF converters/tools cannot recognize (such as in Driver 1/2, which use tracker format, Syphon Filter uses something completely different etc). But I have managed to rip out fully working and correct PSF's from game like Resident Evil 1-3 and Silent Hill, and converted those into Midi as well, to be used with pretty much any software.

So, if it looks like the music track sequence data is stored in some sort of unknown custom format, better forget it then, it's not reasonable to work on this for months in that case :D

Yagotzirck
Verified
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 131
Joined: Jul 17, 2013

Post by Yagotzirck » August 10th, 2020, 4:55 am

Yeah, this is a tough nut to crack; I noticed that the same "GT20" signature appears inside the executable file as well, which means there are compressed files embedded in the exe (which may or may not be the sequence files, and not necessarily in the standard .SEQ format).
I even made a RAM dump, and the fact that I couldn't find any .SEQ file in there (by searching for "SEQp"/"pQES" signatures) isn't particularly promising either :(

All in all I don't think it would be particularly difficult to extract PSFs manually, just extremely time consuming, especially considering I don't have any experience in ripping PSFs and I'm a bit rusty in file format reverse engineering in general; I might give it another try in the future though :)

Oh and by the way, there's a PC version of this game as well, did you give it a look already?
*Maybe* there are standard MIDIs for that.

User avatar
DD-Indeed
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 165
Joined: Jun 07, 2014
I am a: Average DIY-guy
PlayStation Model: SCPH-5502

Post by DD-Indeed » August 10th, 2020, 5:10 am

The PC version of this game has different mixing and actually different compositions, so they're two different OST sets basically. Also, in the PC version, the music files are saved in weird .ATF format, and I have no clue what the heck those are, been searching for hours on Google.

But, since the PC version came first and PS1 version a year later, the PC version music stuff could lead us to the system what was used with PS1 version 8-)

User avatar
DD-Indeed
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 165
Joined: Jun 07, 2014
I am a: Average DIY-guy
PlayStation Model: SCPH-5502

Post by DD-Indeed » August 10th, 2020, 7:29 am

Oh wait! I downloaded this software to extract those .ATF format packages:

https://sourceforge.net/projects/dragonunpacker/


I noticed that it has this raw unpacker, and it could seek out tracker/midi files and it turns out that the PS1 version has .XM files in that UNI3.DAT. And those are FastTracker 2 files, which was quite popular tracker software, that was also used in Driver 1/2.

I also managed to find .WAV files from those .ATF packages within the PC version, so I can finally rip out those songs from it too.

Yagotzirck
Verified
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 131
Joined: Jul 17, 2013

Post by Yagotzirck » August 10th, 2020, 7:54 am

Nice find!
I can confirm that - Found the .xm tracker files as well:
Image
They were in plain sight since the beginning, but I was so stubborn on trying to find .SEQ files that they completely went under the radar :evil:

Are you able to build .PSFs if I pass you both .XM and .VAB files, or you're fine with those you got from the PC version?

User avatar
DD-Indeed
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 165
Joined: Jun 07, 2014
I am a: Average DIY-guy
PlayStation Model: SCPH-5502

Post by DD-Indeed » August 10th, 2020, 8:31 am

The PC version has straight up .WAV files packed into those .ATF files + some instructions for handling subtitles or timings etc on .ATH files, so there's no tracker data available propably, but the PS1 has those.
Unfortunanently I couldn't get those .XM files to work, so we need to investigate more. I may try with the Driver 1/2 tracker file converter I have in my collection, maybe that can help with this.

What I don't understand that why didn't they use those .WAV files on PS1 too. Instead, they took some songs and mixed/rearranged them differently on PS1 and there's even couple completely different tracks on PS1 version.

Yagotzirck
Verified
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 131
Joined: Jul 17, 2013

Post by Yagotzirck » August 10th, 2020, 9:47 am

PSX's SPU only supports samples in Sony's custom ADPCM format, therefore converting them to single .VAG samples or to a .VAB soundbank is pretty much a forced step.
I haven't checked thoroughly, but the only difference I noticed is that PSX's soundtrack lacks some bass/sounds higher-pitched compared to the PC's one, can't tell if that's a result of samples' compression to ADPCM or if they were arranged differently.

Either way, if VGMToolbox only supports SEQ tracks I could try writing a custom driver using XMPlay2 (a library included with the PsyQ CD, not quite confident in the fact they used that for the game's music player but worth a try) and we'll see the result.

User avatar
DD-Indeed
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 165
Joined: Jun 07, 2014
I am a: Average DIY-guy
PlayStation Model: SCPH-5502

Post by DD-Indeed » August 11th, 2020, 12:29 am

Well if you want to, go ahead. I could also try to make sense with those .XM and .VAB, if you can send those, maybe I'll get them into playable state.

Gotta check out something. This game was originally developed by Creative Reality, and UK-based team, which went bankrupt at around '03, and this was their last and only 3D-title.
But, the PS1 port of this game was handled by Coyote Developments Ltd, so we propably should look for other games that they did, to see whether they used similar stuff for this game as well in audio department.


Edit: Nope, checked their European Super League football game from same year, and that had straight XA-tracks that can be played normally.


Edit2: DragonUnpacker discovered these .669 format files from that UNI3.DAT, and those are UNIS Composer tracker files, which is apparently obsolete old 8-track tracker software used for compositions even back in Amiga days. Investigating those now.

User avatar
DD-Indeed
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 165
Joined: Jun 07, 2014
I am a: Average DIY-guy
PlayStation Model: SCPH-5502

Post by DD-Indeed » August 11th, 2020, 7:19 am

Okay, so both of the game versions, PC and PS1, have .XM and .669 format files, PS1 have less of those, but I think that they cut out certain tracks from it. But I find it weird that the PC version has its music already rendered into functional WAV file, which is just packed into these custom containers. However, PS1 propably uses just the tracker file, which is then injected into the sound processing unit alongside soundfonts.

Post Reply

Who is online

Users browsing this forum: nocash and 5 guests