New user! Curious about how to reverse PS1 data files

Post a topic about yourself to let others know your skills, hobbies, etc.
Post Reply
User avatar
Lollie
Curious PSXDEV User
Curious PSXDEV User
Posts: 10
Joined: Oct 18, 2019
Contact:

New user! Curious about how to reverse PS1 data files

Post by Lollie » October 20th, 2019, 3:52 am

Hi all, I'm Lollie — primarily a visual artist, but I tinker and toy with games/gamedev a fair bit. I'm hoping for some pointers in the right direction!

I'm interested in ripping game data (currently, textures and models) from a few early PS1 games, but I know this tends to be easier said than done. Unless a game happens to use one of the more common formats (TIM, TMD, PMD, HMD), the only real option is to dig through game files and figure out how to parse them.


So, some questions:
1) Are there any resources that detail currently-known PSX file formats? Preferably header/structure information and examples. I'd like to try my hand at reading/parsing common formats first, before tackling any unknown formats. I figure that would give me a better understanding of what I should be looking for in a file.

2) What kind of tools are going to be most useful for reading through data files? I know that a hex editor is essential (I have Hexplorer, though it feels kind of limited), but is there anything else that I should consider?

3) What will I need in order to actually parse/convert a PSX data file into a useable file format? This is one area where I'm just completely clueless about where to start. I'd be grateful for any resources that cover this subject.

Thanks for your time!

Yagotzirck
Verified
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 131
Joined: Jul 17, 2013

Post by Yagotzirck » October 21st, 2019, 8:18 am

1) Your best bet is the PSYQ documentation/examples, but there aren't many games implementing the common formats you mentioned; the majority of them implemented their own headers/formats.
The good news is that, unlike in DOS where everyone pretty much implemented their own custom data formats, there isn't much flexibility in psx data formats, e.g. audio must necessarily be ADPCM, textures/sprites/etc must be in one of the formats supported by the GPU(4/8/16/24 bpp, though the latter is quite rare), and so on... this might not apply to 3D models though, but I'll leave that part to someone more knowledgeable than myself since I never tried to extract 3D stuff.

2) PSound, TIM2View and JPSXDec do a pretty good job scanning known formats/data patterns, but if they fail you pretty much have to figure things out on your own; a while ago I extracted most of Disruptor's sprites/sounds, what I did was examining the exe with IDA PRO looking for known PSYQ functions like LoadImage(), CdRead() etc, noting the addresses of interest, setting breakpoints at those addresses in NO$PSX to understand through live debugging the offsets at which the headers were being read from, what was the decompression algorithm used for the sprites... you get the idea.
Of course a hex editor is also mandatory, but you need to have some general patterns in your head first to use it efficiently(and those patterns will only come with experience).

3) Some half-decent programming skills :)
You pretty much have to write your own extractor once you have documented the original files; afterall, if you have to reverse the formats yourself it's because there wasn't any available tool to extract them in the first place :roll:


All in all, in my opinion reversing data formats is a matter of
  1. Good programming skills / assembly knowledge / ability at recognizing data structures
  2. Patience
  3. Perseverance
b) and c) are the most important points, so if you are the "I want it all, I want it now" type of guy you might as well forget about it, since reversing a file format can take from a few hours to months, depending on your level of experience and the file format itself.

User avatar
Lollie
Curious PSXDEV User
Curious PSXDEV User
Posts: 10
Joined: Oct 18, 2019
Contact:

Post by Lollie » October 25th, 2019, 4:20 am

It's good to know that there's some amount of standardization at least, even with custom format headers thrown into the mix. If anyone happens to have experience ripping 3D models and can share how they tend to work, that'd be awesome. In the meantime, I'll try PSYQ docs! I'm assuming they're part of the Technical Reference CDs in PSXdev's Downloads page?

I've played with PSound & JPSXDec, they seem pretty good at what they do! PSXPrev does a solid job sometimes as well, with textures at least - I've had way less luck with models. Honestly, that alone might answer whether models play by the same rules or not, lol.

IDA PRO is a little out of my price range. I did end up finding a PSX library for Ghidra though, and the release for Ghidra 9.1 Beta appears to work. (I'm assuming IDA PRO's libraries are going to be better overall, but I'll work with what I can afford)
Yagotzirck wrote: October 21st, 2019, 8:18 am 3) Some half-decent programming skills :)
You pretty much have to write your own extractor once you have documented the original files; afterall, if you have to reverse the formats yourself it's because there wasn't any available tool to extract them in the first place :roll:
I have... a little programming experience! Although I have a feeling that my limited experience with C# in Unity isn't going to help me out a whole lot. Patterns are in my ballpark though.

Thank you for all that! I already knew that this sort of task would be a long-term thing, so I'm treating it as such. Already had some success reading a smaller format (Ridge Racer Type 4's custom decals in memory card save files), so I'm interested in seeing how deep I can get into this.

Yagotzirck
Verified
Extreme PSXDEV User
Extreme PSXDEV User
Posts: 131
Joined: Jul 17, 2013

Post by Yagotzirck » October 27th, 2019, 3:39 am

Lollie wrote: October 25th, 2019, 4:20 am It's good to know that there's some amount of standardization at least, even with custom format headers thrown into the mix. If anyone happens to have experience ripping 3D models and can share how they tend to work, that'd be awesome. In the meantime, I'll try PSYQ docs! I'm assuming they're part of the Technical Reference CDs in PSXdev's Downloads page?
That's one way to get them, but I'd suggest to get PSY-Q SDK as well since it includes docs updated to version 4.6 if I remember correctly (though you probably don't care about that if you only want docs for the file formats you mentioned and nothing else).
I've played with PSound & JPSXDec, they seem pretty good at what they do! PSXPrev does a solid job sometimes as well, with textures at least - I've had way less luck with models. Honestly, that alone might answer whether models play by the same rules or not, lol.
Pretty much :D
You could study how the GPU/GTE operates and reverse engineer the game's exe to check the commands the game sends to them after reading/parsing the models, but honestly that's a bit too extreme if you're just starting out, lol.
IDA PRO is a little out of my price range. I did end up finding a PSX library for Ghidra though, and the release for Ghidra 9.1 Beta appears to work. (I'm assuming IDA PRO's libraries are going to be better overall, but I'll work with what I can afford)
Unless you have some personal ethic against "extreme sports" or you aren't willing to take the (albeit minimal) risks associated with them, you won't believe what you can find for free while wading fast-moving rivers... word to the wise ;)


Anyway good job with the RR decals - shows you aren't afraid to actually get your hands dirty, unlike the majority of one-posters around here opening threads asking how to start and then disappearing into the void as soon as they realize the amount of effort required :lol:

As for the extractor part, it's not that difficult... once you've documented the file format you reversed(or the parts of it you're interested in) you've done 80% of the job; shameless plug, but here you can find some examples of how an extractor is like.
Quite barebone, but they get the job done :)

Good luck with your journey!

User avatar
Lollie
Curious PSXDEV User
Curious PSXDEV User
Posts: 10
Joined: Oct 18, 2019
Contact:

Post by Lollie » October 28th, 2019, 10:34 pm

Yagotzirck wrote: October 27th, 2019, 3:39 amUnless you have some personal ethic against "extreme sports" or you aren't willing to take the (albeit minimal) risks associated with them, you won't believe what you can find for free while wading fast-moving rivers... word to the wise ;)
Y'know, I didn't even consider it for some reason, lol. I'll have to go for a swim sometime.
Yagotzirck wrote: October 27th, 2019, 3:39 amshameless plug, but here you can find some examples of how an extractor is like.
Quite barebone, but they get the job done :)
More than happy to take a plug, these are all very useful - I really just had no idea what an extractor looks like in code. Cheers!

MikeArden
What is PSXDEV?
What is PSXDEV?
Posts: 1
Joined: Jun 16, 2020

Post by MikeArden » June 16th, 2020, 8:15 pm

hey! as I am a new user here & literally amazed while going throug your hardwork for providing me these URLs.Thanks!

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 4 guests