NO$PSX V2.0 Released (12th Aug 2017)

Problems or feedback about the NO$PSX emulator by Martin Korth
User avatar
nocash
Verified
PSX Aficionado
PSX Aficionado
Posts: 541
Joined: Nov 12, 2012
Contact:

NO$PSX V2.0 Released (12th Aug 2017)

Post by nocash » August 12th, 2017, 11:04 am

Just released no$psx v2.0 (psx emulator/debugger), and also updated the SPXSPX playstation specs:
http://problemkaputt.de/psx.htm
There are a bunch of new things here and there. A lot of new stuff related to peripherals. And alongsides, two new features for the kernel clone:
* It's now working as Expansion ROM (not needed a BIOS ROM replacement), see Kernel Clone for details.
* It's now including a VCD player, see VCD Player for PSX for details.


Code: Select all

12 Aug 2017 - no$psx v2.0
- help: added PAL/NTSC Color Mods chapter (with info for older and newer boards)
- help: namco lightgun added U1 chip pinout & link to schematic (thanks nicolas)
- help: namco lightgun added U2 chip part number (BA7071F aka sync separator)
- help: scph-1150 notes: need delay, no rumble cfg cmds, different motor ctrl
- help: scph-1150 notes: field of motion 00,00-FF,FF (unlike scph-1200/scph-110)
- help: added/updated chipset/pinouts for scph-1150,scph-1180,scph-1200,scph-110
- emu/reset/init: forces [104Ah].bit6-style joypad reset (needed for deathball)
- bios clone: bugfixed chdir and findfirst functions (needed for deathball)
- bios clone: workaround on edge-triggered cdrom irqs (needed for deathball)
- bios clone: re-ordered cdrom cmd=setloc vs task=setloc (more reliable timing)
- bios clone: forces param fifo clearing (after unsupported vcd/secret cmds)
- bios clone: implemented screen centering option (formerly wasn't functional)
- xplorer/help: lots of info on Xplorer db25 protocol, memory map, chipset, etc
- datel/help: some more details on Datel db25 protocol, memory map, chipset, etc
- xplorer/datel/help: added FLASH commands, chip IDs, board detection, etc.
- help: chipless modchips chapter, with circuits/pinouts for pu-7 thru pm-41(2)
- spu/emu: allows reverb-output with DISABLED reverb-writes (raw pcm playback)
- spu/emu: supports MUTE flag from SPUCNT (mutes SPU audio, and also VCD audio)
- cdrom/help: caution Unsupported GetQ,VCD,SecretUnlock do need clr param fifo
- bios-clone: forces clearing param fifo (after SecretUnlock and VCD detection)
- bios clone: increased joypad-slot-select-delay (needed for SCPH-1150)
- bios clone: added video cd VCD software player (monochrome 10fps, 11kHz mono)
- bios clone: uses compression for GUI and VCD player (for 128Kbyte expansion)
- bios clone: uses tighter flush cache code (with loop instead of unrolled loop)
- cdrom loader: fixed crash upon newer .nrg files (those with "CUEX" chunks)
- vcd/help: specs for VCD format (iso, mpeg, mp2) (still lacks lowlevel details)
- vcd/help: added pinout and component list for scph-5903 (video cd console)
- mdec/help: q_scale=0 disables zagzig (can be useful for raw YUV-to-RGB)
- mdec/help: added note on EOB not being required after full blocks
- mdec/emu: reproduces zagzig-disable, and eob being not required on full blocks
- gpu/help: added GPU Versions chapter (all known old 160-pin gpu differences)
- card/help: updated Memory Card Notes (vmem, yaroze, compression, etc.)
- cdrom/help: added description for "cu2/bin files" (psio cdrom-image format)
- gui: fixed window positions/fullscr when taskbar at upper/left (thanx joseph)
- mips/mem: emulates main ram waitstates for unaligned mem (lwl/swl/lwr/swr)
- expansion/kernel: added installer functions in menu, utility, remote access
- expansion/kernel: new nocash bios clone version for xplorer and datel carts
- expansion/emu: prevents flash commands to destroy memory at 2AAAh/5555h
- xplorer: supports Xplorer on 80x86 side (for PC parallel port to PSX) (xboo)
- xplorer: supports Xplorer on MIPS side (in expansion version of bios clone)
- gpu/help: added pinouts for old 160-pin GPU, dual-ported VRAM, and RGB chips
- poc/help: added details on garbage_byte for memory areas without ldrb support
- poc/help: memory access times for ram/flash arm/thumb opcode/data reads
- poc/help: when rtc paused: rtc irq fires at 4096Hz (instead 1Hz)
- bios/mem: dynamically allocates psx bios (allows sizes bigger than 512Kbyte)
- mips/cop: somewhat emulates breakflags in bit0/bit1 of DCIC cop0r7 register
- snapshot: faster compression (omits large zerofilled areas from look-up-tree)
- gpu: fixed crash on NTSC-auto-center in .INI file (pic.ysiz=0) (thanx ricardo)
- help: added info on joy_mode.bit8 being clk polarity (thanx charles macdonald)

rama3
Verified
/// PSXDEV | ELITE ///
/// PSXDEV | ELITE ///
Posts: 510
Joined: Apr 16, 2017

Post by rama3 » August 12th, 2017, 11:35 am

An amazing amount of new stuff. I'll enjoy reading through it all!
And for hardware people, this reads like the bios clone for expansion cards is ready!

User avatar
LameGuy64
Verified
Psy-Q Enthusiast
Psy-Q Enthusiast
Posts: 388
Joined: Apr 10, 2013
I am a: Hobbyist Game Developer
Motto: Commercial or not, play it!
PlayStation Model: H2000/7000
Location: Philippines
Contact:

Post by LameGuy64 » August 12th, 2017, 1:32 pm

Any plans on supporting PsyQ/Programmer's Tool .sym files for debugging? Support of that would've made no$psx an excellent alternative to owning a functioning DTL-H2000 for doing debugging work. Even basic support such as symbols only would help tremendously.

Also, I would love an option that'll display cop2 instructions in the debugger as GTE commands (RTPT, NCLIP, etc) as well as displaying cop2 registers as regular r0-r31 registers or C2_* macros that represents the register's purpose (C2_XY0, C2_Z0, etc like when writing GTE routines in assembly with the macro headers in the official development kit).

I'm not really demanding much, I just thought those would be good suggestions for the next update (if there ever will be).
Please don't forget to include my name if you share my work around. Credit where it is due.

Dev. Console: SCPH-7000 with SCPH-7501 ROM, MM3, PAL color fix, Direct AV ports, DB-9 port for Serial I/O, and a Xplorer FX with Caetla 0.35.

DTL-H2000 PC: Dell Optiplex GX110, Windows 98SE & Windows XP, Pentium III 933MHz, 384MB SDRAM, ATI Radeon 7000 VE 64MB, Soundblaster Audigy, 40GB Seagate HDD, Hitachi Lite-on CD-RW Drive, ZIP 250 and 3.5" Floppy.

User avatar
nocash
Verified
PSX Aficionado
PSX Aficionado
Posts: 541
Joined: Nov 12, 2012
Contact:

Post by nocash » August 12th, 2017, 3:20 pm

Yes, I could try to add support for those .sym files. I've never used any psx SDK's, and until today I haven't even known that they can store debug info in .sym files (though I've wondered if they might have some such feature).

Do you have any specs for that .sym file format? Having some sample package with exe+sym (or cue+bin+sym) would be also useful. And are there other formats that should be also supported (for psxsdk, or whatever)?

Currently supported are my own .sym files (simple ascii/text files containing address+label pairs, plus some support for defining data fields; to see how it looks like: get the VCD Player source code, assemble it with no$psx, then view the .sym file in a text editor).
And, as no$psx is based on no$gba, it's theoretically also supporting elf/dwarf2, but I have no idea if that kind of debug info is used in psx world at all (?) and, if so, it would probably still need some small adjustments to get it working in no$psx.

Ah, yes, supporting GTE opcode/register names would be a good idea, too. I've never used the GTE too much, and wasn't really aware that my assembler/disassembler doesn't yet support it. Thanks for reminding me about it!

User avatar
gwald
Verified
Net Yaroze Enthusiast
Net Yaroze Enthusiast
Posts: 282
Joined: Sep 18, 2013
I am a: programmer/DBA
PlayStation Model: Net Yaroze
Contact:

Post by gwald » August 13th, 2017, 12:42 pm

"New no$psx version should work with the DeathBall - but only with one joypad (which makes the game rather unplayable; no$psx has some support for up to 12 joypads when emulating 12 consoles at once, but it still lacks support for more than (1) joypad per console)."

:lol: :naughty :naughty :naughty
:crying

Someone
Curious PSXDEV User
Curious PSXDEV User
Posts: 20
Joined: Aug 07, 2016

Post by Someone » August 13th, 2017, 11:47 pm

Just checked new version and found some bugs:

1. Corrupted startup "PlayStation" logo.
1.jpg
2. Doom doesn't render walls.
2.jpg
3. Reverberation also quite bugged. Best game to check it - Vagrant Story, when you walk into saving point there is sound loop anomaly appears.
You do not have the required permissions to view the files attached to this post.

User avatar
LameGuy64
Verified
Psy-Q Enthusiast
Psy-Q Enthusiast
Posts: 388
Joined: Apr 10, 2013
I am a: Hobbyist Game Developer
Motto: Commercial or not, play it!
PlayStation Model: H2000/7000
Location: Philippines
Contact:

Post by LameGuy64 » August 14th, 2017, 12:43 pm

Unfortunately, I don't know much about the file specification of .sym files but I imagine it just contains program symbols, addresses and line numbers for debuggers to use. I can provide you sample binaries, map file and source code when I have the time to help you figure out the file specification.

I think the best you could implement to no$psx's debugger for the meantime is support for .map files which you can use as a very basic symbols file. map files that the PsyQ compiler can output are just plain text files that lists all the symbols of the compiled program and their respective memory addresses. Not enough data for source-level debugging but should be good enough for basic symbols support.

By the way, can you also add an emulator option to have exceptions handling done by the emulator instead of the BIOS? So that when the program crashes, the emulator just stays at the exact point where the exception occurred so I can analyze exactly what happened on the debugger like you would on a official development system. The exception handler breaks that convenience to me especially when it overwrites some registers.

And one more thing, can you also add support for SN break commands for PC-side file access (PCDRV)? It basically works by setting certain registers (usually a0,a1,a2,a3 and v0 depending on function) and then executing a break instruction with a specific break code which tells the debugger kernel to perform a specific PC side file operation using the certain registers as parameters. I think details about this are documented in the caetla/catflap documentation because caetla supports PCDRV (albeit read only if I remember correctly) and if you want, I can write additional details about PCDRV when needed as I have a official development system with working PCDRV access.

You may also want to make the emulator ignore break instructions with the code of 1024 because the SN debugger uses that as a polling instruction that must be placed somewhere in a main loop of a program to keep the SN debugger in sync with the development unit. That way, prototype/beta software that was built using the SN tools and features said break instruction can be run on no$psx along with PCDRV support.

Actually, no$psx already has some GTE support albeit very limited. It displays it as generic cop2 instructions/registers but doesn't exactly match the conventions of the official MIPS assembler for the PSX (mostly register names). By the way, when analyzing header files from the official SDK containing GTE macros, be aware that they use invalid 32-bit words as GTE instructions and the way it works is that once you've compiled your GTE program into a object file, you need to run it through dmpsx which basically translates the words into correct cop2 instructions. I don't understand why Sony made it work like this, perhaps as an attempt to prevent Net Yaroze/leaked SDK users from being able to take advantage of the GTE?
Last edited by LameGuy64 on August 14th, 2017, 11:28 pm, edited 1 time in total.
Please don't forget to include my name if you share my work around. Credit where it is due.

Dev. Console: SCPH-7000 with SCPH-7501 ROM, MM3, PAL color fix, Direct AV ports, DB-9 port for Serial I/O, and a Xplorer FX with Caetla 0.35.

DTL-H2000 PC: Dell Optiplex GX110, Windows 98SE & Windows XP, Pentium III 933MHz, 384MB SDRAM, ATI Radeon 7000 VE 64MB, Soundblaster Audigy, 40GB Seagate HDD, Hitachi Lite-on CD-RW Drive, ZIP 250 and 3.5" Floppy.

Someone
Curious PSXDEV User
Curious PSXDEV User
Posts: 20
Joined: Aug 07, 2016

Post by Someone » August 14th, 2017, 6:39 pm

I've seen lots of SYM files on "PsyQ DevKit Samples Addon CD" and as far as I know there were different versions of their format during PS1 lifespan.

User avatar
nocash
Verified
PSX Aficionado
PSX Aficionado
Posts: 541
Joined: Nov 12, 2012
Contact:

Post by nocash » August 20th, 2017, 9:11 am

SYM Files
Different versions sounds nasty, good that you mention that!
Having some small homebrew sample for the "current/newest" version would be also great.
That Samples Addon CD might be also useful (if it's for the newest version?) (I don't know where to get that CD though).
There isn't any specs or source code for sym files yet?
I've found this https://github.com/BigEvilCorporation/ASM68KSymbolDump but it's for 68K (not mips/psx). But maybe it's better than nothing. Going by that code, the sym files would contain filename prefixed by 88h (plus a filename length value), is that similar on psx?
Oh, and what should I know about CPE files? Are they important for something? Are there any specs on the fileformat?

GTE Opcodes
How would those opcodes look like exactly in "official" source code? The registers are prefixed with "C2_", and all uppercase? And the corresponding opcode would be then something like "load r1,C2_XXX" or the like?
And the actual maths opcodes like RTPS etc. some of them can have parameters (with sf flag, and/or selecting a matrix/vector), how do those paramters look like, and how are they ordered when using multiple parameters?
Some source code and/or disassembly would be nice.

Exception Handling
Did you try the "Exception" settings in no$psx options? Don't know if they are covering what you want, but I would hope so. You can also use Ctrl+E in debugger to enable/disable those settings.

User avatar
Shadow
Verified
Admin / PSXDEV
Admin / PSXDEV
Posts: 2670
Joined: Dec 31, 2012
PlayStation Model: H2000/5502
Discord: Shadow^PSXDEV

Post by Shadow » August 20th, 2017, 7:24 pm

"Samples Addon CD" is available in the 'Downloads' section of PSXDEV (not the forums). Just grab the latest SDK and compile a simple example and you'll generate a *.SYM file during compilation with CCPSX.
Development Console: SCPH-5502 with 8MB RAM, MM3 Modchip, PAL 60 Colour Modification (for NTSC), PSIO Switch Board, DB-9 breakout headers for both RGB and Serial output and an Xplorer with CAETLA 0.34.

PlayStation Development PC: Windows 98 SE, Pentium 3 at 400MHz, 128MB SDRAM, DTL-H2000, DTL-H2010, DTL-H201A, DTL-S2020 (with 4GB SCSI-2 HDD), 21" Sony G420, CD-R burner, 3.25" and 5.25" Floppy Diskette Drives, ZIP 100 Diskette Drive and an IBM Model M keyboard.

User avatar
LameGuy64
Verified
Psy-Q Enthusiast
Psy-Q Enthusiast
Posts: 388
Joined: Apr 10, 2013
I am a: Hobbyist Game Developer
Motto: Commercial or not, play it!
PlayStation Model: H2000/7000
Location: Philippines
Contact:

Post by LameGuy64 » August 20th, 2017, 11:50 pm

I've attached a little archive containing a pre-compiled hello world program made using the official PsyQ SDK (though I'm using the 'clean' Programmer's Tool SDK but they're both identical) with source and symbol files (.sym, .map) included.

I think the different sym file versions might refer to the differences between the SN systems toolchain (the one included in the PsyQ/Programmer's Tool SDK) and the CodeWarrior toolchain (which has drastically different conventions than SN's toolchain). I'm not exactly sure though because I have not been able to compile a program with CodeWarrior as the leaked IDE requires some cracking to get it to compile an executable.

I've also included an assembler source file from a 3D game engine project I'm working on which uses macros for accessing GTE registers and executing GTE operations. It is pretty much the closest you can get to an assembler source file written by a licensed developer.

As for the CPE file format, I don't know of any documentation of the file format as Sony never released any documentation of it. You can try studying the source code of Orion's version of cpe2x. I'm not sure if cpe contains additional data used for debugging purposes.

As for the GTE opcodes and registers. If you use the macro headers when writing your assembler program with the official SDK (gtereg.h & inline_a.h), GTE registers can be specified by using the C2_* macros (ie. C2_VXY0 -> r0, C2_VZ0 -> r1, C2_OTZ -> r7) which makes writing assembly source that uses the GTE much more convenient. The macros for GTE opcodes are just simple macros that adds a word with a specific value to your program which are converted to cop2 instructions when running the object file of the compiled assembly program through dmpsx.

In the official debuggers. GTE registers are referenced by conventional register names (r0 - r31) and GTE opcodes are referenced by cop2 followed by an opcode (cop2 $00280030). The C2_* macro names and opcodes only comes to play when you're doing source-level debugging with assembler files that uses said macros.
You do not have the required permissions to view the files attached to this post.
Please don't forget to include my name if you share my work around. Credit where it is due.

Dev. Console: SCPH-7000 with SCPH-7501 ROM, MM3, PAL color fix, Direct AV ports, DB-9 port for Serial I/O, and a Xplorer FX with Caetla 0.35.

DTL-H2000 PC: Dell Optiplex GX110, Windows 98SE & Windows XP, Pentium III 933MHz, 384MB SDRAM, ATI Radeon 7000 VE 64MB, Soundblaster Audigy, 40GB Seagate HDD, Hitachi Lite-on CD-RW Drive, ZIP 250 and 3.5" Floppy.

Gh0stBlade
Interested PSXDEV User
Interested PSXDEV User
Posts: 8
Joined: Aug 21, 2017

Post by Gh0stBlade » August 21st, 2017, 12:38 am

nocash wrote:SYM Files
Different versions sounds nasty, good that you mention that!
Having some small homebrew sample for the "current/newest" version would be also great.
That Samples Addon CD might be also useful (if it's for the newest version?) (I don't know where to get that CD though).
There isn't any specs or source code for sym files yet?
I've found this https://github.com/BigEvilCorporation/ASM68KSymbolDump but it's for 68K (not mips/psx). But maybe it's better than nothing. Going by that code, the sym files would contain filename prefixed by 88h (plus a filename length value), is that similar on psx?
Oh, and what should I know about CPE files? Are they important for something? Are there any specs on the fileformat?

GTE Opcodes
How would those opcodes look like exactly in "official" source code? The registers are prefixed with "C2_", and all uppercase? And the corresponding opcode would be then something like "load r1,C2_XXX" or the like?
And the actual maths opcodes like RTPS etc. some of them can have parameters (with sf flag, and/or selecting a matrix/vector), how do those paramters look like, and how are they ordered when using multiple parameters?
Some source code and/or disassembly would be nice.

Exception Handling
Did you try the "Exception" settings in no$psx options? Don't know if they are covering what you want, but I would hope so. You can also use Ctrl+E in debugger to enable/disable those settings.
Hi,

First off, thanks for the awesome work on No$psx! I've been using it to debug Tomb Raider: Chronicles in an attempt to decompile the entire game back to C. https://github.com/Gh0stBlade/TOMB5

Stohrendorf wrote a symdumper in C# which can be downloaded here: https://github.com/stohrendorf/symdump/ It works really well, supports Tomb Raider: Chronicles, Soul Reaver and Resident Evil 2 (I've not tried any other games).

Edit: Just attatched Tomb Raider: Chronicles MAIN.EXE, MAIN.SYM, MAIN.MAP. Unfortunately this version of the game will not run without official hardware or an emulator that emulators sio parralel port to retrieve files from a PC (pcopen/pcclose/pcread).
You do not have the required permissions to view the files attached to this post.

User avatar
nocash
Verified
PSX Aficionado
PSX Aficionado
Posts: 541
Joined: Nov 12, 2012
Contact:

Post by nocash » August 24th, 2017, 7:36 am

I've reverse-engineered most of the .SYM file chunks. The five chunk numbers from the ASM68K version do match for PSX version, too (but with slightly different parameter values). But there are a lot more than five chunks... lameguy's sample uses 12 different chunks, and the tomb5.zip package contains yet four more chunks, so, as by now, there are 16 known chunks. The whole .sym file consists of a 8byte fileheader, which is then followed by the various chunks.

Code: Select all

PsyQ .SYM Files

Fileheader
  00h 4  File ID ("MND",01h)
  04h 4  Whatever (0,0,0,0)    ;TOMB5: 0,02h,0,0

Chunk 01h: Symbol (Immediate?) (eg. memsize, or membase)
Chunk 02h: Symbol (Function Address for External Function?)
Chunk 05h: Symbol (?)
Chunk 06h: Symbol (?)
  00h 4   Address/Value
  04h 1   Chunk ID (01h/02h/05h/06h)
  05h 1   Symbol Length (LEN)
  06h LEN Symbol (eg. "VSync")

Chunk 80h: Source Code Line Numbers: Address for 1 Line
  00h 4   Address (for current line)
  04h 1   Chunk ID (80h)

Chunk 82h: Source Code Line Numbers: Address for N Lines
  00h 4   Address (for N lines, starting at current line)
  04h 1   Chunk ID (82h)
  05h 1   Number of Lines (00h=None, or 02h and up?)

Chunk 84h: Source Code Line Numbers: Address for N Lines (16bit?)
  00h 4   Address (for N lines, starting at current line)
  04h 1   Chunk ID (84h)
  05h 2   Number of Lines (?)

Chunk 86h: Source Code Line Numbers: Address for Line (32bit???)
  00h 4   Address (for N lines, starting at current line)
  04h 1   Chunk ID (84h)
  05h 4   Absolute Line Number (rather than number of lines) (?)

Chunk 88h: Source Code Line Numbers: Start with Filename
  00h 4   Address (start address)
  04h 1   Chunk ID (88h=Filename)
  05h 4   First Line Number (after comments/definitions) (32bit?)
  09h 1   Filename Length (LEN)
  0Ah LEN Filename (eg. "C:\path\main.c")

Chunk 8Ah: Source Code Line Numbers: End of Source Code
  00h 4   Address (end address)
  04h 1   Chunk ID (8Ah)

Chunk 8Ch: Internal Function: Start with Filename
  00h 4    Address
  04h 1    Chunk ID (8Ch)
  05h 4    Whatever (1Eh,00h,20h,00h)  ;or 1Eh,00h,18h,00h
  09h 4    Whatever (00h,00h,1Fh,00h)
  0Dh 4    Whatever (00h,00h,00h,C0h)
  11h 4    Whatever (FCh,FFh,FFh,FFh)  ;mask? neg.offset?
  15h 4    Whatever (10h,00h,00h,00h)  <-- line number (32bit?)
  19h 1    Filename Length (LEN1)
  1Ah LEN1 Filename (eg. "C:\path\main.c")
  xxh 1    Symbol Length (LEN2)
  xxh LEN2 Symbol (eg. "VSync")

Chunk 8Eh: Internal Function: End of Function (end of chunk 8Ch)
  00h 4   Address
  04h 1   Chunk ID (8Eh)
  05h 4   Line Number                 <-- line number (32bit?)

Chunk 90h: Internal Function:Whatever90h... first instruction in main func?
Chunk 92h: Internal Function:Whatever92h... last instruction in main func?
Maybe line numbers? Or end of definitions for incoming parameters?
  00h 4   Address
  04h 1   Chunk ID (90h/92h)
  05h 4   Whatever (1Fh,00h,00h,00h)  <-- line number relative to main.start?

Chunk 94h: Type/Symbol (Simple Types?)
  00h 4   Offset (when used within a structure, or stack-N, or otherwise zero)
  04h 1   Chunk ID (94h)
  05h 2   Class (000Dh=Type.alias, 000Ah=Address, 0001h=Stack, 0002h=Addr)
  07h 2   Type (XX = 8bit,16bit,signed,etc.?)
  09h 4   Zero, or Size in Bytes (for "memblocks")
  0xh 1   Symbol Name Length (LEN)
  0xh LEN Symbol Name (eg. "size_t")

Chunk 96h: Type/Symbol (Complex Structures/Arrays?)
  00h 4   Offset (when used within a structure, otherwise zero)
  04h 1   Chunk ID (96h)
  05h 2   Class (02h=Array,08h=RefToStruct,0Dh=DefineAlias,66h=StructEnd)
  07h 2   Type (0xh=Small, 3xh=WithArrayStuff?) (same/similar as in chunk 94h)
  09h 4     Struct Size in Bytes
  0Dh 2     Array Dimensions (DIM) (0=none) ;eg. [3][4] --> 0002h
  0Fh DIM*4 Array Entries per Dimension     ;eg. [3][4] --> 00000003h,00000004h
  xxh 1     Internal Fake Name Length (LEN1) (0=none)
  xxh LEN1  Internal Fake Name  (eg. ".1fake")
  xxh 1     Symbol Name Length (LEN2)
  xxh LEN2  Symbol Name (eg. "r")

Class definition (in chunk 94h) (and somewhat same/similar in chunk 96h)
(looks same/similar as C_xxx class values in COFF files!)
  0001h = Local variable              (with Offset = negative stack offset)
  0002h = Global variable or Function (with Offset = address)
  0008h = Item in Structure           (with Offser = offset within struct)
  0009h = Incoming Function param     (with Offset = index; 0,4,8,etc.)
  000Ah = Type address / struc start? (with Offset = zero)
  000Dh = Type alias                  (with Offset = zero)

Type definition (in chunk 94h/96h)
(maybe lower 4bit=type, and next 4bit=usage/variant?)
(looks same/similar as T_xxx type values in COFF files!)
  0000h =
  0001h =
  0002h =
  0003h =                (16bit signed?)
  0004h = int            (32bit signed?)
  0005h =
  0006h =
  0007h =
  0008h = (address)      (32bit unsigned?) (with Definition=000Ah)
  0009h =
  000Ah =
  000Bh =
  000Ch = u_char         (8bit unsigned?)
  000Dh = u_short,ushort (16bit unsigned?)
  000Eh = u_int          (32bit unsigned?)
  000Fh = u_long         (64bit unsigned?) (or rather SAME as above?)
  0021h = function with 0 params, and/or return="nothing"?
  0024h = main function with 2 params, and/or return="int"?
  0052h = argv           (string maybe?)
  0038h = GsOT           (huh?)
  00F8h = GsOT_TAG       (huh?)
  00FCh = PACKET         (huh?)
  ??    = float,bool,string,ptr,packet,(un-)signed8/16/32/64bit,etc
  ??    = custom struct
What is that Stohrendorf for Tomb Raider and other games? I am double-confused...

Do you mean that some/all (?) PSX games were shipped with .SYM files on the retail discs?
Or did you find a tomb raider .SYM file from some non-retail prototype version?

Looking at the stohrendorf/symdump source code...
The Block.cs file seems to be somehow dealing with chunk ID numbers, but it's masking them to 7bit values, and then deals with them in decimal. For example, value "20" (aka 14h) seems to refer to chunk 94h in 8bit notation.
Or so, I couldn't really say that I understand what the stohrendorf source code is supposed to do. The chunk numbers seem to be also processed in various files other than Blocks.cs, but I haven't yet found if or where it's doing the "real" stuff (like processing the parameters that belong to those chunks).

*** SYM FILE CONTENTS ***
The stuff in the chunks consists of three main parts:
- Source code: filenames/line numbers
- Type Info: variables/structures/arrays
- Symbol: immediates/function labels

**** SYMBOLS ****
That part looks fairly simple, just assigning label strings to addresses (and immediates).

**** TYPE INFO ****
I am not really planning to support that in no$psx. Anyways, this stuff allows to define structures, and to assign those structures (or predefined basic types) to variables.
The "Class/Type" values seem to be same as in COFF files (apart from that, the "chunk-based" .SYM files are quite different than "section-based" COFF files).
Anyways, it would be nice to verify if the Type values are really same as in COFF. Could somebody make a .SYM file the defines variables with all possible basic types? Ie. boolean, strings, and signed & unsigned 8bit/16bit/32bit values, pointers, and 64bit values and float values (if that's supported), etc?
Best would be if the name of the variables is reflecting the type (eg. "my_int = int"). Plus some explanatation what it's doing (for example, "int"... I guess... that means a "signed 32it" value... or is that wrong?)
And, also make a structure, where one of the structure elements refers to another structure!

**** LINE NUMBERS ****
Decoding the line numbers works quite well for lameguy's sample. But for tomb5, I am ending up with DoGameflow in Line=1771 but going by the https://github.com/Gh0stBlade/TOMB5/blo ... GAMEFLOW.C file, it should be around Line=93. Any ideas what's going wrong there?
Is the .sym file based on the GAMEFLOW.C source code? Is it an older prototype .sym that doesn't really belong to the decompiled source code? Or would be possible that leading include files do "insert" extra lines?

Also, I am not sure how the more complex line number cases are supposed to work, for example, this snippet from within Tomb5\GAME\CONTROL.C (the line numbers seem to suffer the same problem as mentioned above) (CONTROL.C has only 943 lines):

Code: Select all

Src.Line32    Addr=0001E38C, Line=451            ;set absolute line
Src.Line8     Addr=0001E3A4, Line=452..456       ;assign several lines (or assign one line, and SKIP the other lines?)
Src.Line      Addr=0001E3B8, Line=457            ;assign one line (and increment to next line)
Src.Line16    Addr=0001E3B8, Line=458..1113      ;assign several lines (or assign one line, and SKIP the other lines?)
Src.Line      Addr=0001E3E4, Line=1114           ;assign one line (and increment to next line)
Src.Line16    Addr=0001E3E4, Line=1115..1377     ;assign several lines (or assign one line, and SKIP the other lines?)
Src.Line8     Addr=0001E3F4, Line=1378..1379     ;assign several lines (or assign one line, and SKIP the other lines?)
Src.Line32    Addr=0001E3F8, Line=1377           ;set absolute line
Src.Line8     Addr=0001E400, Line=1378..1384     ;assign several lines (or assign one line, and SKIP the other lines?)
Src.Line32    Addr=0001E410, Line=1377           ;set absolute line
That's somehow assigning multiple lines to address 0001E3F4.
And the other way around, line 1378 is assingned to multiple addresses.
I guess such thinks might happen if a HLL read-modify-write operation gets scattered to several ASM opcodes (and the opcodes getting ordered noncontinously for load-delay and branch-delay reasons).
But I am not sure how that stuff could/should be displayed in a debugger window. A screenshot showing how the code near 0001E3F4 is getting displayed in PsyQ debugger would be nice.

Btw. people will probably hate that... but I like to keep displaying ASM code in no$psx, and have it displayed "mixed" with HLL code (ie. HLL source code lines, followed by the corresponding disassembled ASM opcodes) (however that would work out when the ASM opcodes are having different ordering than the HLL code lines).
Last edited by nocash on August 25th, 2017, 1:42 pm, edited 2 times in total.

Gh0stBlade
Interested PSXDEV User
Interested PSXDEV User
Posts: 8
Joined: Aug 21, 2017

Post by Gh0stBlade » August 24th, 2017, 12:56 pm

nocash wrote:
What is that Stohrendorf for Tomb Raider and other games? I am double-confused...
Stohrendorf is the name of the programmer who wrote the PSX Sym file dumper I'm using for my project.
nocash wrote: Do you mean that some/all (?) PSX games were shipped with .SYM files on the retail discs?
Or did you find a tomb raider .SYM file from some non-retail prototype version?
No, Tomb Raider 2 (PAL) shipped with a corrupt .sym file (PALMAIN.SYM) unfortunately it's 0KB. The .sym file for Tomb Raider: Chronicles (TR5) is from a leaked Tomb Raider zip file containing various tools for the game, used to create levels for the PlayStation version. The other games i mentioned had .sym files included in beta versions which are publicly available. Though the files I attached aren't retail majority of the code is the same and I'm adding the differences from the final NTSC 1.0 version in my repository too.
nocash wrote: **** LINE NUMBERS ****
Decoding the line numbers works quite well for lameguy's sample. But for tomb5, I am ending up with DoGameflow in Line=1771 but going by the https://github.com/Gh0stBlade/TOMB5/blo ... GAMEFLOW.C file, it should be around Line=93. Any ideas what's going wrong there?
Is the .sym file based on the GAMEFLOW.C source code? Is it an older prototype .sym that doesn't really belong to the decompiled source code? Or would be possible that leading include files do "insert" extra lines?

Also, I am not sure how the more complex line number cases are supposed to work, for example, this snippet from within Tomb5\GAME\CONTROL.C (the line numbers seem to suffer the same problem as mentioned above) (CONTROL.C has only 943 lines):
No, the MAIN.EXE, MAIN.MAP, MAIN.SYM files I attached are not the result of compiling my TOMB5 repository on GitHub. The attached .sym file has been produced from the original Tomb Raider code. The TOMB5 repository i linked does not contain any of the original code. It's translated to C by hand from the attached binary (MAIN.EXE) disassembled in IDA Pro. Variable/method names all come from MAIN.SYM's contents. This is why line numbers do not match up, TOMB5 repo lacks comments etc which will have pushed the line numbers in the original code. The linked repo is not the original code so the lines won't be 1:1 for the files I attached : - ).

User avatar
Shadow
Verified
Admin / PSXDEV
Admin / PSXDEV
Posts: 2670
Joined: Dec 31, 2012
PlayStation Model: H2000/5502
Discord: Shadow^PSXDEV

Post by Shadow » August 24th, 2017, 6:00 pm

It's better to reverse the symbol files from the Psy-Q example NO$CASH that LameGuy64 shared. That way you can link and match it to a working and 1:1 copy of the source code which was used to generate the actual *.SYM file.
Development Console: SCPH-5502 with 8MB RAM, MM3 Modchip, PAL 60 Colour Modification (for NTSC), PSIO Switch Board, DB-9 breakout headers for both RGB and Serial output and an Xplorer with CAETLA 0.34.

PlayStation Development PC: Windows 98 SE, Pentium 3 at 400MHz, 128MB SDRAM, DTL-H2000, DTL-H2010, DTL-H201A, DTL-S2020 (with 4GB SCSI-2 HDD), 21" Sony G420, CD-R burner, 3.25" and 5.25" Floppy Diskette Drives, ZIP 100 Diskette Drive and an IBM Model M keyboard.

User avatar
nocash
Verified
PSX Aficionado
PSX Aficionado
Posts: 541
Joined: Nov 12, 2012
Contact:

Post by nocash » August 25th, 2017, 12:40 pm

Gh0stBlade wrote:The linked repo is not the original code so the lines won't be 1:1 for the files I attached : - ).
Good to know, then I don't have to worry about the mismatching line numbers. Even without matching source code, the the huge 1.5Mbyte .SYM file from tomb5 is very useful to make sure that the debugger won't stumble over unknown chunk numbers in bigger projects, and also interesting to see that "commercial" games used the same .SYM format as homebrew ones. And the smaller .SYM file from LameGuy is also very useful for getting started with some minimalistic sample. Many thanks for both of the .SYM files!

Now, a few more sample with source code, and a bit bigger than LameGuy's sample (with some more code, and more than one source code file) would be nice. Can somebody compile some bigger/medium sized projects? Like from homebrew open source stuff, or samples from old CDs, or, are there even "big" commercial titles that became open source? I would need the source files + EXE + SYM.

And a sample with some uncommon corner cases would be interesting...
* Something that uses variables with all basic types (like boolean, float, etc, as mentioned in my previous post).
* And variables that contain pointers to such basic types, plus arrays of such basic types.
* Something with "nested" source code files (eg. code that uses "include" to insert code from another file in the middle of the file... or even in the middle of a function).
* And maybe a file that starts with more than 65536 blanks lines (and/or comment lines), just to see if linenumbers can really exceed 16bit space.
* A comment with around 140 lines in the middle of a function (which might be represented as unsigned 8bit line step, or as signed 16bit step, depending on whether the .SYM files uses (un-)signed values).
* And some small sample that has one opcode derived from multiple source code lines (eg. I guess you could spell a simple instruction like "x=5" across multiple lines, with "x" and "=" and "5" in separate lines, plus blank lines inserted inbetween of it).
* Some ASM Code
* Some Macros
* Some code with GOTO statement
* Functions with more than 4 parameters (=which have the extra params on stack instead of in cpu registers)
* Anything else you could think of.

CPE Files vs EXE Files
Would it make any sense to support .CPE files in no$psx? The .CPE files don't contain any debug info or other useful stuff, so they are essentially same as .EXE files. The sample from LameGuy contained both .EXE and .CPE, so it would be pointless to add .CPE support for it (and at worst, it would cause useless confusion when showing both file(extensions)s in the Open File dialog).
Or are there any cases where you have only the .CPE file, and can't easily convert it to .EXE format yourself?

Btw. the CPE format is, based on source code for "cpebin" and "pxe2cpe" tools from "SurfSmurf / Movement^[damn] Rednex" (and also based on examining LameGuy's sample):

Code: Select all

PsyQ .CPE Files

Fileheader
  00h 4   File ID ("CPE",01h)

Chunk 00h: End of File
  00h 1   Chunk ID (00h)

Chunk 01h: Load Data
  00h 1   Chunk ID (01h)
  01h 4   Address (usually 80010000h and up)
  05h 4   Size (LEN)
  09h LEN Data (binary EXE code/data)
Theoretically, this could contain the whole EXE body in a single chunk.
However, the PsyQ files are usually containing hundreds of small chunks (with
each function and each data item in a separate chunk). For converting CPE to
EXE, use "ExeOffset = (CpeAddress AND 1FFFFFFFh)-10000h+800h".

Chunk 02h: Run Address (whatever, optional, usually not used in CPE files)
  00h 1   Chunk ID (02h)
  01h 4   Address
Unknown what this is. It's not the entrypoint (which is set via chunk 03h).
Maybe intended to change the default load address (usually 80010000h)?

Chunk 03h: Set Reg X to longword y (32bit) (LEN=4)
Chunk 04h: Set Reg X to word y     (16bit) (LEN=2)
Chunk 05h: Set Reg X to byte y     (8bit) (LEN=1)
Chunk 06h: Set Reg X to triplet y  (24bit) (LEN=3)
  00h 1   Chunk ID (03h..06h)
  01h 2   Register (usually 0090h=Initial PC, aka Entrypoint)
  03h LEN Value (8bit..32bit)

Chunk 07h: Select Workspace (whatever, optional, usually not used in CPE)
  00h 1   Chunk ID (07h)
  01h 4   Workspace number (usually 00000000h)

Chunk 08h: Select Unit (whatever, usually first chunk in CPE file)
  00h 1   Chunk ID (08h)
  01h 1   Unit (usually 00h)

Example from lameguy's sample.cpe:
  0000h 4    File ID ("CPE",01h)
  0004h 2    Select Unit 0            (08h,00h)
  0006h 7    Set Entrypoint 8001731Ch (03h,0090h,8001731Ch)
  000Dh 0Dh  Load   (01h,800195F8h,00000004h,0,0,0,0)
  001Ah ..   Load   (01h,80010000h,0000002Bh,...)
  004Eh ..   Load   (01h,8001065Ch,00000120h,...)
  0177h ...  Load   (01h,8001077Ch,0000012Ch,...)
  02ACh ...  Load   (01h,800108A8h,000000A4h,...)
  ...   ...  Load   (...)
  98F4h ...  Load   (01h,800195F0h,00000008h,...)
  9905h 1    End    (00h)
So far, it doesn't have anything that isn't found in EXE files. In fact, it lacks some things like the initial stackpointer value from EXE headers, as well as extra parameters that are normally defined in SYSTEM.CNF (unless those things can be somehow defined in CPE, too, maybe via further "Set Register" chunks).
Last edited by nocash on August 26th, 2017, 1:32 am, edited 1 time in total.

User avatar
LameGuy64
Verified
Psy-Q Enthusiast
Psy-Q Enthusiast
Posts: 388
Joined: Apr 10, 2013
I am a: Hobbyist Game Developer
Motto: Commercial or not, play it!
PlayStation Model: H2000/7000
Location: Philippines
Contact:

Post by LameGuy64 » August 26th, 2017, 12:16 am

Great to see PsyQ sym support is slowly coming to fruition.

I've attached another archive containing a piece of a project that I've been working on for a long while as of late. Its a 3D game engine of my very own (Project Scarlet) and it comes with multiple C and assembler (not in-line) files so it should be a pretty nice sample to test your debugger implementation on more complex projects. I don't mind releasing the sources as I plan to open source it soon anyway.

As for CPE support, I think it would be pretty nice to have it anyway even though it doesn't have much of a use when it is more or less the same as a PS-EXE minus the ability to set initial register values (such as sp). Catflap supports CPE and I don't think it would make much effort to implement support for loading CPE files anyway.

I don't think there's ever a case where you cannot convert a CPE into a PS-EXE considering how there's hardly anything on the format that may cause that I think.
You do not have the required permissions to view the files attached to this post.
Please don't forget to include my name if you share my work around. Credit where it is due.

Dev. Console: SCPH-7000 with SCPH-7501 ROM, MM3, PAL color fix, Direct AV ports, DB-9 port for Serial I/O, and a Xplorer FX with Caetla 0.35.

DTL-H2000 PC: Dell Optiplex GX110, Windows 98SE & Windows XP, Pentium III 933MHz, 384MB SDRAM, ATI Radeon 7000 VE 64MB, Soundblaster Audigy, 40GB Seagate HDD, Hitachi Lite-on CD-RW Drive, ZIP 250 and 3.5" Floppy.

User avatar
nocash
Verified
PSX Aficionado
PSX Aficionado
Posts: 541
Joined: Nov 12, 2012
Contact:

Post by nocash » August 30th, 2017, 2:37 am

Thanks for the scarlet sample! With the multiple source code files it looks good for testing some more complicated cases.

Having a PSXSDK sample from somebody would be also nice! I assume that it does have debug info stored in ELF files (?) which would be even easier to implement because the no$gba-based code in no$psx is already containing support for that format. And it might also help for testing the GUI's source line display before adding SYM file format.

In theory, PsyQ SYM support should be easy, it might require some nasty patchwork in 10-20 places though. The debugger is currently supporting my own ASCII .SYM files, the no$gba-style ELF files, and the PsyQ .SYM files as 3rd format might make the code really messy. Well, I guess I'll just have to start doing it step by step, and maybe clean-up some code.

Xavi92
Verified
C Programming Expert
C Programming Expert
Posts: 161
Joined: Oct 06, 2012
PlayStation Model: SCPH-5502
Contact:

Post by Xavi92 » August 31st, 2017, 8:34 am

Hi nocash,

Sorry for the delay - I've been quite busy at work and had no time to dedicate on this. I've recompiled my game with "-g" compiler enabled regarding the game part. Source code can be identified from output ELF file by using mipsel-unknown-elf-objdump -S

https://github.com/XaviDCR92/Airport/bl ... IRPORT.elf

Full source code can be found on the repository:
https://github.com/XaviDCR92/Airport/tree/master/Source

Hope it helps!

User avatar
nocash
Verified
PSX Aficionado
PSX Aficionado
Posts: 541
Joined: Nov 12, 2012
Contact:

Post by nocash » October 26th, 2017, 7:02 am

I am still working on this. Displaying source code lines for SYM and ELF files seems to be working okay.

SYM file research/support is rather unsatisfying since the basic "class/type" constants are still unknown. I don't know what I could do about it... I've already asked twice... so well, asking another time: Can someboy make a SYM/EXE/source code sample that does assign all possible types, like assingning my_int = int, my_short = short, etc.? If you know what those types are referring to, please also add that in the comments (eg. signed/unsiged 8bit/16bit/32bit etc).

For the COP2/GTE, I am just adding support for register names like C2_OTZ. Or actually, currently I have it named "cop2_otz" instead of "C2_OTZ", which looks better to me, but I could change it to use the "official" uppercase name (if that's needed?)

Now I am wondering if there's some common ASM syntax for COP0 registers, too? Eg. SR, EPC, CAUSE etc. or should they be called C0_SR, C0_EPC, C0_CAUSE like GTE ones?
In case of BadVaddr I am unsure if there's an official name for it, or if it's BadVaddr itself being the official name. I've seen some document that seems to have abbreviated it to BADV, but I don't know if that's common/official, or if it's just slang?

And for GTE opcode/macros... I haven't extracted them for the SDK yet. Would be cool if somebody could post the relevant code/definitions, or even better: make a SYM/EXE/source code sample that contains all possible GTE opcodes. Including different variations (assuming that some of the macros do have parameters... or are there separate different macros (instead of using one macro+parameter) for that purpose?)

Post Reply

Who is online

Users browsing this forum: No registered users and 4 guests