HC16 Processor: managing the value of XK, YK, ZK

[rimwall]

Hoping for help with an issue that has me stumped. Have written a processor language for the HC16. First time writing a language. Have used HCS08 as a starting point. Language is written, compiles ok & disassembles ok. The decompilation looks quite messy, but I haven't turned my mind to decompilation yet.

The HC16 (20 bit) is backward compatible with the HC11 (16 bit). Manual here: https://www.nxp.com/docs/en/reference-manual/CPU16RM.pdf. The HC16 constructs 20 bit addresses by a combination of 16 bit index registers (IX, IY, IZ) and 4 bit prepended extension registers (XK, YK, ZK). In the language spec I have modelled the 4 bit extension registers as 1 byte registers.

My problem boils down to what values the disassembler uses for XK, YK, ZK. If I don't specify a value, the disassembler assumes a nil value which means it gets most RAM references wrong because the processor RAM is in addresses 0x020000 to 0x028000. If I use <context_data> (in the cspec file) to set a value of 2, the disassembler uses the specified value for a little while and gets the addresses correct. But the diassembler then seems to 'forget' the value of 2 and reverts back to a value of zero after any operation on that extension register.

See below example for the value of ZK... In this example, <context_data> has specified ZK to be 2 at all addresses. At 0x34cc, in the LDAB instruction, the disassembler uses the 16 bit immediate value of 0x1628 with the ZK value of 0x02 to get the correct reference at 0x021628, which is in RAM. However, lower down, at 0x34e4 in the BSET instruction, the disassembler uses the immediate value of 0x1638 with a ZK value of 0x00 to get the incorrect reference 0x001638 which is in ROM, not RAM.

The only intervening operation on ZK is the PULM instruction which pulls the value of ZK (and other registers) from the stack, but the value of ZK is unchanged from when it was pushed onto the stack by the PSHM instruction at 0x34c2.

My understanding is that Ghidra is a 'static' view of the code and does not track the values of the registers. "Set Register Value" may help, but I am assuming this has the same effect as <context_data> (ie) the setting will be 'forgotten'. I have tried "Set Register Value" but weirdly it doesn't seem to change the disassembled address. But, even if "Set Register Value" worked, this seems like a very manual approach. Hence, I am wondering if there is some way to systematically help the disassembler get the correct value for XK, YK, ZK, and therefore get the correct address references?

I am using Ghidra 10.1 if that has any impact.

Relevant excerpts from the slaspec file:

oprx16_8_Z:		imm16,IZ  is imm16 & IZ { addr:3 = (zext(ZK) << 16) + zext(IZ) + sext(imm16:2); export *:1 addr; }

# LDAB IND16,Z gggg
:LDAB oprx16_8_Z            is Prebyte=0x17 & op8=0xE5; oprx16_8_Z {
	B = oprx16_8_Z;
	$(ZF) = (B == 0);
	$(N) = (B s< 0);
	$(V) = 0;
}

# BSET IND16,Z mm gggg
:BSET msk8, oprx16_8_Z	      is Prebyte=0x0 & op8=0x29; msk8; oprx16_8_Z {
	local result:1 = oprx16_8_Z | msk8;
	oprx16_8_Z = result;
	$(N) = (result s< 0);
	$(ZF) = (result == 0);
	$(V) = 0;
}

# PSHM IMM8 ii
:PSHM iopr8i						is Prebyte=0x0 & op8=0x34; iopr8i {
	local tempEK:2 = zext(EK[0,4]) << 12;
	local tempXK:2 = zext(XK[0,4]) << 8;
	local tempYK:2 = zext(YK[0,4]) << 4;
	local tempZK:2 = zext(ZK[0,4]);
	local temp:2 = tempEK + tempXK + tempYK + tempZK;
	if (iopr8i[0,1] == 0) goto <next1>;
		Push2(D);
	<next1>
	if (iopr8i[1,1] == 0) goto <next2>;
		Push2(E);
	<next2>
	if (iopr8i[2,1] == 0) goto <next3>;
		Push2(IX);
	<next3>
	if (iopr8i[3,1] == 0) goto <next4>;
		Push2(IY);
	<next4>
	if (iopr8i[4,1] == 0) goto <next5>;
		Push2(IZ);
	<next5>
	if (iopr8i[5,1] == 0) goto <next6>;
		Push2(temp);
	<next6>
	if (iopr8i[6,1] == 0) goto <next7>;
		Push2(CCR);
	<next7>
}

# PULM IMM8 ii
:PULM iopr8i				is Prebyte=0x0 & op8=0x35; iopr8i {
	local temp:2;
	if (iopr8i[0,1] == 0) goto <next1>;
		Pull2(CCR);
	<next1>
	if (iopr8i[1,1] == 0) goto <next2>;
		Pull2(temp);
		EK = zext(temp[12,4]);
		XK = zext(temp[8,4]);
		YK = zext(temp[4,4]);
		ZK = zext(temp[0,4]);
	<next2>
	if (iopr8i[2,1] == 0) goto <next3>;
		Pull2(IZ);
	<next3>
	if (iopr8i[3,1] == 0) goto <next4>;
		Pull2(IY);
	<next4>
	if (iopr8i[4,1] == 0) goto <next5>;
		Pull2(IX);
	<next5>
	if (iopr8i[5,1] == 0) goto <next6>;
		Pull2(E);
	<next6>
	if (iopr8i[6,1] == 0) goto <next7>;
		Pull2(D);
	<next7>
}

macro Push2(operand) {
	paddr:3 = SKSP;
	*paddr = operand;
	SKSP = SKSP - 2;
}

macro Pull2(operand) {
	SKSP = SKSP + 2;
	paddr:3 = SKSP;
	operand = *paddr;
}

Disassembly...

[rimwall]

And yet, I have now found some examples where the disassembler does correctly adjust the value of XK, YK, ZK. See below example. In this example the value of YK is set to 0xf using <context_data> in the cspec file (with no address restrictions) :
LDAB - The value of 0x2 is loaded into register B.
TBYK - transfer value of B into YK - so YK becomes 0x2 (so the <context_data> value is ignored)
LDY - transfer the operand 0x1608 into IY
STAA - store the value at YK:IY + 0xb (address 0x21613) in register A

Strange. It seems it is possible to get the right values for XK, YK, ZK. How do I get it to do this consistently?

Is it to do with immediate values -vs- values from pointers?

iopr8i:			"#"imm8  is imm8 { export *[const]:1 imm8; }
iopr16i:		"#"imm16  is imm16 { export *[const]:2 imm16; }
oprx16_8_Y:		imm16,IY  is imm16 & IY { addr:3 = (zext(YK) << 16) + zext(IY) + sext(imm16:2); export *:1 addr; }

# LDAB IMM8 ii
:LDAB iopr8i         	   is Prebyte=0x0 & op8=0xF5; iopr8i {
	B = iopr8i;
	$(ZF) = (B == 0);
	$(N) = (B s< 0);
	$(V) = 0;
}

# TBYK
:TBYK						is Prebyte=0x37 & op8=0x9D {
	YK[0,4] = B[0,4];
}

# LDY IMM16 jj kk
:LDY iopr16i            is Prebyte=0x37 & op8=0xBD; iopr16i {
	IY = iopr16i;
	$(ZF) = (IY == 0);
	$(N) = (IY s< 0);
	$(V) = 0;
}

# STAA IND16,Y gggg
:STAA oprx16_8_Y            is Prebyte=0x17 & op8=0x5A; oprx16_8_Y {
	oprx16_8_Y = A;
	$(ZF) = (A == 0);
	$(N) = (A s< 0);
	$(V) = 0;
}

[GhidorahRex]

Can you copy your cspec and pspec here as well? Are the XK, YK, ZK registers listed as unaffected in the cspec?

[rimwall]

Hi, thanks for helping out!

cspec and pspec shown below.

When the cspec has no settings for XK, YK, ZK the disassembler seems to keep their value at zero (I haven't checked this exhaustively)

The examples in the first two posts are consistent with the settings shown below in the cspec (ie) YK set to 0xf and ZK set to 2 for all addresses

.cspec (note that I don't know the proper default function prototype - this is just a guess)

<?xml version="1.0" encoding="UTF-8"?>

<compiler_spec>
  <data_organization>  <!-- These tags need to be verified -->
     <absolute_max_alignment value="0" />
     <machine_alignment value="2" />
     <default_alignment value="2" />
     <pointer_size value="2" />
     <wchar_size value="4" />
     <short_size value="2" />
     <integer_size value="4" />
     <long_size value="4" />
     <long_long_size value="8" />
     <float_size value="4" />
     <double_size value="8" />
     <long_double_size value="8" />
  </data_organization>

  <global>
    <range space="RAM"/>
  </global>
  
  <context_data>
	<tracked_set space="RAM">
		<set name="YK" val="0xf"/> 
		<set name="ZK" val="2"/>
	</tracked_set>
  </context_data>    
  
  <stackpointer register="SKSP" space="RAM" growth="negative"/>
  
  <default_proto>
      <prototype name="__asmA" extrapop="2" stackshift="2" strategy="register">
      <input>
        <pentry minsize="1" maxsize="1">
          <register name="A"/>
        </pentry>
        <pentry minsize="1" maxsize="1">
          <register name="B"/>
        </pentry>
        <pentry minsize="2" maxsize="2">
          <register name="D"/>
        </pentry>
        <pentry minsize="2" maxsize="2">
          <register name="E"/>
        </pentry>
        <pentry minsize="1" maxsize="2">
          <register name="IX"/>
        </pentry>
        <pentry minsize="1" maxsize="2">
          <register name="IY"/>
        </pentry>
        <pentry minsize="1" maxsize="2">
          <register name="IZ"/>
        </pentry>
        <pentry minsize="1" maxsize="500" align="2">
          <addr offset="2" space="stack"/>
        </pentry>
      </input>
      <output>
        <pentry minsize="1" maxsize="2">
          <register name="D"/>
        </pentry>
      </output>
      <unaffected>
        <register name="SKSP"/>
      </unaffected>
    </prototype>
  </default_proto>

</compiler_spec>

.pspec

<processor_spec>
  <programcounter register="PKPC"/>

  <default_symbols> <!-- names not updated, no functionality -->
	<symbol name="VECTOR_00000" address="00000" entry="true" type="code_ptr"/>

[snip multiple rows]

	<symbol name="VECTOR_001FC" address="001FC" entry="true" type="code_ptr"/>
	<symbol name="VECTOR_001FE" address="001FE" entry="true" type="code_ptr"/>

  </default_symbols>
</processor_spec>

[GhidorahRex]

If you enable the pcode field in the instruction listing you should be able to see the individual pcode operations for each instruction. In the Listing display while highlighting an instruction, click on "Edit Listing Fields" then right-click on Pcode and Enable Field.

It could be that the pcode for the PSHM or PULM isn't quite right. That might be the best way to tell.

[rimwall]

Hi, I've started looking at the decompilation now. The PSHM and PULM instructions always result in an error such as "WARNING: Removing unreachable block (RAM,0x001c4c)". Not sure why?

[rimwall]

Thanks, I've had a look at the pcode. I can see how PULM from the slaspec file is reflected in the pcode. If I understand the syntax correctly, perhaps the problem is where LOAD is attempting to put 8 bytes into a 2 byte register (eg: CCR). I don't know where it is getting 8 bytes from? pcode and regsiter definitions below. PULM definition and associated macros are earlier in the thread.

          001c4c 35 01           PULM       #0x1
               (unique, 0xf0a80, 1) = INT_AND (const, 0x1, 1), (const, 0x1, 1)
               (unique, 0xf0b00, 1) = INT_EQUAL (unique, 0xf0a80, 1), (const, 0x0, 1)
               CBRANCH <0>, (unique, 0xf0b00, 1)
               (register, 0x23, 3) = INT_ADD (register, 0x23, 3), (const, 0x2, 3)
               (register, 0x30, 2) = LOAD (const, 0x1f1, 8), (register, 0x23, 3)
             <0>
               (unique, 0xf0c00, 1) = INT_RIGHT (const, 0x1, 1), (const, 0x1, 4)
               (unique, 0xf0c80, 1) = INT_AND (unique, 0xf0c00, 1), (const, 0x1, 1)
               (unique, 0xf0d00, 1) = INT_EQUAL (unique, 0xf0c80, 1), (const, 0x0, 1)
               CBRANCH <1>, (unique, 0xf0d00, 1)
               (register, 0x23, 3) = INT_ADD (register, 0x23, 3), (const, 0x2, 3)
               (unique, 0xf0d80, 2) = LOAD (const, 0x1f1, 8), (register, 0x23, 3)
               (unique, 0xf0e00, 2) = INT_RIGHT (unique, 0xf0d80, 2), (const, 0xc, 4)
               (register, 0x40, 1) = SUBPIECE (unique, 0xf0e00, 2), (const, 0x0, 4)
               (unique, 0xf0f80, 1) = SUBPIECE (unique, 0xf0d80, 2), (const, 0x1, 4)
               (register, 0x41, 1) = INT_AND (unique, 0xf0f80, 1), (const, 0xf, 1)
               (unique, 0xf1100, 2) = INT_RIGHT (unique, 0xf0d80, 2), (const, 0x4, 4)
               (unique, 0xf1180, 1) = SUBPIECE (unique, 0xf1100, 2), (const, 0x0, 4)
               (register, 0x42, 1) = INT_AND (unique, 0xf1180, 1), (const, 0xf, 1)
               (unique, 0xf1300, 1) = SUBPIECE (unique, 0xf0d80, 2), (const, 0x0, 4)
               (register, 0x43, 1) = INT_AND (unique, 0xf1300, 1), (const, 0xf, 1)
             <1>
               (unique, 0xf1500, 1) = INT_RIGHT (const, 0x1, 1), (const, 0x3, 4)
               (unique, 0xf1580, 1) = INT_AND (unique, 0xf1500, 1), (const, 0x1, 1)
               (unique, 0xf1600, 1) = INT_EQUAL (unique, 0xf1580, 1), (const, 0x0, 1)
               CBRANCH <2>, (unique, 0xf1600, 1)
               (register, 0x23, 3) = INT_ADD (register, 0x23, 3), (const, 0x2, 3)
               (register, 0x14, 2) = LOAD (const, 0x1f1, 8), (register, 0x23, 3)
             <2>
               (unique, 0xf1700, 1) = INT_RIGHT (const, 0x1, 1), (const, 0x4, 4)
               (unique, 0xf1780, 1) = INT_AND (unique, 0xf1700, 1), (const, 0x1, 1)
               (unique, 0xf1800, 1) = INT_EQUAL (unique, 0xf1780, 1), (const, 0x0, 1)
               CBRANCH <3>, (unique, 0xf1800, 1)
               (register, 0x23, 3) = INT_ADD (register, 0x23, 3), (const, 0x2, 3)
               (register, 0x12, 2) = LOAD (const, 0x1f1, 8), (register, 0x23, 3)
             <3>
               (unique, 0xf1900, 1) = INT_RIGHT (const, 0x1, 1), (const, 0x5, 4)
               (unique, 0xf1980, 1) = INT_AND (unique, 0xf1900, 1), (const, 0x1, 1)
               (unique, 0xf1a00, 1) = INT_EQUAL (unique, 0xf1980, 1), (const, 0x0, 1)
               CBRANCH <4>, (unique, 0xf1a00, 1)
               (register, 0x23, 3) = INT_ADD (register, 0x23, 3), (const, 0x2, 3)
               (register, 0x10, 2) = LOAD (const, 0x1f1, 8), (register, 0x23, 3)
             <4>
               (unique, 0xf1b00, 1) = INT_RIGHT (const, 0x1, 1), (const, 0x6, 4)
               (unique, 0xf1b80, 1) = INT_AND (unique, 0xf1b00, 1), (const, 0x1, 1)
               (unique, 0xf1c00, 1) = INT_EQUAL (unique, 0xf1b80, 1), (const, 0x0, 1)
               CBRANCH <5>, (unique, 0xf1c00, 1)
               (register, 0x23, 3) = INT_ADD (register, 0x23, 3), (const, 0x2, 3)
               (register, 0x0, 2) = LOAD (const, 0x1f1, 8), (register, 0x23, 3)
             <5>
               (unique, 0xf1d00, 1) = INT_RIGHT (const, 0x1, 1), (const, 0x7, 4)
               (unique, 0xf1d80, 1) = INT_AND (unique, 0xf1d00, 1), (const, 0x1, 1)
               (unique, 0xf1e00, 1) = INT_EQUAL (unique, 0xf1d80, 1), (const, 0x0, 1)
               CBRANCH <6>, (unique, 0xf1e00, 1)
               (register, 0x23, 3) = INT_ADD (register, 0x23, 3), (const, 0x2, 3)
               (register, 0x2, 2) = LOAD (const, 0x1f1, 8), (register, 0x23, 3)
             <6>

The space and register definitions are...

define endian=big;

define alignment=2;

define space RAM      type=ram_space      size=3  default;
define space register type=register_space size=3;

@define VECTOR_SWI 	"0x000C"

################################################################
# Registers
################################################################

define register offset=0x00 size=4 [ ED      ];
define register offset=0x00 size=2 [ E   D   ];
define register offset=0x00 size=1 [ _ _ A B ];

# index registers
define register offset=0x10 size=2 [ IX      IY      IZ      ];
define register offset=0x10 size=1 [ IXH IXL IYH IYL IZH IZL ];

# use PKPC as PK : PC, same for SP
define register offset=0x20 size=3 [ PKPC       SKSP       ];
define register offset=0x21 size=2 [    PC                 ];
define register offset=0x24 size=2 [               SP      ];
define register offset=0x20 size=1 [ PK PCH PCL SK SPH SPL ];

# CCR definition 
define register offset=0x30 size=2 [ CCR ];

# define address extension registers EK, XK, YK, ZK (treat each 4 bit item as 8 bits)
define register offset=0x40 size=1 [ EK XK YK ZK ];

define register offset=0x50 size=2 [ HR IR ];

# MAC accumulator is 4.5 bytes long
define register offset=0x60 size=5 [ AM ];
define register offset=0x60 size=1 [ AM4 AM3 AM2 AM1 AM0 ];

define register offset=0x70 size=2 [ XYMSK     ];
define register offset=0x70 size=1 [ XMSK YMSK ];

[GhidorahRex]

Where is that pcode from? It doesn't look like the pcode from the listing. It also doesn't look like it's assigning an 8-byte value to CCR, it looks like the constant 0x1f1 is loaded as an 8-byte value. I'm not sure where that constant comes from.

[rimwall]

It's the pcode from a single PULM instruction.

This pcode at the start...

               (unique, 0xf0a80, 1) = INT_AND (const, 0x1, 1), (const, 0x1, 1)
               (unique, 0xf0b00, 1) = INT_EQUAL (unique, 0xf0a80, 1), (const, 0x0, 1)

...correlates to...
if (iopr8i[0,1] == 0)

...and the following pcode...

               (register, 0x23, 3) = INT_ADD (register, 0x23, 3), (const, 0x2, 3)
               (register, 0x30, 2) = LOAD (const, 0x1f1, 8), (register, 0x23, 3)

...correlates to...
Pull2(CCR);

...which, via the macro, expands to...

macro Pull2(operand) {
	SKSP = SKSP + 2;
	paddr:3 = SKSP;
	operand = *paddr;
}

The value being put in register 0x30 (CCR) should be coming from the stack pointer register 0x23 (SKSP). Perhaps 0x1f1 is the value on the modelled stack? Still not sure why it thinks it's an 8 byte value. The next logical branch (if entered) would pull 4 x 2 byte values... does this requirement for 8 bytes in one branch somehow get applied to each branch?

[rimwall]

By playing around with it, I have figured out why the "WARNING: Removing unreachable block" error is displayed. It is saying that because there are some of the PSHM / PULM conditional branches that are not executed based on the immediate operand. For example, if I reduce the instruction to:

# PULM IMM8 ii
:PULM iopr8i				is Prebyte=0x0 & op8=0x35; iopr8i {
	local temp:2;
	if (iopr8i[0,1] == 0) goto <next1>;
		Pull2(CCR);
	<next1>
}

Then wherever there is a PULM #0x01 instruction no WARNING occurs. If there is a PULM #0x02 instruction then the WARNING occurs.

So, bottom line, it's not really an error.

That's solved the WARNINGs. Still puzzling over how to handle these extension registers and their impact on addresses...

[rimwall]

Next mystery, it seems like the decomplier somehow cuts out branches of the code even though those branches could occur.

Example:

          0078b6 dc  18           LDX        offset  DAT_0ffe18 ,IY                            = ??
          0078b8 b6  f8           BNE        LAB_0078b6
          0078ba 37  b5  55  55    LDD        #0x5555
          0078be 37  da  01  e8    STD        0x1e8 ,IY=>DAT_0fffe8                            = ??
          0078c2 37  bc  30  00    LDX        #0x3000
          0078c6 9c  18           STX        offset  DAT_0ffe18 ,IY                            = ??

The LDX loads the value at RAM location 0x0ffe18 (uninitialised) into the IX register. The address 0xffe18 came from YK being 0x0f, IY being 0xfe00 and the instruction offset of 0x18 giving 0xffe18. If the value at 0xffe18 is nil, the LDX would set the Z flag. The BNE then branches based on the value of the Z flag. However, the decompiler says "WARNING: Removing unreachable block (RAM,0x0078ba)" and the decompiled function doesn't reflect the code from 0x0078ba onwards.

Why does the decompiler cut out this code when it can't know the value of the Z flag?

pcode of the LDX and BNE follows...

                   LAB_0078b6                                      XREF[1]:     0078b8 (j)   
          0078b6 dc  18           LDX        offset  DAT_0ffe18 ,IY                            = ??
             (unique, 0x2b80, 3)  =  INT_ZEXT  (register, 0x42, 1)
             (unique, 0x2c00, 3)  =  INT_LEFT  (unique, 0x2b80, 3) , (const, 0x10, 4)
             (unique, 0x2c80, 3)  =  INT_ZEXT  (register, 0x12, 2)
             (unique, 0x2d00, 3)  =  INT_ADD  (unique, 0x2c00, 3) , (unique, 0x2c80, 3)
             (unique, 0x2d80, 3)  =  INT_ZEXT  (const, 0x18, 1)
             (unique, 0x2e80, 3)  =  INT_ADD  (unique, 0x2d00, 3) , (unique, 0x2d80, 3)
             (unique, 0x2f00, 2)  =  LOAD  (const, 0x1f1, 4) , (unique, 0x2e80, 3)
             (register, 0x10, 2)  =  COPY  (unique, 0x2f00, 2)
             (unique, 0xc2780, 2)  =  INT_AND  (register, 0x30, 2) , (const, 0xfbff, 2)
             (unique, 0xc2700, 1)  =  INT_EQUAL  (register, 0x10, 2) , (const, 0x0, 2)
             (unique, 0xc2800, 2)  =  INT_ZEXT  (unique, 0xc2700, 1)
             (unique, 0xc2880, 2)  =  INT_LEFT  (unique, 0xc2800, 2) , (const, 0xa, 4)
             (register, 0x30, 2)  =  INT_OR  (unique, 0xc2780, 2) , (unique, 0xc2880, 2)
             (unique, 0xc2980, 2)  =  INT_AND  (register, 0x30, 2) , (const, 0xf7ff, 2)
             (unique, 0xc2900, 1)  =  INT_SLESS  (register, 0x10, 2) , (const, 0x0, 2)
             (unique, 0xc2a00, 2)  =  INT_ZEXT  (unique, 0xc2900, 1)
             (unique, 0xc2a80, 2)  =  INT_LEFT  (unique, 0xc2a00, 2) , (const, 0xb, 4)
             (register, 0x30, 2)  =  INT_OR  (unique, 0xc2980, 2) , (unique, 0xc2a80, 2)
             (unique, 0xc2b00, 2)  =  INT_AND  (register, 0x30, 2) , (const, 0xfdff, 2)
             (unique, 0xc2b80, 2)  =  INT_ZEXT  (const, 0x0, 1)
             (unique, 0xc2c00, 2)  =  INT_LEFT  (unique, 0xc2b80, 2) , (const, 0x9, 4)
             (register, 0x30, 2)  =  INT_OR  (unique, 0xc2b00, 2) , (unique, 0xc2c00, 2)
          0078b8 b6  f8           BNE        LAB_0078b6
             (unique, 0x4fd80, 2)  =  INT_RIGHT  (register, 0x30, 2) , (const, 0xa, 4)
             (unique, 0x4fe00, 1)  =  SUBPIECE  (unique, 0x4fd80, 2) , (const, 0x0, 4)
             (unique, 0x4fe80, 1)  =  INT_AND  (unique, 0x4fe00, 1) , (const, 0x1, 1)
             (unique, 0x4ff00, 1)  =  INT_EQUAL  (unique, 0x4fe80, 1) , (const, 0x0, 1)
             CBRANCH  (RAM, 0x78b6, 1) , (unique, 0x4ff00, 1)

Thanks!

[rimwall]

Any thoughts on the current questions?

1. Is there a better way to track the value of XK, YK, ZK etc? In order to do this properly would I need to write a script that tracks the values as the code would execute, and then apply these values as register constraints to each function? Or can Ghidra do this automatically somehow?
2. why does the decomplier cut out some branches of the code?

A new question, why does the disassembler simplify an address, but the decompiler keeps it shown in its various pieces?

For example, disassembler expresses the address succinctly like this...
00ad4e 17 60 18 80 SUBA offset DAT_021880 ,IZ

while the decompiler turns the address into this...

bVar8 = 2;
bVar2 = *(byte *)((uint3)bVar8 * 0x10000 + (uint3)in_IZ + 0x1880);

[rimwall]

Hi, I tried the different approach to PSHM, but now it pushes all registers onto the stack regardless of the immediate value.

Updated constructors:

define token data8 (8)
	imm8  = (0,7)
	imm8_0 = (0,0)
	imm8_1 = (1,1)
	imm8_2 = (2,2)
	imm8_3 = (3,3)
	imm8_4 = (4,4)
	imm8_5 = (5,5)
	imm8_6 = (6,6)
	imm8_7 = (7,7)
	simm8 = (0,7) signed
	rel   = (0,7) signed
;

PshCCR: is imm8_6=1 {
	Push2(CCR);
}

PshCCR: is imm8_6=0 {
}

PshKs: is imm8_5=1 & PshCCR {
	local tempEK:2 = zext(EK[0,4]) << 12;
	local tempXK:2 = zext(XK[0,4]) << 8;
	local tempYK:2 = zext(YK[0,4]) << 4;
	local tempZK:2 = zext(ZK[0,4]);
	local temp:2 = tempEK + tempXK + tempYK + tempZK;
	Push2(temp);
	build PshCCR;
}

PshKs: is imm8_5=0 & PshCCR {
	build PshCCR;
}

PshIZ: is imm8_4=1 & PshKs {
	Push2(IZ);
	build PshKs;
}

PshIZ: is imm8_4=0 & PshKs {
	build PshKs;
}

PshIY: is imm8_3=1 & PshIZ {
	Push2(IY);
	build PshIZ;
}

PshIY: is imm8_3=0 & PshIZ {
	build PshIZ;
}

PshIX: is imm8_2=1 & PshIY {
	Push2(IX);
	build PshIY;
}

PshIX: is imm8_2=0 & PshIY {
	build PshIY;
}

PshE: is imm8_1=1 & PshIX {
	Push2(E);
	build PshIX;
}

PshE: is imm8_1=0 & PshIX {
	build PshIX;
}

PshD: is imm8_0=1 & PshE {
	Push2(D);
	build PshE;
}

PshD: is imm8_0=0 & PshE {
	build PshE;
}

:PSHM iopr8i						is Prebyte=0x0 & op8=0x34; iopr8i & PshD {
	build PshD;
}

resultant pcode (for this example of PSHM #0x02) has 6 STORE commands even though it should only be storing one item on the stack:

          001e26 34 02           PSHM       offset PTR_DAT_00ff23                            = 00ffff
                                                      STORE (const, 0x1f1, 8), (register, 0x23, 3), (register, 0x2, 2)
                                                      (register, 0x23, 3) = INT_SUB (register, 0x23, 3), (const, 0x2, 3)
                                                      STORE (const, 0x1f1, 8), (register, 0x23, 3), (register, 0x0, 2)
                                                      (register, 0x23, 3) = INT_SUB (register, 0x23, 3), (const, 0x2, 3)
                                                      STORE (const, 0x1f1, 8), (register, 0x23, 3), (register, 0x10, 2)
                                                      (register, 0x23, 3) = INT_SUB (register, 0x23, 3), (const, 0x2, 3)
                                                      STORE (const, 0x1f1, 8), (register, 0x23, 3), (register, 0x12, 2)
                                                      (register, 0x23, 3) = INT_SUB (register, 0x23, 3), (const, 0x2, 3)
                                                      STORE (const, 0x1f1, 8), (register, 0x23, 3), (register, 0x14, 2)
                                                      (register, 0x23, 3) = INT_SUB (register, 0x23, 3), (const, 0x2, 3)
                                                      (unique, 0xd980, 1) = INT_AND (register, 0x40, 1), (const, 0xf, 1)
                                                      (unique, 0xda00, 2) = INT_ZEXT (unique, 0xd980, 1)
                                                      (unique, 0xdb00, 2) = INT_LEFT (unique, 0xda00, 2), (const, 0xc, 4)
                                                      (unique, 0xdb80, 1) = INT_AND (register, 0x41, 1), (const, 0xf, 1)
                                                      (unique, 0xdc00, 2) = INT_ZEXT (unique, 0xdb80, 1)
                                                      (unique, 0xdd00, 2) = INT_LEFT (unique, 0xdc00, 2), (const, 0x8, 4)
                                                      (unique, 0xdd80, 1) = INT_AND (register, 0x42, 1), (const, 0xf, 1)
                                                      (unique, 0xde00, 2) = INT_ZEXT (unique, 0xdd80, 1)
                                                      (unique, 0xdf00, 2) = INT_LEFT (unique, 0xde00, 2), (const, 0x4, 4)
                                                      (unique, 0xdf80, 1) = INT_AND (register, 0x43, 1), (const, 0xf, 1)
                                                      (unique, 0xe080, 2) = INT_ZEXT (unique, 0xdf80, 1)
                                                      (unique, 0xe100, 2) = INT_ADD (unique, 0xdb00, 2), (unique, 0xdd00, 2)
                                                      (unique, 0xe180, 2) = INT_ADD (unique, 0xe100, 2), (unique, 0xdf00, 2)
                                                      (unique, 0xe280, 2) = INT_ADD (unique, 0xe180, 2), (unique, 0xe080, 2)
                                                      STORE (const, 0x1f1, 8), (register, 0x23, 3), (unique, 0xe280, 2)
                                                      (register, 0x23, 3) = INT_SUB (register, 0x23, 3), (const, 0x2, 3)

[GhidorahRex]

I had some issues getting your language spec to run. I think you might be using an outdated version of ghidra. Your memory blocks in the pspec are missing initialized="false" But I was able to get it running after a minute. I don't see the same issues you have. The slaspec is identical except I added the above PSHM changes. The pcode only has a single push. Notice how the formatting of the pcode is more readable in the listing as well.

[rimwall]

Thanks, very helpful! Thanks for fixing that issue with PSHM. I have fixed PULM in a similar fashion.

The PCode still looks the same With Ghidra 11.1.2 - is there an option somewhere to make it more human readable (like your example above)?

As an aside, the auto-analysis in Ghidra 11.1.2 doesn't seem as "auto" as it was in Ghidra 10.1. With 10.1, one execution of the auto-analysis (default options) would unravel pretty much the whole binary. With 11.1.2 it takes multiple attempts because the auto-analysis often stops when it has identified function entries and defined that address as a function, but it does not proceed to dissassemble the function code (default options again). Weird. It seems to bomb out in the stack analysis.

The improved constructors seem to have solved the 'WARING: removed code' issue. Be great to get some help with the other issues:

1. Is there a better way to track the value of XK, YK, ZK etc? In order to do this properly would I need to write a script that tracks the values as the code would execute, and then apply these values as register constraints to each function? Or can Ghidra do this automatically somehow?

2. why does the disassembler simplify an address, but the decompiler keeps it shown in its various pieces? Can this be changed to make the decomplier more readable?

For example, disassembler expresses the address succinctly like this...
00ad4e 17 60 18 80 SUBA offset DAT_021880 ,IZ

while the decompiler turns the address into this...

bVar8 = 2;
bVar2 = *(byte *)((uint3)bVar8 * 0x10000 + (uint3)in_IZ + 0x1880);

[GhidorahRex]

As far as tracking the values of XK, YK, and ZK, I don't see the issue that you have above where the values are not being tracked properly. I think it may have had to do with the if ... goto sets in the pshm and pulm instructions. Can you double-check your original instructions and verify that they're being tracked properly now?

Make sure that you have something like

  <context_data>
    <tracked_set space="RAM">
      <set name="XK" val="2"/>
      <set name="YK" val="2"/>
      <set name="ZK" val="2"/>
    </tracked_set>
  </context_data>

in your pspec file, too.

[GhidorahRex]

To fix the pcode:
Edit > Tool Options > Listing Fields > Pcode Field > uncheck Display Raw Pcode

[rimwall]

Did you mean have the <context_data> in both the cspec and the pspec files?

I now have that. The disassembly has changed, but now instead of a reference to the incorrect address of 0x1638 (should be 0x21638 with value of ZK of 0x02), there is now an unreferenced immediate value of 0x1638.

Latest disassembly follows (compare to snip in the first post):

[rimwall]

Ignore this entry. I thought I had the order of PULM reversed, but the order is correct, so the mystery of the reference in the BSET instruction remains.

[emteere]

The default constant propagation should follow the registers values of ZK and IZ and produce a reference.
Although it might be having trouble on the small registers unless you are using 11.1.2.

You need to move <context_data> from your .cspec file to your .pspec file.

<processor_spec>
  <programcounter register="PKPC"/>

  <context_data>
	<tracked_set space="RAM">
		<set name="EK" val="2"/>
		<set name="XK" val="2"/>
		<set name="YK" val="0xf"/> 
		<set name="ZK" val="2"/>
	</tracked_set>
  </context_data> 
  ....

An analyzer can be made to propagate the EK, XK, YK, ZK registers if they are globally set, or set before calling a function.

In the .cspec file, the decompiler will assume the EK, XK, YK, ZK are unknown after a call unless they are in the unaffected list

The stack params/locals are not set correctly. I believe this .cspec will work much better. I'm still seeing other issues.

<?xml version="1.0" encoding="UTF-8"?>

<compiler_spec>
  <data_organization>  <!-- These tags need to be verified -->
     <absolute_max_alignment value="0" />
     <machine_alignment value="2" />
     <default_alignment value="2" />
     <pointer_size value="2" />
     <wchar_size value="4" />
     <short_size value="2" />
     <integer_size value="4" />
     <long_size value="4" />
     <long_long_size value="8" />
     <float_size value="4" />
     <double_size value="8" />
     <long_double_size value="8" />
  </data_organization>

  <global>
    <range space="RAM"/>
  </global>  
  
  <stackpointer register="SKSP" space="RAM" growth="negative"/>
  <returnaddress>
    <varnode space="stack" offset="4" size="2"/>
  </returnaddress>
  
  <default_proto>
      <prototype name="__asmA" extrapop="4" stackshift="4" strategy="register">
      <input>
        <pentry minsize="1" maxsize="1">
          <register name="A"/>
        </pentry>
        <pentry minsize="1" maxsize="1">
          <register name="B"/>
        </pentry>
        <pentry minsize="2" maxsize="2">
          <register name="D"/>
        </pentry>
        <pentry minsize="2" maxsize="2">
          <register name="E"/>
        </pentry>
        <pentry minsize="1" maxsize="2">
          <register name="IX"/>
        </pentry>
        <pentry minsize="1" maxsize="2">
          <register name="IY"/>
        </pentry>
        <pentry minsize="1" maxsize="2">
          <register name="IZ"/>
        </pentry>
        <pentry minsize="1" maxsize="500" align="2">
          <addr offset="6" space="stack"/>
        </pentry>
      </input>
      <output>
        <pentry minsize="1" maxsize="2">
          <register name="D"/>
        </pentry>
      </output>
      <unaffected>
        <register name="SKSP"/>
        <register name="EK"/>
        <register name="XK"/>
        <register name="YK"/>
        <register name="ZK"/>
      </unaffected>
    </prototype>
  </default_proto>

</compiler_spec>

[emteere]

I'm also seeing the split of the XK and IX register incremented with the AIX instruction causing trouble for the decompiler following it as a combined value as implemented. I'm not sure this is easily fixable. Will have to think about it.

[rimwall]

Hi, thanks! I have made those updates to the cspec and pspec. Happy to hear your advice on any other settings - I was having trouble finding reference material for the cspec and pspec files.

Confirming I’m using Ghidra 11.1.2

The issue above (references not including value of extension registers) still seems to occur. See the BSET instruction at 0x34e4 below. The only thing in between 0x34cc and 0x34e4 that affects ZK is the PULM instruction. Is it somehow pulling a 0x0 off the stack? Or is the stack management somehow not correct, so the pushes aren't lining up with the pulls? Not sure how to see how Ghidra is tracking the stack...

[GhidorahRex]

Is it incorrect in both the listing and decompiler? Can you try clearing and re-disassembling the BSET instruction? I had an issue when I was looking at this where the decompiler output would be correct, but the reference in the BSET was wrong until I cleared it and re-disassembled it.

[rimwall]

How odd. I tried two approaches:

1. Fresh startup of Ghidra 11.1.2

2. Brand new import, set up memory blocks, then went to start of function that contains the BSET (function start at 0x3474), pressed 'D' to dissassemble, pressed 'F' to set as a function. The BSET instruction in the listing window now reflects a reference, but it points to 0x01638, so it's missing the ZK value of 0x02. Cleared the BSET instruction, and pressed 'D' again. Still reflects a reference to 0x01638. Tried clearing the whole function, pressed 'D' again. No change. Still a reference to 0x01638
   

3. Opened previous import which was as per the image above. Cleared the BSET instruction, and pressed 'D' again. Now I get the correct reference to 0x21638.
   

Why would there be two different outcomes? Is there a way to consistently get the correct reference?

The decompiler output is the same in all scenarios, and is not wrong. It's just not simplified. Instead of showing the address as 0x21638 it will show it as the following for case 2...

        pbVar1 = (byte *)((uint3)bVar10 * 0x10000 + (uint3)param_1 + 0x1638);
        *pbVar1 = *pbVar1 | 0x80;

...and the following for case 1...

        (&DAT_001638)[(uint3)bVar12 * 0x10000 + (uint3)param_1] =
             (&DAT_001638)[(uint3)bVar12 * 0x10000 + (uint3)param_1] | 0x80;

where bVar10/12 has the value of 0x2 and param_1 is IZ

[emteere]

Reference are computed and laid down by an analyzer that runs when new code is added.
There has been some work in re-analyzing for some change of information. However it could be very expensive to figure out what needs to be fixed from a change constantly. In this case setting the default value in the .pspec isn't even an event that is detected.
You can re-run analysis and the pointers should be fixed, similar to the event that is generated when you cleared the instruction and re-disassembled it.

The decompiler is computing always, well mostly, on the fly each time.

[emteere]

For the decompiler simplification, it comes down to the keeping of the pointer in two pieces that have too much manipulation going on to simplify them each time they are incremented. Will need to think about how to deal with that. It occurs in other processors that have been extended with extra "segment" or "page" registers in a similar way.

[Wall-AF]

For the decompiler simplification, it comes down to the keeping of the pointer in two pieces that have too much manipulation going on to simplify them each time they are incremented. Will need to think about how to deal with that. It occurs in other processors that have been extended with extra "segment" or "page" registers in a similar way.

@emteere Not just pointers using segments, values using several registers too (#5900)!

[rimwall]

You can re-run analysis and the pointers should be fixed, similar to the event that is generated when you cleared the instruction and re-disassembled it.

I have tried, but no matter how many times I run the analyser (and/or clear the code), this particular BSET example refuses to fix itself. I also tried manually changing the reference to an offset reference from 0x20000. This gives correct reference. Then I cleared the instruction, pressed 'D' again. Still get reference to 0x01638. Then I deleted all references, which turns it into an immediate value 0x1638. Then I cleared the instruction, pressed 'D' again. Still get reference to 0x01638.

The only way I get the right reference is by opening the file created earlier (from the post starting 'Hi, thanks..'). This file started life with an unreferenced immediate value but was changed to 0x21638 by clearing and pressing D. If I clear it again, and press D again, the correct reference to 0x21638 is generated. Something about this file consistently gives the expected reference of 0x21638. Something about the other file consistently gives 0x01638.

it comes down to the keeping of the pointer in two pieces that have too much manipulation going on to simplify them each time they are incremented. Will need to think about how to deal with that. It occurs in other processors that have been extended with extra "segment" or "page" registers in a similar way.

Ah, ok. I was wondering whether something about the way I'd written the various spec files was causing the issue.

[rimwall]

May not be related, but this issue I just stumbled across also has an element of 'stickiness' in the disassembler: #6957