Boot fails with Cannot find any partition table for 00030000 after flashing

[apbr]

We are flashing devices with Jetson Orin NX 8GB modules using initrd-flash.
On some devices we observe that they do not boot properly after flashing, even though the flashing process is successful.
Booting fails with the following message in the MB2 step:

I> Task: Partition Manager Init
I> strt_pg_num(1) num_of_pgs(1) read_buf(0x273801000)
I> strt_pg_num(131071) num_of_pgs(1) read_buf(0x273801000)
W> Cannot find any partition table for 00030000
W> PARTITION_MANAGER: Failed to publish partition.
... (some lines left out) ...
I> Partition name: A_spe-fw
E> Cannot find partition A_spe-fw
E> Partition A_spe-fw not found
E> MB2_FW_LOAD: Failed to load FW.
C> Task 0x0 failed (err: 0x224d090d)
E> Top caller module: LOADER, error module: PARTITION_MANAGER, reason: 0x0d, aux_info: 0x09

Those modules were brand new and never flashed before by us.
Does anyone has an idea what could cause this issue?
When flashing with initrd-flash, do we flash the partition table onto the QSPI or is that left untouched?

Another difference we noticed:
Devices that boot successfully after flashing have the following section in the flash log:

[   1.1341 ] Dumping customer Info
[   1.1344 ] tegrarcm_v2 --instance 1-4 --chip 0x23 0 --oem dump bct tmp.bct
[   1.1346 ] MB2 Applet version 01.00.0000
[   1.1407 ] 0000000054220302: E> NV3P_SERVER: Failed to get address for br bct from nv3p helper.
[   1.2198 ] 
[   1.2199 ] 
[   1.2199 ] Error: try getting custinfo fail. Moving on

Devices that don't boot properly after flashing instead contain this lines for the same step:

[   1.1259 ] Dumping customer Info
[   1.1262 ] tegrarcm_v2 --instance 1-4 --chip 0x23 0 --oem dump bct tmp.bct
[   1.1265 ] MB2 Applet version 01.00.0000
[   1.1325 ] Saved bct in tmp.bct
[   1.1710 ] tegrabct_v2 --brbct tmp.bct --chip 0x23 0 --custinfo /root/temp/image-nx-factory_548_1533_1943.tegraflash/custinfo_out.bin
[   1.1712 ] C[   1.1714 ] ustomer data saved in /root/temp/image-nx-factory_548_1533_1943.tegraflash/custinfo_out.bin successfully

[madisongh]

When flashing with initrd-flash, do we flash the partition table onto the QSPI or is that left untouched?

By default it does - everything the QSPI flash should get updated. The script has an option --skip-bootloader to omit this step.

Which branch are you using, and is your machine definition based off the p3768-0000-p3767-0001 or something else? My guess is that there are some settings for the machine that are wrong for that particular SKU.

[apbr]

I already did a diff between flashing logs of modules that worked and ones that didn't (Though it was probably not the first flashing attempt for them):
This is a diff from the log of Step 1 (working vs non-working):

 [timestamps] Dumping customer Info
 [timestamps] tegrarcm_v2 --instance 1-4 --chip 0x23 0 --oem dump bct tmp.bct
 [timestamps] MB2 Applet version 01.00.0000
-[timestamps] 0000000054220302: E> NV3P_SERVER: Failed to get address for br bct from nv3p helper.
-[timestamps] 
-[timestamps] 
-[timestamps] Error: try getting custinfo fail. Moving on
+[timestamps] Saved bct in tmp.bct
+[timestamps] tegrabct_v2 --brbct tmp.bct --chip 0x23 0 --custinfo /root/temp/our-image-devel-nx-factory_548_1533_1943.tegraflash/custinfo_out.bin
+[timestamps] C[   1.1714 ] ustomer data saved in /root/temp/our-image-devel-nx-factory_548_1533_1943.tegraflash/custinfo_out.bin successfully
 [timestamps] Rebooting to recovery mode
 [timestamps] tegrarcm_v2 --instance 1-4 --chip 0x23 0 --ismb2
 [timestamps] tegrarcm_v2 --instance 1-4 --chip 0x23 0 --ismb2applet
 [timestamps] MB2 Applet version 01.00.0000
 [timestamps] Booting to recovery mode
 [timestamps] tegrarcm_v2 --instance 1-4 --chip 0x23 0 --reboot recovery
 [timestamps] MB2 Applet version 01.00.0000

There is also an interesting part in the diff from the serial console. This is the first boot when flashing, even before booting the initrd Linux (working vs non-working):

 I> QSPI Flash: Macronix 64MB
 I> QSPI-0l initialized successfully
 I> Secondary storage device: SDMMC_BOOT instance: 3
-E> Error in command_complete 18000 int_status
+E> Error in command_complete 18001 int_status
 E> OCR failed, error = 39390706
 E> STORAGE: Failed to open SDMMC: 3.
 W> Ignoring init failure for device 0-3
....
 E> LOADER: Invalid value Header magic.
 E> Validation failed for 1 copy of BRBCT @ 0
 I> strt_pg_num(512) num_of_pgs(16) read_buf(0x40071738)
-E> LOADER: Invalid value Header magic.
-E> Validation failed for 2 copy of BRBCT @ 262144
-I> strt_pg_num(1024) num_of_pgs(16) read_buf(0x40071738)
-E> LOADER: Invalid value Header magic.
-E> Validation failed for 3 copy of BRBCT @ 524288
-I> strt_pg_num(1536) num_of_pgs(16) read_buf(0x40071738)
-E> LOADER: Invalid value Header magic.
-E> Validation failed for 4 copy of BRBCT @ 786432
-E> NV3P_SERVER: Failed to get address for br bct from nv3p helper.
+I> BRBCT integrity check is success
+I> Validation passed for 2 copy of BRBCT @ 262144
 I> Rebooting : reboot-recovery

Apart from that I did not notice any relevant differences.
Anything specific to look out for?

[madisongh]

Look for any difference at all in the names of the files used by the scripts to put together the boot firmware blobs (in the log.initrd-flash.<datetime> file and/or the device side logs in the device-logs-<datetime> subdirectory.

It might be easier to check for differences by using ./doflash --spi-only to do the boot firmware flashing. And you'd get even more detail by adding a set -x to the top of the tegra-flash.sh script before doing that. That's the usual approach I take when trying to troubleshoot this kind of thing.

[apbr]

After some investigations, it seems that the initrd-flash process does not properly write the QSPI flash:
I created QSPI flash dumps from various of our devices.
When comparing the BCT binary with the 64 instances in the QSPI, I can see that some of them do not match. For devices where we see the boot issue, the BCT does not match in a lot of cases.
When flashing with the Nvidia SDK on the other hand, all 64 BCT entries in QSPI look fine on a first glance.
A workaround to fix the "broken" Jetsons, was to first flash them with the Nvidia SDK and then flash them with our meta-tegra setup.
So it seems the Nivida SDK somehow manages to properly write into the QSPI, while meta-tegra scripts do not.
Any idea what could be the issue?

Regarding running ./doflash --spi-only:
When trying that, I get

[  10.1042 ] Start flashing
[  10.1045 ] tegradevflash_v2 --pt flash.xml.bin --create
[  10.1047 ] Bootloader version 01.00.0000
[  10.1062 ] Erasing spi: 0 ......... [Done]
[  11.1139 ] Writing partition secondary_gpt with gpt_secondary_3_0.bin [ 16896 bytes ]
[  11.1152 ] [................................................] 100%
[  11.1184 ] 000000004d4d2c01: E> NV3P_SERVER: Failed to initialize partition table from GPT.
[  11.3508 ] 
[  11.3508 ] 
Error: Return value 1
Command tegradevflash_v2 --pt flash.xml.bin --create

(Similar to #1423, the recommended ./doflash.sh --no-flash --rcm-boot --external-device did not help)

[apbr]

Another observation:
I entered the shell of the initrd Linux during the flash process (Before the initrd Linux executs init-flash.sh).
Then I tried to manually erase parts of the flash using mtd_debug erase /dev/mtd0 $offset 8192 and flash_erase /dev/mtd0 $offset 2.
For some parts none of the commands were able to erase that part of the flash. So it seems like the "meta-tegra tools" have an issue there.

[madisongh]

If the doflash method of flashing the QSPI also results in boot failure, that confirms my suspicion that the problem is with the host-side scripts selecting the wrong binaries for flashing, rather than anything on the device side.

If you could provide logs of the output of the flashing scripts from the Yocto build and the stock L4T flashing, it would really help in troubleshooting this further. I'd do it myself, but I don't have an Orin NX 8GB board.

[apbr]

So it seems the issue is that some Jetson modules come from the factory with write/erase protection enabled (see Nvidia forum) which caused the initrd flashing to only write the QSPI flash incompletely.

The main reason the protection was an issue for us is that we use a mainline Linux 6.6 kernel. This kernel lacked support for the specific QSPI chip on the Jetson Orin NX. We ported OE4T/linux-jammy-nvidia-tegra@fd1d6ec and OE4T/linux-jammy-nvidia-tegra@092a0a1 to our Kernel. Additionally we added flash_unlock -u /dev/mtd0 at the start of program-boot-device.sh.
With that changes we could successfully flash the "broken" Jetsons.