Booting NVME with PCIE Gen3

[yoramrotbach]

Describe the bug

Sabrent Rocket 2230 NVME SSD, works great with pcie-gen3 speed once the system is already booted from uSD card.
As this drive drive most likely doesn't support pcie-Gen2 speeds, the eeprom code doesn't see it and the loading process is stuck. Please add a switch to enable pcie-gen3 switch in eeprom. The dtparam in config.txt is too late for the boot process.

Steps to reproduce the behaviour

Use any pcie hat for raspberry pi 5. I'm using Geekworm x1003.
Use NVME SSD drive that doesn't support gen2 speeds. I'm using Sabrent rocket 2230, supporting gen3 and up speeds.
The NVME has been formatted to boot. The source of the image was a working uSD card.
Try to boot - will not succeed for many minutes. Black screen. after several minutes some text appears letting me know that all possible boot devices failed.
Turn PCIE switch in eeprom config - no effect.
boot from uSD card with pcie-gen3 configuration - NVME drive works great.

Device (s)

Raspberry Pi 5

Bootloader configuration.

[all]
BOOT_UART=1
POWER_OFF_ON_HALT=0
BOOT_ORDER=0xf461
PCIE_PROBE=1

BOOTLOADER: up to date
CURRENT: Wed Jun 5 03:41:49 PM UTC 2024 (1717602109)
LATEST: Wed Jun 5 03:41:49 PM UTC 2024 (1717602109)
RELEASE: latest (/lib/firmware/raspberrypi/bootloader-2712/latest)
Use raspi-config to change the release.

System

Latest Raspberry pi os 64 bit
bootloader latest 2712
firmware version 6fe0b091
Linux raspberrypi 6.6.31+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.31-1+rpt1 (2024-05-29) aarch64 GNU/Linux

Bootloader logs

No response

USB boot

No response

NVMe boot

Node Generic SN Model Namespace Usage Format FW Rev

---

/dev/nvme0n1 /dev/ng0n1 48803179100692 Sabrent SB-2130-256 1 256.06 GB / 256.06 GB 512 B + 0 B R21B47.1

---

NVME Identify Controller:
vid : 0x1987
ssvid : 0x1987
sn : 48803179100692
mn : Sabrent SB-2130-256
fr : R21B47.1
rab : 4
ieee : 6479a7
cmic : 0
[3:3] : 0 ANA not supported
[2:2] : 0 PCI
[1:1] : 0 Single Controller
[0:0] : 0 Single Port

mdts : 6
cntlid : 0
ver : 0x10400
rtd3r : 0x186a0
rtd3e : 0x4c4b40
oaes : 0x200
[31:31] : 0 Discovery Log Change Notice Not Supported
[27:27] : 0 Zone Descriptor Changed Notices Not Supported
[15:15] : 0 Normal NSS Shutdown Event Not Supported
[14:14] : 0 Endurance Group Event Aggregate Log Page Change Notice Not Supported
[13:13] : 0 LBA Status Information Notices Not Supported
[12:12] : 0 Predictable Latency Event Aggregate Log Change Notices Not Supported
[11:11] : 0 Asymmetric Namespace Access Change Notices Not Supported
[9:9] : 0x1 Firmware Activation Notices Supported
[8:8] : 0 Namespace Attribute Changed Event Not Supported

ctratt : 0x2
[19:19] : 0 Flexible Data Placement Not Supported
[15:15] : 0 Extended LBA Formats Not Supported
[14:14] : 0 Delete NVM Set Not Supported
[13:13] : 0 Delete Endurance Group Not Supported
[12:12] : 0 Variable Capacity Management Not Supported
[11:11] : 0 Fixed Capacity Management Not Supported
[10:10] : 0 Multi Domain Subsystem Not Supported
[9:9] : 0 UUID List Not Supported
[8:8] : 0 SQ Associations Not Supported
[7:7] : 0 Namespace Granularity Not Supported
[6:6] : 0 Traffic Based Keep Alive Not Supported
[5:5] : 0 Predictable Latency Mode Not Supported
[4:4] : 0 Endurance Groups Not Supported
[3:3] : 0 Read Recovery Levels Not Supported
[2:2] : 0 NVM Sets Not Supported
[1:1] : 0x1 Non-Operational Power State Permissive Supported
[0:0] : 0 128-bit Host Identifier Not Supported

rrls : 0
cntrltype : 1
[7:2] : 0 Reserved
[1:0] : 0x1 I/O Controller
fguid : 00000000-0000-0000-0000-000000000000
crdt1 : 0
crdt2 : 0
crdt3 : 0
nvmsr : 0
[1:1] : 0 NVM subsystem Not part of an Enclosure
[0:0] : 0 NVM subsystem Not part of an Storage Device

vwci : 0
[7:7] : 0 VPD Write Cycles Remaining field is Not valid.
[6:0] : 0 VPD Write Cycles Remaining

mec : 0
[1:1] : 0 NVM subsystem Not contains a Management Endpoint on a PCIe port
[0:0] : 0 NVM subsystem Not contains a Management Endpoint on an SMBus/I2C port

oacs : 0x17
[10:10] : 0 Lockdown Command and Feature Not Supported
[9:9] : 0 Get LBA Status Capability Not Supported
[8:8] : 0 Doorbell Buffer Config Not Supported
[7:7] : 0 Virtualization Management Not Supported
[6:6] : 0 NVMe-MI Send and Receive Not Supported
[5:5] : 0 Directives Not Supported
[4:4] : 0x1 Device Self-test Supported
[3:3] : 0 NS Management and Attachment Not Supported
[2:2] : 0x1 FW Commit and Download Supported
[1:1] : 0x1 Format NVM Supported
[0:0] : 0x1 Security Send and Receive Supported

acl : 3
aerl : 3
frmw : 0x14
[5:5] : 0 Multiple FW or Boot Update Detection Not Supported
[4:4] : 0x1 Firmware Activate Without Reset Supported
[3:1] : 0x2 Number of Firmware Slots
[0:0] : 0 Firmware Slot 1 Read/Write

lpa : 0x1e
[6:6] : 0 Telemetry Log Data Area 4 Not Supported
[5:5] : 0 LID 0x0, Scope of each command in LID 0x5, 0x12, 0x13 Not Supported
[4:4] : 0x1 Persistent Event log Supported
[3:3] : 0x1 Telemetry host/controller initiated log page Supported
[2:2] : 0x1 Extended data for Get Log Page Supported
[1:1] : 0x1 Command Effects Log Page Supported
[0:0] : 0 SMART/Health Log Page per NS Not Supported

elpe : 62
[7:0] : 62 (0's based) Error Log Page Entries (ELPE)

npss : 4
[7:0] : 4 (0's based) Number of Power States Support (NPSS)

avscc : 0x1
[0:0] : 0x1 Admin Vendor Specific Commands uses NVMe Format

apsta : 0x1
[0:0] : 0x1 Autonomous Power State Transitions Supported

wctemp : 356
[15:0] : 83°C (356 Kelvin) Warning Composite Temperature Threshold (WCTEMP)

cctemp : 358
[15:0] : 85°C (358 Kelvin) Critical Composite Temperature Threshold (CCTEMP)

mtfa : 100
hmpre : 16384
hmmin : 16384
tnvmcap : 256,060,514,304
[127:0] : 256,060,514,304
Total NVM Capacity (TNVMCAP)

unvmcap : 0
[127:0] : 0
Unallocated NVM Capacity (UNVMCAP)

rpmbs : 0
[31:24]: 0 Access Size
[23:16]: 0 Total Size
[5:3] : 0 Authentication Method
[2:0] : 0 Number of RPMB Units

edstt : 30
dsto : 0
fwug : 4
kas : 0
hctma : 0x1
[0:0] : 0x1 Host Controlled Thermal Management Supported

mntmt : 273
[15:0] : 0°C (273 Kelvin) Minimum Thermal Management Temperature (MNTMT)

mxtmt : 356
[15:0] : 83°C (356 Kelvin) Maximum Thermal Management Temperature (MXTMT)

sanicap : 0xa0000002
[31:30] : 0x2 Media is additionally modified after sanitize operation completes successfully
[29:29] : 0x1 No-Deallocate After Sanitize bit in Sanitize command Not Supported
[2:2] : 0 Overwrite Sanitize Operation Not Supported
[1:1] : 0x1 Block Erase Sanitize Operation Supported
[0:0] : 0 Crypto Erase Sanitize Operation Not Supported

hmminds : 1024
hmmaxd : 16
nsetidmax : 0
endgidmax : 1
anatt : 0
anacap : 0
[7:7] : 0 Non-zero group ID Not Supported
[6:6] : 0 Group ID does change
[4:4] : 0 ANA Change state Not Supported
[3:3] : 0 ANA Persistent Loss state Not Supported
[2:2] : 0 ANA Inaccessible state Not Supported
[1:1] : 0 ANA Non-optimized state Not Supported
[0:0] : 0 ANA Optimized state Not Supported

anagrpmax : 0
nanagrpid : 0
pels : 96
domainid : 0
megcap : 0
sqes : 0x66
[7:4] : 0x6 Max SQ Entry Size (64)
[3:0] : 0x6 Min SQ Entry Size (64)

cqes : 0x44
[7:4] : 0x4 Max CQ Entry Size (16)
[3:0] : 0x4 Min CQ Entry Size (16)

maxcmd : 256
nn : 1
oncs : 0x5f
[8:8] : 0 Copy Not Supported
[7:7] : 0 Verify Not Supported
[6:6] : 0x1 Timestamp Supported
[5:5] : 0 Reservations Not Supported
[4:4] : 0x1 Save and Select Supported
[3:3] : 0x1 Write Zeroes Supported
[2:2] : 0x1 Data Set Management Supported
[1:1] : 0x1 Write Uncorrectable Supported
[0:0] : 0x1 Compare Supported

fuses : 0
[0:0] : 0 Fused Compare and Write Not Supported

fna : 0
[3:3] : 0 Format NVM Broadcast NSID (FFFFFFFFh) Supported
[2:2] : 0 Crypto Erase Not Supported as part of Secure Erase
[1:1] : 0 Crypto Erase Applies to Single Namespace(s)
[0:0] : 0 Format Applies to Single Namespace(s)

vwc : 0x7
[2:1] : 0x3 The Flush command supports NSID set to FFFFFFFFh
[0:0] : 0x1 Volatile Write Cache Present

awun : 255
awupf : 0
icsvscc : 1
[0:0] : 0x1 NVM Vendor Specific Commands uses NVMe Format

nwpc : 0
[2:2] : 0 Permanent Write Protect Not Supported
[1:1] : 0 Write Protect Until Power Supply Not Supported
[0:0] : 0 No Write Protect and Write Protect Namespace Not Supported

acwu : 0
ocfs : 0
[1:1] : 0 Controller Copy Format 1h Not Supported
[0:0] : 0 Controller Copy Format 0h Not Supported

sgls : 0
[15:8] : 0 SGL Descriptor Threshold
[1:0] : 0 Scatter-Gather Lists Not Supported

mnan : 0
maxdna : 0
maxcna : 0
subnqn : nqn.2020-11.com.phison:nvme:PS5021:48803179100692
ioccsz : 0
iorcsz : 0
icdoff : 0
fcatt : 0
[0:0] : 0 Dynamic Controller Model

msdbd : 0
ofcs : 0
[0:0] : 0 Disconnect command Not Supported

ps 0 : mp:5.00W operational enlat:0 exlat:0 rrt:0 rrl:0
rwt:0 rwl:0 idle_power:- active_power:-
active_power_workload:-
ps 1 : mp:2.40W operational enlat:0 exlat:0 rrt:1 rrl:1
rwt:1 rwl:1 idle_power:- active_power:-
active_power_workload:-
ps 2 : mp:1.92W operational enlat:0 exlat:0 rrt:2 rrl:2
rwt:2 rwl:2 idle_power:- active_power:-
active_power_workload:-
ps 3 : mp:0.0700W non-operational enlat:5000 exlat:10000 rrt:3 rrl:3
rwt:3 rwl:3 idle_power:- active_power:-
active_power_workload:-
ps 4 : mp:0.0050W non-operational enlat:6000 exlat:44000 rrt:4 rrl:4
rwt:4 rwl:4 idle_power:- active_power:-
active_power_workload:-

---

---

NVME Identify Namespace 1:
nsze : 0x1dcf32b0
ncap : 0x1dcf32b0
nuse : 0x1dcf32b0
nsfeat : 0
[4:4] : 0 NPWG, NPWA, NPDG, NPDA, and NOWS are Not Supported
[3:3] : 0 NGUID and EUI64 fields if non-zero, Reused
[2:2] : 0 Deallocated or Unwritten Logical Block error Not Supported
[1:1] : 0 Namespace uses AWUN, AWUPF, and ACWU
[0:0] : 0 Thin Provisioning Not Supported

nlbaf : 1
flbas : 0
[6:5] : 0 Most significant 2 bits of Current LBA Format Selected
[4:4] : 0 Metadata Transferred in Separate Contiguous Buffer
[3:0] : 0 Least significant 4 bits of Current LBA Format Selected

mc : 0
[1:1] : 0 Metadata Pointer Not Supported
[0:0] : 0 Metadata as Part of Extended Data LBA Not Supported

dpc : 0
[4:4] : 0 Protection Information Transferred as Last 8 Bytes of Metadata Not Supported
[3:3] : 0 Protection Information Transferred as First 8 Bytes of Metadata Not Supported
[2:2] : 0 Protection Information Type 3 Not Supported
[1:1] : 0 Protection Information Type 2 Not Supported
[0:0] : 0 Protection Information Type 1 Not Supported

dps : 0
[3:3] : 0 Protection Information is Transferred as Last 8 Bytes of Metadata
[2:0] : 0 Protection Information Disabled

nmic : 0
[0:0] : 0 Namespace Multipath Not Capable

rescap : 0
[7:7] : 0 Ignore Existing Key - Used as defined in revision 1.2.1 or earlier
[6:6] : 0 Exclusive Access - All Registrants Not Supported
[5:5] : 0 Write Exclusive - All Registrants Not Supported
[4:4] : 0 Exclusive Access - Registrants Only Not Supported
[3:3] : 0 Write Exclusive - Registrants Only Not Supported
[2:2] : 0 Exclusive Access Not Supported
[1:1] : 0 Write Exclusive Not Supported
[0:0] : 0 Persist Through Power Loss Not Supported

fpi : 0
[7:7] : 0 Format Progress Indicator Not Supported

dlfeat : 9
[4:4] : 0 Guard Field of Deallocated Logical Blocks is set to 0xFFFF
[3:3] : 0x1 Deallocate Bit in the Write Zeroes Command is Supported
[2:0] : 0x1 Bytes Read From a Deallocated Logical Block and its Metadata are 0x00

nawun : 0
nawupf : 0
nacwu : 0
nabsn : 0
nabo : 0
nabspf : 0
noiob : 0
nvmcap : 0
mssrl : 0
mcl : 0
msrc : 0
nulbaf : 0
anagrpid: 0
nsattr : 0
nvmsetid: 0
endgid : 0
nguid : 00000000000000016479a74820c00937
eui64 : 6479a77a20c00937
LBA Format 0 : Metadata Size: 0 bytes - Data Size: 512 bytes - Relative Performance: 0x1 Better (in use)
LBA Format 1 : Metadata Size: 0 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best

Network (TFTP boot)

No response

[pelwell]

As this drive drive most likely doesn't support pcie-Gen2 speeds, the eeprom code doesn't see it and the loading process is stuck

Is that anything other than a random guess?

Gathering some evidence from the boot diagnostics via the debug serial port (or a few photos) would help your case.

[lurch]

boot from uSD card with pcie-gen3 configuration - NVME drive works great.

It would also be interesting to see if you get any errors in dmesg if you boot from uSD card with pcie-gen2 configuration.

[peterharperuk]

I think I have a sabrent rocket. Let me check if I can get it to work.

[yoramrotbach]

@lurch - when gen=2, the nvme isn't seen at all by the system.
nothing in dmesg.
sudo nvme list - returns nothing.

@pelwell - working on getting the serial debug information as suggested

[timg236]

I have Sabrent Rocket 2230 and it works fine at Gen 2 with the Raspberry Pi M.2 HAT+