Bring Your Own Kernel

[eligavril]

With the release of Jetpack 6, Nvidia claims that choosing a kernel version (other than the default 5.15) is now possible, and provides a guide for it:
https://docs.nvidia.com/jetson/archives/r36.3/DeveloperGuide/SD/Kernel/BringYourOwnKernel.html

Now that r36.3 is supported here in the scarthgap branch, I was wondering how hard would it be to use the meta-tegra layer but build the 6.6 LTS kernel instead of what is provided in recipes-kernel/linux/linux-jammy-nvidia-tegra_5.15.bb. Is there any guide on the subject (other than what Nvidia provided, which is unrelated to yocto)?

Also, do you intend to add and maintain recipes for building kernel versions other than 5.15? I think it would be really cool if there was support for at least the LTS versions.

[madisongh]

The person I know who has been playing around with this has been on vacation this week, hopefully he can comment here when he gets back.

As for adding/maintaining recipes for other kernel versions, I have no plans to do so at the moment. If it's something that enough folks need, and we can get help with testing and maintenance, then maybe.

[ichergui]

Hi @eligavril
I'm back from holidays. I hope that I can help.
I worked on the BYOK feature a couple weeks ago. I was able to have Linux kernel v6.1 and v6.6 working on the Jetson Orin series.
I added around 94 patches on top of Linux kernel v6.1 to make it work. This is based on the following link:
https://docs.nvidia.com/jetson/archives/r36.3/DeveloperGuide/SD/Kernel/BringYourOwnKernel.html#upstream-patches

I didn't add any patch for Linux kernel v6.6
I hope that we can add it to the meta-tegra layer. but I agree with @madisongh about the request from enough folks also we need help with the testing and maintenance.

• Linux kernel v6.1 on Jetson Orin AGX Industrial

root@jetson-agx-orin-devkit-industrial:~# hostnamectl 
 Static hostname: jetson-agx-orin-devkit-industrial
       Icon name: computer
      Machine ID: f3908c5a18a44b5d937f807c0ac95261
         Boot ID: b3d54458188840049f1f10b7c7fbf94f
Operating System: OE4Tegra Demonstration Distro 4.4+snapshot-b5624ee5643d881afa004571a096a189ab5389b5 (master)
     CPE OS Name: cpe:/o:openembedded:tegrademo:4.4+snapshot-b5624ee5643d881afa004571a096a189ab5389b5
          Kernel: Linux 6.1.87-yocto-standard+g6741e066ec76
    Architecture: arm64
 Hardware Vendor: Unknown
  Hardware Model: NVIDIA Jetson AGX Orin Developer Kit
Firmware Version: v36.2.0
   Firmware Date: Wed 2023-12-06
    Firmware Age: 4month 2w 3d
root@jetson-agx-orin-devkit-industrial:~# uname -a 
Linux jetson-agx-orin-devkit-industrial 6.1.87-yocto-standard+g6741e066ec76 #1 SMP PREEMPT Wed Apr 17 09:18:29 UTC 2024 aarch64 GNU/Linux
root@jetson-agx-orin-devkit-industrial:~# 

• Linux kernel v6.6 on Jetson Orin AGX Industrial

root@jetson-agx-orin-devkit-industrial:~# 
root@jetson-agx-orin-devkit-industrial:~# hostnamectl 
 Static hostname: jetson-agx-orin-devkit-industrial
       Icon name: computer
      Machine ID: 0681af3103c247ffa8bea6eb2f2638e1
         Boot ID: e27aeede83bd4ca3867ab75dc00530c2
Operating System: OE4Tegra Demonstration Distro 4.4+snapshot-eb723d4a1082b10d14554f799c6c31cee620152d (master)
     CPE OS Name: cpe:/o:openembedded:tegrademo:4.4+snapshot-eb723d4a1082b10d14554f799c6c31cee620152d
          Kernel: Linux 6.6.29-l4t-r36.3
    Architecture: arm64
 Hardware Vendor: NVIDIA
  Hardware Model: NVIDIA Jetson AGX Orin Developer Kit
Firmware Version: 36.3.0-gcid-36106755
   Firmware Date: Thu 2024-04-25
    Firmware Age: 5d
root@jetson-agx-orin-devkit-industrial:~# 

[eligavril]

Hi @ichergui

It's great news that you got kernel v6.6 working!

I've made my own attempt at building the image with kernel v6.6 and ran into some problems. While the kernel itself did compile, I've encountered issues with nvidia kernel modules (related to nvidia-kernel-oot recipe, I think?).

Could you by any chance provide me with the changes you've made to the meta-tegra layer to get it to work? Even if kernel v6.6 isn't maintained right now in meta-tegra, perhaps I can add\override the necessary stuff in my own custom layer.

It's important to note that I'm working with Orin NX 16GB module.

Any help on the matter would be greatly appreciated!

[ichergui]

Hi @eligavril
I will sync up with @madisongh and see the best option we can go with.
I will keep you posted

[albertbu]

Hi @ichergui @eligavril
This feature is really important for us too, we'd like to see some effort in this direction
Thanks!

[holgerson97]

Is there also an explanation on how to bring a custom 5.x kernel with meta-tegra?

[ichergui]

Hi @holgerson97
Do you mean upgrade Linux kernel version from v5.10 to v5.15, right ?

[holgerson97]

@ichergui I currently have a custom Kernel based on v5.15. I would like to include that one, also I would like to bump this Kernel to 6.6 anytime soon.

What do you think would be required to do so?

[ichergui]

Hi @holgerson97
Are you using the following Linux kernel source ?
https://github.com/OE4T/linux-jammy-nvidia-tegra
or
the upstream Linux kernel

[holgerson97]

@ichergui the upstream kernel

[aknaub]

Hello @ichergui, @madisongh and everyone!

Is there any progress on the BYOK project? Maybe some insights that you are able to share, or an unmerged branch that we could use as reference?

I am making some effort with using a Kernel v5.19 or higher on a Jetson-based board.
Following the documentation by nvidia (https://docs.nvidia.com/jetson/archives/r36.3/DeveloperGuide/SD/Kernel/BringYourOwnKernel.html#upstream-patches), I have the choice to either use 5.19 from Mainline and apply the list of patches, referenced by their mainline commit IDs, or to use 6.6 and have a much shorter list of patches.

It was mentioned here by @eligavril (and I ran into this myself - see below) that using Kernel v6.6 brings inconsistencies with nvidia-kernel-oot, so I started with patching v5.19 according to the Nvidia list of commits. I gathered 146 of them, but quickly found out that they don't apply because some intermediates are missing. After cherry-picking another 11 patches from mainline, I was only able apply up to patch #103 of 157 (somewhere in area of 6.3), gave up and switched back to v6.6.

To start off Kernel v6.6, I used v6.6.23, the same SRC_URI in linux-yocto recipe from poky, in a bbappend to linux-jammy-nvidia-tegra.
I looked into the build issues of nvidia-kernel-oot to see how severe is the incompatibility.
I could fix the build issue caused by the slightly changed signature of iommu_map() introduced in 1369459b2e219a6f4c861404c4f195cd81dcbb40
Now I face the issue that an interface in pci-epf.h, used by nvscic2c-pcie OOT module has significantly diverged between linux-jammy-nvidia-tegra and mainline v6.x.

I will try to handle that as well, but if you can offer a shortcut, it will be highly appreciated.

[kekiefer]

I've had this effort on the backburner for a few weeks since I've been working on Xavier with the same linux-yocto kernel, and got that to a place where it needs the linux-yocto support in here for the Orins.

The WIP builds fine and boots. I haven't done a lot of testing on Orin as I got stuck for my use-case where egl/gbm display output isn't working yet (kms works). The boot looks pretty sane so I hope problems will be limited.

I'll start a WIP pull request branch so we can all work together on this.

[kekiefer]

See WIP in #1680

[ichergui]

Hi @aknaub
We decide to use the linux-yocto in master branch for now in our last monthly meeting instead of using the Linux kernel from upstream for different reasons.
Thanks @kekiefer that put together the changes in the PR #1680
I will help on the review and testing on Orin series.