s390x enablement tracker

[cgwalters]

I think right now this is probably just porting some code from e.g. https://github.com/coreos/coreos-installer around setting up partitions etc. to enable install to-disk.

What also clearly needs to happen here is fixes for https://github.com/osbuild/bootc-image-builder

(Which also leads into the overlap with Anaconda problem)

[yoheiueda]

@cgwalters thank you very much for creating this tracker.

So far, I have identified the following issues.

• Add the s390x architecture in image.go of bootc-image-builder
• Add the partition table for s390x in partition_tables.go of bootc-image-builder
• /boot/efi partitions is required as follows while this partition is not necessary on s390x

panic: required mounts for bootupd stage [/boot/efi] missing

• Disable this architecture check in install.rs of bootc

  bootc/lib/src/install.rs

  Lines 1057 to 1059 in 282ec39

  	if cfg!(target_arch = "s390x") {
  	anyhow::bail!("Installation is not supported on this architecture yet");
  	}

• Install zipl, otherwise the following error occurs when performing image conversion

ERROR Installing to filesystem: Creating ostree deployment: Performing deployment: Final bootloader swap: Failed to execute child process “zipl” (No such file or directory)

I am currently working on the last issue.

[cgwalters]

Add the s390x architecture in image.go of bootc-image-builder

Yep, I filed osbuild/bootc-image-builder#460 as that code is in a distinct project.

[cgwalters]

Install zipl, otherwise the following error occurs when performing image conversion

Right. A lot of useful code in https://github.com/coreos/coreos-installer/blob/main/src/s390x/zipl.rs that I think we can just copy into this project.

[cgwalters]

Also https://gitlab.com/fedora/bootc/tracker/-/issues/15 is related to this

[yoheiueda]

After updating the base image to include bootupd on s390x, I got the following error.

device/disk (org.osbuild.loopback): loop0 acquired (locked: False)
mount/part2 (org.osbuild.xfs): mounting /dev/loop0p2 -> /store/tmp/buildroot-tmp-fcpd6h9i/mounts/
mount/part1 (org.osbuild.ext4): mounting /dev/loop0p1 -> /store/tmp/buildroot-tmp-fcpd6h9i/mounts/boot
mount/part3 (org.osbuild.fat): mounting /dev/loop0p3 -> /store/tmp/buildroot-tmp-fcpd6h9i/mounts/boot/efi
Host kernel does not have SELinux support, but target enables it by default; this is less well tested.  See https://github.com/containers/bootc/issues/419
Installing image: docker://localhost/centos-stream-9-bootc:latest
Initializing ostree layout
Initializing sysroot
ostree/deploy/default initialized as OSTree stateroot
Deploying container image...Error: Config file '/lib/s390-tools/zipl.conf': Cannot build automenu: no IPL entries available
Looking for components in '/lib/s390-tools'
Using config file '/lib/s390-tools/zipl.conf'
done
ERROR Installing to filesystem: Creating ostree deployment: Performing deployment: Final bootloader swap: Child process exited with code 1
Traceback (most recent call last):
  File "/run/osbuild/bin/org.osbuild.bootc.install-to-filesystem", line 53, in <module>
    r = main(args["options"], args["inputs"], args["paths"])
  File "/run/osbuild/bin/org.osbuild.bootc.install-to-filesystem", line 48, in main
    subprocess.run(pargs, env=env, check=True)
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['bootc', 'install', 'to-filesystem', '--source-imgref', 'containers-storage:[overlay@/run/osbuild/containers/storage+/run/containers/storage]f276b7c877e9e130a787dc53a31fdc714edc95456a4fc10e67ff57c1a8038133', '--skip-fetch-check', '--generic-image', '--karg', 'rw', '--karg', 'console=tty0', '--karg', 'console=ttyS0', '--target-imgref', 'localhost/centos-stream-9-bootc:latest', '/run/osbuild/mounts']' returned non-zero exit status 1.

The error occurs during executing zipl at

bootc/lib/src/install.rs

Line 1184 in f152bfe

let aleph = initialize_ostree_root_from_self(state, rootfs).await?;

using the ostree library.
https://github.com/ostreedev/ostree/blob/ac6ba4392285d4f30a937fddf2963b879fe129bd/src/libostree/ostree-bootloader-zipl.c#L477-L478

I guess we need to generate a proper zipf.conf here.

@cgwalters do we need to generate zipl.conf before calling initialize_ostree_root_from_self in bootc? Or, can we use the ostree libbary to genearte zipl.conf. I don't see such generation code in _ostree_bootloader_zipl_write_config.

https://github.com/ostreedev/ostree/blob/ac6ba4392285d4f30a937fddf2963b879fe129bd/src/libostree/ostree-bootloader-zipl.c#L101-L115

crate::bootloader::install_via_bootupd is called after initialize_ostree_root_from_self.

bootc/lib/src/install.rs

Line 1194 in f152bfe

crate::bootloader::install_via_bootupd(&rootfs.device, &rootfs.rootfs, &state.config_opts)?;

It looks like bootupd does no-op for s390x. coreos/bootupd#556

I also think we need a zipl_helper script for loop device like this.
https://github.com/openstack/diskimage-builder/blob/ac180b9ac61b7c8577fe0d489a33b0dd805d8acb/diskimage_builder/elements/zipl/finalise.d/50-zipl#L29-L52

[yoheiueda]

I further read the source code, and I think that zipl needs to recognize the BLS config file at
/store/tmp/buildroot-tmp-ctk6b4of/mounts/boot/loader/entries/ostree-1.conf

[cgwalters]

See https://github.com/coreos/coreos-assembler/blob/43a9c80e1f548269d71d6d586f0d5754c60f6144/src/create_disk.sh#L511

We need to match some of the logic there. Now zipl is somewhat special so far in the ostree/bootc architecture because it basically always acts like a UKI, rolling the bootloader,kernel,initrd together and hence crosses the domain of bootupd and bootc.

Going a bit forward I'd like to merge bootupd into this project coreos/bootupd#432 which would help a bit in making coordinated changes.

[yoheiueda]

@cgwalters thank you for the information.

I managed to create a QCOW2 disk using bootc. To work around the issues I reported above, I need to prepare a wrapper script for zipl, and update centos-bootc image to include it. I don't think this is a good solution, but I just confirmed that the created image works as expected on an s390x machine.

Next, I will investigate what changes are necessary in the bootc and bootupd source code to support zipl.

[cgwalters]

I further read the source code, and I think that zipl needs to recognize the BLS config file at
/store/tmp/buildroot-tmp-ctk6b4of/mounts/boot/loader/entries/ostree-1.conf

This is also related to ostreedev/ostree#3104 perhaps?

This is quite a complex thing going on as osbuild (as used by bootc-image-builder) is also doing its own container stuff, then we have podman, and we also have bubblewrap involved here.

I managed to create a QCOW2 disk using bootc. To work around the issues I reported above, I need to prepare a wrapper script for zipl, and update centos-bootc image to include it. I don't think this is a good solution, but I just confirmed that the created image works as expected on an s390x machine.

Can you post that script?

Also, did you try a direct bootc install to-disk?

[yoheiueda]

@cgwalters sorry for the late reply. I was on vacation.

My script is this one.
https://gist.github.com/yoheiueda/c328ca7eccf313dd03292aeca71fbd02

I directly put this script in the centos bootc base image, and replaced with the original zipl binary in it.

This script bind-mounts /run/osbuild/mounts/boot to /boot, and also temporarily installs a helper script at /run/osbuild/mounts/s390-tools/zipl_helper.blkext. The target disk is

A helper script is called for disk device types that are not natively supported by zipl.
https://github.com/ibm-s390-linux/s390-tools/blob/0e3034c0f699cb311c0f63d0bb475a816f5fba91/zipl/src/disk.c#L359-L384

In the case of osbuild installation, the target disk type is loop, and the disk type of its partition is blkext, so a helper script named zipl_helper.blkext will be called.

This work around needs to update the centos bootc base image, so I don't think it is an appropriate solution for permanent fix, but I just confirmed that QCOW2 image generated with this workaroud, so I now believe there is not other blockers for s390x.

This is also related to ostreedev/ostree#3104 perhaps?
Yes, I think so.

https://github.com/ostreedev/ostree/blob/8f559e94b775b96adbf2bdd2462266bce3c42185/src/libostree/ostree-bootloader-zipl.c#L480-L503

If target_deployment is set and /boot is appropriately mounted, bind-mounting of /run/osbuild/mounts/boot is not necessary. But, it looks like target_deployment is set to NULL, and zipl is directly executed.

Also, did you try a direct bootc install to-disk?

I have only tried install-to-filesystem via bootc-image-builder. I will try a direct bootc install to-disk next.

[cgwalters]

A helper script is called for disk device types that are not natively supported by zipl.
https://github.com/ibm-s390-linux/s390-tools/blob/0e3034c0f699cb311c0f63d0bb475a816f5fba91/zipl/src/disk.c#L359-L384

Ahhh of course. This is the big difference between what we were doing in coreos-assembler using qemu (providing a native dasd disk to the build) vs what bib (and bootc can do) with loopback. Hmm...that said, it looks like osbuild's zipl stage isn't doing anything too special here?

If target_deployment is set and /boot is appropriately mounted, bind-mounting of /run/osbuild/mounts/boot is not necessary. But, it looks like target_deployment is set to NULL, and zipl is directly executed.

I think that may be one of the bugs here.

[yoheiueda]

@cgwalters

The osbuild's zipl stage explicitly set parameters targetbase, targettype, targettype, targetblocksize and targetoffset. In this case, the helper script is not necessary. The helper script is used to identify these values when the parameters are not specified.

So, is it an option to use the osbuild's zipl stage instead of ostree by generating a s390x-specific manifest by bootc?

Regarding bootc install to-disk, I needed to modify the code to prepare partitions for s390x.

bootc/lib/src/install/baseline.rs

Line 234 in 01f8d76

let mut sgdisk = Task::new("Initializing partitions", "sgdisk");

Then, I got the following error. I guess the cause of the error is that BLS configs under /boot/loader/entries are not visible from zipl. In this case, I didn't use my zipl wrapper script here.

sudo podman run --rm --privileged --pid=host -v /var/lib/containers:/var/lib/containers -v /dev:/dev -v $PWD/output:/output --security-opt label=type:unconfined_t localhost/centos-stream-9-bootc:latest env RUST_LOG=debug bootc install to-disk --skip-fetch-check --via-loopback --wipe /output/disk.img

Error: Config file '/lib/s390-tools/zipl.conf': Cannot build automenu: no IPL entries available
Looking for components in '/lib/s390-tools'
Using config file '/lib/s390-tools/zipl.conf'
done
DEBUG exec: "losetup" "-d" "/dev/loop0"
ERROR Installing to disk: Creating ostree deployment: Performing deployment: Final bootloader swap: Child process exited with code 1

In the case of bootc install to-disk, this command should work as a standalone tool, so use of osbuild's zipl stage may not be an option.

As you mentioned earlier, if bootupd code is integrated into bootc, we can directly add zipl support in bootc. I think this is a best option, but it will take some time?

[cgwalters]

The osbuild's zipl stage explicitly set parameters targetbase, targettype, targettype, targetblocksize and targetoffset. In this case, the helper script is not necessary. The helper script is used to identify these values when the parameters are not specified.

OK. Of those...do you understand where targetoffset comes from? I looked through the code but I don't understand that.

So, is it an option to use the osbuild's zipl stage instead of ostree by generating a s390x-specific manifest by bootc?

We have a logistical problem in that so far, osbuild is something designed for build-time only; we don't ship it onto the end system. IOW it doesn't exist by default in a bootc container. That doesn't mean it can't, but it would be a big change to add it.

There's a larger picture problem around code sharing between osbuild, anaconda, bootc (and for that matter, a wide array of other tools in this space, some distro specific, others more generic like Kiwi etc.)

Looking at that osbuild stage, in the end it's just parsing some bootloader configs and invoking zipl mostly, and we already have all that code here too, right?

Regarding bootc install to-disk, I needed to modify the code to prepare partitions for s390x.

Can you send a patch for that? Even if it's not all complete, nothing stops us from merging it now.

Then, I got the following error. I guess the cause of the error is that BLS configs under /boot/loader/entries are not visible from zipl.

Hmm, yes. There's a whole lot of namespacing and mounts going on here. It's just complex since zipl so far is the only case where we need to run code to update the bootloader.

As you mentioned earlier, if bootupd code is integrated into bootc, we can directly add zipl support in bootc. I think this is a best option, but it will take some time?

I don't think we need to block on merging bootupd; it would help a bit, but it's not the core problem here.

[yoheiueda]

OK. Of those...do you understand where targetoffset comes from? I looked through the code but I don't understand that.

According to this page, targetoffset needs to be set to 2048 when the first partition is the boot partition.
https://bugs.launchpad.net/diskimage-builder/+bug/1730641

MBR consumes 512 Byte. Due to the alignment the next (1MiB -512 Bytes) are unused. The first partition starts at 1 MiB which is block 2048 ((1024 * 1024)B / 512 Bytes per Block).

-> zipls targetoffset should be set to 2048 when the first partition is the boot partition.

Currently, I specify the whole disk (/dev/loop0) to targetbase, and its first partition (/dev/loop0p1) is /boot, so we need to specify targetoffset=2048. If I directly specify the first partition (/dev/loop0p1) to targetbase, do I need to specify targetoffset=0? I will confirm it later.

Looking at that osbuild stage, in the end it's just parsing some bootloader configs and invoking zipl mostly, and we already have all that code here too, right?

zipl has a mechanism to find kernel and ramdisk paths using BLS info under /boot/loader/entries/, but the mechanism is only used when the command line options (targetbase, targetoffset, ...) are not specified if I remember correctly. When the target options are specified, the options --image, --ramdisk, and --parameters are also needed to be specified directly.

So, I think we may need to copy the code that searchs BLS info from https://github.com/coreos/coreos-installer/blob/main/src/s390x/zipl.rs to identify kernel and ramdisk paths.

Regarding bootc install to-disk, I needed to modify the code to prepare partitions for s390x.

Can you send a patch for that? Even if it's not all complete, nothing stops us from merging it now.

Sure. I currently did quick dirty hacks to suppress errors on s390x. I'll prepare a cleaner patch to fix the errors.

[yoheiueda]

I managed to run bootc install to-disk. I updated my zipl wrapper script to calculate a targetoffset of /boot partition.

https://gist.github.com/yoheiueda/c328ca7eccf313dd03292aeca71fbd02

[yoheiueda]

@cgwalters

Regarding bootc install to-disk, I needed to modify the code to prepare partitions for s390x.

Can you send a patch for that? Even if it's not all complete, nothing stops us from merging it now.

I raised a PR for this issue. #623

[yoheiueda]

@cgwalters I raised an PR to run zipl with correct options on a loop device. #665

[yoheiueda]

@cgwalters I created an PR to enable s390x. #705

[cgwalters]

Great! So...I think we can keep osbuild/bootc-image-builder#460 open to track the bib work.

The remaining bits I think are probably just to turn on ppc64le/s390x on the builds of quay.io/centos-bootc/bootc-image-builder which should likely be tracked there too.

OK to close this one?

[yoheiueda]

Yes. It is OK to close this one. Thank you very much for your support!

I'll submit a PR to bootc-image-builder soon.