Bootstrap issues with 4.9.0 image #12

saschagrunert · 2021-12-22T13:30:43Z

Hey, I have the issue that the bootstrap node is not bootable any more. The installations seems to be fine, then it restarts the machine from the iPXE process. Then the reboot got stuck with:

Booting from Hard drive C:
..
error: ../../grub-core/disk/i386/pc/biosdisk.c:498:failure reading sector 0x0
from `cd'.

I tried provisioning multiple facilities (da11, ams6, fra1) without any success.

It also happens that the CoreOS kernel boot screen of grub appears, but then the screen turns black via the out of band console. Pinging the machine is possible but not accessing any service like ssh.

The text was updated successfully, but these errors were encountered:

displague · 2022-01-03T20:04:32Z

I ran into different problems with the bootstrap node on reboot, #10 - these were not OS boot related.

What device plan were you using, @saschagrunert ?

saschagrunert · 2022-01-04T10:17:32Z

@displague do you mean the machine type? I recently tried c3.small as well as c3.medium (Dell R6515) and in both cases the nodes encounter a black screen followed by a reboot after the coreos boot screen selector. The iPXE installation exits with a success indicator.

So I assume it’s something in the kernel boot parameters. 🤔

orenc1 · 2022-01-11T10:54:31Z

I'm now encountering the very same issue as well while trying to spin up an OCP 4.9 on Equinix Metal in DC13 facility.
The Out-of-Band console shows a black screen for all servers (except the lb), or stuck with:

$ ssh d186e620-c6ba-44c1-88e2-b6de75031825@sos.dc13.platformequinix.com
[SOS Session Ready. Use ~? for help.]
[Note: You may need to press RETURN or Ctrl+L to get a prompt.]

This is the bootstrap.ipxe configuration that is being used:

#!ipxe

set release 4.9
set zstream 0
set arch x86_64
set coreos-url http://147.28.129.183:8080
set coreos-img ${coreos-url}/rhcos-${release}.${zstream}-${arch}-live-rootfs.${arch}.img
set console console=ttyS1,115200n8
 
kernel ${coreos-url}/rhcos-${release}.${zstream}-${arch}-live-kernel-${arch} initrd=main coreos.live.rootfs_url=${coreos-img} coreos.inst.install_dev=sda coreos.inst.ignition_url=http://147.28.129.183:8080/bootstrap-append.ign ${console} console=tty0 console=ttyS0,115200n8 ip=dhcp
initrd --name main ${coreos-url}/rhcos-${release}.${zstream}-${arch}-live-initramfs.${arch}.img
boot

Please advise,
Thanks

displague · 2022-01-11T18:17:00Z

I see the console is repeated in the kernel args there - on both ttyS0 and ttyS1. That doesn't sound right, but I don't know if that should create a problem (other than logs from the "getty")

displague · 2022-01-11T19:00:48Z

I'm also getting black screens on the SoS console for the control plane and worker nodes. SSH is not responsive either.

I see that the control plane nodes are configured with an IPXE Script URL of http://{lb-0 address}:8080/master.ipxe - this URL 404s

displague · 2022-01-11T19:17:50Z

The ipxe scripts are not found on the bastion node (/usr/share/nginx/html/*.ipxe)

displague · 2022-01-11T20:39:32Z

From an empty state, I applied the following individually

terraform apply -target 'module.bastion.null_resource.ignition_append_files["master"]'
terraform apply -target 'module.bastion.null_resource.ipxe_files'
terraform apply -target 'module.bastion.null_resource.ocp_install_ignition'

With this approach, /usr/share/nginx/html/ contained the files that were not present in the previous pass. I didn't target all of the files that are supposed to be in this directory.

I then ran a full terraform apply and observed the following warnings or errors:

module.prepare_openshift.null_resource.ocp_installer: Creating...
module.prepare_openshift.null_resource.ocp_installer: Provisioning with 'file'...
module.prepare_openshift.null_resource.ocp_installer: Provisioning with 'remote-exec'...
module.prepare_openshift.null_resource.ocp_installer (remote-exec): Connecting to remote host via SSH...
...
module.prepare_openshift.null_resource.ocp_installer (remote-exec): gzip: stdin: not in gzip format
module.prepare_openshift.null_resource.ocp_installer (remote-exec): tar: Child returned status 1
module.prepare_openshift.null_resource.ocp_installer (remote-exec): tar: Error is not recoverable: exiting now

module.prepare_openshift.null_resource.ocp_installer (remote-exec): gzip: stdin: not in gzip format
module.prepare_openshift.null_resource.ocp_installer (remote-exec): tar: Child returned status 1
module.prepare_openshift.null_resource.ocp_installer (remote-exec): tar: Error is not recoverable: exiting now
module.prepare_openshift.null_resource.ocp_installer (remote-exec): cp: cannot stat ‘oc’: No such file or directory
module.prepare_openshift.null_resource.ocp_installer: Creation complete after 2s [id=894906645089249242]
module.prepare_openshift.null_resource.ocp_pullsecret: Creating...
module.prepare_openshift.null_resource.ocp_pullsecret: Provisioning with 'file'...
module.prepare_openshift.null_resource.ocp_pullsecret: Provisioning with 'remote-exec'...
module.prepare_openshift.null_resource.ocp_pullsecret (remote-exec): (output suppressed due to sensitive value in config)
module.prepare_openshift.null_resource.ocp_pullsecret (remote-exec): (output suppressed due to sensitive value in config)
module.prepare_openshift.null_resource.ocp_pullsecret: Creation complete after 3s [id=2213855489334895954]
module.prepare_openshift.data.template_file.installer_config: Reading...
module.prepare_openshift.data.template_file.installer_config: Read complete after 0s [id=a974cd42f9ea73442c095c53a67da5fd85a58726cde2a4f1aaa6a6d05324e2d6]
module.prepare_openshift.null_resource.ocp_install_config: Creating...
module.prepare_openshift.null_resource.ocp_install_config: Provisioning with 'file'...
module.prepare_openshift.null_resource.ocp_install_config: Provisioning with 'remote-exec'...
module.prepare_openshift.null_resource.ocp_install_config (remote-exec): Connecting to remote host via SSH...
...
module.prepare_openshift.null_resource.ocp_install_config (remote-exec): Connected!
module.prepare_openshift.null_resource.ocp_install_config: Creation complete after 2s [id=2694546928241472463]
module.prepare_openshift.null_resource.ocp_install_manifests: Creating...
module.prepare_openshift.null_resource.ocp_install_manifests: Provisioning with 'remote-exec'...
module.prepare_openshift.null_resource.ocp_install_manifests (remote-exec): Connecting to remote host via SSH...
...
module.prepare_openshift.null_resource.ocp_install_manifests (remote-exec): Connected!
module.prepare_openshift.null_resource.ocp_install_manifests (remote-exec): /tmp/terraform_1825922558.sh: line 6: /tmp/artifacts/openshift-install: No such file or directory
module.prepare_openshift.null_resource.ocp_install_manifests (remote-exec): sed: can't read /tmp/artifacts/install/manifests/cluster-scheduler-02-config.yml: No such file or directory
module.prepare_openshift.null_resource.ocp_install_manifests (remote-exec): /tmp/terraform_1825922558.sh: line 8: /tmp/artifacts/openshift-install: No such file or directory
module.prepare_openshift.null_resource.ocp_install_manifests (remote-exec): cp: cannot stat ‘/tmp/artifacts/install/*.ign’: No such file or directory
module.prepare_openshift.null_resource.ocp_install_manifests (remote-exec): 506 Cannot talk to daemon
module.prepare_openshift.null_resource.ocp_install_manifests: Still creating... [10s elapsed]
module.prepare_openshift.null_resource.ocp_install_manifests (remote-exec): 200 OK
module.prepare_openshift.null_resource.ocp_install_manifests: Creation complete after 12s [id=5925476240016676298]
null_resource.get_kubeconfig: Creating...
null_resource.get_kubeconfig: Provisioning with 'local-exec'...
null_resource.get_kubeconfig (local-exec): Executing: ["/bin/sh" "-c" "mkdir -p ./auth; scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /Users/marques/.ssh/id_rsa_mos-v8nqj [email protected]:/tmp/artifacts/install/auth/* ./auth/"]
module.openshift_controlplane.metal_device.node[2]: Creating...
module.openshift_workers.metal_device.node[0]: Creating...
module.openshift_workers.metal_device.node[1]: Creating...
module.openshift_controlplane.metal_device.node[1]: Creating...
module.openshift_controlplane.metal_device.node[0]: Creating...
module.openshift_bootstrap.metal_device.node[0]: Creating...
null_resource.get_kubeconfig (local-exec): Warning: Permanently added '139.178.84.39' (ED25519) to the list of known hosts.
null_resource.get_kubeconfig (local-exec): scp: /tmp/artifacts/install/auth/*: No such file or directory
module.openshift_controlplane.metal_device.node[2]: Still creating... [10s elapsed]

The control planes nodes do not seem to be accessible again.

orenc1 · 2022-01-11T20:52:54Z

When I ran terraform with:

ocp_version=4.9
ocp_version_zstream=0

or:

ocp_version=4.8
ocp_version_zstream=14

which are corresponding to RHCOS 4.9.0 and 4.8.14 respectively, the /usr/share/nginx/html folder on the bastion/lb host was populated with these files:

root@lb-0 ~]# ll /usr/share/nginx/html/
total 1000852
-rwxr-xr-x. 1 root root      3971 Oct  7  2019 404.html
-rwxr-xr-x. 1 root root      4020 Oct  7  2019 50x.html
-rwxr-xr-x. 1 root root      1175 Jan 11 11:41 bootstrap-append.ign
-rwxr-xr-x. 1 root root       606 Jan 11 11:41 bootstrap.ipxe
-rwxr-xr-x. 1 root root      1172 Jan 11 11:41 master-append.ign
-rwxr-xr-x. 1 root root       603 Jan 11 11:41 master.ipxe
-rwxr-xr-x. 1 root root  89362572 Jan 11 11:41 rhcos-4.8.14-x86_64-live-initramfs.x86_64.img
-rwxr-xr-x. 1 root root  10030448 Jan 11 11:41 rhcos-4.8.14-x86_64-live-kernel-x86_64
-rwxr-xr-x. 1 root root 925434368 Jan 11 11:41 rhcos-4.8.14-x86_64-live-rootfs.x86_64.img
-rwxr-xr-x. 1 root root      1172 Jan 11 11:41 worker-append.ign
-rwxr-xr-x. 1 root root       603 Jan 11 11:41 worker.ipxe

as expected, and the files are indeed accessible, e.g. http://147.28.129.183:8080/bootstrap.ipxe

and I've also seen the following error regarding copying from /tmp/artifacts/install/auth/* in the local host. What should populate that folder?

Error: Error running command 'mkdir -p ./auth; scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /home/ocohen/.ssh/id_rsa_metal-4cfi5 [email protected]:/tmp/artifacts/install/auth/* ./auth/': exit status 1. Output: Warning: Permanently added '147.28.129.183' (ECDSA) to the list of known hosts.
scp: /tmp/artifacts/install/auth/*: No such file or directory

saschagrunert · 2022-01-12T08:08:45Z

@displague I had 404's when running terraform apply sequentially without a git clean -fdx in between. I think it does not download the image correctly when running multiple times. From a clean state it always downloads the image and I never encountered 404 errors. 🤷

displague · 2023-04-18T12:15:52Z

It looks like #20 will be addressing some of these concerns.

displague · 2024-06-13T19:38:56Z

A number of the problems discussed here have been previously resolved in #20.

Error: Error running command 'mkdir -p ./auth; scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /home/ocohen/.ssh/id_rsa_metal-4cfi5 [email protected]:/tmp/artifacts/install/auth/* ./auth/': exit status 1. Output: Warning: Permanently added '147.28.129.183' (ECDSA) to the list of known hosts.
scp: /tmp/artifacts/install/auth/*: No such file or directory

This was experienced and fixed in #31

displague added the bug Something isn't working label Jan 3, 2022

This was referenced Jun 13, 2024

Some fixes and update OCP version to 4.12 #20

Merged

chore: update the module for June 2024 current TF Provider and upstream module behaviors #31

Merged

displague closed this as completed in #31 Jun 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bootstrap issues with 4.9.0 image #12

Bootstrap issues with 4.9.0 image #12

saschagrunert commented Dec 22, 2021 •

edited

Loading

displague commented Jan 3, 2022 •

edited

Loading

saschagrunert commented Jan 4, 2022

orenc1 commented Jan 11, 2022

displague commented Jan 11, 2022

displague commented Jan 11, 2022 •

edited

Loading

displague commented Jan 11, 2022 •

edited

Loading

displague commented Jan 11, 2022 •

edited

Loading

orenc1 commented Jan 11, 2022

saschagrunert commented Jan 12, 2022 •

edited

Loading

displague commented Apr 18, 2023

displague commented Jun 13, 2024 •

edited

Loading

Bootstrap issues with 4.9.0 image #12

Bootstrap issues with 4.9.0 image #12

Comments

saschagrunert commented Dec 22, 2021 • edited Loading

displague commented Jan 3, 2022 • edited Loading

saschagrunert commented Jan 4, 2022

orenc1 commented Jan 11, 2022

displague commented Jan 11, 2022

displague commented Jan 11, 2022 • edited Loading

displague commented Jan 11, 2022 • edited Loading

displague commented Jan 11, 2022 • edited Loading

orenc1 commented Jan 11, 2022

saschagrunert commented Jan 12, 2022 • edited Loading

displague commented Apr 18, 2023

displague commented Jun 13, 2024 • edited Loading

saschagrunert commented Dec 22, 2021 •

edited

Loading

displague commented Jan 3, 2022 •

edited

Loading

displague commented Jan 11, 2022 •

edited

Loading

displague commented Jan 11, 2022 •

edited

Loading

displague commented Jan 11, 2022 •

edited

Loading

saschagrunert commented Jan 12, 2022 •

edited

Loading

displague commented Jun 13, 2024 •

edited

Loading