Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1 master & 0 workers setup requires "create manifest" step twice #238

Closed
pjbrzozowski opened this issue Jul 1, 2020 · 35 comments
Closed
Labels
lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.

Comments

@pjbrzozowski
Copy link

pjbrzozowski commented Jul 1, 2020

Describe the bug
OKD 4.4 installer updates FCOS image to version 32

Version
4.4.0-0.okd-2020-07-01-045420 (but I also checked it with a lot of other 4.4 builds)

How reproducible

  1. Follow standard OKD procedures for Bare-metal installation.
  2. Notice FCOS version being initially 31 and then it is updated to 32.
  3. Waiting for boostrap is never finished.

This was working fine 100% before FCOS 32 was released and marked as STABLE.

Kernel PXE:
fedora-coreos-31.20200517.3.0-live-initramfs.x86_64.img
fedora-coreos-31.20200517.3.0-live-kernel-x86_64

Baremetal CoreOS image:
fedora-coreos-31.20200505.3.0-metal.x86_64.raw.xz

Log bundle
[must-gather ] OUT Get https://api.ocp.domain.net:6443/apis/image.openshift.io/v1/namespaces/openshift/imagestreams/must-gather: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-apiserver-lb-signer") [must-gather ] OUT [must-gather ] OUT Using must-gather plugin-in image: quay.io/openshift/origin-must-gather:latest Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-apiserver-lb-signer")

$ openshift-install wait-for bootstrap-complete --log-level=debug DEBUG OpenShift Installer 4.4.0-0.okd-2020-07-01-045420 DEBUG Built from commit ddd989504d76ae25b0a020db9b29f9375d5ce242 INFO Waiting up to 20m0s for the Kubernetes API at https://api.ocp.domain.net:6443... DEBUG Still waiting for the Kubernetes API: Get https://api.ocp.domain.net:6443/version?timeout=32s: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-apiserver-lb-signer")

Related issues
Possibly:
#229
#227

Thank you!

@vrutkovs
Copy link
Member

vrutkovs commented Jul 1, 2020

@vrutkovs vrutkovs added the triage/needs-information Indicates an issue needs more information in order to work on it. label Jul 1, 2020
@pjbrzozowski
Copy link
Author

log-bundle-20200701135351.tar.gz

Thank you @vrutkovs !

@vrutkovs
Copy link
Member

vrutkovs commented Jul 1, 2020

Notice FCOS version being initially 31 and then it is updated to 32.

That's expected - initial FCOS is being updated to the version in the release (https://origin-release.apps.ci.l2s4.p1.openshiftapps.com/releasestream/4.4.0-0.okd/release/4.4.0-0.okd-2020-07-01-045420).

Control plane never requested master ignition from bootstrap - ensure your LB is setup correctly. Did any of the masters booted? Can they access LB address and fetch ignition?

@cgruver
Copy link

cgruver commented Jul 1, 2020

I am seeing a similar issue with FCOS 32 and OKD 4.5 bare-metal UPI.

The Bootstrap node starts fine, the API comes up, but the master nodes are unable to retrieve ignition config.

Using curl to attempt to retrieve the ignition config results in a 503.

I've got a log bundle that I will upload as soon as I double check to ensure that I haven't done something silly to cause the issue.

@vrutkovs
Copy link
Member

vrutkovs commented Jul 1, 2020

I suspect Zincati has been fixed recently and we don't disable it, so master nodes are being updated to latest stable instead of the payload we expect

@cgruver
Copy link

cgruver commented Jul 1, 2020

I've got a long weekend with the US holiday, so I'll have some time in my lab to tinker.

@pjbrzozowski pjbrzozowski changed the title OKD 4.4 installer (all builds) updating FCOS image to 32 - no pinning to 31 OKD 4.4 installer (all builds) started to fail Jul 1, 2020
@pjbrzozowski
Copy link
Author

@vrutkovs The output I have generated was actually for just a beginning of installation, control node was never started. Sorry about that, I thought that FCOS should not update for higher than 31 for 4.4.0 and I assumed this is the problem I am seeing since 29 Jun.

I used to destroy and recreate cluster few times per day, and at some point, even with exactly the same source files, kernel images, openshift-installer version I am not longer able to bootstrap the cluster any more. I hoped it was related to FCOS version not pinned, but the problem must be elsewhere.

@cgruver For 4.5.0 I was also seeing what you are describing #229 by the way.

Probably this bug does not make sense any more, as @vrutkovs explained how it is fixed between openshift-installer and FCOS versions. I will open a new ticket and I will provide more information there - I hope this is ok @vrutkovs.

Thank you!

@vrutkovs
Copy link
Member

vrutkovs commented Jul 1, 2020

Feel free to reuse this ticket if you like

@pjbrzozowski
Copy link
Author

pjbrzozowski commented Jul 1, 2020

Thanks @vrutkovs!

The version (which still uses FCOS 31 and which was battle tested for me) was 4.4.0-0.okd-2020-05-23-055148-beta5 with fedora-coreos-31.20200517.3.0-live-kernel-x86_64 PXE kernel.

Please find the attached archive (this time control node was bootstrapped completely):
https://drive.google.com/file/d/1NoP-EMtWczvVTygtqVfEj3ahXU2qfMUa/view?usp=sharing (I could not attach it here, because of size limitation).

In bootkube.sh I see that it should be completed:
Jul 01 13:12:12 bootstrap bootkube.sh[8683]: Sending bootstrap-finished event.Tearing down temporary bootstrap co ntrol plane... Jul 01 13:12:12 bootstrap bootkube.sh[8683]: Waiting for CEO to finish... Jul 01 13:12:12 bootstrap bootkube.sh[8683]: I0701 13:12:12.585853 1 waitforceo.go:64] Cluster etcd operato r bootstrapped successfully Jul 01 13:12:12 bootstrap bootkube.sh[8683]: I0701 13:12:12.588486 1 waitforceo.go:58] cluster-etcd-operato r bootstrap etcd Jul 01 13:12:12 bootstrap bootkube.sh[8683]: bootkube.service complete Jul 01 13:12:12 bootstrap systemd[1]: bootkube.service: Succeeded.

But in wait-for bootstrap-commplete ends up with waiting for Kubernetes API: context deadline exceeded.

I can not find anything interesting in POD logs, kube-apiserver is complaining about certificates:
I0701 14:39:30.644963 1 log.go:172] http: TLS handshake error from 10.4.20.10:53422: remote error: tls: bad certificate I0701 14:39:30.722111 1 log.go:172] http: TLS handshake error from 10.4.20.10:53424: remote error: tls: bad certificate I0701 14:39:30.794873 1 log.go:172] http: TLS handshake error from 10.4.20.10:53430: remote error: tls: bad certificate I0701 14:39:30.870435 1 log.go:172] http: TLS handshake error from 10.4.20.10:53432: remote error: tls: bad certificate I0701 14:39:30.948731 1 log.go:172] http: TLS handshake error from 10.4.20.10:53434: remote error: tls: bad certificate I0701 14:39:55.861784 1 log.go:172] http: TLS handshake error from 10.4.20.10:53624: remote error: tls: unknown certificate authority

Just like my kubectl command:
$ export KUBECONFIG=~/Lab/infra/openshift/auth/kubeconfig $ kubectl get nodes Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-apiserver-lb-signer")

I am in dead end now, I see PODs bouncing. I don't know anything what could be changed except for the external dependencies, this I why I was hoping for this to be related to FCOS upgrade.

If you can take a look and tell me what I am missing, that would be great.

Thanks!

@vrutkovs
Copy link
Member

vrutkovs commented Jul 2, 2020

Few notes from log-bundle:

  1. beta5, single master, which is not marked as worker.
    Are you planning to add workers later? If not then master needs to be marked as a worker too (workers: 0 in install-config).

  2. bootstrap completed. Can your machine access the LB correctly? context deadline exceeded might mean LB misconfiguration - apparently cluster API is working correctly but api. lb is not setup correctly, so you get certificate signed by unknown authority. See https://github.com/openshift/okd/tree/master/Guides/UPI for example LB config

@vrutkovs vrutkovs changed the title OKD 4.4 installer (all builds) started to fail OKD 4.4 beta5 installer - lb error after the cluster has bootstrapped Jul 2, 2020
@pjbrzozowski
Copy link
Author

1. beta5, single master, which is not marked as worker.
   Are you planning to add workers later? If not then master needs to be marked as a worker too (`workers: 0` in install-config).

I wanted to add more when bootstrap was ready, I had workers: 0 as I got this from example 4.4 Bare Metal installation and I wanted to check if this will make any difference.

2. bootstrap completed. Can your machine access the LB correctly? `context deadline exceeded` might mean LB misconfiguration - apparently cluster API is working correctly but `api.` lb is not setup correctly, so you get `certificate signed by unknown authority`. See https://github.com/openshift/okd/tree/master/Guides/UPI for example LB config

I have just checked and it seems like api and master-01 endpoint are providing exactly the same certificates:

pbrzozowski@navi:~ $ openssl s_client -connect services.ocp.ontux.net:6443 -servername api.ocp.ontux.net -showcerts </dev/null 2>/dev/null | openssl x509 -text | md5sum
87b389418bd69160350bf47d8467b35d  -
pbrzozowski@navi:~ $ openssl s_client -connect master-01.ocp.ontux.net:6443 -servername api.ocp.ontux.net -showcerts </dev/null 2>/dev/null | openssl x509 -text | md5sum
87b389418bd69160350bf47d8467b35d  -

pbrzozowski@navi:~ $ openssl s_client -connect master-01.ocp.ontux.net:6443 -servername master-01.ocp.ontux.net -showcerts </dev/null 2>/dev/null | openssl x509 -text | md5sum
e05221052fadba6abbee5247b8b1c8d7  -
pbrzozowski@navi:~ $ openssl s_client -connect api.ocp.ontux.net:6443 -servername master-01.ocp.ontux.net -showcerts </dev/null 2>/dev/null | openssl x509 -text | md5sum
e05221052fadba6abbee5247b8b1c8d7  -

Beside, this configuration was working fine for over 3 weeks:

[root@services html]# ls -la /etc/haproxy/haproxy.cfg 
-rw-r--r--. 1 root root 2693 Jun 15 16:25 /etc/haproxy/haproxy.cfg

It is almost 1 to 1 example from haproxy.cfg example.

Actually this is like some kind of sorcery to me, I think I have checked every possible scenario. But perhaps I am missing something super obvious.

This is how the api certificate looks like:

(ansible) pbrzozowski@navi:infra/openshift git:(master) ✗$ openssl s_client -connect api.ocp.ontux.net:6443 -servername api.ocp.ontux.net -showcerts </dev/null 2>/dev/null | openssl x509 -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 9211289488554104315 (0x7fd512fc808e95fb)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: OU = openshift, CN = kube-apiserver-lb-signer
        Validity
            Not Before: Jul  2 08:58:38 2020 GMT
            Not After : Aug  1 08:58:39 2020 GMT
        Subject: CN = api.ocp.ontux.net
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:b4:6f:f2:69:86:3a:6f:0b:a0:c6:c3:82:a3:f2:
                    9e:86:e9:3b:98:da:4a:17:13:9e:6f:26:2b:f2:d8:
                    1c:8a:44:0b:32:2d:c8:53:73:97:20:30:db:dd:1e:
                    d6:fe:46:03:b3:48:46:34:71:af:8e:68:fd:c5:d3:
                    15:09:fd:10:2e:8c:e6:c7:21:f0:f9:dc:57:fe:4c:
                    13:ba:c8:47:b1:dc:30:05:4a:df:2a:cb:f2:fb:ce:
                    51:21:20:d1:20:7d:a9:89:18:1f:37:94:d8:e4:c4:
                    52:21:90:b0:e9:c3:ac:42:c7:74:d3:1b:c5:3c:94:
                    5c:0a:ce:24:ba:d3:f5:1d:b7:4e:c3:f0:6f:01:ed:
                    b6:bc:37:0f:b0:91:16:b8:83:8a:4d:8d:36:cd:f6:
                    04:f2:b4:4c:a6:cb:70:5b:ba:c2:d1:ce:f4:c5:61:
                    43:49:d3:54:2f:7c:da:aa:6d:dd:c0:a6:7b:2a:ae:
                    db:86:e0:37:54:cf:cb:7e:df:13:5b:38:d8:9e:e4:
                    71:08:12:95:1f:24:19:2b:d4:4d:05:1e:3e:16:ac:
                    0a:0b:63:db:b4:3e:c0:d0:b0:0d:a1:8d:11:0e:99:
                    35:ff:4b:68:ce:b1:66:9d:cd:c5:55:70:66:6b:c5:
                    a2:fa:93:88:cc:00:17:b6:22:86:65:47:65:39:0b:
                    23:2f
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                7C:76:C0:2A:AF:DA:85:8F:32:17:3E:C7:52:C1:3A:24:01:45:CC:A6
            X509v3 Authority Key Identifier:
                keyid:19:0A:B1:20:9D:55:3E:7C:AA:A8:02:5A:75:35:A3:82:7D:EA:58:68

            X509v3 Subject Alternative Name:
                DNS:api.ocp.ontux.net
    Signature Algorithm: sha256WithRSAEncryption
         bc:25:c4:3b:e0:1d:d3:62:65:e1:fc:42:b5:fc:ec:1c:a3:20:
         df:d7:64:5f:01:fa:c3:21:94:88:70:9f:b9:e7:cd:61:fe:68:
         85:af:99:66:fc:80:bd:bc:69:c6:36:03:f3:0f:ed:0a:de:d9:
         dd:e5:f3:47:d2:7a:5d:fe:9c:b7:41:ef:57:dc:9e:b6:a4:46:
         66:82:93:db:53:15:d0:0b:a5:6a:da:03:20:33:68:a3:07:09:
         24:c0:9b:cd:08:13:e1:60:4a:51:af:fe:1c:77:13:71:19:c3:
         b0:a2:b4:98:07:1d:d8:6c:ae:51:7e:db:09:97:49:1f:ee:1a:
         07:67:a2:b8:fa:67:af:0e:2d:2e:85:6f:3f:a1:5a:98:7f:2b:
         20:03:42:dc:93:27:d8:6b:3c:09:7e:8a:4f:72:fa:07:07:c3:
         da:00:5d:ec:4a:d3:26:bf:0d:3e:09:00:ae:28:fe:8f:b7:f3:
         78:02:c2:51:8e:bb:78:ed:e7:14:6a:b2:09:e2:a1:71:35:93:
         84:98:55:b1:11:05:d5:e4:c1:5d:08:f8:dd:77:78:87:0e:bf:
         6c:48:79:83:97:18:0e:dc:96:a1:ad:b9:f4:9e:74:ee:fe:d5:
         0e:45:fa:a8:71:79:10:f4:29:a3:a7:74:77:af:b1:2f:75:84:
         c9:fe:68:9b
-----BEGIN CERTIFICATE-----
MIIDajCCAlKgAwIBAgIIf9US/ICOlfswDQYJKoZIhvcNAQELBQAwNzESMBAGA1UE
CxMJb3BlbnNoaWZ0MSEwHwYDVQQDExhrdWJlLWFwaXNlcnZlci1sYi1zaWduZXIw
HhcNMjAwNzAyMDg1ODM4WhcNMjAwODAxMDg1ODM5WjAcMRowGAYDVQQDExFhcGku
b2NwLm9udHV4Lm5ldDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALRv
8mmGOm8LoMbDgqPynobpO5jaShcTnm8mK/LYHIpECzItyFNzlyAw290e1v5GA7NI
RjRxr45o/cXTFQn9EC6M5sch8PncV/5ME7rIR7HcMAVK3yrL8vvOUSEg0SB9qYkY
HzeU2OTEUiGQsOnDrELHdNMbxTyUXArOJLrT9R23TsPwbwHttrw3D7CRFriDik2N
Ns32BPK0TKbLcFu6wtHO9MVhQ0nTVC982qpt3cCmeyqu24bgN1TPy37fE1s42J7k
cQgSlR8kGSvUTQUePhasCgtj27Q+wNCwDaGNEQ6ZNf9LaM6xZp3NxVVwZmvFovqT
iMwAF7YihmVHZTkLIy8CAwEAAaOBlDCBkTAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l
BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUfHbAKq/ahY8y
Fz7HUsE6JAFFzKYwHwYDVR0jBBgwFoAUGQqxIJ1VPnyqqAJadTWjgn3qWGgwHAYD
VR0RBBUwE4IRYXBpLm9jcC5vbnR1eC5uZXQwDQYJKoZIhvcNAQELBQADggEBALwl
xDvgHdNiZeH8QrX87ByjIN/XZF8B+sMhlIhwn7nnzWH+aIWvmWb8gL28acY2A/MP
7Qre2d3l80fSel3+nLdB71fcnrakRmaCk9tTFdALpWraAyAzaKMHCSTAm80IE+Fg
SlGv/hx3E3EZw7CitJgHHdhsrlF+2wmXSR/uGgdnorj6Z68OLS6Fbz+hWph/KyAD
QtyTJ9hrPAl+ik9y+gcHw9oAXexK0ya/DT4JAK4o/o+383gCwlGOu3jt5xRqsgni
oXE1k4SYVbERBdXkwV0I+N13eIcOv2xIeYOXGA7clqGtufSedO7+1Q5F+qhxeRD0
KaOndHevsS91hMn+aJs=
-----END CERTIFICATE-----

When I am trying to login using kubeadmin password from /auth directory:

(ansible) pbrzozowski@navi:infra/openshift git:(master) ✗$ oc login -u kubeadmin -p QPtM3-cMPoj-KzTtZ-sGGE8 https://api.ocp.ontux.net:6443
The server uses a certificate signed by an unknown authority.
You can bypass the certificate check, but any data you send to the server could be intercepted by others.
Use insecure connections? (y/n): y

error: couldn't get https://api.ocp.ontux.net:6443/.well-known/oauth-authorization-server: unexpected response status 404

The same if I try to connect to master directly:

(ansible) pbrzozowski@navi:infra/openshift git:(master) ✗$ oc login -u kubeadmin -p QPtM3-cMPoj-KzTtZ-sGGE8 https://master-01.ocp.ontux.net:6443
The server is using a certificate that does not match its hostname: x509: certificate is valid for kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, openshift, openshift.default, openshift.default.svc, openshift.default.svc.cluster.local, 172.30.0.1, not master-01.ocp.ontux.net
You can bypass the certificate check, but any data you send to the server could be intercepted by others.
Use insecure connections? (y/n): y

error: couldn't get https://master-01.ocp.ontux.net:6443/.well-known/oauth-authorization-server: unexpected response status 404

I have also tried to rebuild the cluster with PXE kernel from FCOS 32 (as stated in #229 (comment)), but this failed the same way. And this time I had 3 workers in config.

Please find the log archive attached:
https://drive.google.com/file/d/1N8ASDt6zS2LsPWrz2Wn_PrZAvalulCmj/view?usp=sharing

Vadim, please let me know if there is anything else you can think of, what I can check.

Thanks a lot for your help!

@pjbrzozowski
Copy link
Author

@vrutkovs I managed to deploy it. I will let you know soon.

@pjbrzozowski
Copy link
Author

So, now it works every time, but I need to run a command to create manifests twice for some reason:

#!/bin/bash

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

echo "Preparing build directory..."

rm -rf ${DIR}/build
mkdir ${DIR}/build

cp ${DIR}/install-config.yaml ${DIR}/build

echo "Creating manifests..."
openshift-install create manifests --dir ${DIR}/build

echo "Creating manifests (2)..."
openshift-install create manifests --dir ${DIR}/build

echo "Creating ignition files..."
openshift-install create ignition-configs --dir ${DIR}/build

echo "Pushing ingition files to HTTP server..."
scp ${DIR}/build/*.ign [email protected]:/var/www/html/okd4/ ; ssh [email protected] chown apache -R /var/www/html

The key here is to run create manifests twice. This is the output I am getting:

Preparing build directory...
Creating manifests...
INFO Consuming Install Config from target directory
WARNING Discarding the Openshift Manifests that was provided in the target directory because its dependencies are dirty and it needs to be regenerated
Creating manifests (2)...
INFO Consuming OpenShift Install (Manifests) from target directory
Creating ignition files...
INFO Consuming Openshift Manifests from target directory
INFO Consuming Common Manifests from target directory
INFO Consuming Worker Machines from target directory
INFO Consuming OpenShift Install (Manifests) from target directory
INFO Consuming Master Machines from target directory
Pushing ingition files to HTTP server...
bootstrap.ign                                                                  100%  290KB   1.3MB/s   00:00
master.ign                                                                     100% 1844    49.1KB/s   00:00
worker.ign                                                                     100% 1844    45.6KB/s   00:00

And now it is perfectly fine!

This is debug output from the first iteration, with the warning:

Creating manifests...
DEBUG OpenShift Installer 4.4.0-0.okd-2020-05-23-055148-beta5
DEBUG Built from commit 0f0142e7261349b93c3dd3dd02a9ce164dfd2d4f
DEBUG Fetching Master Machines...
DEBUG Loading Master Machines...
DEBUG   Loading Cluster ID...
DEBUG     Loading Install Config...
DEBUG       Loading SSH Key...
DEBUG       Loading Base Domain...
DEBUG         Loading Platform...
DEBUG       Loading Cluster Name...
DEBUG         Loading Base Domain...
DEBUG         Loading Platform...
DEBUG       Loading Pull Secret...
DEBUG       Loading Platform...
DEBUG     Using Install Config loaded from target directory
DEBUG   Loading Platform Credentials Check...
DEBUG     Loading Install Config...
DEBUG   Loading Install Config...
DEBUG   Loading Image...
DEBUG     Loading Install Config...
DEBUG   Loading Master Ignition Config...
DEBUG     Loading Install Config...
DEBUG     Loading Root CA...
DEBUG   Fetching Cluster ID...
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG   Generating Cluster ID...
DEBUG   Fetching Platform Credentials Check...
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG   Generating Platform Credentials Check...
DEBUG   Fetching Install Config...
DEBUG   Reusing previously-fetched Install Config
DEBUG   Fetching Image...
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG   Generating Image...
DEBUG   Fetching Master Ignition Config...
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG     Fetching Root CA...
DEBUG     Generating Root CA...
DEBUG   Generating Master Ignition Config...
DEBUG Generating Master Machines...
INFO Consuming Install Config from target directory
DEBUG Purging asset "Install Config" from disk
DEBUG Fetching Worker Machines...
DEBUG Loading Worker Machines...
DEBUG   Loading Cluster ID...
DEBUG   Loading Platform Credentials Check...
DEBUG   Loading Install Config...
DEBUG   Loading Image...
DEBUG   Loading Worker Ignition Config...
DEBUG     Loading Install Config...
DEBUG     Loading Root CA...
DEBUG   Fetching Cluster ID...
DEBUG   Reusing previously-fetched Cluster ID
DEBUG   Fetching Platform Credentials Check...
DEBUG   Reusing previously-fetched Platform Credentials Check
DEBUG   Fetching Install Config...
DEBUG   Reusing previously-fetched Install Config
DEBUG   Fetching Image...
DEBUG   Reusing previously-fetched Image
DEBUG   Fetching Worker Ignition Config...
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG     Fetching Root CA...
DEBUG     Reusing previously-fetched Root CA
DEBUG   Generating Worker Ignition Config...
DEBUG Generating Worker Machines...
DEBUG Fetching Common Manifests...
DEBUG Loading Common Manifests...
DEBUG   Loading Cluster ID...
DEBUG   Loading Install Config...
DEBUG   Loading Ingress Config...
DEBUG     Loading Install Config...
DEBUG   Loading DNS Config...
DEBUG     Loading Install Config...
DEBUG     Loading Cluster ID...
DEBUG     Loading Platform Credentials Check...
DEBUG   Loading Infrastructure Config...
DEBUG     Loading Cluster ID...
DEBUG     Loading Install Config...
DEBUG     Loading Cloud Provider Config...
DEBUG       Loading Install Config...
DEBUG       Loading Cluster ID...
DEBUG       Loading Platform Credentials Check...
DEBUG     Loading Additional Trust Bundle Config...
DEBUG       Loading Install Config...
DEBUG   Loading Network Config...
DEBUG     Loading Install Config...
DEBUG     Loading Network CRDs...
DEBUG   Loading Proxy Config...
DEBUG     Loading Install Config...
DEBUG     Loading Network Config...
DEBUG   Loading Scheduler Config...
DEBUG     Loading Install Config...
DEBUG   Loading Image Content Source Policy...
DEBUG     Loading Install Config...
DEBUG   Loading Root CA...
DEBUG   Loading Certificate (etcd-signer)...
DEBUG   Loading Certificate (etcd-ca-bundle)...
DEBUG     Loading Certificate (etcd-signer)...
DEBUG   Loading Certificate (etcd-client)...
DEBUG     Loading Certificate (etcd-signer)...
DEBUG   Loading Certificate (etcd-metric-ca-bundle)...
DEBUG     Loading Certificate (etcd-metric-signer)...
DEBUG   Loading Certificate (etcd-metric-signer)...
DEBUG   Loading Certificate (etcd-metric-signer-client)...
DEBUG     Loading Certificate (etcd-metric-signer)...
DEBUG   Loading Certificate (mcs)...
DEBUG     Loading Root CA...
DEBUG     Loading Install Config...
DEBUG   Loading CVOOverrides...
DEBUG   Loading EtcdCAConfigMap...
DEBUG   Loading EtcdClientSecret...
DEBUG   Loading EtcdHostServiceEndpoints...
DEBUG   Loading EtcdHostService...
DEBUG   Loading EtcdMetricClientSecret...
DEBUG   Loading EtcdMetricServingCAConfigMap...
DEBUG   Loading EtcdMetricSignerSecret...
DEBUG   Loading EtcdNamespace...
DEBUG   Loading EtcdService...
DEBUG   Loading EtcdSignerSecret...
DEBUG   Loading KubeCloudConfig...
DEBUG   Loading EtcdServingCAConfigMap...
DEBUG   Loading KubeSystemConfigmapRootCA...
DEBUG   Loading MachineConfigServerTLSSecret...
DEBUG   Loading OpenshiftConfigSecretPullSecret...
DEBUG   Loading OpenshiftMachineConfigOperator...
DEBUG   Fetching Cluster ID...
DEBUG   Reusing previously-fetched Cluster ID
DEBUG   Fetching Install Config...
DEBUG   Reusing previously-fetched Install Config
DEBUG   Fetching Ingress Config...
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG   Generating Ingress Config...
DEBUG   Fetching DNS Config...
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG     Fetching Cluster ID...
DEBUG     Reusing previously-fetched Cluster ID
DEBUG     Fetching Platform Credentials Check...
DEBUG     Reusing previously-fetched Platform Credentials Check
DEBUG   Generating DNS Config...
DEBUG   Fetching Infrastructure Config...
DEBUG     Fetching Cluster ID...
DEBUG     Reusing previously-fetched Cluster ID
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG     Fetching Cloud Provider Config...
DEBUG       Fetching Install Config...
DEBUG       Reusing previously-fetched Install Config
DEBUG       Fetching Cluster ID...
DEBUG       Reusing previously-fetched Cluster ID
DEBUG       Fetching Platform Credentials Check...
DEBUG       Reusing previously-fetched Platform Credentials Check
DEBUG     Generating Cloud Provider Config...
DEBUG     Fetching Additional Trust Bundle Config...
DEBUG       Fetching Install Config...
DEBUG       Reusing previously-fetched Install Config
DEBUG     Generating Additional Trust Bundle Config...
DEBUG   Generating Infrastructure Config...
DEBUG   Fetching Network Config...
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG     Fetching Network CRDs...
DEBUG     Generating Network CRDs...
DEBUG   Generating Network Config...
DEBUG   Fetching Proxy Config...
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG     Fetching Network Config...
DEBUG     Reusing previously-fetched Network Config
DEBUG   Generating Proxy Config...
DEBUG   Fetching Scheduler Config...
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG   Generating Scheduler Config...
DEBUG   Fetching Image Content Source Policy...
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG   Generating Image Content Source Policy...
DEBUG   Fetching Root CA...
DEBUG   Reusing previously-fetched Root CA
DEBUG   Fetching Certificate (etcd-signer)...
DEBUG   Generating Certificate (etcd-signer)...
DEBUG   Fetching Certificate (etcd-ca-bundle)...
DEBUG     Fetching Certificate (etcd-signer)...
DEBUG     Reusing previously-fetched Certificate (etcd-signer)
DEBUG   Generating Certificate (etcd-ca-bundle)...
DEBUG   Fetching Certificate (etcd-client)...
DEBUG     Fetching Certificate (etcd-signer)...
DEBUG     Reusing previously-fetched Certificate (etcd-signer)
DEBUG   Generating Certificate (etcd-client)...
DEBUG   Fetching Certificate (etcd-metric-ca-bundle)...
DEBUG     Fetching Certificate (etcd-metric-signer)...
DEBUG     Generating Certificate (etcd-metric-signer)...
DEBUG   Generating Certificate (etcd-metric-ca-bundle)...
DEBUG   Fetching Certificate (etcd-metric-signer)...
DEBUG   Reusing previously-fetched Certificate (etcd-metric-signer)
DEBUG   Fetching Certificate (etcd-metric-signer-client)...
DEBUG     Fetching Certificate (etcd-metric-signer)...
DEBUG     Reusing previously-fetched Certificate (etcd-metric-signer)
DEBUG   Generating Certificate (etcd-metric-signer-client)...
DEBUG   Fetching Certificate (mcs)...
DEBUG     Fetching Root CA...
DEBUG     Reusing previously-fetched Root CA
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG   Generating Certificate (mcs)...
DEBUG   Fetching CVOOverrides...
DEBUG   Generating CVOOverrides...
DEBUG   Fetching EtcdCAConfigMap...
DEBUG   Generating EtcdCAConfigMap...
DEBUG   Fetching EtcdClientSecret...
DEBUG   Generating EtcdClientSecret...
DEBUG   Fetching EtcdHostServiceEndpoints...
DEBUG   Generating EtcdHostServiceEndpoints...
DEBUG   Fetching EtcdHostService...
DEBUG   Generating EtcdHostService...
DEBUG   Fetching EtcdMetricClientSecret...
DEBUG   Generating EtcdMetricClientSecret...
DEBUG   Fetching EtcdMetricServingCAConfigMap...
DEBUG   Generating EtcdMetricServingCAConfigMap...
DEBUG   Fetching EtcdMetricSignerSecret...
DEBUG   Generating EtcdMetricSignerSecret...
DEBUG   Fetching EtcdNamespace...
DEBUG   Generating EtcdNamespace...
DEBUG   Fetching EtcdService...
DEBUG   Generating EtcdService...
DEBUG   Fetching EtcdSignerSecret...
DEBUG   Generating EtcdSignerSecret...
DEBUG   Fetching KubeCloudConfig...
DEBUG   Generating KubeCloudConfig...
DEBUG   Fetching EtcdServingCAConfigMap...
DEBUG   Generating EtcdServingCAConfigMap...
DEBUG   Fetching KubeSystemConfigmapRootCA...
DEBUG   Generating KubeSystemConfigmapRootCA...
DEBUG   Fetching MachineConfigServerTLSSecret...
DEBUG   Generating MachineConfigServerTLSSecret...
DEBUG   Fetching OpenshiftConfigSecretPullSecret...
DEBUG   Generating OpenshiftConfigSecretPullSecret...
DEBUG   Fetching OpenshiftMachineConfigOperator...
DEBUG   Generating OpenshiftMachineConfigOperator...
DEBUG Generating Common Manifests...
DEBUG Fetching Openshift Manifests...
DEBUG Loading Openshift Manifests...
DEBUG   Loading Install Config...
DEBUG   Loading Cluster ID...
DEBUG   Loading Kubeadmin Password...
DEBUG   Loading OpenShift Install (Manifests)...
DEBUG   Loading CloudCredsSecret...
DEBUG   Loading KubeadminPasswordSecret...
DEBUG   Loading RoleCloudCredsSecretReader...
DEBUG   Loading Private Cluster Outbound Service...
DEBUG   Loading Baremetal Config CR...
DEBUG   Loading Community Operators setting...
DEBUG   Loading Image...
WARNING Discarding the Openshift Manifests that was provided in the target directory because its dependencies are dirty and it needs to be regenerated
DEBUG   Fetching Install Config...
DEBUG   Reusing previously-fetched Install Config
DEBUG   Fetching Cluster ID...
DEBUG   Reusing previously-fetched Cluster ID
DEBUG   Fetching Kubeadmin Password...
DEBUG   Generating Kubeadmin Password...
DEBUG   Fetching OpenShift Install (Manifests)...
DEBUG   Generating OpenShift Install (Manifests)...
DEBUG   Fetching CloudCredsSecret...
DEBUG   Generating CloudCredsSecret...
DEBUG   Fetching KubeadminPasswordSecret...
DEBUG   Generating KubeadminPasswordSecret...
DEBUG   Fetching RoleCloudCredsSecretReader...
DEBUG   Generating RoleCloudCredsSecretReader...
DEBUG   Fetching Private Cluster Outbound Service...
DEBUG   Generating Private Cluster Outbound Service...
DEBUG   Fetching Baremetal Config CR...
DEBUG   Generating Baremetal Config CR...
DEBUG   Fetching Community Operators setting...
DEBUG   Generating Community Operators setting...
DEBUG   Fetching Image...
DEBUG   Reusing previously-fetched Image
DEBUG Generating Openshift Manifests...

@vrutkovs Have you seen something like this?

Thank you!

@vrutkovs
Copy link
Member

vrutkovs commented Jul 2, 2020

WARNING Discarding the Openshift Manifests that was provided in the target directory because its dependencies are dirty and it needs to be regenerated

That's weird. Are you reusing the same directory across attempts? If yes you should remove hidden files (".openshift_*") and/or use --dir switch to create a new install directory every time

@pjbrzozowski
Copy link
Author

I am not reusing it, it is always fresh directory:

echo "Preparing build directory..."

rm -rf ${DIR}/build
mkdir ${DIR}/build

Super weird stuff, and I don't really understand why this was not causing the issue before.

Thanks Vadim!

@pjbrzozowski
Copy link
Author

Here it is with completely empty directory:

pbrzozowski@navi:Work/try1 $ ls -la
total 12
drwxrwxr-x. 2 pbrzozowski pbrzozowski 4096 Jul  2 18:13 .
drwxrwxr-x. 8 pbrzozowski pbrzozowski 4096 Jul  2 18:12 ..
-rw-rw-r--. 1 pbrzozowski pbrzozowski 1163 Jul  2 18:13 install-config.yaml
pbrzozowski@navi:Work/try1 $ openshift-install create manifests                                             
INFO Consuming Install Config from target directory 
WARNING Discarding the Openshift Manifests that was provided in the target directory because its dependencies are dirty and it needs to be regenerated 
pbrzozowski@navi:Work/try1 $ ls -la 
total 192
drwxrwxr-x. 4 pbrzozowski pbrzozowski   4096 Jul  2 18:13 .
drwxrwxr-x. 8 pbrzozowski pbrzozowski   4096 Jul  2 18:12 ..
drwxr-x---. 2 pbrzozowski pbrzozowski   4096 Jul  2 18:13 manifests
drwxr-x---. 2 pbrzozowski pbrzozowski   4096 Jul  2 18:13 openshift
-rw-rw-r--. 1 pbrzozowski pbrzozowski  24832 Jul  2 18:13 .openshift_install.log
-rw-r-----. 1 pbrzozowski pbrzozowski 148368 Jul  2 18:13 .openshift_install_state.json

@vrutkovs
Copy link
Member

vrutkovs commented Jul 3, 2020

What's the output of openshift-install create manifests --log-level debug?

@pjbrzozowski
Copy link
Author

Creating manifests...
DEBUG OpenShift Installer 4.4.0-0.okd-2020-05-23-055148-beta5
DEBUG Built from commit 0f0142e7261349b93c3dd3dd02a9ce164dfd2d4f
DEBUG Fetching Master Machines...
DEBUG Loading Master Machines...
DEBUG   Loading Cluster ID...
DEBUG     Loading Install Config...
DEBUG       Loading SSH Key...
DEBUG       Loading Base Domain...
DEBUG         Loading Platform...
DEBUG       Loading Cluster Name...
DEBUG         Loading Base Domain...
DEBUG         Loading Platform...
DEBUG       Loading Pull Secret...
DEBUG       Loading Platform...
DEBUG     Using Install Config loaded from target directory
DEBUG   Loading Platform Credentials Check...
DEBUG     Loading Install Config...
DEBUG   Loading Install Config...
DEBUG   Loading Image...
DEBUG     Loading Install Config...
DEBUG   Loading Master Ignition Config...
DEBUG     Loading Install Config...
DEBUG     Loading Root CA...
DEBUG   Fetching Cluster ID...
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG   Generating Cluster ID...
DEBUG   Fetching Platform Credentials Check...
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG   Generating Platform Credentials Check...
DEBUG   Fetching Install Config...
DEBUG   Reusing previously-fetched Install Config
DEBUG   Fetching Image...
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG   Generating Image...
DEBUG   Fetching Master Ignition Config...
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG     Fetching Root CA...
DEBUG     Generating Root CA...
DEBUG   Generating Master Ignition Config...
DEBUG Generating Master Machines...
INFO Consuming Install Config from target directory
DEBUG Purging asset "Install Config" from disk
DEBUG Fetching Worker Machines...
DEBUG Loading Worker Machines...
DEBUG   Loading Cluster ID...
DEBUG   Loading Platform Credentials Check...
DEBUG   Loading Install Config...
DEBUG   Loading Image...
DEBUG   Loading Worker Ignition Config...
DEBUG     Loading Install Config...
DEBUG     Loading Root CA...
DEBUG   Fetching Cluster ID...
DEBUG   Reusing previously-fetched Cluster ID
DEBUG   Fetching Platform Credentials Check...
DEBUG   Reusing previously-fetched Platform Credentials Check
DEBUG   Fetching Install Config...
DEBUG   Reusing previously-fetched Install Config
DEBUG   Fetching Image...
DEBUG   Reusing previously-fetched Image
DEBUG   Fetching Worker Ignition Config...
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG     Fetching Root CA...
DEBUG     Reusing previously-fetched Root CA
DEBUG   Generating Worker Ignition Config...
DEBUG Generating Worker Machines...
DEBUG Fetching Common Manifests...
DEBUG Loading Common Manifests...
DEBUG   Loading Cluster ID...
DEBUG   Loading Install Config...
DEBUG   Loading Ingress Config...
DEBUG     Loading Install Config...
DEBUG   Loading DNS Config...
DEBUG     Loading Install Config...
DEBUG     Loading Cluster ID...
DEBUG     Loading Platform Credentials Check...
DEBUG   Loading Infrastructure Config...
DEBUG     Loading Cluster ID...
DEBUG     Loading Install Config...
DEBUG     Loading Cloud Provider Config...
DEBUG       Loading Install Config...
DEBUG       Loading Cluster ID...
DEBUG       Loading Platform Credentials Check...
DEBUG     Loading Additional Trust Bundle Config...
DEBUG       Loading Install Config...
DEBUG   Loading Network Config...
DEBUG     Loading Install Config...
DEBUG     Loading Network CRDs...
DEBUG   Loading Proxy Config...
DEBUG     Loading Install Config...
DEBUG     Loading Network Config...
DEBUG   Loading Scheduler Config...
DEBUG     Loading Install Config...
DEBUG   Loading Image Content Source Policy...
DEBUG     Loading Install Config...
DEBUG   Loading Root CA...
DEBUG   Loading Certificate (etcd-signer)...
DEBUG   Loading Certificate (etcd-ca-bundle)...
DEBUG     Loading Certificate (etcd-signer)...
DEBUG   Loading Certificate (etcd-client)...
DEBUG     Loading Certificate (etcd-signer)...
DEBUG   Loading Certificate (etcd-metric-ca-bundle)...
DEBUG     Loading Certificate (etcd-metric-signer)...
DEBUG   Loading Certificate (etcd-metric-signer)...
DEBUG   Loading Certificate (etcd-metric-signer-client)...
DEBUG     Loading Certificate (etcd-metric-signer)...
DEBUG   Loading Certificate (mcs)...
DEBUG     Loading Root CA...
DEBUG     Loading Install Config...
DEBUG   Loading CVOOverrides...
DEBUG   Loading EtcdCAConfigMap...
DEBUG   Loading EtcdClientSecret...
DEBUG   Loading EtcdHostServiceEndpoints...
DEBUG   Loading EtcdHostService...
DEBUG   Loading EtcdMetricClientSecret...
DEBUG   Loading EtcdMetricServingCAConfigMap...
DEBUG   Loading EtcdMetricSignerSecret...
DEBUG   Loading EtcdNamespace...
DEBUG   Loading EtcdService...
DEBUG   Loading EtcdSignerSecret...
DEBUG   Loading KubeCloudConfig...
DEBUG   Loading EtcdServingCAConfigMap...
DEBUG   Loading KubeSystemConfigmapRootCA...
DEBUG   Loading MachineConfigServerTLSSecret...
DEBUG   Loading OpenshiftConfigSecretPullSecret...
DEBUG   Loading OpenshiftMachineConfigOperator...
DEBUG   Fetching Cluster ID...
DEBUG   Reusing previously-fetched Cluster ID
DEBUG   Fetching Install Config...
DEBUG   Reusing previously-fetched Install Config
DEBUG   Fetching Ingress Config...
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG   Generating Ingress Config...
DEBUG   Fetching DNS Config...
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG     Fetching Cluster ID...
DEBUG     Reusing previously-fetched Cluster ID
DEBUG     Fetching Platform Credentials Check...
DEBUG     Reusing previously-fetched Platform Credentials Check
DEBUG   Generating DNS Config...
DEBUG   Fetching Infrastructure Config...
DEBUG     Fetching Cluster ID...
DEBUG     Reusing previously-fetched Cluster ID
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG     Fetching Cloud Provider Config...
DEBUG       Fetching Install Config...
DEBUG       Reusing previously-fetched Install Config
DEBUG       Fetching Cluster ID...
DEBUG       Reusing previously-fetched Cluster ID
DEBUG       Fetching Platform Credentials Check...
DEBUG       Reusing previously-fetched Platform Credentials Check
DEBUG     Generating Cloud Provider Config...
DEBUG     Fetching Additional Trust Bundle Config...
DEBUG       Fetching Install Config...
DEBUG       Reusing previously-fetched Install Config
DEBUG     Generating Additional Trust Bundle Config...
DEBUG   Generating Infrastructure Config...
DEBUG   Fetching Network Config...
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG     Fetching Network CRDs...
DEBUG     Generating Network CRDs...
DEBUG   Generating Network Config...
DEBUG   Fetching Proxy Config...
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG     Fetching Network Config...
DEBUG     Reusing previously-fetched Network Config
DEBUG   Generating Proxy Config...
DEBUG   Fetching Scheduler Config...
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG   Generating Scheduler Config...
DEBUG   Fetching Image Content Source Policy...
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG   Generating Image Content Source Policy...
DEBUG   Fetching Root CA...
DEBUG   Reusing previously-fetched Root CA
DEBUG   Fetching Certificate (etcd-signer)...
DEBUG   Generating Certificate (etcd-signer)...
DEBUG   Fetching Certificate (etcd-ca-bundle)...
DEBUG     Fetching Certificate (etcd-signer)...
DEBUG     Reusing previously-fetched Certificate (etcd-signer)
DEBUG   Generating Certificate (etcd-ca-bundle)...
DEBUG   Fetching Certificate (etcd-client)...
DEBUG     Fetching Certificate (etcd-signer)...
DEBUG     Reusing previously-fetched Certificate (etcd-signer)
DEBUG   Generating Certificate (etcd-client)...
DEBUG   Fetching Certificate (etcd-metric-ca-bundle)...
DEBUG     Fetching Certificate (etcd-metric-signer)...
DEBUG     Generating Certificate (etcd-metric-signer)...
DEBUG   Generating Certificate (etcd-metric-ca-bundle)...
DEBUG   Fetching Certificate (etcd-metric-signer)...
DEBUG   Reusing previously-fetched Certificate (etcd-metric-signer)
DEBUG   Fetching Certificate (etcd-metric-signer-client)...
DEBUG     Fetching Certificate (etcd-metric-signer)...
DEBUG     Reusing previously-fetched Certificate (etcd-metric-signer)
DEBUG   Generating Certificate (etcd-metric-signer-client)...
DEBUG   Fetching Certificate (mcs)...
DEBUG     Fetching Root CA...
DEBUG     Reusing previously-fetched Root CA
DEBUG     Fetching Install Config...
DEBUG     Reusing previously-fetched Install Config
DEBUG   Generating Certificate (mcs)...
DEBUG   Fetching CVOOverrides...
DEBUG   Generating CVOOverrides...
DEBUG   Fetching EtcdCAConfigMap...
DEBUG   Generating EtcdCAConfigMap...
DEBUG   Fetching EtcdClientSecret...
DEBUG   Generating EtcdClientSecret...
DEBUG   Fetching EtcdHostServiceEndpoints...
DEBUG   Generating EtcdHostServiceEndpoints...
DEBUG   Fetching EtcdHostService...
DEBUG   Generating EtcdHostService...
DEBUG   Fetching EtcdMetricClientSecret...
DEBUG   Generating EtcdMetricClientSecret...
DEBUG   Fetching EtcdMetricServingCAConfigMap...
DEBUG   Generating EtcdMetricServingCAConfigMap...
DEBUG   Fetching EtcdMetricSignerSecret...
DEBUG   Generating EtcdMetricSignerSecret...
DEBUG   Fetching EtcdNamespace...
DEBUG   Generating EtcdNamespace...
DEBUG   Fetching EtcdService...
DEBUG   Generating EtcdService...
DEBUG   Fetching EtcdSignerSecret...
DEBUG   Generating EtcdSignerSecret...
DEBUG   Fetching KubeCloudConfig...
DEBUG   Generating KubeCloudConfig...
DEBUG   Fetching EtcdServingCAConfigMap...
DEBUG   Generating EtcdServingCAConfigMap...
DEBUG   Fetching KubeSystemConfigmapRootCA...
DEBUG   Generating KubeSystemConfigmapRootCA...
DEBUG   Fetching MachineConfigServerTLSSecret...
DEBUG   Generating MachineConfigServerTLSSecret...
DEBUG   Fetching OpenshiftConfigSecretPullSecret...
DEBUG   Generating OpenshiftConfigSecretPullSecret...
DEBUG   Fetching OpenshiftMachineConfigOperator...
DEBUG   Generating OpenshiftMachineConfigOperator...
DEBUG Generating Common Manifests...
DEBUG Fetching Openshift Manifests...
DEBUG Loading Openshift Manifests...
DEBUG   Loading Install Config...
DEBUG   Loading Cluster ID...
DEBUG   Loading Kubeadmin Password...
DEBUG   Loading OpenShift Install (Manifests)...
DEBUG   Loading CloudCredsSecret...
DEBUG   Loading KubeadminPasswordSecret...
DEBUG   Loading RoleCloudCredsSecretReader...
DEBUG   Loading Private Cluster Outbound Service...
DEBUG   Loading Baremetal Config CR...
DEBUG   Loading Community Operators setting...
DEBUG   Loading Image...
WARNING Discarding the Openshift Manifests that was provided in the target directory because its dependencies are dirty and it needs to be regenerated
DEBUG   Fetching Install Config...
DEBUG   Reusing previously-fetched Install Config
DEBUG   Fetching Cluster ID...
DEBUG   Reusing previously-fetched Cluster ID
DEBUG   Fetching Kubeadmin Password...
DEBUG   Generating Kubeadmin Password...
DEBUG   Fetching OpenShift Install (Manifests)...
DEBUG   Generating OpenShift Install (Manifests)...
DEBUG   Fetching CloudCredsSecret...
DEBUG   Generating CloudCredsSecret...
DEBUG   Fetching KubeadminPasswordSecret...
DEBUG   Generating KubeadminPasswordSecret...
DEBUG   Fetching RoleCloudCredsSecretReader...
DEBUG   Generating RoleCloudCredsSecretReader...
DEBUG   Fetching Private Cluster Outbound Service...
DEBUG   Generating Private Cluster Outbound Service...
DEBUG   Fetching Baremetal Config CR...
DEBUG   Generating Baremetal Config CR...
DEBUG   Fetching Community Operators setting...
DEBUG   Generating Community Operators setting...
DEBUG   Fetching Image...
DEBUG   Reusing previously-fetched Image
DEBUG Generating Openshift Manifests...

@vrutkovs
Copy link
Member

vrutkovs commented Jul 3, 2020

hmm, that's very odd. WARNING Discarding the Openshift Manifests that was provided in the target directory because its dependencies are dirty and it needs to be regenerated should not have happened.

Could you also attach the contents of install-config.yaml?

@pjbrzozowski
Copy link
Author

You are right, this is super odd.

This is my install-config.yaml file:

apiVersion: v1
baseDomain: ontux.net
compute:
- hyperthreading: Enabled
  name: worker
  replicas: 3
controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 1
metadata:
  name: ocp
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
platform:
  none: {}
fips: false
pullSecret: '{"auths":{"fake":{"auth": "bar"}}}' 
sshKey: 'ssh-rsa AAAAB3NzaC1yc....'

@pjbrzozowski
Copy link
Author

pjbrzozowski commented Jul 3, 2020

It works fine if I use openshift-installer from OpenShift:

pbrzozowski@ibm:Work/install-test $ openshift-install version                               
openshift-install 4.4.9
built from commit 1541bf917973186bbab6a5f895f08db4334a5d9a
release image quay.io/openshift-release-dev/ocp-release@sha256:15280aba8f1c82fe39180a617e3b4886401f6c2aef63c7962203aa10530d1db8
pbrzozowski@ibm:Work/install-test $ ls -la
total 12
drwxrwxr-x.  2 pbrzozowski pbrzozowski 4096 Jul  3 12:31 .
drwxrwxr-x. 22 pbrzozowski pbrzozowski 4096 Jul  3 12:23 ..
-rw-r--r--.  1 pbrzozowski pbrzozowski 2248 Jul  3 12:31 install-config.yaml
pbrzozowski@ibm:Work/install-test $ openshift-install create manifests                      
INFO Consuming Install Config from target directory 
pbrzozowski@ibm:Work/install-test $ ls -la
total 188
drwxrwxr-x.  4 pbrzozowski pbrzozowski   4096 Jul  3 12:32 .
drwxrwxr-x. 22 pbrzozowski pbrzozowski   4096 Jul  3 12:23 ..
drwxr-x---.  2 pbrzozowski pbrzozowski   4096 Jul  3 12:32 manifests
drwxr-x---.  2 pbrzozowski pbrzozowski   4096 Jul  3 12:32 openshift
-rw-rw-r--.  1 pbrzozowski pbrzozowski  24326 Jul  3 12:32 .openshift_install.log
-rw-r-----.  1 pbrzozowski pbrzozowski 150708 Jul  3 12:32 .openshift_install_state.json

I have also tried on the same machine, to use OKD openshift-installer:

pbrzozowski@ibm:Work/install-test $ ls -la
total 12
drwxrwxr-x.  2 pbrzozowski pbrzozowski 4096 Jul  3 12:27 .
drwxrwxr-x. 22 pbrzozowski pbrzozowski 4096 Jul  3 12:23 ..
-rw-r--r--.  1 pbrzozowski pbrzozowski 2248 Jul  3 12:27 install-config.yaml
pbrzozowski@ibm:Work/install-test $ openshift-install create manifests 
INFO Consuming Install Config from target directory 
WARNING Discarding the Openshift Manifests that was provided in the target directory because its dependencies are dirty and it needs to be regenerated 
pbrzozowski@ibm:Work/install-test $ ls -la
total 196
drwxrwxr-x.  4 pbrzozowski pbrzozowski   4096 Jul  3 12:27 .
drwxrwxr-x. 22 pbrzozowski pbrzozowski   4096 Jul  3 12:23 ..
drwxr-x---.  2 pbrzozowski pbrzozowski   4096 Jul  3 12:27 manifests
drwxr-x---.  2 pbrzozowski pbrzozowski   4096 Jul  3 12:27 openshift
-rw-rw-r--.  1 pbrzozowski pbrzozowski  24826 Jul  3 12:27 .openshift_install.log
-rw-r-----.  1 pbrzozowski pbrzozowski 155528 Jul  3 12:27 .openshift_install_state.json
pbrzozowski@ibm:Work/install-test $ openshift-install version
openshift-install 4.5.0-0.okd-2020-07-02-103539
built from commit 7d88671b2f20cc71c55fc9d5fe71818cdd029bd0
release image registry.svc.ci.openshift.org/origin/release@sha256:a8632750477f4ebd396cdacf916c64351307c4a587935d9d1039bb22cbf839b9

And it has the warning, which creates "unbootstrapable" ignition files.

@vrutkovs vrutkovs removed the triage/needs-information Indicates an issue needs more information in order to work on it. label Jul 3, 2020
@vrutkovs
Copy link
Member

vrutkovs commented Jul 3, 2020

Okay, must be some OKD-specific change broke that. Not quite sure which change exactly

@cgruver
Copy link

cgruver commented Jul 3, 2020

@pjbrzozowski

I noticed that you are creating a cluster with one master and 3 workers.

It would be interesting to see if you still see the warning in the following two scenarios:

  1. 1 master & 0 workers
  2. 3 masters and 3 workers

What happens if you run openshift-install create manifests with those two changes?

If you don't get the warning, that might help narrow the search.

@pjbrzozowski
Copy link
Author

Hello!

Case 1:

pbrzozowski@navi:Work/okd-versions $ openshift-install create manifests
INFO Consuming Install Config from target directory 
WARNING Making control-plane schedulable by setting MastersSchedulable to true for Scheduler cluster settings 
WARNING Discarding the Openshift Manifests that was provided in the target directory because its dependencies are dirty and it needs to be regenerated

Case 2:

pbrzozowski@navi:Work/okd-versions $ openshift-install create manifests                                   
INFO Consuming Install Config from target directory 

And it actually did narrow the search :)

Thanks!

@vrutkovs
Copy link
Member

vrutkovs commented Jul 3, 2020

Right, I see, so its single master install code breaking things

@vrutkovs vrutkovs changed the title OKD 4.4 beta5 installer - lb error after the cluster has bootstrapped 1 master & 0 workers setup requires "create manifest" step twice Jul 3, 2020
@vrutkovs vrutkovs added this to the GA milestone Jul 3, 2020
@kai-uwe-rommel
Copy link

I'm seeing exactly the same behaviour.

@kai-uwe-rommel

This comment has been minimized.

@vrutkovs vrutkovs pinned this issue Jul 13, 2020
@vrutkovs
Copy link
Member

This is a pretty important issue, but since it has a workaround we won't block GA on it

@openshift-bot
Copy link

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

@openshift-ci-robot openshift-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 27, 2020
@kai-uwe-rommel
Copy link

Any plans to fix that?

@kai-uwe-rommel
Copy link

/remove-lifecycle stale

@openshift-ci-robot openshift-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 27, 2020
@vrutkovs vrutkovs unpinned this issue Dec 1, 2020
@openshift-bot
Copy link

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

@openshift-ci-robot openshift-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 25, 2021
@openshift-bot
Copy link

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

@openshift-ci-robot openshift-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Feb 24, 2021
@openshift-bot
Copy link

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

@openshift-ci-robot
Copy link

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

binnes added a commit to binnes/okd that referenced this issue Jul 11, 2023
…fter switch to MkDocs site (okd-project#238)

* move from master to main branch

* remove blog entry re creating blog on old site

* Add site README

* add CNAME in docs so gets published

* Delete CNAME

* fix CNAME for production site

Signed-off-by: Brian Innes <[email protected]>

* fixed links re MkDocs going live + container tooling instructions

Signed-off-by: Brian Innes <[email protected]>

* added Windows support for docker based tooling

Signed-off-by: Brian Innes <[email protected]>

* added podman instructions

Signed-off-by: Brian Innes <[email protected]>

* tidy up commands

Signed-off-by: Brian Innes <[email protected]>

Co-authored-by: Brian Innes <[email protected]>
binnes added a commit to binnes/okd that referenced this issue Jul 11, 2023
…fter switch to MkDocs site (okd-project#238) (okd-project#2)

* move from master to main branch

* remove blog entry re creating blog on old site

* Add site README

* add CNAME in docs so gets published

* Delete CNAME

* fix CNAME for production site

Signed-off-by: Brian Innes <[email protected]>

* fixed links re MkDocs going live + container tooling instructions

Signed-off-by: Brian Innes <[email protected]>

* added Windows support for docker based tooling

Signed-off-by: Brian Innes <[email protected]>

* added podman instructions

Signed-off-by: Brian Innes <[email protected]>

* tidy up commands

Signed-off-by: Brian Innes <[email protected]>

Co-authored-by: Brian Innes <[email protected]>

Co-authored-by: Brian Innes <[email protected]>
binnes added a commit to binnes/okd that referenced this issue Jul 11, 2023
…fter switch to MkDocs site (okd-project#238) (okd-project#2) (okd-project#239)

* move from master to main branch

* remove blog entry re creating blog on old site

* Add site README

* add CNAME in docs so gets published

* Delete CNAME

* fix CNAME for production site

Signed-off-by: Brian Innes <[email protected]>

* fixed links re MkDocs going live + container tooling instructions

Signed-off-by: Brian Innes <[email protected]>

* added Windows support for docker based tooling

Signed-off-by: Brian Innes <[email protected]>

* added podman instructions

Signed-off-by: Brian Innes <[email protected]>

* tidy up commands

Signed-off-by: Brian Innes <[email protected]>

Co-authored-by: Brian Innes <[email protected]>

Co-authored-by: Brian Innes <[email protected]>

Co-authored-by: Brian Innes <[email protected]>
binnes added a commit to binnes/okd that referenced this issue Jul 11, 2023
* Add docker/podman instructions for site update and fix broken links after switch to MkDocs site (okd-project#238) (okd-project#2)

* move from master to main branch

* remove blog entry re creating blog on old site

* Add site README

* add CNAME in docs so gets published

* Delete CNAME

* fix CNAME for production site

Signed-off-by: Brian Innes <[email protected]>

* fixed links re MkDocs going live + container tooling instructions

Signed-off-by: Brian Innes <[email protected]>

* added Windows support for docker based tooling

Signed-off-by: Brian Innes <[email protected]>

* added podman instructions

Signed-off-by: Brian Innes <[email protected]>

* tidy up commands

Signed-off-by: Brian Innes <[email protected]>

Co-authored-by: Brian Innes <[email protected]>

Co-authored-by: Brian Innes <[email protected]>

* added primary working group information

Signed-off-by: Brian Innes <[email protected]>

Co-authored-by: Brian Innes <[email protected]>
binnes added a commit to binnes/okd that referenced this issue Jul 11, 2023
* Add docker/podman instructions for site update and fix broken links after switch to MkDocs site (okd-project#238) (okd-project#2)

* move from master to main branch

* remove blog entry re creating blog on old site

* Add site README

* add CNAME in docs so gets published

* Delete CNAME

* fix CNAME for production site

Signed-off-by: Brian Innes <[email protected]>

* fixed links re MkDocs going live + container tooling instructions

Signed-off-by: Brian Innes <[email protected]>

* added Windows support for docker based tooling

Signed-off-by: Brian Innes <[email protected]>

* added podman instructions

Signed-off-by: Brian Innes <[email protected]>

* tidy up commands

Signed-off-by: Brian Innes <[email protected]>

Co-authored-by: Brian Innes <[email protected]>

Co-authored-by: Brian Innes <[email protected]>

* added primary working group information

Signed-off-by: Brian Innes <[email protected]>

* add missing links

Signed-off-by: Brian Innes <[email protected]>

* fix branch name

Signed-off-by: Brian Innes <[email protected]>

* convert pdf links to relative links

Signed-off-by: Brian Innes <[email protected]>

* update working group overview

Signed-off-by: Brian Innes <[email protected]>

Co-authored-by: Brian Innes <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.
Projects
None yet
Development

No branches or pull requests

6 participants