[Bug] Requirements for deployment couldn't be found

[DavidMachacek]

I cannot deploy Strimzi operator on IBM Cloud Kubernetes service.

Steps to reproduce:
kubectl create -f https://operatorhub.io/install/strimzi-kafka-operator.yaml

Expected behavior
CSV and istallplan results in operator being deployed

Environment

• Strimzi version: strimzi-cluster-operator.v0.20.0
• Installation method: OperatorHub.io
• Kubernetes cluster: v1.18.10+IKS
• Infrastructure: IBM Cloud

Actual behavior
CSV get stuck on pending with status "one or more requirements couldn't be found" and nothing deployed. InstallPlan is Approved.

YAML files and logs
CSV status from description
Status:
Conditions:
Last Transition Time: 2020-11-06T21:33:29Z
Last Update Time: 2020-11-06T21:33:29Z
Message: requirements not yet checked
Phase: Pending
Reason: RequirementsUnknown
Last Transition Time: 2020-11-06T21:33:29Z
Last Update Time: 2020-11-06T21:33:29Z
Message: one or more requirements couldn't be found
Phase: Pending
Reason: RequirementsNotMet
Last Transition Time: 2020-11-06T21:33:29Z
Last Update Time: 2020-11-06T21:33:29Z
Message: one or more requirements couldn't be found
Phase: Pending
Reason: RequirementsNotMet
Requirement Status:
Group: operators.coreos.com
Kind: ClusterServiceVersion
Message: CSV minKubeVersion (1.11.0) less than server version (v1.18.10+IKS)
Name: strimzi-cluster-operator.v0.20.0
Status: Present
Version: v1alpha1
Group: apiextensions.k8s.io
Kind: CustomResourceDefinition
Message: CRD is present and Established condition is true
Name: kafkabridges.kafka.strimzi.io
Status: Present
Uuid: bd04d4d0-1ddc-4b54-9cbf-0269019ecb43
Version: v1beta1
Group: apiextensions.k8s.io
Kind: CustomResourceDefinition
Message: CRD is present and Established condition is true
Name: kafkaconnectors.kafka.strimzi.io
Status: Present
Uuid: a239ccf1-6aff-4cd5-b3b1-987b5dfd2625
Version: v1beta1
Group: apiextensions.k8s.io
Kind: CustomResourceDefinition
Message: CRD is present and Established condition is true
Name: kafkaconnects.kafka.strimzi.io
Status: Present
Uuid: 59e2e5ae-9777-4bd3-af72-f612835d28f4
Version: v1beta1
Group: apiextensions.k8s.io
Kind: CustomResourceDefinition
Message: CRD is present and Established condition is true
Name: kafkaconnects2is.kafka.strimzi.io
Status: Present
Uuid: 8a831e0a-c14e-454a-a8c2-643f7a3e1fc4
Version: v1beta1
Group: apiextensions.k8s.io
Kind: CustomResourceDefinition
Message: CRD is present and Established condition is true
Name: kafkamirrormaker2s.kafka.strimzi.io
Status: Present
Uuid: eeb3ad84-a6fb-4efa-ba56-445776e90354
Version: v1beta1
Group: apiextensions.k8s.io
Kind: CustomResourceDefinition
Message: CRD is present and Established condition is true
Name: kafkamirrormakers.kafka.strimzi.io
Status: Present
Uuid: 2996aefa-f994-41ea-b77c-4b83422df776
Version: v1beta1
Group: apiextensions.k8s.io
Kind: CustomResourceDefinition
Message: CRD is present and Established condition is true
Name: kafkarebalances.kafka.strimzi.io
Status: Present
Uuid: 4d1af7f6-74c3-40ad-a991-d9eec2c43e10
Version: v1beta1
Group: apiextensions.k8s.io
Kind: CustomResourceDefinition
Message: CRD is present and Established condition is true
Name: kafkas.kafka.strimzi.io
Status: Present
Uuid: bc62eb3c-8f94-41c0-b668-ec7f37e9a32b
Version: v1beta1
Group: apiextensions.k8s.io
Kind: CustomResourceDefinition
Message: CRD is present and Established condition is true
Name: kafkatopics.kafka.strimzi.io
Status: Present
Uuid: dcbf4f17-2cf3-4d83-a6cf-4d11d0b93eee
Version: v1beta1
Group: apiextensions.k8s.io
Kind: CustomResourceDefinition
Message: CRD is present and Established condition is true
Name: kafkausers.kafka.strimzi.io
Status: Present
Uuid: 3baf622e-af83-4500-8164-973630b0d70d
Version: v1beta1
Group:
Kind: ServiceAccount
Message: Service account does not exist
Name: strimzi-cluster-operator
Status: NotPresent
Version: v1
Events:
Type Reason Age From Message

---

Normal RequirementsUnknown 9m8s operator-lifecycle-manager requirements not yet checked
Normal RequirementsNotMet 9m7s operator-lifecycle-manager one or more requirements couldn't be found

catalog operator logs
time="2020-11-06T21:51:13Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=ibm-system/c1621dc504650f3e13082e654b1a70108da25de4e93768d2fcd9646977d2223 key=kafkaconnects.kafka.strimzi.io.crd.yaml
time="2020-11-06T21:51:13Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/operators/subscriptions/strimzi-kafka-operator
time="2020-11-06T21:51:13Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=ibm-system/c1621dc504650f3e13082e654b1a70108da25de4e93768d2fcd9646977d2223 key=kafkas.kafka.strimzi.io.crd.yaml
time="2020-11-06T21:51:13Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=ibm-system/c1621dc504650f3e13082e654b1a70108da25de4e93768d2fcd9646977d2223 key=kafkatopics.kafka.strimzi.io.crd.yaml
time="2020-11-06T21:51:13Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=ibm-system/c1621dc504650f3e13082e654b1a70108da25de4e93768d2fcd9646977d2223 key=kafkausers.kafka.strimzi.io.crd.yaml
time="2020-11-06T21:51:13Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=ibm-system/c1621dc504650f3e13082e654b1a70108da25de4e93768d2fcd9646977d2223 key=kafkabridges.kafka.strimzi.io.crd.yaml
time="2020-11-06T21:51:13Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=ibm-system/c1621dc504650f3e13082e654b1a70108da25de4e93768d2fcd9646977d2223 key=kafkaconnectors.kafka.strimzi.io.yaml
time="2020-11-06T21:51:13Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=ibm-system/c1621dc504650f3e13082e654b1a70108da25de4e93768d2fcd9646977d2223 key=kafkaconnects2is.kafka.strimzi.io.crd.yaml
time="2020-11-06T21:51:13Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=ibm-system/c1621dc504650f3e13082e654b1a70108da25de4e93768d2fcd9646977d2223 key=kafkamirrormaker2s.kafka.strimzi.io.crd.yaml
time="2020-11-06T21:51:13Z" level=warning msg="status not equal, updating..." id=J7Awc ip=install-fwbss namespace=operators phase=Installing
time="2020-11-06T21:51:16Z" level=info msg="error updating subscription status" channel=stable error="Operation cannot be fulfilled on subscriptions.operators.coreos.com "strimzi-kafka-operator": the object has been modified; please apply your changes to the latest version and try again" id=QVp2H namespace=operators pkg=strimzi-kafka-operator source=operatorhubio-catalog sub=strimzi-kafka-operator
E1106 21:51:16.894615 1 queueinformer_operator.go:282] sync "operators" failed: error updating Subscription status: Operation cannot be fulfilled on subscriptions.operators.coreos.com "strimzi-kafka-operator": the object has been modified; please apply your changes to the latest version and try again
time="2020-11-06T21:51:17Z" level=info msg="error transitioning InstallPlan" id=RI/d6 ip=install-fwbss namespace=operators phase=Installing syncError="the InstallPlan contains invalid data" updateError="Operation cannot be fulfilled on installplans.operators.coreos.com "install-fwbss": the object has been modified; please apply your changes to the latest version and try again"
E1106 21:51:17.552303 1 queueinformer_operator.go:282] sync {"update" "operators/install-fwbss"} failed: error transitioning InstallPlan: the InstallPlan contains invalid data and error updating InstallPlan status: error updating InstallPlan status: Operation cannot be fulfilled on installplans.operators.coreos.com "install-fwbss": the object has been modified; please apply your changes to the latest version and try again
time="2020-11-06T21:51:17Z" level=info msg=syncing id=Z6+Ee ip=install-fwbss namespace=operators phase=Failed

[scholzj]

That looks the same as what is described in #3905. But I didn't managed to reproduce it in any of my environments. And I do not have any IBM Cloud access to try it there. I would suggest to look into the OLM logs, since the service account would need to be created by OLM. Maybe the logs contain some errors why it failed.

[DavidMachacek]

I added some log lines from catalog operator logs

[scholzj]

I do not think the errors

time="2020-11-06T22:35:27Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"my-strimzi-kafka-operator\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/operators/subscriptions/my-strimzi-kafka-operator
E1106 22:35:27.788188       1 queueinformer_operator.go:290] sync {"update" "operators/my-strimzi-kafka-operator"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "my-strimzi-kafka-operator": the object has been modified; please apply your changes to the latest version and try again

is related. I can see it as well during my installation which succeeds just fine. Can you check if the service account and the related (cluster)rolebindings are created or not?

[DavidMachacek]

I did check, and indeed they not created. Commands like "k get sa/role/rolebinding -n operators | grep strimzi" dont return anything.
How can I force operator to stay with 0.19 version and to use the 0.20?

[scholzj]

Well, at least we know that they were really not created. But I have no idea why would OLM not create them. There were no changes to this between 0.19 and 0.20.

You can choose the channel for 0.19. You could also install it from YAML files.

[DavidMachacek]

everything suddenly starts working with channel strimzi-0.19.x