Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ScyllaDB installation on openshift #713

Closed
gautam-borkar opened this issue Jul 21, 2021 · 4 comments · May be fixed by #1955
Closed

ScyllaDB installation on openshift #713

gautam-borkar opened this issue Jul 21, 2021 · 4 comments · May be fixed by #1955
Assignees
@tnozicka
Labels
kind/feature Categorizes issue or PR as related to a new feature. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.

Comments

@gautam-borkar
Copy link

gautam-borkar commented Jul 21, 2021 •
edited
Loading

Describe the bug
Trying to install scylladb cluster on Openshift using help. However the installation is failing with the error message in the description.

To Reproduce
Steps to reproduce the behavior:

  1. helm repo add scylla https://scylla-operator-charts.storage.googleapis.com/stable
  2. helm repo update
  3. kubectl apply -f examples/common/cert-manager.yaml
  4. kubectl wait -n cert-manager --for=condition=ready pod -l app=cert-manager --timeout=60s
  5. helm install scylla-operator scylla/scylla-operator --values examples/helm/openshit/values.operator.yaml --create-namespace --namespace scylla-operator
  6. kubectl wait -n scylla-operator --for=condition=ready pod -l app.kubernetes.io/name=scylla-operator --timeout=240s
  7. helm install scylla scylla/scylla --values examples/helm/openshift/values.cluster.yaml --create-namespace --namespace scylla

Expected behavior
ScyllaDB successfully installed on Openshift.

Logs
Operator logs :-

{"L":"ERROR","T":"2021-07-21T15:55:29.960Z","N":"cluster-controller","M":"An error occurred during cluster 
reconciliation","cluster":"scylla/scylla-incident-mgmt","resourceVersion":"1062782","error":"failed to sync headless service: 
error syncing headless service scylla-incident-mgmt-client: services \"scylla-incident-mgmt-client\" is forbidden: cannot set 
blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>","_trace_id":"_3epK41ZSS-
1WYf-RF_mjQ","errorStack":"github.com/scylladb/scylla-operator/pkg/controllers/cluster.
(*ClusterReconciler).sync\n\tgithub.com/scylladb/scylla-
operator/pkg/controllers/cluster/sync.go:55\ngithub.com/scylladb/scylla-operator/pkg/controllers/cluster.
(*ClusterReconciler).Reconcile\n\tgithub.com/scylladb/scylla-
operator/pkg/controllers/cluster/cluster_controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.
(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-
[email protected]/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.
(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-
[email protected]/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.
(*Controller).Start.func1.2\n\tsigs.k8s.io/controller-
[email protected]/pkg/internal/controller/controller.go:216\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\tk8
s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\tk8s.io/apimachin
[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\tk8s.io/[email protected]/pkg/util
/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tk8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.
io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\tk8s.io/[email protected]/pkg/util/wait/wait.go:185\nk8s.io/apimac
hinery/pkg/util/wait.UntilWithContext\n\tk8s.io/[email protected]/pkg/util/wait/wait.go:99\nruntime.goexit\n\truntime/asm
_amd64.s:1371\n"}

Environment:

  • Platform: Openshift
  • Kubernetes version:
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:59:11Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0+2817867", GitCommit:"2817867655bb7b68215b4e77873a8facf82bee06", GitTreeState:"clean", BuildDate:"2021-06-02T22:14:22Z", GoVersion:"go1.15.7", Compiler:"gc", Platform:"linux/amd64"}
  • Scylla version: 4.4.3
  • Openshift version: 4.7.16

Additional context
I have added privilege access to serviceaccount and user following yam file for role bindings roles.txt
Note :- Please rename the file with .yaml extension
@gautam-borkar gautam-borkar added the kind/bug Categorizes issue or PR as related to a bug. label Jul 21, 2021
@tnozicka
Copy link
Member

we don't support OpenShift yet :( although except for tuning permissions it should work
this looks like a permissions issue, it needs to be able to set finalizers

failed to sync headless service: 
error syncing headless service scylla-incident-mgmt-client: services \"scylla-incident-mgmt-client\" is forbidden: cannot set 
blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on

@tnozicka tnozicka added this to the v1.8 milestone Dec 2, 2021
@Choraden
Copy link
Contributor

Choraden commented Nov 8, 2022

In order to run Scylla on Openshift I did:

  • allowed clusterrole to create/update/delete finalizers of the following resources
    (file: helm/scylla-operator/templates/clusterrole_def.yaml)
    • peristentvolumes
    • secrets
    • services
    • statefulsets
    • scyllaclusters
    • configmaps
    • poddisruptionbudgets
    • daemonsets
    • nodeconfigs
    • serviceaccounts
    • jobs
  • allowed clusterrole to create/update/delete finalizer of configmaps
    (file: helm/scylla-operator/templates/scyllacluster_member_clusterrole_def.yaml)
  • created security context constraint
allowedCapabilities:
  - SYS_NICE
allowPrivilegeEscalation: true
allowPrivilegedContainer: true
allowHostDirVolumePlugin: true
allowHostIPC: true
allowHostPID: true
apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: scylla-operator-scc
allowHostNetwork: true
allowHostPorts: true
readOnlyRootFilesystem: false
runAsUser:
  type: RunAsAny
seLinuxContext:
  type: RunAsAny
fsGroup:
  type: RunAsAny
users:
  - system:serviceaccount:scylla-operator:scylla-operator
  - system:serviceaccount:scylla-operator:webhook-server
  - system:serviceaccount:scylla:simple-cluster-member
  - system:serviceaccount:scylla-operator-node-tuning:scylla-node-config
  • Changed NodeConfig Clusterrole to give permission to create daemonset/finalizers (file: pkg/controller/nodeconfig/resource.go)
{
        APIGroups: []string{"apps"},
	Resources: []string{"daemonsets", "daemonsets/finalizers"},
	Verbs:     []string{"create", "delete", "get", "list", "patch", "update", "watch"},
}
  • Added ServiceAccount for the jobs (file: pkg/controller/nodeconfigdaemon/resource.go)
func makePerftuneJobForNode(...) {
... 
      Spec: corev1.PodSpec{
            ServiceAccountName: naming.NodeConfigAppName,
            ...
      }
}
func makePerftuneJobForContainers(...) {
... 
      Spec: corev1.PodSpec{
            ServiceAccountName: naming.NodeConfigAppName,
            ...
      }
}

@mykaul
Copy link
Contributor

mykaul commented Nov 13, 2022

Nice work! Currently we don't aim to support OCP/OKD. We'll focus on EKS first.

@mykaul mykaul added kind/feature Categorizes issue or PR as related to a new feature. and removed kind/bug Categorizes issue or PR as related to a bug. labels Dec 27, 2022
@zimnx zimnx modified the milestones: v1.8, v1.9 Jan 27, 2023
@mykaul mykaul mentioned this issue Mar 14, 2023
Open
@tnozicka tnozicka removed this from the v1.9 milestone Aug 16, 2023
@tnozicka tnozicka mentioned this issue Feb 20, 2024
Open
@tnozicka tnozicka added the priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. label May 3, 2024
@tnozicka tnozicka self-assigned this May 24, 2024
@tnozicka tnozicka added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. labels May 24, 2024
@tnozicka tnozicka mentioned this issue Jun 5, 2024
Open
@tnozicka
Copy link
Member

tracked in #424

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tnozicka tnozicka

Labels
kind/feature Categorizes issue or PR as related to a new feature. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[WIP] Fix OpenShift [Part 1] tnozicka/scylla-operator
5 participants
@zimnx @mykaul @tnozicka @gautam-borkar @Choraden