Null pointer exception when trying to create a Cassandra Cluster locally #610
Labels
bug
Something isn't working
Comments
5 tasks
|
If you run the operator locally, you need to define where the configFile is (there's a cmdline parameter --config for that). |
|
I am running into the same issue in my Cassandra cluster in an 11 node production cluster, however, I don't think this has to do with the For me, the Cassandra cluster has been running just fine for the past 4-5 weeks. However, I just did a cluster upgrade which required cordoning and uncordoning my cluster nodes one a time to perform an upgrade. However, once the nodes were all back online, the Manifest: Stack Trace: |
|
Something in your deployment has caused the cass-operator to lose its access to the imageConfig. There should be a ConfigMap which has suffix If you can check that ConfigMap and the Pod definition of cass-operator to see that they still match and are correctly deployed there. It's mounted as /configs/image_config.yaml in the cass-operator pod. |
|
Here's the relevant parts of my pod: And the configmap: I set my startup command to be: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happened?
The operator config file was not loaded, when I started the operator with
make run2024-01-21T13:59:22.773-0300 INFO setup Oper config file = {"configfile": ""}
2024-01-21T13:59:22.773-0300 INFO setup Oper config = {"operConfig": {"metrics":{},"health":{},"webhook":{}}}
This made the
imageConfigFileparameter be recognized as the empty string, which in turn propagated and triggered a null pointer exception during the cassandra pod creation (as demonstrated in the logs below)Having said that, is there a recommended way to run the operator locally and properly load the config file?
What did you expect to happen?
I expected the operator to create the Cassandra cluster successfully
How can we reproduce it (as minimally and precisely as possible)?
Clone repository
make manifests
make install
make run
k apply -f config/samples/example-cassdc-three-nodes-single-rack.yaml
cass-operator version
commit 609325a on master branch
Kubernetes version
Client Version: v1.28.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.27.3
Method of installation
make run
Anything else we need to know?
2024-01-21T13:34:13.769-0300 INFO controllers.CassandraDatacenter Reconcile loop completed {"cassandradatacenter": "default/dc1", "requestNamespace": "default", "requestName": "dc1", "loopID": "b863ad02-c6f1-4e4e-a786-5dda8e710144", "duration": 29.483785625}
2024-01-21T13:34:13.770-0300 INFO Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference {"controller": "cassandradatacenter_controller", "controllerGroup": "cassandra.datastax.com", "controllerKind": "CassandraDatacenter", "CassandraDatacenter": {"name":"dc1","namespace":"default"}, "namespace": "default", "name": "dc1", "reconcileID": "cc59a724-9886-4887-b96f-d59355ea82da"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x40 pc=0x101595ec0]
goroutine 242 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
/Users/guidi/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119 +0x1a4
panic({0x1019722a0?, 0x1026ca660?})
/opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/panic.go:914 +0x218
github.com/k8ssandra/cass-operator/pkg/images.AddDefaultRegistryImagePullSecrets(...)
/Users/guidi/projects/cass-operator/pkg/images/images.go:184
github.com/k8ssandra/cass-operator/pkg/reconciliation.buildPodTemplateSpec(0x14000230500, {{0x14000163c89, 0x5}, {0x0, 0x0}, 0x0, 0x0}, 0x2?)
/Users/guidi/projects/cass-operator/pkg/reconciliation/construct_podtemplatespec.go:797 +0x220
github.com/k8ssandra/cass-operator/pkg/reconciliation.newStatefulSetForCassandraDatacenter(0x140006ba000, {0x14000163c89, 0x5}, 0x14000230500, 0x1)
/Users/guidi/projects/cass-operator/pkg/reconciliation/construct_statefulset.go:115 +0x480
github.com/k8ssandra/cass-operator/pkg/reconciliation.(*ReconciliationContext).GetStatefulSetForRack(0x1400024e500, 0x140006b25a0)
/Users/guidi/projects/cass-operator/pkg/reconciliation/reconcile_racks.go:1386 +0x108
github.com/k8ssandra/cass-operator/pkg/reconciliation.(*ReconciliationContext).CheckRackCreation(0x1400024e500)
/Users/guidi/projects/cass-operator/pkg/reconciliation/reconcile_racks.go:141 +0x9c
github.com/k8ssandra/cass-operator/pkg/reconciliation.(*ReconciliationContext).ReconcileAllRacks(0x1400024e500)
/Users/guidi/projects/cass-operator/pkg/reconciliation/reconcile_racks.go:2324 +0x460
github.com/k8ssandra/cass-operator/pkg/reconciliation.(*ReconciliationContext).CalculateReconciliationActions(0x1400024e500)
/Users/guidi/projects/cass-operator/pkg/reconciliation/handler.go:68 +0xe8
github.com/k8ssandra/cass-operator/internal/controllers/cassandra.(*CassandraDatacenterReconciler).Reconcile(0x1400010e870, {0x101b767f0, 0x140006b0330}, {{{0x14000163c60, 0x7}, {0x14000163c5c, 0x3}}})
/Users/guidi/projects/cass-operator/internal/controllers/cassandra/cassandradatacenter_controller.go:147 +0x67c
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x101b767f0?, {0x101b767f0?, 0x140006b0330?}, {{{0x14000163c60?, 0x1018ce360?}, {0x14000163c5c?, 0x1400040ae08?}}})
/Users/guidi/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:122 +0x8c
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0x14000438b40, {0x101b76828, 0x1400010e7d0}, {0x1019c8ae0?, 0x140003ba760?})
/Users/guidi/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:323 +0x2a0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0x14000438b40, {0x101b76828, 0x1400010e7d0})
/Users/guidi/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274 +0x198
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
/Users/guidi/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235 +0x74
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 85
/Users/guidi/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:231 +0x43c
Exiting.
┆Issue is synchronized with this Jira Story by Unito
┆Issue Number: CASS-12
The text was updated successfully, but these errors were encountered: