[Bug] Unable to reconcile Grafana(Dashboard|Datasource) resources - tls: failed to verify certificate: x509: certificate signed by unknown authority

[sarahbx]

Describe the bug
Unable to import Grafana(Dashboard|Datasource) resources

As reported by the ODF QE team, dashboard import is failing for a recent OCP nightly version.
redhat-performance/odf-grafana#28

Also seen on OCP v4.17.2:
from grafana controller v5: (grafana v5.14.0)

ERROR GrafanaDashboardReconciler error reconciling dashboard {"controller": "grafanadashboard", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDashboard", "GrafanaDashboard": {"name":"odf-performance-analysis","namespace":"perfscale"}, "namespace": "perfscale", "name": "odf-performance-analysis", "reconcileID": "{uuid}", "dashboard": "odf-performance-analysis", "grafana": "grafana", "error": "Get \"https://grafana-route-perfscale.apps.{CLUSTER}/api/folders?limit=1000&page=1\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}

Version
Full semver version of the operator being used e.g. v4.10.0, v5.0.0-rc0
v5.14.0

To Reproduce
Steps to reproduce the behavior:

1. Install grafana-operator v5.14.0
2. Create GrafanaDashboard or datasource to be discovered by grafana
3. Check grafana controller logs for error

Expected behavior
Dashboards and datasources imported to grafana

Suspect component/Location where the bug might be occurring
Controller-v5

Screenshots
n/a

Runtime (please complete the following information):

• OS: [e.g. Linux,Fedora,Mac] RHEL9
• Grafana Operator Version [e.g. v5.0.0] v5.14.0
• Environment: [e.g Openshift,Kubernetes,minikube etc. please specify versions] OpenShift v4.16.14 & v4.17.2
• Deployment type: [e.g Openshift OLM/Helm/kustomize] OLM
• Other: [Other variables/things that might be relevant to this bug, versions of other services e.g. operator-sdk]

Additional context
redhat-performance/odf-grafana#28

[theSuess]

Thanks for the report. This is due to us fixing a longstanding security issue in which we did not verify TLS certificates. With the newest version we just released, you can configure TLS settings in .spec.client.tls. To solve for this specific use case, either add a matching CA bundle or disable TLS verification to return to the old behavior.

See these issues for more details:
#1683
#1675