Jupyter Controller reconcileHandler incorrectly reports ERRORs when reconciling

[nishant-dash]

Bug Description

I see that on various notebook reconciliations, the reconcile handler reports the same error

However the error is not critical, since it eventually reconciles it ok.
This red herring in the logs messes with the alerts from JupyterControllerRuntimeReconciliationErrorsExceedThreshold which results in it almost always firing.

I filed an issue upstream kubeflow/notebooks#62

To Reproduce

N/A

Environment

ckf 1.9/stable
jupyter controller 1.9/stable 1038

Relevant Log Output

2024-10-01T13:07:25.847Z [jupyter-controller] 1.7277880458479142e+09	ERROR	controller.notebook	Reconciler error	{"reconciler group": "kubeflow.org", "reconciler kind": "Notebook", "name": "mini-notebook-gpu", "namespace": "<REDACTED>", "error": "Operation cannot be fulfilled on notebooks.kubeflow.org \"mini-notebook-gpu\": the object has been modified; please apply your changes to the latest version and try again"}

Additional Context

No response

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-6343.

This message was autogenerated

[orfeas-k]

#412 removes this alert rule, since it will always go to Firing as long as the upstream controller produces those errors. It will be backported to 1.9 too. Once upstream issue is fixed, we should put the alert rule back (and revert the changes from the PR). Here's the removed alert rule:

alert: JupyterControllerRuntimeReconciliationErrorsExceedThreshold
expr: rate(controller_runtime_reconcile_errors_total[5m]) > 0
for: 0m
labels:
  severity: critical
annotations:
  summary: Total number of reconciliation errors per controller
  description: >
    Total number of reconciliation errors per controller
    LABELS = {{ $labels }}

We will keep this issue open to track the upstream one.

[orfeas-k]

@nishant-dash We commented out the alert rule, the change has been promoted to 1.9/stable so you should refresh and the rule won't be there anymore. As mentioned above, keeping this open to track the upstream issue.