konnect: handle deletions first

[pmalek]

What this PR does / why we need it:

This PR makes the generic Konnect entity reconciler to handle deletions first so that deletions of multiple entities are handled faster.

This was observed during heavier, local performance tests.

This PR:

controller_runtime_active_workers{controller="KongConsumer"} 0
controller_runtime_max_concurrent_reconciles{controller="KongConsumer"} 8
controller_runtime_reconcile_errors_total{controller="KongConsumer"} 0
controller_runtime_reconcile_panics_total{controller="KongConsumer"} 0
controller_runtime_reconcile_total{controller="KongConsumer",result="error"} 0
controller_runtime_reconcile_total{controller="KongConsumer",result="requeue"} 0
controller_runtime_reconcile_total{controller="KongConsumer",result="requeue_after"} 50
controller_runtime_reconcile_total{controller="KongConsumer",result="success"} 150
controller_runtime_terminal_reconcile_errors_total{controller="KongConsumer"} 0
workqueue_adds_total{controller="KongConsumer",name="KongConsumer"} 200
workqueue_depth{controller="KongConsumer",name="KongConsumer"} 0
workqueue_retries_total{controller="KongConsumer",name="KongConsumer"} 50


kubectl wait --timeout=-1s --for=delete kongconsumer --all  0.11s user 0.05s system 1% cpu 10.374 total

controller_runtime_active_workers{controller="KongConsumer"} 0
controller_runtime_max_concurrent_reconciles{controller="KongConsumer"} 8
controller_runtime_reconcile_errors_total{controller="KongConsumer"} 0
controller_runtime_reconcile_panics_total{controller="KongConsumer"} 0
controller_runtime_reconcile_total{controller="KongConsumer",result="error"} 0
controller_runtime_reconcile_total{controller="KongConsumer",result="requeue"} 0
controller_runtime_reconcile_total{controller="KongConsumer",result="requeue_after"} 50
controller_runtime_reconcile_total{controller="KongConsumer",result="success"} 350
controller_runtime_terminal_reconcile_errors_total{controller="KongConsumer"} 0
workqueue_adds_total{controller="KongConsumer",name="KongConsumer"} 400
workqueue_depth{controller="KongConsumer",name="KongConsumer"} 0
workqueue_retries_total{controller="KongConsumer",name="KongConsumer"} 50

---

main:

controller_runtime_active_workers{controller="KongConsumer"} 0
controller_runtime_max_concurrent_reconciles{controller="KongConsumer"} 8
controller_runtime_reconcile_errors_total{controller="KongConsumer"} 0
controller_runtime_reconcile_panics_total{controller="KongConsumer"} 0
controller_runtime_reconcile_total{controller="KongConsumer",result="error"} 0
controller_runtime_reconcile_total{controller="KongConsumer",result="requeue"} 0
controller_runtime_reconcile_total{controller="KongConsumer",result="requeue_after"} 50
controller_runtime_reconcile_total{controller="KongConsumer",result="success"} 150
controller_runtime_terminal_reconcile_errors_total{controller="KongConsumer"} 0
workqueue_adds_total{controller="KongConsumer",name="KongConsumer"} 200
workqueue_depth{controller="KongConsumer",name="KongConsumer"} 0
workqueue_retries_total{controller="KongConsumer",name="KongConsumer"} 50

kubectl wait --timeout=-1s --for=delete kongconsumer --all  0.12s user 0.05s system 0% cpu 19.466 total

controller_runtime_active_workers{controller="KongConsumer"} 0
controller_runtime_max_concurrent_reconciles{controller="KongConsumer"} 8
controller_runtime_reconcile_errors_total{controller="KongConsumer"} 0
controller_runtime_reconcile_panics_total{controller="KongConsumer"} 0
controller_runtime_reconcile_total{controller="KongConsumer",result="error"} 0
controller_runtime_reconcile_total{controller="KongConsumer",result="requeue"} 0
controller_runtime_reconcile_total{controller="KongConsumer",result="requeue_after"} 50
controller_runtime_reconcile_total{controller="KongConsumer",result="success"} 350
controller_runtime_terminal_reconcile_errors_total{controller="KongConsumer"} 0
workqueue_adds_total{controller="KongConsumer",name="KongConsumer"} 400
workqueue_depth{controller="KongConsumer",name="KongConsumer"} 0
workqueue_retries_total{controller="KongConsumer",name="KongConsumer"} 50

Conclusion

The same amount of reconciliations with nearly halved deletion time.

Creation of 50 consumers, decreased ever so slightly but that might be a network latency effect. Non goal for this PR.

main:

kubectl -n default wait --for=condition=Programmed=true kongconsumer --all  1  0.18s user 0.08s system 0% cpu 26.805 total

This PR:

kubectl -n default wait --for=condition=Programmed=true kongconsumer --all  1  0.18s user 0.08s system 1% cpu 18.907 total

[pmalek]

This would require the change in behavior described in #788: so that the operator does not set the ownership relations between object and objects living inside the CP are not scheduled for deletion when CP gets deleted.

Closing until that changes.