Add coordinator fault tolerance check

[johscheuer]

Description

Adding a fault tolerance check for coordinators and only allow operations if all coordinators are up and running. In a fallow up we could use those fault tolerances and only validate them for affected Pods, e.g. only validate the coordinator fault tolerance for coordinator changes.

Type of change

Please select one of the options below.

• New feature (non-breaking change which adds functionality)

Discussion

Currently the requirement is that all coordinators are up and running. This is the most strict policy.

Testing

I added new unit tests. Some of the existing tests are broken and they need some modification (or the mock setup).

Documentation

Follow-up

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: e5d30c0
• Duration 2:47:44
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 2ae7024
• Duration 2:43:58
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[09harsh]

LGTM. Just a minor comment.