Initial support for three data hall replication #1651
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
manfontan
reviewed
May 30, 2023
Result of fdb-kubernetes-operator-pr on Linux CentOS 7
|
Result of fdb-kubernetes-operator-pr-kind on Linux CentOS 7
|
johscheuer
commented
Jun 1, 2023
brownleej
requested changes
Jun 1, 2023
johscheuer
force-pushed
the
add-support-three-data-hall
branch
from
fe562f1 to
a8c689c
Compare
johscheuer
requested review from
brownleej,
09harsh and
manfontan
and removed request for
sbodagala
Result of fdb-kubernetes-operator-pr on Linux CentOS 7
|
Result of fdb-kubernetes-operator-pr on Linux CentOS 7
|
Result of fdb-kubernetes-operator-pr on Linux CentOS 7
|
johscheuer
commented
Oct 11, 2023
Result of fdb-kubernetes-operator-pr on Linux CentOS 7
|
Result of fdb-kubernetes-operator-pr on Linux CentOS 7
|
Result of fdb-kubernetes-operator-pr on Linux CentOS 7
|
johscheuer
commented
Oct 12, 2023
brownleej
reviewed
Oct 12, 2023
johscheuer
commented
Oct 12, 2023
johscheuer
force-pushed
the
add-support-three-data-hall
branch
from
7508317 to
892c6a9
Compare
johscheuer
commented
Oct 12, 2023
Result of fdb-kubernetes-operator-pr on Linux CentOS 7
|
Result of fdb-kubernetes-operator-pr on Linux CentOS 7
|
johscheuer
commented
Oct 13, 2023
There was a problem hiding this comment.
Result of fdb-kubernetes-operator-pr on Linux CentOS 7
- Commit ID: 892c6a9
- Duration 2:49:03
- Result: ❌ FAILED
- Error:
Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1- Build Log terminal output (available for 30 days)
- Build Workspace zip file of the working directory (available for 30 days)
We've seen this failure a few times:
2023/10/12 16:52:33 reconciled name=fdb-cluster-ccrfs4x5, namespace=pr-464-x6hz2s62, generation:2
• [FAILED] [1121.240 seconds]
Operator Migrations when a migration is triggered and the namespace quota is limited [BeforeEach] should add the prefix to all instances [e2e, pr]
[BeforeEach] /codebuild/output/src2181315438/src/github.com/FoundationDB/fdb-kubernetes-operator/e2e/test_operator_migrations/operator_migration_test.go:77
[It] /codebuild/output/src2181315438/src/github.com/FoundationDB/fdb-kubernetes-operator/e2e/test_operator_migrations/operator_migration_test.go:104
[FAILED] Timed out after 600.000s.
Expected
<int64>: 2
to be zero-valued
In [BeforeEach] at: /codebuild/output/src2181315438/src/github.com/FoundationDB/fdb-kubernetes-operator/e2e/test_operator_migrations/operator_migration_test.go:100 @ 10/12/23 16:52:33.427
------------------------------
I'm going to open an issue for it to fix it.
Same for:
• [FAILED] [1937.014 seconds]
Operator Upgrades upgrading a cluster without chaos [It] Upgrade from 7.1.37 to 7.3.15 [e2e, pr]
/codebuild/output/src2181315438/src/github.com/FoundationDB/fdb-kubernetes-operator/e2e/fixtures/upgrade_test_configuration.go:115
[FAILED] Unexpected error:
<*errors.errorString | 0xc00029fb40>:
timed out waiting for the condition
{
s: "timed out waiting for the condition",
}
occurred
In [It] at: /codebuild/output/src2181315438/src/github.com/FoundationDB/fdb-kubernetes-operator/e2e/test_operator_upgrades_variations/operator_upgrades_variations_test.go:119 @ 10/12/23 16:54:29.772
There were additional failures detected. To view them in detail run ginkgo -vv
------------------------------
This was referenced Oct 13, 2023
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Fixes: #348
Type of change
Please select one of the options below.
Discussion
When running a FoundationDB cluster on public cloud providers it can be useful to use three data hall as redundancy more and spread Pods across multiple availability zones. So far the operator was not supporting this mode. Now the operator supports this mode with minimal code changes, the drawback is, that a user has to create 3
FoundationDBClusterresources.Testing
I did some manual testing:
and
fdbclishow the correct config:Documentation
Added to this PR.
Follow-up
We can improve the three data hall setup in the future to only require one single
FoundationDBClusterresource, but that will require additional changes in the operator.As a follow up we could think about adding support for three_data_center as well. Based on my current understanding those modes are similar.