Add docs for Nexus circuit breaker #3220
Open
+59
−46
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rodrigozhou
commented
Nov 26, 2024
Comment on lines
+149
to
+158
| ### Circuit Breaker {#circuit-breaker} | ||
|
|
||
| The circuit breaker kicks in when requests fail with a [retryable error](https://github.com/temporalio/temporal/blob/13d6cd8cf7a4ba0c4660cf98f672bbd645dca3e7/components/nexusoperations/executors.go#L659) | ||
| consecutively as it might indicate that the destination (eg: Nexus service to start operation, or | ||
| the caller for callback request) is down or unable to process the request. The default behavior of | ||
| the circuit breaker is to open after 5 consecutive failed requests. Once in open state, Nexus taskk | ||
| will fail early and requests won't be sent to destination. After a minute in open state, it will | ||
| change to half-open state, which will allow only 1 request to be made. If the request is successful, | ||
| then the circuit breaker changes its state to closed, and allows all requests to pass through. | ||
|
|
There was a problem hiding this comment.
This is the only change I made, all other changes was yarn build.
bergundy
reviewed
Nov 27, 2024
| @@ -146,6 +146,16 @@ As mentioned above, a synchronous Nexus Operation handler has less than 10 secon | |||
| Once the caller Workflow schedules an Operation with the caller’s Temporal cluster, the caller’s Nexus Machinery keeps trying to start the Operation, with automatic retries and exponential backoff. | |||
| If a Nexus Operation returns a [retryable error](https://github.com/temporalio/temporal/blob/13d6cd8cf7a4ba0c4660cf98f672bbd645dca3e7/components/nexusoperations/executors.go#L659) when attempting to start, the Operation it will be retried up to the [default Retry Policy’s](https://github.com/temporalio/temporal/blob/de7c8879e103be666a7b067cc1b247f0ac63c25c/components/nexusoperations/config.go#L111) max attempts and expiration interval. | |||
|
|
|||
| ### Circuit Breaker {#circuit-breaker} | |||
|
|
|||
| The circuit breaker kicks in when requests fail with a [retryable error](https://github.com/temporalio/temporal/blob/13d6cd8cf7a4ba0c4660cf98f672bbd645dca3e7/components/nexusoperations/executors.go#L659) | |||
There was a problem hiding this comment.
I would explicitly call out request timeouts and how that may relate to when a worker is down.
| ### Circuit Breaker {#circuit-breaker} | ||
|
|
||
| The circuit breaker kicks in when requests fail with a [retryable error](https://github.com/temporalio/temporal/blob/13d6cd8cf7a4ba0c4660cf98f672bbd645dca3e7/components/nexusoperations/executors.go#L659) | ||
| consecutively as it might indicate that the destination (eg: Nexus service to start operation, or |
There was a problem hiding this comment.
The endpoint would be down, not the service.
| The circuit breaker kicks in when requests fail with a [retryable error](https://github.com/temporalio/temporal/blob/13d6cd8cf7a4ba0c4660cf98f672bbd645dca3e7/components/nexusoperations/executors.go#L659) | ||
| consecutively as it might indicate that the destination (eg: Nexus service to start operation, or | ||
| the caller for callback request) is down or unable to process the request. The default behavior of | ||
| the circuit breaker is to open after 5 consecutive failed requests. Once in open state, Nexus taskk |
There was a problem hiding this comment.
Not sure users will understand what nexus tasks are in this context and this applies to callback "tasks" as well. This is all internal details, you should use "the Nexus machinery" instead.
| consecutively as it might indicate that the destination (eg: Nexus service to start operation, or | ||
| the caller for callback request) is down or unable to process the request. The default behavior of | ||
| the circuit breaker is to open after 5 consecutive failed requests. Once in open state, Nexus taskk | ||
| will fail early and requests won't be sent to destination. After a minute in open state, it will |
There was a problem hiding this comment.
Suggested change
| will fail early and requests won't be sent to destination. After a minute in open state, it will | |
| will fail early and requests won't be sent to that destination. After a minute in open state, it will |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Add docs for Nexus circuit breaker
Notes to reviewers