How do I know if a tasklist is overloaded #4403
-
|
How can I realized the tasklist backlog much more task, while the workers polls task from it too slowly? Are there some metrics can helping this? |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 15 replies
-
|
There is two metric of scheduled to start latency for both decision and activity task: https://github.com/uber-go/cadence-client/blob/b09692f6838f3c45fe02fde689787c87d925bdaf/internal/common/metrics/constants.go#L46 if you are sure that workers are enough and idle, than the only possibility of high latency is tasklist being overloaded |
Beta Was this translation helpful? Give feedback.
-
|
I find this metric is reported by worker. |
Beta Was this translation helpful? Give feedback.
-
|
I find the cadence-0.20.0/common/service/config/metrics.go file in cadence-server.It contains a NewScope function, this function set server's tally's scope,but why this feature dont include in cadence-client? |
Beta Was this translation helpful? Give feedback.
-
|
Check this docs https://cadenceworkflow.io/docs/operation-guide/monitor/ |
Beta Was this translation helpful? Give feedback.
-
|
I red this, this is a Prometheus metrics, I implement a statsd metrics by my self. Another question is that what is the different between WorkerMetricScope and ServiceMetricScope? Does the cadence server's metrics can emit in worker machine too? |
Beta Was this translation helpful? Give feedback.
-
|
Server and client metrics are separate. The two scopes you implemented are both client side metrics. WorkerScope is for worker operation metrics, like polling and processing activity and decision tasks. ServiceScope is for non worker , like starting workflow |
Beta Was this translation helpful? Give feedback.
-
|
In this pieces of cadence client code func (atp *activityTaskPoller) pollWithMetrics(ctx context.Context,
pollFunc func(ctx context.Context) (*s.PollForActivityTaskResponse, time.Time, error)) (interface{}, error) {
response, startTime, err := pollFunc(ctx)
if err != nil {
return nil, err
}
if response == nil || len(response.TaskToken) == 0 {
return &activityTask{}, nil
}
workflowType := response.WorkflowType.GetName()
activityType := response.ActivityType.GetName()
metricsScope := getMetricsScopeForActivity(atp.metricsScope, workflowType, activityType)
metricsScope.Counter(metrics.ActivityPollSucceedCounter).Inc(1)
metricsScope.Timer(metrics.ActivityPollLatency).Record(time.Now().Sub(startTime))
scheduledToStartLatency := time.Duration(response.GetStartedTimestamp() - response.GetScheduledTimestampOfThisAttempt())
metricsScope.Timer(metrics.ActivityScheduledToStartLatency).Record(scheduledToStartLatency)
return &activityTask{task: response, pollStartTime: startTime}, nil
}the metrics scope is atp's(*activityTaskPoller) scope, is this atp's scope derived from the worker.Options's metrics scope? workerOptions := worker.Options{
MetricsScope: h.WorkerMetricScope,
Logger: h.Logger,
MaxConcurrentActivityTaskPollers: 10,
MaxConcurrentDecisionTaskPollers: 10,
}
h.StartWorkers(domain, taskListName, workerOptions) |
Beta Was this translation helpful? Give feedback.
-
|
I already solved it.Please ignore this question. |
Beta Was this translation helpful? Give feedback.
-
|
I had catch the worker's metrics. The metrics 'cadence-activity-poll-transient-failed' and 'cadence-decision-poll-transient-failed' counter is very high? Why the poll always failed? |
Beta Was this translation helpful? Give feedback.
-
|
It depends on the machine perf. Check cpu/memory and network capacity |
Beta Was this translation helpful? Give feedback.
-
|
I check the metrics on the server side.There is a metric named 'cadence_client_requests', the tag with this metric cadence_role='matching_client',cadence_service=('cadence_frontend' and 'cadence_matching') has almostly 150 every second, is this normal with an only has 10+ workflows cluster? |
Beta Was this translation helpful? Give feedback.
-
|
It depends on the workflow code and yes, that’s possible. |
Beta Was this translation helpful? Give feedback.
-
|
I accomplished my worker's metrics emiting. Some latency metrics are very high. |
Beta Was this translation helpful? Give feedback.
-
|
No, the high latency is expected for poll API because it’s doing long polling. Also GetWorkflowHistory also have long polling options (which is confusing and we are working separate it ) . |
Beta Was this translation helpful? Give feedback.
-
|
another scene is I have this stack trace of the workflow that Timeout START_TO_CLOSE in my cadence-web UI. the code in cadence-worker/logic/cronWorkflow.go:211 is only a activity execute: err = workflow.ExecuteActivity(ctx, SendJsonToDDMQV1Activity, featureValue, params).Get(ctx, nil)why it pending in the Get funciton, I dont need the return value of this activity. |
Beta Was this translation helpful? Give feedback.
-
|
It’s pending on Get. Get is a blocking call to wait for activity results |
Beta Was this translation helpful? Give feedback.
-
|
If you don’t need to wait, then remove the Get. ExecuteActivity will return a future and it’s not blocking |
Beta Was this translation helpful? Give feedback.
There is two metric of scheduled to start latency for both decision and activity task: https://github.com/uber-go/cadence-client/blob/b09692f6838f3c45fe02fde689787c87d925bdaf/internal/common/metrics/constants.go#L46
if you are sure that workers are enough and idle, than the only possibility of high latency is tasklist being overloaded