High latency in starting decision task[load test]

[alexfjw]

Hi there, any idea what might contribute to high latency in starting a decision task?

We've been running a load test on a single task list, aiming to reach 100 wfs/s.
The workflow only runs 1 activity, which is to return the variables passed to it. (i.e. minimal processing)

I notice that the number of workflows created can hit 100 wfs/s after enabling scalable tasks lists. However, the activities created are less than the number of workflows triggered & hovers around 60/s. The number of pending workflows eventually increases and times out.

The only error log I get is RemoteSyncMatchedError{Message: remote sync match failed} from matching service, which I am unsure how to fix.

Any idea what I might have missed?

Other info:

• On Mysql for persistence, but did not see any metrics relating to high load on the db. We are using 2048 shards.
• I do not think it is an issue with workflow worker count as some of our workers were momentarily idle during the test.
• We are using some of the dynamic configs shared by others in the slack channel for performance but did not see much of a difference.
• We are running this on a 4 replica cluster (4 of matching, history, frontend, workers) and also tried scaling history up to 8 but did not see a difference. Did not see high load on the pods.

Dynamic config used

    frontend.persistenceMaxQPS:
      - value: 10000
        constraints: {}
    frontend.rps:
      - value: 6000
        constraints: {}
    frontend.domainrps:
      - value: 6000
        constraints: {}
    matching.persistenceMaxQPS:
      - value: 15000
        constraints: {}
    matching.rps:
      - value: 6000
        constraints: {}
    history.taskProcessRPS:
      - value: 5000
        constraints: {}
    history.persistenceMaxQPS:
      - value: 45000
        constraints: {}
    history.rps:
      - value: 15000
        constraints: {}
    history.cacheMaxSize:
      - value: 1024
        constraints: {}
    history.taskSchedulerWorkerCount:
      - value: 800
        constraints: {}
    history.taskSchedulerShardQueueSize:
      - value: 400
        constraints: {}
    history.timerTaskWorkerCount:
      - value: 20
        constraints: {}
    history.transferTaskWorkerCount:
      - value: 20
        constraints: {}
    history.replicatorTaskWorkerCount:
      - value: 20
        constraints: {}
    matching.numTasklistWritePartitions:
      - value: 10
        constraints: {}
    matching.numTasklistReadPartitions:
      - value: 10
        constraints: {}

[longquanzheng]

The error of remote sync match failed error means the task is not able to match with a worker within the time required ( a few seconds).

What is your sever version? I know there is a bug in scalable task list in the earlier version when the feature was just released, which could cause issue like that.

Also, I don’t see you have enabled scalable tasklist in the dynamic config . Can you confirm?

Scalable tasklist is not related to how many wf can be created in parallel. The factor of it is number of shards, db perf, and num of history hosts.

Scalable tasklist only contribute to how many tasks(including decision and activity) can be started in parallel, after a workflow is created.

[meiliang86]

Does create more worker help in this case?

[alexfjw]

Thanks for the help everyone! After retesting multiple tasklists on one worker with more history/matching/frontend hosts, workflows were no longer dropped.

TPS of a single tasklist/partition depends on the perf of the perf of a matching instance

My mistake was to limit to just 10 - 15 task lists, as well as just adding a small number of hosts. For now we are satisfied with the existing performance and have a better idea of what to tweak if more performance is required.

[zhangkyou]

After retesting multiple tasklists on one worker with more history/matching/frontend hosts, workflows were no longer dropped.

How can I use multiple tasklist on one worker?
Can you describe more details of your solution? I would appreciate it！Thanks！

[alexfjw]

As mentioned by Quan Zheng, it is possible to create a tasklist & append a numerical suffix to it.
I also had to dynamically name the workflow and activity attached to the tasklist. Assigning the same activity to multiple workflows did not work, had an error saying it the activity was already registered.

[zhangkyou]

the worker started with only one TaskListName parameter，but how can i use multiple tasklist on one worker?
worker := worker.New( service, Domain, TaskListName, workerOptions) err := worker.Start()

[alexfjw]

We started multiple workers, each in its own goroutine

…

On Sat, Aug 21, 2021 at 11:26 AM Kun Zhang ***@***.***> wrote: the worker started with only one TaskListName parameter，but how can i use multiple tasklist on one worker? worker := worker.New( service, Domain, TaskListName, workerOptions) err := worker.Start() — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4363 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACWTL23RAGTW3D4BOWLF5FDT54MHVANCNFSM5CEYD77Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[zhangkyou]

Got it！

After retesting multiple tasklists on one worker with more history/matching/frontend hosts, workflows were no longer dropped.

Another question, “with more history/matching/frontend hosts” how do this scaled.
In cadence-sample project, a common.SampleHelper only can associate to one service like this

SampleHelper struct {
		Service            workflowserviceclient.Interface // only associate to one service
		WorkerMetricScope  tally.Scope
		ServiceMetricScope tally.Scope
		Logger             *zap.Logger
		Config             Configuration
		Builder            *WorkflowClientBuilder
		DataConverter      encoded.DataConverter
		CtxPropagators     []workflow.ContextPropagator
		workflowRegistries []registryOption
		activityRegistries []registryOption

		configFile string
}

[alexfjw]

The history/matching/frontend hosts were scaled using the banzai cloud helm chart. The worker only needs to register to 1 endpoint. We had a load balancer pointing to our k8s nodes.

…

On Sat, Aug 21, 2021 at 12:35 PM Kun Zhang ***@***.***> wrote: Got it！ After retesting multiple tasklists on one worker with more history/matching/frontend hosts, workflows were no longer dropped. Another question, “with more history/matching/frontend hosts” how do this scaled. In cadence-sample project, a common.SampleHelper only can associate to one service like this `SampleHelper struct { Service workflowserviceclient.Interface // only assocaite one service WorkerMetricScope tally.Scope ServiceMetricScope tally.Scope Logger *zap.Logger Config Configuration Builder *WorkflowClientBuilder DataConverter encoded.DataConverter CtxPropagators []workflow.ContextPropagator workflowRegistries []registryOption activityRegistries []registryOption configFile string }` — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4363 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACWTL25HQLBVIBX5NZJLUHDT54UKPANCNFSM5CEYD77Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .