Race condition when you make 2 parallel steps wait_for the same previous step in a workflow

[Harshith-umesh]

Describe the bug

There appears to be a concurrency issue in the engine which is causing a race condition when multiple steps wait for the same previous step using the wait_for condition. A sample workflow is present here

To reproduce

run go test arcaflow-engine/workflow/workflow_test.go -v go test and the TestWaitForParallel testcase will fail randomly.

Additional context

Error logs:

2023-07-24T21:26:54+05:30	error		no steps running, no more executable steps, cannot construct any output (outputs have the following dependencies: success: steps.third_wait.outputs, steps.second_wait.outputs)

2023-07-24T21:26:54+05:30	warning		Plugin step wait start failed. step closed while waiting for run configuration

2023-07-24T21:26:54+05:30	info		There are currently 0 steps starting, 0 waiting for input, 0 running, 3 finished

2023-07-24T21:26:54+05:30	error		no steps running, no more executable steps, cannot construct any output (outputs have the following dependencies: success: steps.second_wait.outputs, steps.third_wait.outputs)

    workflow_test.go:288: Unexpected error: no steps running, no more executable steps, cannot construct any output (outputs have the following dependencies: success: steps.third_wait.outputs, steps.second_wait.outputs)

--- FAIL: TestWaitForParallel (0.00s)```