[FEATURE]: Multiple Task Update overlap with each other (Conductor + MySQL)

[ssureez]

Describe the Feature Request

Conductor + MySQL persistence

We are facing issue in conductor workflow orchestration where workflows are stuck in RUNNING state.

Reason is Worker's Task update overlaps with the previous worker update so workflows stayed as RUNNING state even after completed the processing

Scenario:
Consider we have two worker (worker1, worker2).

When worker1 completes its execution it call task update api. Conductor server receives the request and decider service decides there is worker2 to be scheduled. So worker2 scheduled and worker 1 task update is continue the processing. Within few milliseconds when worker2 also completes its execution then worker2 also call task update API.

In this scenario both Worker1 and Worker2 task updates are processing in parallel. Consider Worker2 task update first even before Worker1 task update. When Worker2 completes its processing then we see workflow moved to completed state. After that Worker1 task update updating workflow to Running state. This result in workflow stuck in running state forever.

Describe Preferred Solution

In mysql-persistence library, we should accept workflow update only when workflows are in intermediate state. If workflow status moved to terminal state means then updates should be dropped.