Skip to content
This repository has been archived by the owner on Dec 13, 2023. It is now read-only.

Workflow not going forward after DO_WHILE execution completion #3880

Open
appunni-old opened this issue Dec 4, 2023 · 5 comments
Open

Workflow not going forward after DO_WHILE execution completion #3880

appunni-old opened this issue Dec 4, 2023 · 5 comments
Labels
type: bug bugs/ bug fixes

Comments

@appunni-old
Copy link

appunni-old commented Dec 4, 2023

Describe the bug

Details
Conductor version: 3.15.0
Persistence implementation: MySQL
Queue implementation: MySql
Lock: Redis
Workflow definition:

{
  "createTime": 1701669713675,
  "updateTime": 1701669746737,
  "createdBy": "[email protected]",
  "updatedBy": "[email protected]",
  "accessPolicy": {},
  "name": "test_do_while",
  "description": "Workflow details",
  "version": 2,
  "tasks": [
    {
      "name": "default_do_while",
      "taskReferenceName": "task_1__loop_databricks",
      "inputParameters": {},
      "type": "DO_WHILE",
      "startDelay": 0,
      "optional": false,
      "asyncComplete": false,
      "loopCondition": "if ($.task_1__loop_databricks['iteration'] < 1) { true; } else { false; }",
      "loopOver": [
        {
          "name": "default_sleep",
          "taskReferenceName": "task_1__wait_databricks",
          "inputParameters": {
            "duration": "20 seconds",
            "tenantId": "csit"
          },
          "type": "WAIT",
          "startDelay": 0,
          "optional": false,
          "asyncComplete": false
        }
      ]
    },
    {
      "name": "default_sleep",
      "taskReferenceName": "task_2__wait_databricks",
      "inputParameters": {
        "duration": "20 seconds",
        "tenantId": "csit"
      },
      "type": "WAIT",
      "startDelay": 0,
      "optional": false,
      "asyncComplete": false
    }
  ],
  "inputParameters": [],
  "outputParameters": {},
  "schemaVersion": 2,
  "restartable": true,
  "workflowStatusListenerEnabled": false,
  "ownerEmail": "[email protected]",
  "timeoutPolicy": "ALERT_ONLY",
  "timeoutSeconds": 0,
  "variables": {},
  "inputTemplate": {}
}

Task definition: System
Event handler definition:

To Reproduce
Steps to reproduce the behavior:

-> Execute the above workflow
-> Wait for 20 seconds for the WAIT task completion

Expected behavior
WAIT task outside the DO_WHILE was supposed to execute

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Testing was sone after merging #3878

@appunni-old appunni-old added the type: bug bugs/ bug fixes label Dec 4, 2023
@appunni-old
Copy link
Author

This may be related to #3876 . Again Double underscore may be causing issue

@appunni-old
Copy link
Author

{
  "createTime": 1701670338519,
  "updateTime": 1701669746737,
  "createdBy": "[email protected]",
  "updatedBy": "[email protected]",
  "accessPolicy": {},
  "name": "test_do_while_2",
  "description": "Workflow details",
  "version": 1,
  "tasks": [
    {
      "name": "default_do_while",
      "taskReferenceName": "task_1_loop_databricks",
      "inputParameters": {},
      "type": "DO_WHILE",
      "startDelay": 0,
      "optional": false,
      "asyncComplete": false,
      "loopCondition": "if ($.task_1_loop_databricks['iteration'] < 1) { true; } else { false; }",
      "loopOver": [
        {
          "name": "default_sleep",
          "taskReferenceName": "task_1_wait_databricks",
          "inputParameters": {
            "duration": "20 seconds",
            "tenantId": "csit"
          },
          "type": "WAIT",
          "startDelay": 0,
          "optional": false,
          "asyncComplete": false
        }
      ]
    },
    {
      "name": "default_sleep",
      "taskReferenceName": "task_2_wait_databricks",
      "inputParameters": {
        "duration": "20 seconds",
        "tenantId": "csit"
      },
      "type": "WAIT",
      "startDelay": 0,
      "optional": false,
      "asyncComplete": false
    }
  ],
  "inputParameters": [],
  "outputParameters": {},
  "schemaVersion": 2,
  "restartable": true,
  "workflowStatusListenerEnabled": false,
  "ownerEmail": "[email protected]",
  "timeoutPolicy": "ALERT_ONLY",
  "timeoutSeconds": 0,
  "variables": {},
  "inputTemplate": {}
}

using this worked.

@appunni-old
Copy link
Author

            List<TaskModel> loopOverTaskList =
                    workflow.getTasks().stream()
                            .filter(
                                    t ->
                                            TaskUtils.removeIterationFromTaskRefName(
                                                            t.getReferenceTaskName())
                                                    .equals(taskRefName))
                            .collect(Collectors.toList());

This could be the bug here, as they are using removeIterationFromTaskRefName for checking if it's an iterable task. But the method itself is too fragile implementation.

SimpleActionProcessor:132

appunni-old added a commit to appunni-old/conductor that referenced this issue Dec 4, 2023
@gajendrangnanasekaran
Copy link

HI can you help me to do setup conductor with postgres

@appunni-old
Copy link
Author

appunni-old commented Dec 5, 2023

Hi @gajendrangnanasekaran you can use docker-compose to set it up . Clone this repo and run

docker-compose -f docker/docker-compose-postgres.yaml up -d

To kill run

docker-compose -f docker/docker-compose-postgres.yaml kill

If you want to run local setup for development, remove conductor:server from docker-compose file and create a new docker-compose file and run that docker-compose file using the similar command.

Then start up local server by running below command
CONDUCTOR_CONFIG_FILE=docker/server/config/config-postgres-modified.properties ./gradlew bootRun

config-postgres-modified.properties must be pointing to localhost:6432 and localhost:9201 as postgres and elastic search

if you want to run the local server on intellij here is how it looks like
The second line below jdk selection is the VM options

Screenshot 2023-12-05 at 9 55 10 PM

for mysql support I have added
// community
runtimeOnly("com.netflix.conductor:conductor-mysql-persistence:${revConductor}")

in the server/build.gradle dependencies. That's all

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
type: bug bugs/ bug fixes
Projects
None yet
Development

No branches or pull requests

2 participants