argo-workflows-aws-plugin

Argo Workflows Executor Plugin for AWS Services, e.g. SageMaker Pipelines, Glue, etc.

Table of Contents

• Supported AWS Services
• Getting Started
  • Add IAM Role and Policy
  • Enable Executor Plugins
  • Installation
  • Add Workflow Template
  • Trigger Workflow
  • Uninstall Plugin
• References

Supported AWS Services

The following tables describe the implementation state for the protocol's RPC methods and database operations.

Service Name	Implemented?
Amazon SageMaker Pipelines	✔️
AWS Glue	✔️
AWS Step Functions	✔️
AWS Lambda	🚧

Getting Started

Add IAM Role and Policy

The plugin requires IAM role and policy to execute its operations.

The following CDK code add a role, which is later referenced in plugin.yaml manifest.

    const audClaim = `${cluster.clusterOpenIdConnectIssuer}:aud`;
    const subClaim = `${cluster.clusterOpenIdConnectIssuer}:sub`;

    const k8sConditions = new cdk.CfnJson(this, "KubeOIDCCondition", {
      value: {
        [audClaim]: "sts.amazonaws.com",
        // [subClaim]: "system:serviceaccount:kube-system:aws-node",
        [subClaim]: "system:serviceaccount:argo:awf-aws-executor-plugin",
      },
    });

    const awfPluginRole = new cdk.aws_iam.Role(this, "ArgoWorkflowsExecutorPluginRole", {
      roleName: `${stack.stackName}-awf-aws-executor-plugin`,
      assumedBy: new cdk.aws_iam.WebIdentityPrincipal(
        `arn:aws:iam::${cdk.Aws.ACCOUNT_ID}:oidc-provider/${cluster.clusterOpenIdConnectIssuer}`
      ).withConditions({
        StringEquals: k8sConditions,
      }),
    });

    awfPluginRole.addToPolicy(new cdk.aws_iam.PolicyStatement({
      effect: cdk.aws_iam.Effect.ALLOW,
      resources: ["arn:aws:sagemaker:*:*:pipeline/*"],
      actions: [
        "sagemaker:DescribePipeline",
        "sagemaker:StartPipelineExecution",
        "sagemaker:ListPipelineExecutionSteps",
        "sagemaker:DescribePipelineExecution",
        "sagemaker:ListPipelineExecutions",
        "sagemaker:ListPipelines"
      ]
    }));

Enable Executor Plugins

First, enable Executor Plugins:

kubectl patch deployment \
  workflow-controller \
  --namespace argo \
  --type='json' \
  -p='[{"op": "add", "path": "/spec/template/spec/containers/0/env/0", "value": {
    "name": "ARGO_EXECUTOR_PLUGINS",
    "value": "true",
}}]'

Next, restart:

kubectl -n argo set env deployment/workflow-controller ARGO_EXECUTOR_PLUGINS=true
kubectl rollout restart -n argo deployment workflow-controller

Installation

Download the plugin manifest:

wget https://raw.githubusercontent.com/greenpau/argo-workflows-aws-plugin/main/assets/plugin.yaml

Edit metadata.annotations.eks.amazonaws.com/role-arn in the ServiceAccount. (see DEVELOPMENT.md for more information about associated IAM role and policy)

Next, install the plugin:

kubectl apply -f plugin.yaml

The output follows:

serviceaccount/awf-aws-plugin-sa unchanged
clusterrole.rbac.authorization.k8s.io/argo-plugin-addition-role unchanged
clusterrolebinding.rbac.authorization.k8s.io/awf-aws-plugin-addition-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/awf-aws-plugin-binding unchanged
configmap/awf-aws-plugin created

List Argo Workflows Executor Plugins again:

$ kubectl get cm -l workflows.argoproj.io/configmap-type=ExecutorPlugin -n argo

NAME             DATA   AGE
awf-aws          2      34s

Get details about the plugins:

kubectl describe cm -l workflows.argoproj.io/configmap-type=ExecutorPlugin -n argo

Add Workflow Template

Create a workflow template:

kubectl apply -f https://raw.githubusercontent.com/greenpau/argo-workflows-aws-plugin/main/assets/amz-sagemaker-pipelines-workflow-template.yaml

Trigger Workflow

Start new workflow:

kubectl create -f https://raw.githubusercontent.com/greenpau/argo-workflows-aws-plugin/main/assets/amz-sagemaker-pipelines-workflow.yaml

The output follows:

workflow.argoproj.io/sm-pipelines-tswbm created

Review the status of the workflow by the its name, e.g. sm-pipelines-tswbm:

kubectl describe pod -n argo sm-pipelines-tswbm-1340600742-agent

Review logs of the containers (main, awf-aws) inside the pod:

kubectl logs -n argo -c main sm-pipelines-tswbm-1340600742-agent
kubectl logs -n argo -c awf-aws sm-pipelines-tswbm-1340600742-agent

Uninstall Plugin

If necessary, run the following commands to uninstall the plugin:

kubectl delete -f https://raw.githubusercontent.com/greenpau/argo-workflows-aws-plugin/main/assets/plugin.yaml

References

• Argo Workflows - Plugin Directory