Runners are terminated mid-run

[rkirimov]

I use multi-runner config and have two runners set up: one is using spot instances and the other uses on-demand. Both use custom AMIs. I tried using both spot and on-demand and in each case I had ec2 instances terminated while the workflow was running. It happens quite often. I wonder if I got any settings wrong. Here's my tf file:

module "runners" {
  source  = "philips-labs/github-runner/aws//modules/multi-runner"
  version = "5.12.2"

  multi_runner_config = {
    "ubuntu-x64-spot" = {
      delay_webhook_event = 0
      matcherConfig : {
        labelMatchers = [["self-hosted", "linux", "x64", "wf-ubuntu-spot", "wf-ubuntu"]]
        exactMatch    = false
      }
      runner_config = {
        runner_os                       = "linux"
        runner_architecture             = "x64"
        runner_extra_labels             = ["wf-ubuntu", "wf-ubuntu-spot"]
        runner_name_prefix              = "spot"
        ami_filter                      = { name = ["UBUNTU-GHA-RUNNER-SPOT"], state = ["available"] }
        ami_owners                      = [data.aws_caller_identity.current.account_id]
        create_service_linked_role_spot = true
        enable_ssm_on_runners           = true
        ebs_optimized                   = true
        enable_userdata                 = false
        enable_organization_runners     = true
        runners_maximum_count           = -1
        instance_target_capacity_type   = "spot"
        instance_types                  = ["t3.xlarge"]
        scale_down_schedule_expression  = "cron(*/30 * * * ? *)"
        minimum_running_time_in_minutes = 60
        enable_runner_binaries_syncer   = false
        idle_config = [{
          cron      = "* * 9-17 * * *"
          timeZone  = "America/New_York"
          idleCount = 1
        }]
      }
    }
    "ubuntu-x64-on-demand" = {
      delay_webhook_event = 0
      matcherConfig : {
        labelMatchers = [["self-hosted", "linux", "x64", "wf-ubuntu-on-demand", "wf-ubuntu"]]
        exactMatch    = false
      }
      runner_config = {
        runner_os                       = "linux"
        runner_architecture             = "x64"
        runner_extra_labels             = ["wf-ubuntu", "wf-ubuntu-on-demand"]
        runner_name_prefix              = "on-demand"
        ami_filter                      = { name = ["UBUNTU-GHA-RUNNER-ON-DEMAND"], state = ["available"] }
        ami_owners                      = [data.aws_caller_identity.current.account_id]
        create_service_linked_role_spot = true
        enable_ssm_on_runners           = true
        enable_userdata                 = false
        ebs_optimized                   = true
        minimum_running_time_in_minutes = 60
        enable_organization_runners     = true
        instance_types                  = ["t3.xlarge"]
        runners_maximum_count           = 10
        instance_target_capacity_type   = "on-demand"
        scale_down_schedule_expression  = "cron(*/30 * * * ? *)"
        enable_runner_binaries_syncer   = false
        idle_config = [{
          cron      = "* * 9-17 * * *"
          timeZone  = "America/New_York"
          idleCount = 1
        }]
      }
    }
  }

  aws_region = "us-east-1"
  vpc_id     = "vpc-0..."
  subnet_ids = ["subnet-0..."]

  prefix = local.environment

  github_app = {
    key_base64     = var.github_app.key_base64
    id             = var.github_app.id
    webhook_secret = random_id.random.hex
  }

  webhook_lambda_zip                = "../lambdas/webhook.zip"
  runner_binaries_syncer_lambda_zip = "../lambdas/runner-binaries-syncer.zip"
  runners_lambda_zip                = "../lambdas/runners.zip"
}

[crohr]

Can you show the labels you have used in your 2 workflows?

[npalm]

Runners can be terminated by AWS since the module using by default spot. You can configure non spot (on-demand) as well. Another problem (github) is that the runner agent cannot connect to GitHub by exhausting resources during the build. This will lead that github de-register the runner. This is a GitHub issue.

[rkirimov]

After some investigation, resource exhaustion was the culprit. I didn't think of it at all when troubleshooting and just focusing on terraform config itself. Thanks for the feedback!

[klingenm]

Did you check the EC2 instance in AWS Console / CLI that it wasn't in fact Spot Termination? We also thought it was resource exhaustion, but the default "allocation_strategy"="lowest-price" makes it very likely that your instance gets terminated.