GHES 3.14.0: "lost communication with the server."

[m8t88]

We managed to deploy v5.16.1 in our corporate network and were able to build and use our own ubuntu20 and 22 AMIs. Its a multi-runner setup which uses ephemeral spot runners. The GHES version is 3.14.0. and we are using the latest GitHub runner v2.319.1

During an extended performance and load test, we get the intermittent error:

The self-hosted runner: XYZ lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

No code or infrastructure code was touched during the 24h trial phase.

We tried to check the cloudwatch logs and execution logs but could not find the root cause. We suspect it could be some kind of race condition but are not sure how to find the root cause.

Did someone experience something similar or can point us into the right direction to debug?

[stuartp44]

We see this type of issue if you are fully loading your runner inside of your job. Neither us or GitHub manage the resource allocation inside of the runner so if you burn all the resources and stop the GH action runner from communicating back to the GH this is typically what you describe as the issue.

[m8t88]

Thanks a lot for the fast response and clear explanation! We indeed configured our builds to use up all CPUs since that was the configuration on our previous build systems.

We will see what we can change to avoid this going forward.

[klingenm]

This message is also what you get if the instance is reclaimed. The default setting for allocation strategy is set to "lowest-price", which on the AWS docs has this caveat

instance_allocation_strategy can be changes easily enough.