EKS cluster with private endpoint / vpc-cni ordering

[rauthur]

When deploying a new cluster I wasn't able to get node groups to join the cluster initially. It seems this is because the cluster endpoint is private and the vpc-cni addon isn't deployed yet. By default we have addons_depends_on: true in our EKS catalog, however, that prevents the vpc-cni from being added. Changing it will prevent the workflow from completing successfully b/c coredns, etc. require a node.

What am I missing here?

[milldr]

Just last week I went through this same deployment. In my case, I also had the default configuration. Notably these:

# eks/cluster
components:
  terraform:
    eks/cluster:
      vars:
...
        # private configuration
        cluster_endpoint_private_access: true
        cluster_endpoint_public_access: false
        cluster_private_subnets_only: true

        # addon configuration
        addons_depends_on: true
        cluster_kubernetes_version: "1.29"
        # EKS addons
        # https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html
        # AWS recommends to provision the required EKS addons and not to rely on the managed addons
        addons:
          # https://docs.aws.amazon.com/eks/latest/userguide/cni-iam-role.html
          # https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html
          # https://docs.aws.amazon.com/eks/latest/userguide/cni-iam-role.html#cni-iam-role-create-role
          # https://aws.github.io/aws-eks-best-practices/networking/vpc-cni/#deploy-vpc-cni-managed-add-on
          vpc-cni:
            addon_version: "v1.18.1-eksbuild.3" # set `addon_version` to `null` to use the latest version
          # https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html
          kube-proxy:
            addon_version: "v1.29.3-eksbuild.2" # set `addon_version` to `null` to use the latest version
          # https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html
          coredns:
            addon_version: "v1.11.1-eksbuild.9"
            ## Enable autoscaling for coredns (requires AWS EKS build later than 2024-05-15)
            configuration_values: '{"autoScaling":{"enabled":true}}'
          # https://docs.aws.amazon.com/eks/latest/userguide/csi-iam-role.html
          # https://aws.amazon.com/blogs/containers/amazon-ebs-csi-driver-is-now-generally-available-in-amazon-eks-add-ons
          # https://docs.aws.amazon.com/eks/latest/userguide/managing-ebs-csi.html#csi-iam-role
          # https://github.com/kubernetes-sigs/aws-ebs-csi-driver
          aws-ebs-csi-driver:
            addon_version: "v1.31.0-eksbuild.1" # set `addon_version` to `null` to use the latest version
            # If you are not using [volume snapshots](https://kubernetes.io/blog/2020/12/10/kubernetes-1.20-volume-snapshot-moves-to-ga/#how-to-use-volume-snapshots)
            # (and you probably are not), disable the EBS Snapshotter with:
            configuration_values: '{"sidecars":{"snapshotter":{"forceEnable":false}}}'
          aws-efs-csi-driver:
            addon_version: "v2.0.5-eksbuild.1"
            resolve_conflicts: OVERWRITE

I also had trouble getting the nodes to the join the cluster. In my case, it was because we didn't have any way for the nodes to egress the private subnets. I had to enable a NAT gateway and route 0.0.0.0/0 to that nat. Then redeploying the cluster passed.

It may not be the same for your use-case. I would recommend checking the kubelet logs on one of the nodes that fails to join the cluster. For example, in my case I could see this misleading error message:

Nov 01 14:31:31 ip-10-42-177-7.ec2.internal kubelet[2886]: I1101 14:31:31.242302    2886 csi_plugin.go:884] Failed to contact API server when waiting for CSINode publishing: Unauthorized

@rauthur Do you have egress from your private subnets? Does kubelet reveal anything helpful?

[rauthur]

The route table does have egress via a NAT gateway. I think the underlying issue is that nodes/nodegroups cannot communicate with the cluster endpoint b/c they aren't getting / using a private IP. The security groups seem to require access from VPC IPs only. Here are some hints from the TroubleshootWorkerNode runbook:

[WARNING]: Worker node outbound IP to internet is 44.235.xxx.xxx. It is not allowed in the cluster Public CIDR ranges. Please review this URL for further details: https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html
[WARNING]: No secondary private IP addresses are assigned to worker node i-038546b4xxxxxx, ensure that the CNI plugin is running properly. Please review this URL for further details: https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html
[WARNING]: As SSM agent is not reachable on worker node, this document did not check the status of Containerd, Docker and Kubelet daemons. Ensure that required daemons (containerd, docker, kubelet) are running on the worker node using command "systemctl status <daemon-name>".

Since scaling the desired # of instances to 0 and then deploying allows the CNI plugin to be added and THEN also allows these nodegroups to add instances and become active I'm inclined to continue looking into the secondary private IP warning. Even after getting the nodegroups active they still aren't spinning up any nodes.

I haven't been able to see the kubelet logs since there are no nodes in the cluster.

[GabisCampana]

@milldr

[rauthur]

The issue seems to be specific to our plat-dev account. Using an instance within a private subnet of the VPC I cannot reach the VPC-local DNS server (10.2.0.2), but I am able to ping resources on the internet by IP (e.g., 8.8.8.8). Connecting to VPC-local provided DNS within plat-staging works fine and so does deploying / managing a cluster there. I haven't been able to find a difference (yet) in the configured subnets, route tables, etc. other than the VPC CIDR ranges (10.1 vs. 10.2). DNS is enabled for the VPC + the DHCP option set has the Amazon managed DNS configured.

[rauthur]

Looks like I had deployed block-all behavior to the Route53 DNS resolver firewall ONLY in our dev stage. Whoops.

[milldr]

ah that'll do it! I've been bitten by some network blocker or another so many times, that the first step to debug is always double and triple checking connectivity.

If this comes up again (it has a lot for me), I recommend the AWS Reachability Analyzer, especially with cross-account support enabled. For example in my case, I was able to set the source as the ENI of my VPN client in 1 account and the destination as the private IP of the cluster's control plane in another account. It passed in 1 direction, but failed in the reverse and revealed the issue