Coder Infra

Infrastructure to support a Coder instance, provided by CNCF, for the use of contributors

Project contents

Terraform to provision Kubernetes resources to a cluster

Architecture

Deploy a Kubernetes cluster running on Equinix Metal which runs Coder server.

Running in the cluster will be

Cert-Manager: manage LetsEncrypt TLS certs
Nginx-Ingress: serve services like Coder server
MetalLB: BGP IP allocation
PowerDNS: use to allow setting up of DNS01 validation for TLS certs
distribution(s): use as a proxy cache for each common registry to save bandwidth and time
External-DNS: manage DNS in a Kubernetes-native way
Longhorn: storage!
kubevirt: orchestrate VMs using Kubernetes primitives
cluster-api: orchestrate Kubernetes in Kubernetes
Coder: the server itself

The domain coder.sharing.io is where Coder lives. *.coder.sharing.io is to point also to the same IP as Coder.

An NS and A record links the managed records in the DNS provider to the PowerDNS in the cluster

NS  coder.sharing.io      ns1.coder.sharing.io
A   ns1.coder.sharing.io  x.x.x.x

The A record pointing to the LoadBalancer or floating IP.

Needs

[ ] completion of architecture
[x] a domain to use for hosting
[ ] do cluster-api and kubevirt components live in the management cluster?

todo

[x] configure and deploy powerdns
[x] configure and deploy external-dns, point it to powerdns
[x] create a cert and cluster-issuer for coder.sharing.io and *.coder.sharing.io
[x] set up NS DNS record on sharing.io DNS provider
[x] configure storage (#1)
[x] configure and deploy kubevirt
[x] deploy cluster-api
[ ] configure and deploy Coder

Prerequisites

ytt, for YAML updating

go install github.com/vmware-tanzu/carvel-ytt/cmd/ytt@v0.43.0

Setup

A Kubernetes cluster is required to initialise the target cluster.

Either use an existing cluster or create a kind cluster

kind create cluster

Set credentials for Equinix Metal for initialising the cluster-api provider

read PACKET_API_KEY; export PACKET_API_KEY

Initialise cluster-api with the Packet (Equinix Metal) infrastructure provider

export EXP_CLUSTER_RESOURCE_SET=true
clusterctl init --infrastructure packet --bootstrap talos --control-plane talos

List the variables used in the cluster configuration

clusterctl generate cluster --infrastructure packet --list-variables cluster-template.yaml 2>/dev/null

Generate a static cluster configuration

export CONTROLPLANE_NODE_TYPE=c3.small.x86 \
  WORKER_NODE_TYPE=c3.small.x86 \
  FACILITY=sv15 \
  PROJECT_ID=7a44b778-41d2-49fa-9c92-99148516c600 \
  SSH_KEY=gh:BobyMCbobs \
  TALOS_VERSION=v1.2.6 \
  KUBERNETES_VERSION=v1.24.8 \
  NODE_OS=ubuntu_20_04 \
  SERVICE_CIDR=10.96.0.0/12 \
  POD_CIDR=10.244.0.0/16 \
  CONTROL_PLANE_MACHINE_COUNT=3
clusterctl generate cluster cncf-coder --target-namespace cncf-coder --from ./cluster-template-packet-kubeadm.yaml \
    > ./cluster-cncf-coder-infra.yaml

Installation

kubectl apply -f ./cluster-cncf-coder-namespace.yaml
kubectl -n cncf-coder apply -f ./cluster-cncf-coder-infra.yaml

Watch provisioning

kubectl -n cncf-coder get packetmachine,machine,taloscontrolplane,secret,cm

Fetch Kubeconfig

kubectl -n cncf-coder get secret cncf-coder-kubeconfig -o=jsonpath='{.data.value}' | base64 -d > /tmp/cncf-coder-kubeconfig

Mark as schedulable

kubectl --kubeconfig /tmp/cncf-coder-kubeconfig taint nodes --all node-role.kubernetes.io/master-
kubectl --kubeconfig /tmp/cncf-coder-kubeconfig taint nodes --all node-role.kubernetes.io/control-plane-

Prepare

Cert-Manager

curl -o ./cert-manager.yaml -L https://github.com/cert-manager/cert-manager/releases/download/v1.10.0/cert-manager.yaml

Cilium

helm repo add cilium https://helm.cilium.io/
helm template cilium cilium/cilium \
    --version 1.11.6 \
    --namespace kube-system \
    -f ./values/cilium.yaml \
    > ./cilium.yaml

Multus

curl -o multus.yaml -L https://github.com/k8snetworkplumbingwg/multus-cni/raw/master/deployments/multus-daemonset.yml

Metallb

helm repo add metallb https://metallb.github.io/metallb
helm template --create-namespace -n metallb-system metallb metallb/metallb --version 0.13.7 --values values/metallb.yaml \
    > ./metallb.yaml

ingress-nginx

export LB_IP="$(kubectl --kubeconfig /tmp/cncf-coder-kubeconfig -n kube-system get service cloud-provider-equinix-metal-kubernetes-external -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')"

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm template -n ingress-nginx ingress-nginx ingress-nginx/ingress-nginx --version 4.4.0 --values ./values/ingress-nginx.yaml --set controller.service.externalIPs[0]="$LB_IP" > ./ingress-nginx.yaml

Metrics-Server

helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
helm template \
    metrics-server \
    -n kube-system \
    -f ./values/metrics-server.yaml \
    --version 3.8.2 \
    metrics-server/metrics-server \
      > ./metrics-server.yaml

Longhorn

helm repo add longhorn https://charts.longhorn.io
helm template longhorn longhorn/longhorn --values values/longhorn.yaml --namespace longhorn-system --version 1.3.2 \
    > ./longhorn-pre.yaml
cat ./longhorn-pre.yaml | ytt --ignore-unknown-comments -f overlays/ -f - > ./longhorn.yaml
rm ./longhorn-pre.yaml

Kubevirt

curl -o kubevirt-operator.yaml -L https://github.com/kubevirt/kubevirt/releases/download/v0.58.0/kubevirt-operator.yaml
curl -o kubevirt-cr.yaml -L https://github.com/kubevirt/kubevirt/releases/download/v0.58.0/kubevirt-cr.yaml
curl -o cdi-operator.yaml -L https://github.com/kubevirt/containerized-data-importer/releases/download/v1.55.1/cdi-operator.yaml
curl -o cdi-cr.yaml -L https://github.com/kubevirt/containerized-data-importer/releases/download/v1.55.1/cdi-cr.yaml

Stackgres (postgres)

VERSION=1.2.0
helm template \
    -n stackgres stackgres-operator \
    --set authentication.secretRef.name=stackgres-custom-auth \
    "https://stackgres.io/downloads/stackgres-k8s/stackgres/${VERSION}/helm/stackgres-operator.tgz" \
      > ./stackgres-operator.yaml

Coder

helm template -n coder coder https://github.com/coder/coder/releases/download/v0.13.1/coder_helm_0.13.1.tgz \
    --values ./values/coder.yaml \
    > ./coder.yaml

Install

kubectl --kubeconfig /tmp/cncf-coder-kubeconfig create namespace coder

Cilium

kubectl --kubeconfig /tmp/cncf-coder-kubeconfig apply -f ./cilium.yaml

Multus

kubectl --kubeconfig /tmp/cncf-coder-kubeconfig apply -f ./multus.yaml

Longhorn

kubectl --kubeconfig /tmp/cncf-coder-kubeconfig create namespace longhorn-system --dry-run=client -o yaml \
  | kubectl --kubeconfig /tmp/cncf-coder-kubeconfig apply -f -
kubectl --kubeconfig /tmp/cncf-coder-kubeconfig apply -f ./longhorn.yaml

Cert-Manager

kubectl --kubeconfig /tmp/cncf-coder-kubeconfig apply -f ./cert-manager.yaml

metallb

kubectl --kubeconfig /tmp/cncf-coder-kubeconfig create namespace metallb-system --dry-run=client -o yaml \
  | kubectl --kubeconfig /tmp/cncf-coder-kubeconfig apply -f -
kubectl --kubeconfig /tmp/cncf-coder-kubeconfig -n metallb-system apply -f metallb.yaml

Metrics-Server

kubectl --kubeconfig /tmp/cncf-coder-kubeconfig -n kube-system apply -f ./metrics-server.yaml

ingress-nginx

kubectl --kubeconfig /tmp/cncf-coder-kubeconfig create namespace ingress-nginx --dry-run=client -o yaml \
  | kubectl --kubeconfig /tmp/cncf-coder-kubeconfig apply -f -
kubectl --kubeconfig /tmp/cncf-coder-kubeconfig apply -n ingress-nginx -f ./ingress-nginx.yaml

PowerDNS

kubectl --kubeconfig /tmp/cncf-coder-kubeconfig create namespace powerdns --dry-run=client -o yaml \
  | kubectl --kubeconfig /tmp/cncf-coder-kubeconfig apply -f -
export LB_IP="$(kubectl --kubeconfig /tmp/cncf-coder-kubeconfig -n kube-system get service cloud-provider-equinix-metal-kubernetes-external -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')"
envsubst '${LB_IP}' < ./powerdns.yaml | kubectl --kubeconfig /tmp/cncf-coder-kubeconfig apply -f -

Certs

export LB_IP="$(kubectl --kubeconfig /tmp/cncf-coder-kubeconfig -n kube-system get service cloud-provider-equinix-metal-kubernetes-external -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')"
envsubst '${LB_IP}' < ./certs.yaml | kubectl --kubeconfig /tmp/cncf-coder-kubeconfig apply -f -

External-DNS

export LB_IP="$(kubectl --kubeconfig /tmp/cncf-coder-kubeconfig -n kube-system get service cloud-provider-equinix-metal-kubernetes-external -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')"
kubectl --kubeconfig /tmp/cncf-coder-kubeconfig apply -f ./external-dns-crd.yaml
envsubst '${LB_IP}' < ./external-dns.yaml | kubectl --kubeconfig /tmp/cncf-coder-kubeconfig apply -f -
envsubst '${LB_IP}' < ./dnsendpoint.yaml | kubectl --kubeconfig /tmp/cncf-coder-kubeconfig apply -f -

Kubevirt

kubectl --kubeconfig /tmp/cncf-coder-kubeconfig apply \
    -f kubevirt-operator.yaml \
    -f cdi-operator.yaml
kubectl --kubeconfig /tmp/cncf-coder-kubeconfig apply \
    -f kubevirt-cr.yaml \
    -f cdi-cr.yaml

cluster-api

export PACKET_API_KEY="$(kubectl -n cluster-api-provider-packet-system get secret cluster-api-provider-packet-manager-api-credentials -o=jsonpath='{.data.PACKET_API_KEY}' | base64 -d)"
clusterctl --kubeconfig /tmp/cncf-coder-kubeconfig init --infrastructure packet --infrastructure kubevirt

StackGres

kubectl --kubeconfig /tmp/cncf-coder-kubeconfig create namespace stackgres --dry-run=client -o yaml \
  | kubectl --kubeconfig /tmp/cncf-coder-kubeconfig apply -f -

PASSWORD=$(tr -cd '[:alnum:]' < /dev/urandom | fold -w40 | head -n1)
kubectl --kubeconfig /tmp/cncf-coder-kubeconfig -n stackgres create secret generic stackgres-custom-auth \
    --from-literal=k8sUsername=admin \
    --from-literal=password="$(echo "${PASSWORD}" | sha256sum | awk '{print $1}')" \
    --from-literal=clearPassword="${PASSWORD}" \
    --dry-run=client \
    -o yaml \
    | kubectl --kubeconfig /tmp/cncf-coder-kubeconfig apply -f -

kubectl --kubeconfig /tmp/cncf-coder-kubeconfig -n stackgres apply -f ./stackgres-operator.yaml

Coder

kubectl --kubeconfig /tmp/cncf-coder-kubeconfig -n coder apply -f ./postgresql.yaml
kubectl --kubeconfig /tmp/cncf-coder-kubeconfig -n coder apply -f ./coder.yaml
kubectl --kubeconfig /tmp/cncf-coder-kubeconfig -n coder apply -f ./coder-role+rolebinding.yaml

Switch management

clusterctl move -n cncf-coder --to-kubeconfig /tmp/cncf-coder-kubeconfig

Patch cluster-api-provider-kubevirt

until kubernetes-sigs/cluster-api-provider-kubevirt#212 is merged

kubectl --kubeconfig /tmp/cncf-coder-kubeconfig -n capk-system       patch deployment capk-controller-manager        --type=json -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"ghcr.io/ii/cluster-api-provider-kubevirt@sha256:c1030577c70b4797fa1d3a8b56aa2a7009da2960e4a7d40f0528c3a9f376f604"}]'

Tear down

kubectl -n cncf-coder delete cluster cncf-coder
kubectl delete ns cncf-coder

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.org

README.org

Coder Infra

Project contents

Architecture

Needs

todo

Prerequisites

Setup

Installation

Watch provisioning

Fetch Kubeconfig

Mark as schedulable

Prepare

Install

Switch management

Patch cluster-api-provider-kubevirt

Tear down

Files

README.org

Latest commit

History

README.org

File metadata and controls

Coder Infra

Project contents

Architecture

Needs

todo

Prerequisites

Setup

Installation

Watch provisioning

Fetch Kubeconfig

Mark as schedulable

Prepare

Install

Switch management

Patch cluster-api-provider-kubevirt

Tear down