~/.kube/config for Docker Desktop causes install to fail in invocation image

[glyn]

When using duffle to install software in kubernetes, users expect to pass ~/.kube/config to duffle as a credential (as described here).

Unfortunately, when using docker desktop to run kubernetes, ~/.kube/config contains this:

clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: https://localhost:6443
  name: docker-for-desktop-cluster

which fails inside the invocation image.

A workaround is to change localhost to host.docker.internal, but users really need to treat ~/.kube/config as an opaque credential that they pass to duffle without modification. Also, as noted by @justincormack below, host.docker.internal won't work in the host, so not only would the user need to modify ~/.kube/config, but they'd also need to maintain two versions (or at least two similar contexts in the same config).

Steps to reproduce

1. Install docker desktop and enable kubernetes. Ensure it is selected for kubectl.
2. Use the following example: https://github.com/glyn/duffle/tree/hello-k8s/examples/hellok8s
3. Build the example:

   duffle build ./examples/hellok8s
   

4. Create a credential pointing to ~/.kube/config:

   duffle credentials add examples/hellok8s/myk8s.yaml
   

   where examples/hellok8s/myk8s.yaml has the following contents:

   name: myk8s
   credentials:
     - name: kubeconfig
       source:
         path: $HOME/.kube/config
   

5. Attempt to install the example and observe the error:

   duffle install hellok8s-demo hellok8s:0.1 --credentials myk8s
   Executing install action...
   The connection to the server localhost:6443 was refused - did you specify the right host or port?
   Action install complete for hellok8s-demo
   

[justincormack]

host.docker.internal is not a good solution as it will fail from the host, as this only resolves in Desktop containers. And using the host IP would work but will break if it changes. Adding a different IP address is a problem for Docker Desktop as it requires root privileges. So it is hard to see what a good solution is...

[glyn]

Apparently, docker-app has solved this by rewriting the address.

[radu-matei]

@simonferquel or @silvin-lubecki might have a bit more context there - would it be a good idea?

[glyn]

An alternative would be for Docker Desktop to change so that duffle can use the host's unmodified ~/.kube/config in the invocation image (which seems to work when using minikube). This would simplify docker-app too as it would no longer need the above feature.

@simonferquel @silvin-lubecki @chris-crone would you care to comment on the feasibility of that solution?

[glyn]

Apparently, docker-app has solved this by rewriting the address.

docker-app patches ~/.kube/config for Docker Desktop when creating a CNAB from a docker application package. However, they hit the same problem as duffle when using an CNAB which has not undergone this patching. @silvin-lubecki is investigating whether it is possible to fix Docker Desktop. (A potential workaround is to run invocation images in Docker Desktop's kubernetes so that the invocation image doesn't need ~/.kube/config - see #763.)