The "Print your certificate!" experiment at the cert-manager booth at KubeCon

This experiment was run at the cert-manager booth at KubeCon EU 2022 in Valencia, KubeCon NA 2022 in Detroit, KubeCon EU 2023 in Amsterdam, KubeCon NA 2023 in Chicago, and Kubecon EU 2024 in Paris.

⚠️ Except for the URL <cert-manager.github.io/print-your-cert/> which should work forever, the other URLs and IPs presented in this README are temporary.

• Video and slides
• Description of the experiment
• What's the stack?
• Staff: test things
• Running everything on the Raspberry Pi (on the booth)
  • Booth: Initial set up of the Raspberry Pi
  • Booth: Set Up Tailscale on the Raspberry Pi
  • Booth: Make sure you can SSH into the Rasberry Pi
  • Booth: Set Up Docker, Helm, K3d, and kubectl
  • Booth: Set up the tunnel between the Internet and the Raspberry Pi
  • Prerequisite: install k3s on the Raspberry Pi
  • Booth: Configure kubectl on your laptop to access the Raspberry Pi's cluster
  • Booth: Install cert-manager and the issuers on the Raspberry Pi
  • Booth: Run the UI on the Raspberry Pi
  • Booth: Running the printer controller on the Raspberry Pi
• Local development
  • Local development on the UI
  • Local development on the controller (that creates PNGs and prints them)
    • cert-card
    • Testing the printer
    • Testing cert-card
• Troubleshooting
  • From the CLI: usb.core.USBError: [Errno 13] Access denied (insufficient permissions)
  • From the CLI: usb.core.USBError: [Errno 16] Resource busy
  • From the web UI: No such file or directory: '/dev/usb/lp1'

Video and slides

Here is a short video showing what the experiment looked like on Friday 20 May 2022 at KubeCon Valencia:

Here are the slides Mael presented after KubeCon:

Description of the experiment

When visiting the cert-manager booth, you will be welcomed and one of the staff may suggest to visit a QR code from their phone to participate to the "Print your certificate!" experiment, or to use the Raspberry Pi's keyboard and screen available on the booth.

Upon opening the QR code link (or on the Raspberry Pi's screen), the participant is shown a web page prompting for a name and email:

The issuance takes less than a second, and the participant is redirected to a new page where they can see a receipt of their certificate. A button "Print your certificate" appears:

When clicking on "Print your certificate", the participant is told that their certificate will shortly be printed.

The printer, installed on the booth, starts printing two labels: one for the front side, and one for the back side. The booth staff sticks the two printed labels onto a black-colored card (format A7), and uses the wax gun and the wax stamp to stamp the card.

Because the label is made of plastic, and the wax is hot, it is advised to the staff not to put stamp in contact of the label.

The front-side label looks like this:

The back-side label looks like this:

The person can choose the color of the card onto which the cert-manager booth staff will put the two labels that were automatically printed on. I purchased 200 cards of each color (1400 total), so it should be enough:

Here is what it may look like for real. Since I didn't have the above cards for the prototype, I have cut a piece of cardboard with the A7 size (7.4 x 10.5 cm). The label on the front is 6.2 x 8.7 cm, and the wax stamp is 4 cm large.

The "real" colored cards will be smaller (5.4 x 9.0 cm) meaning that I will have to do a smaller label on both sides.

The back-side labels is a QR code containing the PEM-encoded certificate that was issued. Since we didn't find any good use for TLS, we didn't include the private key.

I wanted the smallest TLS certificate possible. After reading Smallest possible certificate for IoT device, it seems ECDSA is good for small signatures, and RSA is not good. The configuration for the ECDSA signature is shown below in print-your-cert-ca.

The QR code contains a URL of the form:

https://cert-manager.github.io/print-your-cert/?asn1=MIICXDCCAgOgAwIU...O7pAkqhQc%3D)
<--------------------------------->       <------------------------->
      Hosted on GitHub Pages                   The base-64 encoded and
                                               URL-encoded PEM-encoded
                                               certificate without the headers.

For example:

https://cert-manager.github.io/print-your-cert/?asn1=MIICXDCCAgOgAwIBAgIQdPaTuGSUDeosii4dbdLBgTAKBggqhkjOPQQDAjAnMSUwIwYDVQQDExxUaGUgY2VydC1tYW5hZ2VyIG1haW50YWluZXJzMB4XDTIyMDUxNjEzMDkwMFoXDTIyMDgxNDEzMDkwMFowLDEqMCgGA1UEAwwhZm9vIGJhciBmb28gYmFyIDxmb28uYmFyQGJhci5mb28%2BMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtmGM5lil9Vw%2Fy5LhpgO8t5gSb5oUo%2BDp5vWw0Z5C7rjvifi0%2FeD9MbVFkxb%2B%2BhmOaaNCVgqDUio1OBOZyL90KzdnGW7nz1fRM2KCNrDF5Y1mO7uv1ZTZa8cVBjF67KjFuNkvvHp74m65bKwXeCHXJBmO3Z1FH8hudICU74%2BNl6tyjlMOsTHv%2BLY0jPfmAtO6eR%2BEf%2FHvgzwsjKds12vdlRCdHSS6u5zlrZZxF3zTO7YuAM7mN8Wbjq94YcpgsJ5ssNOtMu9FwZtPGQDHPaQyVQ86FfjhmMi1IUOUAAGwh%2FQRv8ksX%2BOupHTNdH06WmIDCaGBjWFgPkwicavMZgZG3QIDAQABo0EwPzAOBgNVHQ8BAf8EBAMCBaAwDAYDVR0TAQH%2FBAIwADAfBgNVHSMEGDAWgBQG5XQnDhOUa748L9H7TWZN2avluTAKBggqhkjOPQQDAgNHADBEAiBXmyJ24PTG76pEyq6AQtCo6TXEidqJhsmK9O5WjGBw7wIgaPbcFI5iMMgfPGEATH2AGGutZ6MlxBmwhEO7pAkqhQc%3D

⁉️ How do we get this URL? First, take a PEM-encoded certificate. It will looks like this:

-----BEGIN CERTIFICATE-----
MIIDBzCCAe+gAyPj/8QWMBQUAMIGLMQswCQYD
wIBAgMIG+LMQswCQYDAOPj/8QAaDMBQEFAwUa
...
-----END CERTIFICATE-----

It takes three steps to turn this PEM-encoded certificate into something that can be given with the query parameter ?asn1=....

1. We remove the header and footer, i.e., we remove the lines -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----). The result looks like this:

   MIIDBzCCAe+gAyPj/8QWMBQUAMIGLMQswCQYD
   wIBAgMIG+LMQswCQYDAOPj/8QAaDMBQEFAwUa
   

2. (optional) We can save a few bytes by removing the newlines. The result is:

   MIIDBzCCAe+gAyPj/8QWMBQUAMIGLMQswCQYDwIBAgMIG+LMQswCQYDAOPj/8QAaDMBQEFAwUa
   

3. At this point, we have the ASN.1 certificate encoded in base 64. We have to URL-encode it, which gives:

   MIIDBzCCAe%2BgAyPj%2F8QWMBQUAMIGLMQswCQYDwIBAgMIG%2BLMQswCQYDAOPj%2F8QAaDMBQEFAwUa%0A
   

4. Copy this into the URL:

   https://cert-manager.github.io/print-your-cert?asn1=MIIDBzCCAe%2BgAyPj%2F8QWMBQUAMIGLMQswCQYDwIBAgMIG%2BLMQswCQYDAOPj%2F8QAaDMBQEFAwUa%0A
   

One-line that takes a PEM-encoded certificate and returns a URL:

cat <<EOF | grep -v CERTIFICATE | tr -d $'\n' | python3 -c "import urllib.parse; print(urllib.parse.quote_plus(open(0).read()))" | (printf "https://cert-manager.github.io/print-your-cert?asn1="; cat)
-----BEGIN CERTIFICATE-----
MIIDBzCCAe+gAyPj/8QWMBQUAMIGLMQswCQYD
wIBAgMIG+LMQswCQYDAOPj/8QAaDMBQEFAwUa
...
-----END CERTIFICATE-----
EOF

On the certificate page, the participant can also see their certificate by clicking on the button "Print your certificate". The PEM-encoded certificate is shown in the browser:

On the booth, we have a 42-inch display showing the list of certificates (https://print-your-cert.cert-manager.io/list):

And that's it: you have a certificate that proves that you were at the KubeCon cert-manager booth! The CA used during the conference will be available at some point so that people can verify the signature.

What's the stack?

https://print-your-cert.cert-manager.io
                |
                |
                v
            VM on GCP
                |
                |  Caddy + Tailscale
                |  (see section below)
                |
                v
+---------------------------------+
|               Pi                |
|  K3s cluster                    |   USB   +-------------------+
|    cert-manager                 | ------> | Brother QL-820NWB |
|    print-your-cert-ui (:8080)   |         +-------------------+
|    print-your-cert-controller   |                (on the booth)
+---------------------------------+
                |    (on the booth)
          HDMI  |
                v
     +-------------------+
     | list of certs     |
     | already printed   | 42" display.
     |                   |
     +-------------------+
            (on the booth)

Staff: test things

For anyone who is authorized and wants to test or debug things:

• Install tailscale.

• Run tailscale up, it should open something in your browser → "Sign in with GitHub" → Authorize Tailscale → Multi-user Tailnet cert-manager.

• If http://print-your-cert.cert-manager.io/ doesn't work, then the frontend UI is at http://100.121.173.5:8080/.

• You can test that the printer works at http://100.121.173.5:8013/.

• You can SSH into the Pi (which runs a Kubernetes cluster) as long as you are a member of the cert-manager org:

  ssh [email protected]

Running everything on the Raspberry Pi (on the booth)

Once on the booth, you will need to perform these ten tasks:

1. Booth: Initial set up of the Raspberry Pi
2. Booth: Set Up Tailscale on the Raspberry Pi
3. Booth: Make sure you can SSH into the Rasberry Pi
4. Booth: Set Up Docker, Helm, K3d, and kubectl
5. Booth: Set up the tunnel between the Internet and the Raspberry Pi
6. Prerequisite: install k3s on the Raspberry Pi
7. Booth: Configure kubectl on your laptop to access the Raspberry Pi's cluster
8. Booth: Install cert-manager and the issuers on the Raspberry Pi
9. Booth: Run the UI on the Raspberry Pi
10. Booth: Running the printer controller on the Raspberry Pi

Booth: Initial set up of the Raspberry Pi

Warning

If you need to upgrade Debian on the Raspberry Pi (apt upgrade), please upgrade it at least a week before KubeCon so that any breakage (e.g., the Raspberry UI) can be fixed before the venue! We mistakenly ran sudo apt upgrade on the first day of KubeCon in Amsterdam and ended up spending half of the day fixing it!

First, unplug the micro SD card from the Raspberry Pi and plug it into your laptop using a micro-SD-to-SD card adaptor.

Then, install Raspberry OS on the microSD card using the Imager program. In the Imager program settings, change the username to certmanager and the password to something secret.

Be sure also to enable SSH with password authentication.

You can also use the imager to setup a WiFi connection, which might be helpful with initial setup at home.

At the conference, you should be able to use the GUI to set up a WiFi connection.

If the Wifi doesn't work, somehow SSH into the Pi and run wpa_cli:

$ sudo wpa_cli status
Selected interface 'p2p-dev-wlan0'
wpa_state=DISCONNECTED
p2p_device_address=e6:5f:01:a6:66:00
address=e6:5f:01:a6:66:00
uuid=0fb4e5b4-b372-5253-93e9-fa6f2c4d8037

To look for the right SSID, run on the Pi:

wpa_cli scan && wpa_cli scan_results

Then edit the file /etc/wpa_supplicant/wpa_supplicant.conf and run:

sudo wpa_cli -i wlan0 reconfigure
sudo ifconfig wlan0 down
sudo ifconfig wlan0 up

Then, unmount the micro SD card from your laptop and plug it into the Raspberry Pi.

Booth: Set Up Tailscale on the Raspberry Pi

Plug a keyboard and mouse to the Raspberry and install Tailscale:

# From the Raspberry Pi:
curl -fsSL https://tailscale.com/install.sh | sh

Then, login with the following command. It will open a browser window, allowing you to log in. Use your GitHub account to log in (Sign In with GitHub -> Authorize Tailscale -> Multi-user Tailnet -> select tailnet cert-manager.org.github).

tailscale up --accept-dns=false

Important

Make sure to disable Tailscale's DNS resolution with --accept-dns=false. We have seen a ton of problems with Tailscale's DNS resolution.

If you prefer staying logged in to your personal tailnet, feel free to share the machine with your Tailnet in https://login.tailscale.com/admin/machines. That way, you can stay logged to your tailnet but still be able to access the Raspberry Pi.

Allowing SSH Access to cert-manager org members

If desired, the script below will allow SSH access to cert-manager org members.

This might be overkill; you can also just download any GitHub user's SSH keys using curl -LO https://github.com/<user>.keys and then append the contents of that file to .ssh/authorized_keys.

curl -sH "Authorization: token $(lpass show github.com -p)" https://api.github.com/orgs/cert-manager/members \
  | jq '.[] | .login' -r \
  | ssh -t [email protected] \
    'set -xe; while read -r i; do curl -LsS https://github.com/$i.keys | tee -a $HOME/.ssh/authorized_keys; done; cat $HOME/.ssh/authorized_keys | sort | sed -re 's/\s+$//' | uniq >a; mv a $HOME/.ssh/authorized_keys'

Booth: Make sure you can SSH into the Rasberry Pi

First, make sure you have Tailscale installed and running. Make sure to be login using your GitHub account, and select the Tailnet cert-manager.org.github. The Pi is shared to that Tailnet.

Then, edit your ~/.ssh/config to add the following:

Host pi
  HostName 100.85.65.38
  User pi

All the commands below assume that you have configured your ~/.ssh/config as above.

Booth: Set Up Docker, Helm, K3d, and kubectl

Make sure you have configured ~/.ssh.config in the section above. Then, SSH into the Pi:

ssh pi

Note

You may see the following error message when SSHing into the Pi:

bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)

To fix this, you can run the following command:

# From the Raspberry Pi.
sudo tee -a /etc/environment <<<"LC_ALL=en_US.UTF-8"
sudo tee /etc/locale.gen <<<"en_US.UTF-8 UTF-8"
sudo tee /etc/locale.conf <<<"LANG=en_US.UTF-8"
sudo locale-gen en_US.UTF-8
EOF

Then, install Docker with the command:

# From the Raspberry Pi:
curl -fsSL https://get.docker.com | sudo bash
sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker

k3s requires the memory cgroup v2. To enable it, add the following flags to /boot/cmdline.txt:

cgroup_memory=1 cgroup_enable=memory

Also install vim and jq:

# From the Raspberry Pi:
sudo apt install -y vim jq

Finally, install k3d, helm, and kubectl:

# From the Raspberry Pi:
curl -Ls https://raw.githubusercontent.com/rancher/k3d/main/install.sh | bash
curl -Ls https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

Booth: Set up the tunnel between the Internet and the Raspberry Pi

We want to expose the print-your-cert UI on the Internet at https://print-your-cert.cert-manager.io. To do that, we use a f1-micro VM on GCP and use Caddy to terminate the TLS connections and to forward the connections to the Raspberry Pi's Tailscale IP.

https://print-your-cert.cert-manager.io
              |
              |
              v  35.241.231.131 (eth0)
    +------------------------+
    |  VM "print-your-cert"  |
    |    Caddy + Tailscale   |
    +------------------------+
              |  100.106.168.42 (tailscale0)
              |
              |
              |
              v  100.85.65.38 (tailscale0)
    +-------------------+
    |       Pi          |
    |                   |
    |     :8080 (UI)    |
    +-------------------+

Before creating the VM, you will need to get access to the GCP project cert-manager-general. You can get access by being a maintainer on the cert-manager project and requesting to be added to the GCP project.

Then, you will need to make sure you are logged into Tailscale (see one of the previous sections).

To create the VM print-your-cert, you can use the following command:

Note

Use the GCP zone closest to the KubeCon venue. The examples below use europe-west1-c (Belgium). Try to pick a zone with low CO2 emissions.

Note

Find out the latest debian image by running:

gcloud compute images list | grep debian

# From your laptop:
gcloud compute firewall-rules create allow-tailscale \
    --project cert-manager-general \
    --network default \
    --action allow \
    --direction ingress \
    --rules udp:41641 \
    --source-ranges 0.0.0.0/0
gcloud compute instances create print-your-cert \
    --project cert-manager-general \
    --network default \
    --machine-type=f1-micro \
    --image-family=debian-12 \
    --image-project=debian-cloud \
    --can-ip-forward \
    --boot-disk-size=10GB \
    --zone=europe-west1-c

Then, copy-paste the IP into the print-your-cert.cert-manager.io zone:

1. Copy the IP:

   IP=$(gcloud compute instances describe print-your-cert \
       --project cert-manager-general \
       --zone=europe-west1-c --format json \
         | jq -r '.networkInterfaces[].accessConfigs[] | select(.type=="ONE_TO_ONE_NAT") | .natIP')

2. The zone print-your-cert.cert-manager.io is a delegated zone meant for print-your-cert. Anyone in the Google group [email protected] can update the A record:

   gcloud dns record-sets update --project cert-manager-io \
     --zone print-your-cert-cert-manager-io \
     --type=A --ttl=300 print-your-cert.cert-manager.io --rrdatas=$IP

Then, install Tailscale and make sure IP forwarding is enabled on the VM:

gcloud compute ssh --project cert-manager-general --zone=europe-west1-c print-your-cert -- 'curl -fsSL https://tailscale.com/install.sh | sh'
gcloud compute ssh --project cert-manager-general --zone=europe-west1-c print-your-cert -- \
    "sudo perl -ni -e 'print if \!/^net.ipv4.ip_forward=1/d' /etc/sysctl.conf; \
     sudo tee -a /etc/sysctl.conf <<<net.ipv4.ip_forward=1; \
     sudo sysctl -w net.ipv4.ip_forward=1"

We found that the default-http and default-https now require tags to be set:

gcloud compute instances add-tags print-your-cert --project cert-manager-general --zone europe-west1-c --tags http-server
gcloud compute instances add-tags print-your-cert --project cert-manager-general --zone europe-west1-c --tags https-server

Then, run the following. Click the link that shows and log into Tailscale using "Login with GitHub", and then select the Tailnet cert-manager.org.github.

gcloud compute ssh --project cert-manager-general --zone=europe-west1-c print-your-cert -- sudo tailscale up

Finally, install Caddy as a systemd unit (these commands are inspired from the official guide):

gcloud compute ssh --project cert-manager-general --zone=europe-west1-c print-your-cert -- bash <<'EOF'
sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy
EOF

gcloud compute ssh --project cert-manager-general --zone=europe-west1-c print-your-cert -- bash <<'EOF'
sudo tee /etc/caddy/Caddyfile <<CADDY
print-your-cert.cert-manager.io:443 {
        reverse_proxy 100.85.65.38:8080
}
CADDY
sudo systemctl restart caddy.service
EOF

Finally, you'll need to ensure that the tailscale ACLs allow traffic to the pi.

Visit the Tailscale admin panel and set the "hosts" key to include the Pi, and ensure that there's an ACL allowing traffic:

"acls": [
    {"action": "accept", "users": ["*"], "ports": ["raspberrypi:*"]},
]

Prerequisite: install k3s on the Raspberry Pi

This prerequisite is useful both for local development and for running the experiment on the Raspberry Pi.

First, install the following tools on the Raspberry Pi:

• Docker,
• K3d,
• Helm.

The first step is to create a cluster with a cert-manager issuer:

# From the Raspberry Pi:
k3d cluster create --k3s-arg="--tls-san=$(tailscale ip -4)@server:*"

Booth: Configure kubectl on your laptop to access the Raspberry Pi's cluster

First, make sure you can SSH to the Raspberry Pi over Tailscale in one of the above sections.

Then, run the following:

ssh pi 'k3d kubeconfig get -a | sed "s/0.0.0.0/$(tailscale ip -4)/g"' >/tmp/kc \
  && KUBECONFIG=/tmp/kc:$HOME/.kube/config k config view --flatten >kc \
  && mv kc ~/.kube/config

Booth: Install cert-manager and the issuers on the Raspberry Pi

First install age to be able to decrypt the secrets:

# From your laptop:
brew install age

Install cert-manager:

# From the Raspberry Pi:
helm repo add jetstack https://charts.jetstack.io --force-update
helm upgrade --install -n cert-manager cert-manager jetstack/cert-manager \
  --create-namespace --set installCRDs=true

Then, apply the ClusterIssuer:

kubectl apply -f root_issuer_prod.yaml
kubectl apply -f cluster_issuer.yaml

Then, decrypt the root CA. The passphrase is available in the Venafi 1Password in the cert-manager-team vault.

# From your laptop:
age -d root-print-your-cert-ca.yaml.age >root-print-your-cert-ca.yaml
scp root-print-your-cert-ca.yaml pi:

Finally, apply the decrypted root CA secret:

# From the Raspberry Pi:
kubectl apply -f root-print-your-cert-ca.yaml

Booth: Run the UI on the Raspberry Pi

The UI doesn't run in Kubernetes (just because...). It runs as a container. It is a simple Go binary that serves an HTML site. Its container image name is ghcr.io/cert-manager/print-your-cert-ui:latest.

The following command will build the image on your laptop (faster than on the Pi) and then load the image on the Pi:

# From your laptop:
KO_DOCKER_REPO=ghcr.io/cert-manager/print-your-cert-ui ko build . --platform linux/arm64 --tarball print-your-cert-ui.tar --push=false --bare
ssh pi docker load <print-your-cert-ui.tar

Now, ssh into the Raspberry Pi and launch the UI:

# From your laptop.
ssh pi docker rm -f print-your-cert-ui
ssh pi docker run -d --restart=always --name print-your-cert-ui --net=host \
  -v '/home/certmanager/.kube/config:/home/nonroot/.kube/config' \
  ghcr.io/cert-manager/print-your-cert-ui:latest \
  --issuer print-your-cert-ca \
  --issuer-kind ClusterIssuer \
  --listen 0.0.0.0:8080 \
  --guestbook-ca=""

Set --guestbook-ca to "" because we manually issued a Let's Encrypt certificate for the guestbook.

Note

We don't actually push the image to GHCR. We just load it directly to the Raspberry Pi.

Note

Why not skip buildx and use ko instead? That's because the base images that ko relies on don't support the Rasberry Pi's arm64/v8 architecture:

$ crane manifest cgr.dev/chainguard/static | jq -r '.manifests[].platform | "\(.os)/\(.architecture)"'
linux/amd64
linux/arm
linux/arm64
linux/ppc64le
linux/s390x

Booth: Running the printer controller on the Raspberry Pi

The printer controller is a simple Bash script (yeah, not Go). It doesn't run in Kubernetes just because it makes it easier to hot-reload everything on the booth. ghcr.io/cert-manager/print-your-cert-controller:latest is the container image name.

Make sure that the k3s cluster is running that cert-manager is installed. If not, follow the section Prerequisite: install k3s on the Raspberry Pi.

You may need to install Qemu if you are on Linux:

# From your laptop, only on Linux:
sudo apt install -y qemu qemu-user-static

Then, create a buildx builder:

docker buildx create --name mybuilder --use

Note

If it says "docker: 'buildx' is not a docker command", you may need to install buildx manually. On macOS, you can do it with the following command:

brew install docker-buildx
mkdir -p ~/.docker/cli-plugins/
ln -sfn $(brew --prefix)/opt/docker-buildx/bin/docker-buildx ~/.docker/cli-plugins/docker-buildx

Then, build the image on your desktop (faster than on the Pi) and then push it to the Pi.

# From your laptop:
docker buildx build -f Dockerfile.controller --platform linux/arm64 \
  -t ghcr.io/cert-manager/print-your-cert-controller:latest \
  -o type=docker,dest=print-your-cert-controller.tar .
ssh pi docker load <print-your-cert-controller.tar

Note

We don't push the image to GHCR. We just load it directly on the Pi.

Now, SSH into the Raspberry Pi and launch the controller:

ssh pi sudo chmod a+r ~/.kube/config
ssh pi docker rm -f print-your-cert-controller
ssh pi docker run -d --restart=always --name print-your-cert-controller --privileged -v /dev/bus/usb:/dev/bus/usb -v /home/certmanager/.kube/config:/root/.kube/config --net=host ghcr.io/cert-manager/print-your-cert-controller:latest

You can also run the "debug" printer UI (brother_ql_web) if you want to make sure that the printer works:

ssh pi docker run -d --restart=always --name brother_ql_web \
  --privileged -v /dev/bus/usb:/dev/bus/usb \
  -p 0.0.0.0:8013:8013 ghcr.io/cert-manager/print-your-cert-controller:latest brother_ql_web

Booth: Running the guestbook on a VM

The VM that runs the guestbook is managed by tofu and is defined in booth.tf.

Note

TBD: document https://litestream.io/ that we use for the sqlite backups.

First, you will need to connect to the Raspberry Pi's cluster to be able to create the guestbook certificate. You can do that by running the following:

# From your laptop:
ssh pi kubectl apply -f - --wait <guestbook/certificate.yaml
ssh pi kubectl get secret -n cert-manager root-print-your-cert-ca -ojson \
  | jq -r '.data."tls.crt" | @base64d' >ca.crt
ssh pi kubectl get secret -n cert-manager guestbook-tls -ojson \
  | jq -r '.data."tls.crt" | @base64d' >tls.crt
ssh pi kubectl get secret -n cert-manager guestbook-tls -ojson \
  | jq -r '.data."tls.key" | @base64d' >tls.key

Copy the root CA that you decrypted in one of the previous steps:

gcloud compute scp --project cert-manager-general --zone=europe-west1-c  \
  ca.crt tls.crt tls.key guestbook:.
gcloud compute ssh --project cert-manager-general --zone=europe-west1-c guestbook -- \
  sudo mkdir -p /var/guestbook
gcloud compute ssh --project cert-manager-general --zone=europe-west1-c guestbook -- \
  sudo mv ca.crt tls.crt tls.key /var/guestbook

Finally, build and push:

GOARCH=amd64 GOOS=linux go build -C guestbook .
gcloud compute scp --project cert-manager-general --zone=europe-west1-c  \
  guestbook/guestbook test:.
gcloud compute ssh --project cert-manager-general --zone=europe-west1-c guestbook -- \
  sudo install guestbook /usr/bin

Then, run the following to create the systemd service:

Note

If guestbook has never run on this machine, you will first need to run:

gcloud compute ssh --project cert-manager-general --zone=europe-west1-c guestbook -- \
  mkdir /var/guestbook
gcloud compute ssh --project cert-manager-general --zone=europe-west1-c guestbook -- \
  guestbook -init-db -db-path /var/guestbook/guestbook.sqlite

Finally, run the following to create the systemd service:

gcloud compute ssh --project cert-manager-general --zone=europe-west1-c guestbook -- bash <<'EOF'
sudo tee /usr/lib/systemd/system/guestbook.service <<'SVC'
[Unit]
Description=cert-manager Booth Guestbook
After=network.target

[Service]
ExecStart=/usr/bin/guestbook -ca-cert /var/guestbook/ca.crt -tls-chain /var/guestbook/tls.crt -tls-key /var/guestbook/tls.key -db-path /var/guestbook/guestbook.sqlite -listen :443 -readonly-listen-insecure :80 -autocert-dir /var/guestbook -prod

StandardOutput=journal
StandardError=journal
Type=simple
Restart=always

[Install]
WantedBy=multi-user.target
SVC
sudo systemctl daemon-reload
EOF

Local development

Local development on the UI

You will need Go.

First, follow the steps in Prerequisite: install k3s on the Raspberry to install k3s on your local machine (it is the same as for the Raspberry Pi).

Then, you will need to create a ClusterIssuer:

kubectl apply -f root_issuer_dev.yaml --wait
kubectl apply -f cluster_issuer.yaml --wait
kubectl get secret -n cert-manager root-print-your-cert-ca -ojson | jq -r '.data."tls.crt" | @base64d' >ca.crt

Then, you can run the UI:

go run . --issuer=print-your-cert-ca --issuer-kind=ClusterIssuer \
  -guestbook-ca=ca.crt -guestbook-url=guestbook.print-your-cert.cert-manager.io:9090

Local development on the guestbook

First, you will need to make sure the domains guestbook.print-your-cert.cert-manager.io and readonly-guestbook.print-your-cert.cert-manager.io point to your machine:

sudo perl -ni -e 'print if !/ guestbook.print-your-cert.cert-manager.io$/' /etc/hosts
sudo tee -a /etc/hosts <<<"127.0.0.1 guestbook.print-your-cert.cert-manager.io"
sudo perl -ni -e 'print if !/ readonly-guestbook.print-your-cert.cert-manager.io$/' /etc/hosts
sudo tee -a /etc/hosts <<<"127.0.0.1 readonly-guestbook.print-your-cert.cert-manager.io"

Then:

kubectl apply -f root_issuer_dev.yaml --wait
kubectl apply -f cluster_issuer.yaml --wait
kubectl apply -f guestbook/certificate.yaml --wait

Grab the root CA and the certificate to serve the guestbook:

kubectl get secret -n cert-manager root-print-your-cert-ca -ojson | jq -r '.data."tls.crt" | @base64d' >ca.crt
kubectl get secret -n cert-manager guestbook-tls -ojson | jq -r '.data."tls.crt" | @base64d' >tls.crt
kubectl get secret -n cert-manager guestbook-tls -ojson | jq -r '.data."tls.key" | @base64d' >tls.key
go run -C guestbook . -init-db
go run -C guestbook . -ca-cert ../ca.crt -tls-chain ../tls.crt -tls-key ../tls.key -listen :9090

To use the guestbook, make sure the UI is running locally too (see above).

1. Go to the UI at http://localhost:8080.
2. Submit an email.
3. Wait for the cert to be ready and click "Sign the guestbook".
4. Go to the guestbook at https://readonly-guestbook.print-your-cert.cert-manager.io:9090. To get a "star" instead of a red cross, you need to use curl (or any HTTP client) to sign the book.
5. To get a star instead of a cross, go back to the page in (1), click "Download cert bundle tar".
6. Open a shell session and go to your ~/Downloads folder.
7. Run:

   tar xf cert-manager-bundle.tar
   curl -k https://guestbook.print-your-cert.cert-manager.io/write \
     --cacert ca.crt --cert chain.pem --key pkey.pem \
     -X POST --data-urlencode message@/dev/stdin \
   <<EOF
   Excellent job, Ash!
   EOF

8. Go back to https://readonly-guestbook.print-your-cert.cert-manager.io:9090 to see the guestbook. Now, you should see a ⭐!

Local development on the controller (that creates PNGs and prints them)

The controller is made in two pieces: cert-card which produces a PNG with the label(s), and print-your-cert-controller that runs cert-card every time a certificate object in Kubernetes becomes ready.

cert-card

cert-card which produces a PNG with the label(s)

brew install imagemagick qrencode step
brew install homebrew/cask-fonts/font-open-sans
brew install homebrew/cask-fonts/font-dejavu

To run it, for example:

./cert-card <<EOF
-----BEGIN CERTIFICATE-----
MIICXDCCAgOgAwIBAgIQdPaTuGSUDeosii4dbdLBgTAKBggqhkjOPQQDAjAnMSUw
IwYDVQQDExxUaGUgY2VydC1tYW5hZ2VyIG1haW50YWluZXJzMB4XDTIyMDUxNjEz
MDkwMFoXDTIyMDgxNDEzMDkwMFowLDEqMCgGA1UEAwwhZm9vIGJhciBmb28gYmFy
IDxmb28uYmFyQGJhci5mb28+MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
AQEAtmGM5lil9Vw/y5LhpgO8t5gSb5oUo+Dp5vWw0Z5C7rjvifi0/eD9MbVFkxb+
+hmOaaNCVgqDUio1OBOZyL90KzdnGW7nz1fRM2KCNrDF5Y1mO7uv1ZTZa8cVBjF6
7KjFuNkvvHp74m65bKwXeCHXJBmO3Z1FH8hudICU74+Nl6tyjlMOsTHv+LY0jPfm
AtO6eR+Ef/HvgzwsjKds12vdlRCdHSS6u5zlrZZxF3zTO7YuAM7mN8Wbjq94Ycpg
sJ5ssNOtMu9FwZtPGQDHPaQyVQ86FfjhmMi1IUOUAAGwh/QRv8ksX+OupHTNdH06
WmIDCaGBjWFgPkwicavMZgZG3QIDAQABo0EwPzAOBgNVHQ8BAf8EBAMCBaAwDAYD
VR0TAQH/BAIwADAfBgNVHSMEGDAWgBQG5XQnDhOUa748L9H7TWZN2avluTAKBggq
hkjOPQQDAgNHADBEAiBXmyJ24PTG76pEyq6AQtCo6TXEidqJhsmK9O5WjGBw7wIg
aPbcFI5iMMgfPGEATH2AGGutZ6MlxBmwhEO7pAkqhQc=
-----END CERTIFICATE-----
EOF

Testing the printer

Test that brother_lp works over USB on Pi:

convert -size 230x30 -background white -font /usr/share/fonts/TTF/OpenSans-Regular.ttf -pointsize 25 -fill black -gravity NorthWest caption:"OK." -flatten example.png
brother_ql --model QL-820NWB --printer usb://0x04f9:0x209d print --label 62 example.png

Testing cert-card

openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -utf8 -subj "/CN=Maël Valais <[email protected]>/O=Jetstack" -reqexts v3_req -extensions v3_ca -out ca.crt
step certificate create "CN=Foo Bar <[email protected]>" foo.crt foo.key --ca ca.crt --ca-key ca.key --password-file /dev/null
cert-card <foo.crt
timg cert-card.png
read
brother_ql --model QL-820NWB --printer usb://0x04f9:0x209d print --label 62 cert-card.png

Troubleshooting

From the CLI: usb.core.USBError: [Errno 13] Access denied (insufficient permissions)

Run:

# From the Raspberry Pi.
sudo tee /etc/udev/rules.d/99-brother-ql.rules <<EOF
SUBSYSTEM=="usb", ATTR{idVendor}=="04f9", ATTR{idProduct}=="209d", MODE="0666"
EOF

Then, reload the udev rules:

# From the Raspberry Pi.
sudo udevadm trigger

Then, unplug and replug the printer.

From the CLI: usb.core.USBError: [Errno 16] Resource busy

On the Pi (over SSH), when running brother_ql with the following command:

docker run --privileged -v /dev/bus/usb:/dev/bus/usb -it --rm ghcr.io/cert-manager/print-your-cert-ui:latest brother_ql

you may hit the following message:

usb.core.USBError: [Errno 16] Resource busy

I found that two reasons lead to this message:

1. The primary reason is that libusb-1.0 is installed on the host (on the Pi, that's Debian) and needs to be removed, and replaced with libusb-0.1. You can read more about this in pyusb/pyusb#391.
2. A second reason is that the label settings aren't correct (e.g., you have select the black/red tape but the black-only tape is installed in the printer).

From the web UI: No such file or directory: '/dev/usb/lp1'

This happened when the printer was disconnected.