Deploying / installing libvirt

[Tenebresus]

Hi all,

Im currently trying to deploy OpenStack on my Talos cluster using the folllowing guide https://docs.openstack.org/openstack-helm/latest/install/openstack.html. For OpenStack to work on Kubernetes I have to deploy libvirt pods, whose jobs it is to start libvirt on the worker nodes (https://github.com/openstack/openstack-helm-infra/blob/master/libvirt/templates/bin/_libvirt.sh.tpl). This script is trying to call systemd-run libvirtd in order to start libvirt, but because talos uses it own init system (machined), this fails. Does anyone have any experience with deploying / installing libvirt on their talos cluster? If so, could you be so kind to help me out? :)

[smira]

Sorry, no experience, so just a generic answer.

Talos doesn't have systemd, so there should be some equivalent way to run libvirtd, I guess it might be a static pod, or a daemonset, or something like that.

[Tenebresus]

Hi @wvh-github, are you still available today? :)

[nitrocb]

Hi @Tenebresus @wvh-github,

I have a similar usecase here. I am trying to deploy Ironcore, which is a k8s driven IaaS Solution utilizing libvirt to build VMs on Kubernetes Nodes. We want to use it on Talos, because of ease of setup and security reasons.

The Ironcore solution does not implement libvirt itself, so I needed to build a container image including libvirt to run it as a DaemonSet, which makes the libvirt socket available on the Worker nodes to be used by Ironcore.

The problem I run into is, when I start my image including libvirt on Talos and try to run virtlogd and libvirtd, the shell just hangs and nothing happens.
When I run the container image on an ubuntu host, everything works fine and I can start VMs.

As you seem to have a bit of experience with libvirt on Talos linux, can you give me a hint, if there are some pre-requisites on Talos Linux, which I need to configure to be able to run libvirt on it?

Thanks a lot in advance!

Best regards

[wvh-github]

Hi @nitrocb ,
We are currently working with @Tenebresus on their specific use case through other means.

I'm sure I can help, but I'm not yet familiar with IronCore. If Ironcore expects libvirt to run on the host, there is no real way of doing that at the moment. I also think there is no good way of doing that, and you shouldn't need to really.

Are you per chance sharing any filesystem locations with the host etc? Is the DaemonSet running priviliged? How about the PID space, is that host based or not?

[nitrocb]

Hi @wvh-github ,

thanks for your answer!

Iconcore does only expect to have the libvirt socket available somewhere, where it can access it. So the idea would be to run libvirt in a pod and create the socket on a hostpath mount, where it can also be accessed by the ironcore pod.

I currently only share /run/dbus/system_bus_socket and /dev/kvm with the host. But I also removed those mounts just to check if this causes the problem.

The DaemonSet is running privileged and is using the host network.

hostPID is not set to true, so I guess the default will be false. I could try setting that to true.

I can not start virtlogd and libvirtd as they only hang and do not output or log anything.
I already created an strace, which I am currently analyzing.

[sjabasti]

Hi @wvh-github @Tenebresus

We are in the exact same situation, openstack-helm on talos and libvirtd write mounting /etc/libvirt and the script requiring systemd-run. How did you solve these problems/running libvirtd inside a container?

Thanks in advance

[wvh-github]

I'm going to get back to you on this. Sharing dbus will confuse libvirt as it has hardcoded things it will try to do that might not actually work. We are not sharing those.
I'm going to check how I can share something workable for you.

[nitrocb]

Thanks a lot, really appreciate it!

[nitrocb]

Hi again @wvh-github,

your hint with the dbus socket mount from the host was indeed helpful.
I removed the hostpath mount and startet a dbus-daemon inside the container, which brought libvirt to work in the end.
I was already able to create a test-VM 😊
I am stil struggeling with the network part (Bridged networking, ipv6 vlan, custom mtu), but I'll figure that out.

Thanks a lot!

Best regards

[wvh-github]

Thanks for the feedback @nitrocb.