systemd[1]: bpf-lsm: Failed to load BPF object: No such process

[jm355]

When I use linux-hardened on arch linux, the following error gets logged: Mar 01 08:12:46 linux systemd[1]: bpf-lsm: Failed to load BPF object: No such process
Searching the logs for "bpf", I see:

Mar 01 08:12:27 archlinux kernel: Linux version 6.7.6-hardened1-1-hardened (linux-hardened@archlinux) (gcc (GCC) 13.2.1 20230801, GNU ld (GNU Binutils) 2.42.0) #1 SMP PREEMPT_DYNAMIC Sat, 24 Feb 2024 23:47:28 +0000
...
Mar 01 08:12:27 archlinux kernel: LSM: initializing lsm=capability,landlock,lockdown,yama,bpf,integrity
...
Mar 01 08:12:27 archlinux kernel: hid_bpf: error while preloading HID BPF dispatcher: -22
...
Mar 01 08:12:46 linux systemd[1]: bpf-lsm: Failed to load BPF object: No such process

Booting with the standard linux kernel, when I search the logs for "bpf" I find:

Feb 29 20:00:06 archlinux kernel: Linux version 6.7.6-arch1-2 (linux@archlinux) (gcc (GCC) 13.2.1 20230801, GNU ld (GNU Binutils) 2.42.0) #1 SMP PREEMPT_DYNAMIC Thu, 29 Feb 2024 19:08:51 +0000
...
Feb 29 20:00:06 archlinux kernel: LSM: initializing lsm=capability,landlock,lockdown,yama,bpf,integrity
...
Feb 29 20:00:45 linux systemd[1]: bpf-lsm: LSM BPF program attached

I found some discussion mentioning that linux-hardened had issues with bpf here systemd/systemd#27705. @RA-Kooi, you mentioned you'd look into why that is. Did you find anything?

[RA-Kooi]

I haven't taken the time to look into that yet. I'm also running into this on my custom kernel config, which is a zen-kernel with hardened patches applied. I'm having a feeling it's the case of an unset kernel config option.
Looking at a regular kernel setup on Arch I notice that CONFIG_BPF_LIRC_MODE2=y and CONFIG_BPF_KPROBE_OVERRIDE=y are set whereas they are unset on a hardened kernel.
Last time I grepped the hardened patch I didn't notice any structural changes that would break this. Unless it so happens that bpf_jit_harden = 2 triggers a broken code path that nobody knew is broken. But I think that that is rather unlikely, so my first step would be to set those config parameters and see if it changes anything.

Next time I'll build my kernel I'll make sure to set those BPF options and report back if needed.

[jm355]

Another potential clue: under https://wiki.archlinux.org/title/Security#Kernel_lockdown_mode it says:

integrity: kernel features that allow userland to modify the running kernel are disabled (kexec, bpf).

i don't think integrity mode is set by default though, when I saw that I was running linux-hardened and integrity mode was off

[RA-Kooi]

It is not, but it would explain why it's not working on my machine, since I use lockdown=confidentiality.

[ImagineRats]

While I cannot specifically recall what if any other changes elsewhere may result in similar breakage the main issue is most likely your config having CONFIG_BPF_LSM disabled; the intention behind why this would be in your config is probably not due to any opinion on that option specifically but because it depends on these additional settings:

CONFIG_FTRACE=y
(CONFIG_KPROBES=y && CONFIG_KPROBE_EVENTS=y) || (CONFIG_UPROBES=y && CONFIG_UPROBE_EVENTS=y)

As indicated by your logs this is something SystemD wants but does not explicitly need, that's because it's used to implement the RestrictFileSystems feature in service units. Last I understood SystemD's dependence on BPF_LSM this was limited to that relatively minor feature. So long as other necessary settings are used all other security relevant service features depending on BPF will function properly without BPF_LSM or the aforementioned settings on which it depends.

That said unlike some other situations these will be silent failures should they occur so I would suggest confirming their functionality with your current configuration via simple service units if this is of particular concern to you.

For reference this document can give you some idea of what kernel functionality SystemD depends on, though as is often the case with such things it's likely not all encompassing:

https://github.com/systemd/systemd/blob/main/README

[jm355]

Interesting, that's good to know. Even if RestrictFileSystems is a relatively minor feature, it seems like linux-hardened would want to enable it if it can improve hardening? Especially considering that the default kernel enables it, unless there's some other tradeoff I'm not aware of?

[ImagineRats]

Unfortunately it's complicated and I'm woefully unfit to offer any meaningful insight on the hardening significance of those changes which will vary depending on your particular threat model. Regardless these are things you'll see recommended against commonly in various places, particularly based on the references used by this project:

https://github.com/a13xp0p0v/kernel-hardening-checker

While this is unrelated to SystemD here are some discussions that center around some of the aforementioned dependencies of BPF_LSM:

lkrg-org/lkrg#27 (comment)
GrapheneOS/os-issue-tracker#408 (comment)

Aside from that though this isn't a bug/issue in the linux-hardened project itself as it's not responsible for the configuration you're using, if you believe the decision to not support BPF_LSM by the maintainer of your system's linux-hardened package should be reconsidered that's something you'd have to handle via your distribution's communication channels for such things.

[jm355]

Ah I see, I'll close this ticket and talk to the maintainer. Thanks for the insight!

[jm355]

Looks like the packager for arch linux is @anthraxx of this repo, so I'll ping them here. @anthraxx, what do you think about this?

[RA-Kooi]

So I recently updated my kernel and config and did a bit of testing.
I did not enable CONFIG_BPF_LIRC_MODE2 because it's for lirc devices, which I do not have.

I did enable the following:

CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_KPROBES=y
CONFIG_OPTPROBES=y
CONFIG_KPROBES_ON_FTRACE=y
CONFIG_UPROBES=y
CONFIG_KRETPROBES=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_KPROBES_ON_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS=y
CONFIG_HAVE_DYNAMIC_FTRACE_NO_PATCHABLE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_FTRACE=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=y
CONFIG_DYNAMIC_FTRACE_WITH_ARGS=y
CONFIG_FTRACE_MCOUNT_RECORD=y
CONFIG_FTRACE_MCOUNT_USE_CC=y
CONFIG_HAVE_SAMPLE_FTRACE_DIRECT=y
CONFIG_HAVE_SAMPLE_FTRACE_DIRECT_MULTI=y

And I also disabled kernel lockdown. And in this case systemd still gives me the error that there's no such process.
I did however use kernel hardening checker to check my config, where the majority of options are enabled, so that may have something to do with it. I think we'd do well to open a thread on the systemd repo to ask what the incompatibility could be.

[jm355]

What about CONFIG_BPF_LSM?

[RA-Kooi]

CONFIG_BPF_LSM has been enabled in my config from the start.

[ImagineRats]

So I recently updated my kernel and config and did a bit of testing. I did not enable CONFIG_BPF_LIRC_MODE2 because it's for lirc devices, which I do not have.

I did enable the following:

CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_KPROBES=y
CONFIG_OPTPROBES=y
CONFIG_KPROBES_ON_FTRACE=y
CONFIG_UPROBES=y
CONFIG_KRETPROBES=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_KPROBES_ON_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS=y
CONFIG_HAVE_DYNAMIC_FTRACE_NO_PATCHABLE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_FTRACE=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=y
CONFIG_DYNAMIC_FTRACE_WITH_ARGS=y
CONFIG_FTRACE_MCOUNT_RECORD=y
CONFIG_FTRACE_MCOUNT_USE_CC=y
CONFIG_HAVE_SAMPLE_FTRACE_DIRECT=y
CONFIG_HAVE_SAMPLE_FTRACE_DIRECT_MULTI=y

And I also disabled kernel lockdown. And in this case systemd still gives me the error that there's no such process. I did however use kernel hardening checker to check my config, where the majority of options are enabled, so that may have something to do with it. I think we'd do well to open a thread on the systemd repo to ask what the incompatibility could be.

Have you first confirmed that this is not also an issue when using the same kernel parameters (sysctl+cmdline) on your distro kernel, i.e. not linux-hardened? Your problem is most likely due to enforcing a lockdown mode considering relevant documentation suggestions and that the same message appears on machines of my own which are not running linux-hardened, but do include additional parameter hardening.

Whether this results in bpf-lsm being unsupported is not something I've tested though doing so would require only a simple service unit to confirm. It's unlikely this issue is related to bpf-lsm being broken due to linux-hardened's patchset.

[RA-Kooi]

I don't use lockdown on non-hardened kernels, so I haven't tested that configuration. I checked my sysctl list for anything that might be interfering with it, but no dice.

I might be able to test an Arch VM with a regular kernel later and enabling lockdown, but to be honest my level of care for this is rapidly dwindling since I have more pressing issues to get to. (Like my kernel panicing very often with GVT-g.)

If there's any interest in my current kernel config I can post it, but it's very specific to my own machine.

[ImagineRats]

Okay, upon briefly searching that error myself while I didn't see anything like a clearly relevant solution the following discussion makes mention of the error in passing, and reminded me of a time when I was seeking to source/resolve syslog messages which may have included yours:

https://bugzilla.redhat.com/show_bug.cgi?id=2084955

The issue was due to initramfs generation failing to include the library file "libbpf.so" (or whatever your distro's versioned name of the library is) based on some set of circumstances. If you check the files in your initramfs do you see that such a file is included? Exactly what the impact of missing this during that stage of the boot process would have is uncertain to me, but I'd assume things would work fine afterwards assuming there is no other conflict.

In my case with Fedora + dracut whatever the complaint I'd been looking at was resolved by installing the "dracut-network" package which happened to result in pulling in that library during initramfs generation, there are of course a number of ways to approach this depending on your preferences.

Should this not be the cause of your issue then yes, attachments of any custom kernel parameters unless you mean you tested these by disabling them and booting with a freshly generated initramfs, as well as the configuration itself would greatly aid in preventing a game of twenty questions between yourself and anyone with a guess on the likely cause.

[jm355]

For what it's worth, adding /usr/lib/libbpf.so to BINARIES or FILES in mkinitcpio.conf didn't work. lsinitcpio showed them, but the error persisted (they weren't there before adding them to BINARIES/FILES)

[ImagineRats]

If you haven't I would ensure that you include every reference to be sure, meaning both 32bit and 64bit versions, the symlinks, and actual files. For example on my current system which has no multilib support this is:

/usr/lib64/libbpf.so.1 -> libbpf.so.1.1.0
/usr/lib64/libbpf.so.1.1.0

Honestly I'm a bit surprised that file exists with such a name on your system, if you confirmed that it does. I was not intending that as something to follow exactly, but to check your lib+lib64 directories for what the actual names would be as this will vary between distros, and when the relevant package was last updated.

[ImagineRats]

As you'd previously said you were on Arch, according to the latest package sources for libbpf following my suggestion would require:

/usr/lib/libbpf.so
/usr/lib/libbpf.so.1
/usr/lib/libbpf.so.1.3.0
/usr/lib32/libbpf.so
/usr/lib32/libbpf.so.1
/usr/lib32/libbpf.so.1.3.0

That is assuming your system is fully updated as well as both the libbpf, and lib32-libbpf packages are installed - if you're on a multilib system, otherwise I'm not implying lib32-libbpf should be necessary. This can likely be narrowed down if it solves the issue, though you should start with everything to see if there's any difference.

https://archlinux.org/packages/core/x86_64/libbpf/files/
https://archlinux.org/packages/multilib/x86_64/lib32-libbpf/files/

If there's no improvement I'll test what is causing the message on my non linux-hardened systems and report back what I found.

[jm355]

Yep, it automatically brought in the symlinked versions, I don't have the lib32 version installed

[ImagineRats]

That's unfortunate, I don't see any reason to believe installing lib32-libbpf would change anything. I'll report back my findings when time allows, so in the next 24-48hrs I should likely have something to say again.

[ImagineRats]

So unfortunately I have relatively little to add from the point of a non linux-hardened system.

Starting from a base line of the test system these were the active custom parameter changes, excluding only sysctl's for networking by which I mean those that exist under either net/ipv4, and net/ipv6 on an updated Fedora 39 installation.

[Kernel parameters] - sysctl's:

abi.vsyscall32                   = 0
dev.tty.ldisc_autoload           = 0
fs.protected_fifos               = 2
fs.protected_regular             = 2
fs.suid_dumpable                 = 0
kernel.core_pattern              =
kernel.core_uses_pid             = 0
kernel.dmesg_restrict            = 1
kernel.io_uring_disabled         = 2
kernel.kexec_load_disabled       = 1
kernel.kptr_restrict             = 2
kernel.panic                     = -1
kernel.panic_on_oops             = 1
kernel.sysrq                     = 0
kernel.unprivileged_bpf_disabled = 1
kernel.yama.ptrace_scope         = 1
net.core.bpf_jit_harden          = 2
vm.memfd_noexec                  = 1
vm.mmap_rnd_bits                 = 32
vm.mmap_rnd_compat_bits          = 16

[Kernel parameters] - cmdline:

audit=1
audit_backlog_limit=8192
debugfs=off
efi=disable_early_pci_dma
hibernate=no
ia32_emulation=0
init_on_alloc=1
init_on_free=1
intel_iommu=on
iommu=force
iommu.strict=1
l1d_flush=on
l1tf=full,force
lockdown=confidentiality
mitigations=auto,nosmt
module.sig_enforce
nosmt
page_alloc.shuffle=1
pti=on
spec_store_bypass_disable=on
spectre_v2=on
vsyscall=none

On this system with only some clearly unrelated, or duplicate messages removed these were what appeared when running journalctl --boot=-0 --grep=bpf:

kernel: LSM: initializing lsm=lockdown,capability,yama,selinux,bpf,landlock,integrity
kernel: LSM support for eBPF active
kernel: hid_bpf: error while preloading HID BPF dispatcher: -22
systemd[1]: systemd 254.10-1.fc39 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
kernel: Lockdown: systemd: use of bpf to read kernel RAM is restricted; see man kernel_lockdown.7
systemd[1]: bpf-lsm: Failed to load BPF object: Invalid argument

Installing "dracut-network" as referenced previously actually did nothing more than add an additional, earlier "bpf-lsm: Failed to load BPF object: Invalid argument" message to the syslog under any circumstances without the following lockdown changes despite including libbpf libraries which weren't there otherwise.

In the end the only change which produced the desired results was either not using lockdown mode at all, or using lockdown=integrity rather than lockdown=confidentiality. In both of those cases the resulting grep of journalctl produced these messages:

kernel: LSM: initializing lsm=lockdown,capability,yama,selinux,bpf,landlock,integrity
kernel: LSM support for eBPF active
systemd[1]: systemd 254.10-1.fc39 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
systemd[1]: bpf-lsm: LSM BPF program attached

(when dracut-network was active in these scenarios it simply produced an additional "bpf-lsm: LSM BPF program attached")

So suffice it to say that if upon a confirmation of cat /sys/kernel/security/lockdown showing either "none", or "integrity" enclosed in brackets, the system's current boot log still shows these errors, and a distro kernel build using the same parameters currently in use does not also produce these failure indications then the answer is all but guaranteed to be within the kernel configuration - though I still very much doubt this has anything to do with the patchset itself.

If the kernel configuration possesses the settings I referenced here: #93 (comment) - thus far despite what was shared previously I have not seen a confirmation that either of the *_EVENTS=y options were active, then I'm not sure what the cause is liable to be. Preferably confirmation of these options would be done using the active kernel via /proc/config.gz, or extracted from vmlinuz using the extract-ikconfig script in the source tree to avoid any potential uncertainty surrounding what settings are indeed being utilized; both of those features will require their relevant kconfig options to be enabled to function.

For reference a number of other commonly recommended parameter hardening settings are active by default in the current Fedora 39 kernel and therefore not explicitly included in my changes, but no more likely to be the source of this issue.

[RA-Kooi]

For what it's worth, adding /usr/lib/libbpf.so to BINARIES or FILES in mkinitcpio.conf didn't work. lsinitcpio showed them, but the error persisted (they weren't there before adding them to BINARIES/FILES)

This didn't work for me either.

the answer is all but guaranteed to be within the kernel configuration

Nit: Please don't use "all but" as it is a confusing idiom (even to some native speakers) and can easily be misconstrued to mean the opposite.

thus far despite what was shared previously I have not seen a confirmation that either of the *_EVENTS=y options were active, then I'm not sure what the cause is liable to be.

From a vanilla linux-hardened (6.6.13) kernel config on Arch:

$ zcat /proc/config.gz | grep -i probe
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_ARCH_SUPPORTS_UPROBES=y
# CONFIG_ARCH_MEMORY_PROBE is not set
CONFIG_KPROBES=y
CONFIG_OPTPROBES=y
CONFIG_KPROBES_ON_FTRACE=y
CONFIG_UPROBES=y
CONFIG_KRETPROBES=y
CONFIG_KRETPROBE_ON_RETHOOK=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_KPROBES_ON_FTRACE=y
CONFIG_ARCH_CORRECT_STACKTRACE_ON_KRETPROBE=y
CONFIG_MODPROBE_PATH="/sbin/modprobe"
# CONFIG_TEST_ASYNC_DRIVER_PROBE is not set
CONFIG_GENERIC_CPU_AUTOPROBE=y
# CONFIG_MTD_JEDECPROBE is not set
CONFIG_DRM_I915_FORCE_PROBE="*"
CONFIG_SND_SOC_INTEL_AVS_MACH_PROBE=m
CONFIG_SND_SOC_SOF_DEBUG_PROBES=m
CONFIG_SND_SOC_SOF_PROBE_WORK_QUEUE=y
CONFIG_SND_SOC_SOF_ACP_PROBES=m
CONFIG_SND_SOC_SOF_HDA_PROBES=m
CONFIG_FPROBE=y
CONFIG_FPROBE_EVENTS=y
CONFIG_KPROBE_EVENTS=y
# CONFIG_KPROBE_EVENTS_ON_NOTRACE is not set
CONFIG_UPROBE_EVENTS=y
CONFIG_PROBE_EVENTS=y
# CONFIG_BPF_KPROBE_OVERRIDE is not set
# CONFIG_KPROBE_EVENT_GEN_TEST is not set

$ cat /sys/kernel/security/lockdown
[none] integrity confidentiality

$ cat /sys/kernel/security/lsm
capability,landlock,lockdown,yama,bpf

$ cat /proc/cmdline
pti=on page_alloc.shuffle=1 rd.luks.name=cab11e14-fe9a-42ab-a72d-642772143447=root root=/dev/mapper/root rw intel_iommu=on

Maybe relevant sysctl settings:

kernel.unprivileged_bpf_disabled=1
net.core.bpf_jit_harden=2