ostree-prepare-root: composefs fails to boot latest scos image

[Prashanth684]

Attached the full secondboot journal.
journal.txt

The specific error is:

Dec 09 17:13:23 localhost systemd[1]: Mounted /sysroot.
Dec 09 17:13:23 localhost systemd[1]: Starting OSTree Prepare OS/...
Dec 09 17:13:23 localhost ostree-prepare-root[675]: Resolved OSTree target to: /sysroot/ostree/deploy/fedora-coreos/deploy/8bd789c8d973d9910e8cc1fc477dd20ac5e975e0663085eeb06a95bfcdba5199.0
Dec 09 17:13:23 localhost ostree-prepare-root[675]: sysroot.readonly configuration value: 1 (fs writable: 1)
Dec 09 17:13:23 localhost ostree-prepare-root[675]: ostree-prepare-root: composefs: failed to mount: No such file or directory
Dec 09 17:13:23 localhost systemd[1]: ostree-prepare-root.service: Main process exited, code=exited, status=1/FAILURE
Dec 09 17:13:23 localhost systemd[1]: ostree-prepare-root.service: Failed with result 'exit-code'.
Dec 09 17:13:23 localhost systemd[1]: Failed to start OSTree Prepare OS/.

[jlebon]

This doesn't look like the first boot of the machine. Which AMI is this BTW? Is this from the CoreOS pipeline, or somewhere else? Let's try to reproduce it outside of OpenShift (presumably by booting the AMI, doing an rpm-ostree rebase or bootc switch and rebooting?).

[Prashanth684]

Yes this is not the first boot. This log is from the e2e test that is run on OKD release nightlies, where the Openshift installer boots the machine with fcos AMis and then on second boot, pivots to scos.

[Prashanth684]

When i tried booting with the AMI that the Openshift installer uses - 39.20231101.3.0, I see this issue. But when booting with a newer AMI - 40.20240602.3.0, I don't see the issue anymore and the rebase works. openshift/installer#8640 updates the AMI for fcos and also tests the okd scos e2e.