Extend template to install packages from lockfiles

[ruffsl]

Extend template to install packages from lockfiles in an another attempt to fix:

• [ros] Provide exact version numbers to ensure rebuilds docker_images#112

---

This PR modifies the ROS Dockerfile templates to install apt packages from a copied text file. Importantly, the copy operation locates this file by checking the TARGETARCH, an automatic platform ARG in the global scope, supported with buildkit:

https://docs.docker.com/engine/reference/builder/#automatic-platform-args-in-the-global-scope

Lastly, these lockfiles also include the full package version identifier, now allowing for our automated CI over on osrf/docker_images to detect minor package updates and syncs, update these lockfiles via automated PRs, and subsequently trigger rebuilds upstream via controlled and selective busting of the docker build cache in the Docker Official Images

[ruffsl]

@mikaelarguedas , I'm finding it a bit of a nuisance that not all meta packages exist for all "supported" platform architectures.

~/git/osrf/docker_images/ros$ ./create_dockerfolders.py dir -a -d .
Generating Dockerfile 'melodic/ubuntu/bionic/ros-core/Dockerfile':
Generating Dockerfile 'melodic/ubuntu/bionic/ros-base/Dockerfile':
Generating Dockerfile 'melodic/ubuntu/bionic/robot/Dockerfile':
Generating Dockerfile 'melodic/ubuntu/bionic/perception/Dockerfile':
Generating Dockerfile 'melodic/ubuntu/bionic/desktop/Dockerfile':
Generating Dockerfile 'melodic/ubuntu/bionic/desktop-full/Dockerfile':
Generating Dockerfile 'noetic/ubuntu/focal/ros-core/Dockerfile':
Generating Dockerfile 'noetic/ubuntu/focal/ros-base/Dockerfile':
Generating Dockerfile 'noetic/ubuntu/focal/robot/Dockerfile':
Generating Dockerfile 'noetic/ubuntu/focal/perception/Dockerfile':
Generating Dockerfile 'noetic/ubuntu/focal/desktop/Dockerfile':
Traceback (most recent call last):
  File "/home/ruffsl/git/osrf/docker_images/ros/./create_dockerfolders.py", line 31, in <module>
    main()
  File "/home/ruffsl/git/osrf/docker_images/ros/./create_dockerfolders.py", line 27, in main
    populate_paths(manifest, args, create_dockerfiles)
  File "/home/ruffsl/git/osrf/docker_templates/docker_templates/folders.py", line 73, in populate_paths
    create_dockerfiles.main(('dir', '-d' + dockerfolder_dir))
  File "/home/ruffsl/git/osrf/docker_images/ros/create_dockerfiles.py", line 58, in main
    expandPackages(data)
  File "/home/ruffsl/git/osrf/docker_templates/docker_templates/packages.py", line 169, in expandPackages
    package_versions = getPackageVersions(data, package_index, data[package_type], package_type)
  File "/home/ruffsl/git/osrf/docker_templates/docker_templates/packages.py", line 126, in getPackageVersions
    package_info = getPackageInfo(package_pattern, package_index)
  File "/home/ruffsl/git/osrf/docker_templates/docker_templates/packages.py", line 91, in getPackageInfo
    package_info = matchs.group(0)
AttributeError: 'NoneType' object has no attribute 'group'

The current template config is not too accommodating for missing packages, as the list of supported platform architectures is indexed by distro, rather than by distro tag. Any suggestions? Would you like to add a few commits here as well?

[mikaelarguedas]

Cool initiative!

update these lockfiles via automated PRs

This will likely require adressing osrf/docker_images#530 to be effective

and subsequently trigger rebuilds upstream via controlled and selective busting of the docker build cache in the Docker Official Images

Just to confirm, the docker farm will have no way to know that the file changed automatically if a job is not triggered. So we would open a PR upstream every time we modify one of these lockfils and that will be what will trigger the rebuilds is that right ?

The current template config is not too accommodating for missing packages, as the list of supported platform architectures is indexed by distro, rather than by distro tag. Any suggestions? Would you like to add a few commits here as well?

Yeah.. Unfortunately I don't know of a simple machine readable way to get this info. Only the buildfarm config knows about them and it doesnt seem convenient to retrieve

Our best shot would be to hack the docker templates python to pass and not crash in case of unfound packages. But this comes with the drawback that then we wont get notified if packages that "should be there" are not found

[mikaelarguedas]

One way of skipping packages if not found: 4ed6033. We'll need to test in details what the implications are, we may end up with a building dockerfile but without the package in it that would mislead users if not caught by us ahead of time

Also added 4b089c8 to support imaged with both ROS1 and ROS2 packages (ros1_bridge)

[ruffsl]

Just to confirm, the docker farm will have no way to know that the file changed automatically if a job is not triggered. So we would open a PR upstream every time we modify one of these lockfils and that will be what will trigger the rebuilds is that right ?

Yea, although I commented over here on how we could use some GitHub actions to automate opening up the PR upstream:

• Automatic PR generation fails to push docker_images#530 (comment)

A question would be on how to connect the two automated PR pipelines. I was thinking of:

• scheduled workflow to update lockfiles
• triggered PR opened to osrf/docker_images
• blocking for manual PR approval
• approved PR merged at osrf/docker_images
• triggered PR opened to ros-infrastructure/official-images
• blocking for manual PR approval
• approved PR merged at ros-infrastructure/official-images
• triggered PR opened to docker-library/official-images

We could simplify this by shorcuruting the causal chain to creating PRs upstream, but I was thinking that the manual blocks would help avoid spamming the librarians incase we needed to manually intervene and make several commits to manifest files directly.

[ruffsl]

Our best shot would be to hack the docker templates python to pass and not crash in case of unfound packages. But this comes with the drawback that then we wont get notified if packages that "should be there" are not found

I added 90aac73 to avoid writing to the lockfile if there are no packages listed for the platform architecture. This would ensure the same Dockerfile behavior, i.e. build time failure, for such cases:

• if the lockfile wasn't updated: the old package version identifier is used, and the build fails when pull from the apt repo
• if the lockfile never existed: the build fails when attempting to copy the lockfile for that platform architecture

There would still be the corner case for lockfiles with multiple packages then simply missing the package(s) that are also missing from the apt repo, but still allowing the Dockerfile build to succeed. Grr...

[mikaelarguedas]

lgtm,

Independant of the automated PR, could we give a shot to merge this and corresponding docker_images PR to submit to the librarian and see is that's an acceptable way to go for them ?

[ruffsl]

Independant of the automated PR, could we give a shot to merge this and corresponding docker_images PR to submit to the librarian and see is that's an acceptable way to go for them ?

I've just asked for their feedback over here, as I'd like to avoid reverting this many commits should this also get rebuffed.

• [ros] Provide exact version numbers to ensure rebuilds docker_images#112 (comment)