Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RUN bind mount not cached when using extra build context #5560

Open
djhoese opened this issue Jun 2, 2024 · 5 comments
Open

RUN bind mount not cached when using extra build context #5560

djhoese opened this issue Jun 2, 2024 · 5 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. stale-issue

Comments

@djhoese
Copy link

djhoese commented Jun 2, 2024

Issue Description

I have a Containerfile that includes a RUN with a bind mount to a file from a second build context specified using --build-context mycon=my_dir to the podman build command. No matter what I do this step is never reused/cached and is always rerun when building. My understanding is that this shouldn't be the case.

If I use the default build context and don't specify a from= in my RUN mount then it works fine. However even if I specify an additional build context that is the base context it fails (see below).

Steps to reproduce the issue

Steps to reproduce the issue

  1. Create directory test_mounting and change to it.

  2. Create test_mounting.Containerfile with content below.

  3. Run:

    echo "TEST" >> test.txt
  4. Run

    podman build --log-level info -t test-mounting:latest --build-context mycon=. -f test_mounting.Containerfile .

Containerfile

FROM centos:7 AS compilers

WORKDIR /work

# prove that some level of caching is working
RUN echo "Current directory: ${PWD}" && ls -lR .

#COPY --from=ccap test.txt new_test.txt

RUN --mount=type=bind,source=test.txt,target=test.txt,from=mycon stat -c '%a %u %U %g %G %W %X %Y %Z' test.txt

RUN ls -la /work && echo "DONE" >> a_file.txt

The non-mounted RUN calls are just to show that things are run and cached or not cached.

Describe the results you received

INFO[0000] podman filtering at log level info
INFO[0000] Using boltdb as database backend
INFO[0000] Setting parallel job count to 121
STEP 1/5: FROM centos:7 AS compilers
STEP 2/5: WORKDIR /work
--> Using cache 288eadf77cb754531263c358b1ca080d8a717c8baf6ef1093bd2edc8c89a4bd1
--> 288eadf77cb7
STEP 3/5: RUN echo "Current directory: ${PWD}" && ls -lR .
--> Using cache 1eb8be4dfbc26c8e66a5ed0ce37b1ae661f845c9febb6a057e9ba507d86d60a0
--> 1eb8be4dfbc2
STEP 4/5: RUN --mount=type=bind,source=test.txt,target=test.txt,from=mycon stat -c '%a %u %U %g %G %W %X %Y %Z' test.txt
644 0 root 0 root 0 1717293066 1717293066 1717293066
--> 339f29b8e127
STEP 5/5: RUN ls -la /work && echo "DONE" >> a_file.txt
total 0
drwxr-xr-x 1 root root  6 Jun  2 01:51 .
dr-xr-xr-x 1 root root 17 Jun  2 01:51 ..
COMMIT test-mounting:latest
--> 8c5f5e0f331e
Successfully tagged localhost/test-mounting:latest
8c5f5e0f331e982352fb257acccae7c6e1e426f911201a483ffa5efca95815e5

Describe the results you expected

I expected the RUN with the mount to be cached. In my real world case my extra build context is not the base context and I'm using --cache-to and --cache-from. The step right after this mount RUN is a lot of compiling so it is very noticeable when it isn't cached.

podman info output

host:
  arch: amd64
  buildahVersion: 1.33.7
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.el9.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: 3ea3d7f99779af0fcd69ec16c211a7dc3b4efb60'
  cpuUtilization:
    idlePercent: 99.69
    systemPercent: 0.08
    userPercent: 0.23
  cpus: 40
  databaseBackend: boltdb
  distribution:
    distribution: rocky
    version: "9.4"
  eventLogger: file
  freeLocks: 2046
  hostname: bumi.ssec.wisc.edu
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 700
      size: 1
    - container_id: 1
      host_id: 1672864
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 53807
      size: 1
    - container_id: 1
      host_id: 1672864
      size: 65536
  kernel: 5.14.0-284.11.1.el9_2.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 12738236416
  memTotal: 135071936512
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-3.el9_4.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-1.el9.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.14.3-1.el9.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.3
      commit: 1961d211ba98f532ea52d2e80f4c20359f241a98
      rundir: /run/user/53807/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: false
    path: /run/user/53807/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.3-1.el9.x86_64
    version: |-
      slirp4netns version 1.2.3
      commit: c22fde291bb35b354e6ca44d13be181c76a0a432
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 4287033344
  swapTotal: 8002727936
  uptime: 5394h 38m 55.00s (Approximately 224.75 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/davidh/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/davidh/.local/share/containers/storage
  graphRootAllocated: 214006284288
  graphRootUsed: 176895717376
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 36
  runRoot: /run/user/53807/containers
  transientStore: false
  volumePath: /home/davidh/.local/share/containers/storage/volumes
version:
  APIVersion: 4.9.4-rhel
  Built: 1714526144
  BuiltTime: Wed May  1 01:15:44 2024
  GitCommit: ""
  GoVersion: go1.21.9 (Red Hat 1.21.9-2.el9_4)
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.4-rhel

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

$ uname -a
Linux bumi.ssec.wisc.edu 5.14.0-284.11.1.el9_2.x86_64 containers/podman#1 SMP PREEMPT_DYNAMIC Tue May 9 17:09:15 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
$ rpm -q podman
podman-4.9.4-3.el9_4.x86_64

Additional information

I originally noticed this in a local gitlab CI runner, but narrowed it down to the above example.

Side question: Does file modification time invalidate the layer cache for a COPY and/or bind mount?

@djhoese djhoese added the kind/bug Categorizes issue or PR as related to a bug. label Jun 2, 2024
@Luap99 Luap99 transferred this issue from containers/podman Jun 3, 2024
@djhoese
Copy link
Author

djhoese commented Jun 11, 2024

I'm unfamiliar with this code base, but if this is expected to work but no one has the time to debug it I'd be grateful if someone could point me to a good starting location in the code. If I find the time maybe I can at least shorten the overall time it takes for this ticket to get resolved.

@rhatdan
Copy link
Member

rhatdan commented Jun 11, 2024

If it works with a docker build then it should work with buildah. Most likely there is no one able to work on this now.

@djhoese
Copy link
Author

djhoese commented Jun 12, 2024

Hm I thought I had tested this with docker, but it looks like you're right @rhatdan. This behaves the same with docker:

$ docker build -t test-mounting:latest --build-context mycon=. -f test_mounting.Containerfile .
[+] Building 0.9s (11/11) FINISHED                                                                                                                                                                                           docker:default
 => [internal] load .dockerignore                                                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                                                        0.0s
 => [internal] load build definition from test_mounting.Containerfile                                                                                                                                                                  0.0s
 => => transferring dockerfile: 395B                                                                                                                                                                                                   0.0s
 => [internal] load metadata for docker.io/library/centos:7                                                                                                                                                                            0.0s
 => [context mycon] load .dockerignore                                                                                                                                                                                                 0.0s
 => => transferring mycon: 2B                                                                                                                                                                                                          0.0s
 => [compilers 1/5] FROM docker.io/library/centos:7                                                                                                                                                                                    0.0s
 => [context mycon] load from client                                                                                                                                                                                                   0.0s
 => => transferring mycon: 437B                                                                                                                                                                                                        0.0s
 => CACHED [compilers 2/5] WORKDIR /work                                                                                                                                                                                               0.0s
 => CACHED [compilers 3/5] RUN echo "Current directory: ${PWD}" && ls -lR .                                                                                                                                                            0.0s
 => [compilers 4/5] RUN --mount=type=bind,source=test.txt,target=test.txt,from=mycon stat -c '%a %u %U %g %G %W %X %Y %Z' test.txt                                                                                                     0.3s
 => [compilers 5/5] RUN ls -la /work && echo "DONE" >> a_file.txt                                                                                                                                                                      0.5s
 => exporting to image                                                                                                                                                                                                                 0.1s
 => => exporting layers                                                                                                                                                                                                                0.1s
 => => writing image sha256:6824f59b7c8a214595432afecfcde008e1ebf0bd715eb58814bf3a385f238bfd                                                                                                                                           0.0s
 => => naming to docker.io/library/test-mounting:latest 

Note no CACHE line for the RUN --mount. If I remove the Containerfile's from=mycon portion and use the base context then I see the CACHE line:

 => CACHED [compilers 2/5] WORKDIR /work                                                                                                                                                                                               0.0s
 => CACHED [compilers 3/5] RUN echo "Current directory: ${PWD}" && ls -lR .                                                                                                                                                            0.0s
 => CACHED [compilers 4/5] RUN --mount=type=bind,source=test.txt,target=test.txt stat -c '%a %u %U %g %G %W %X %Y %Z' test.txt                                                                                                         0.0s
 => CACHED [compilers 5/5] RUN ls -la /work && echo "DONE" >> a_file.txt    

So in that case...I guess I file this with docker somewhere?

EDIT: I found an old terminal session output where in my real world use case the docker output did cache the RUN --mount with the extra context. Now I don't know what's going on.

EDIT 2: Looks like docker caches it if the extra context isn't the current directory (or maybe the same as the base context).

EDIT 3: Podman doesn't seem to cache in any of these situations.

@rhatdan
Copy link
Member

rhatdan commented Jun 24, 2024

@flouthoc PTAL

Copy link

A friendly reminder that this issue had no activity for 30 days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug. stale-issue
Projects
None yet
Development

No branches or pull requests

2 participants