Slower than BuildKit with complex Dockerfile

[AkihiroSuda]

Benchmark:

$ git clone https://github.com/moby/moby.git ./moby
$ docker volume create buildbench-img
$ docker run --privileged -v $(pwd)/moby:/workspace -w /workspace --rm -v buildbench-img:/home/user r.j3ss.co/img:latest build -backend overlayfs -t foo .
$ docker run --privileged -v $(pwd)/moby:/workspace -w /workspace --rm -v buildbench-img:/home/user r.j3ss.co/img:latest build -backend overlayfs -t foo .
$ docker volume rm buildbench-img

Please refer to AkihiroSuda/buildbench#1 for the further information.

I think both BuildKit and img are configured to use overlayfs but maybe needs some investigation.

[jessfraz]

Oh this is awesome

[AkihiroSuda]

Retested and still slow.. AkihiroSuda/buildbench#3
I confirmed the overlayfs snapshotter is explicitly specified. Not sure why it is slow..

[frezbo]

img is really slow to be used in local and CI systems. I really love the rootless builds. Showing some perf details:

img (aws:rean-gov-sd)(kc)$ time img build -t hello .
Building docker.io/library/hello:latest
Setting up the rootfs... this may take a bit.
[+] Building 52.1s (7/7) FINISHED                                                                                                                     
 => local://dockerfile (Dockerfile)                                                                                                              0.4s
 => => transferring dockerfile: 363B                                                                                                             0.0s
 => local://context (.dockerignore)                                                                                                              0.8s
 => => transferring context: 02B                                                                                                                 0.0s
 => docker-image://docker.io/tonistiigi/copy:v0.1.3@sha256:87c46e7b413cdd2c2702902b481b390ce263ac9d942253d366f3b1a3c16f96d6                     24.1s
 => => resolve docker.io/tonistiigi/copy:v0.1.3@sha256:87c46e7b413cdd2c2702902b481b390ce263ac9d942253d366f3b1a3c16f96d6                          4.2s
 => => sha256:87c46e7b413cdd2c2702902b481b390ce263ac9d942253d366f3b1a3c16f96d6 1.36kB / 1.36kB                                                   0.0s
 => => sha256:b5fd220f9e38fc61daf5af13e8a98c18827822428bf6acab1bc1dc67586d6c1a 2.45kB / 2.45kB                                                   0.0s
 => => sha256:7b75ca719cf195ac5ed9863efe1e677c4ed7a0b87db3e2f5daf895f9f84340d8 172.86kB / 172.86kB                                               9.1s
 => => sha256:f5fd20edad0c69902cf702c6b080bcfac264c392f71871eccd39a5f171b01149 751.94kB / 751.94kB                                              16.8s
 => => sha256:58370f0d30e03a5cd4641c95b7e02d2983d1ebf3d6b34f7051d690b9b8512cb6 70.72kB / 70.72kB                                                20.6s
 => => sha256:ed81d31ad9fbd2898dd7b6f490978b3d20c9eb99eed30b509f03977e0096f238 98.68kB / 98.68kB                                                20.5s
 => => sha256:8cd25ee4c3d612d46a5ca115a715a2a1482c5766428c3febb5835296e3e91a16 300.71kB / 300.71kB                                               8.9s
 => => sha256:f5fd20edad0c69902cf702c6b080bcfac264c392f71871eccd39a5f171b01149 751.94kB / 751.94kB                                              16.8s
 => => sha256:ed81d31ad9fbd2898dd7b6f490978b3d20c9eb99eed30b509f03977e0096f238 98.68kB / 98.68kB                                                20.5s
 => => sha256:58370f0d30e03a5cd4641c95b7e02d2983d1ebf3d6b34f7051d690b9b8512cb6 70.72kB / 70.72kB                                                20.6s
 => => unpacking docker.io/tonistiigi/copy:v0.1.3@sha256:87c46e7b413cdd2c2702902b481b390ce263ac9d942253d366f3b1a3c16f96d6                        2.7s
 => docker-image://registry.access.redhat.com/rhel7/rhel:latest@sha256:63e5dc4f05042ffc8e890445  2.3s
 => => resolve registry.access.redhat.com/rhel7/rhel:latest@sha256:63e5dc4f05042ffc8e890445c625  0.0s
 => => sha256:63e5dc4f05042ffc8e890445c6256a96d402704ee4ae5726d4648d9111040ac9 2.04kB / 2.04kB                                                   0.0s
 => => sha256:e5b4081ef6a588a4eff8b3eb8bfc2c072a8dd7b9b744624d8b8f13320211af99 7.16MB / 7.16MB                                                   0.0s
 => => sha256:235652dcceccdd3a7ad45b738efd36c5e364ac4612caa8ef5f9cd46acba3ac19 245B / 245B                                                       0.0s
 => => sha256:c181936b24e2fe9e3c2cc5ab2e90693ae5b120327b42b076241a03c1a454201d 74.86MB / 74.86MB                                                 0.0s
 => => sha256:56090e6c69942902195f07556385b8df9ab351f9d068a940b4fe46f18d5c3426 590B / 590B                                                       0.0s
 => => sha256:4225081786a65b8713fd2e490d7fdd5d0f8af3c88295187baa69c02ea3991654 1.34kB / 1.34kB                                                   0.0s
 => => sha256:ba0314b0ad8d7e281cfa12b0c03b00086741db8d1281c2c57f1379ad69776bcf 3.35kB / 3.35kB                                                   0.0s
 => => sha256:843865e6ede71face95e25065e191e96cd37a6cb46d199a530ad3934a1a5a7cc 204B / 204B                                                       0.0s
 => => sha256:8c31315a3affe93a4215fe027e9ad2a0b09f636b5e8b2240be584a85a6862573 25.22MB / 25.22MB                                                 0.0s
 => => sha256:eeff58259a8d2bec50c4067ed936180ad6bc43b792bac2125436bc437c279e43 25.81MB / 25.81MB                                                 0.0s
 => => sha256:51f3f7e80e81de145949f5d12b960eabecb6069f5aedc5d9f6a349a184f1d694 718B / 718B                                                       0.0s
 => => sha256:3083ffc0ce17c98b6b772e1505fc725d6e4c2973e60cf328998c507ff5b0d93f 375B / 375B                                                       0.0s
 => => sha256:dab9f87f3be2bcb394b128dbff8d952f30d6934e79972a9a6a0597df9af2736e 1.23kB / 1.23kB                                                   0.0s
 => => unpacking registry.access.redhat.com/rhel7/rhel:latest@sha256:63e5dc4f05042ffc8e890445c6  0.8s
 => /bin/sh -c echo hi > /tmp/hello                                                                                                              3.9s
 => copy /src-0/hello opt/                                                                                                                       3.0s
 => exporting to image                                                                                                                           1.8s
 => => exporting layers                                                                                                                          1.4s
 => => exporting manifest sha256:1df16f59ba2f9911984aa64f2ee0f5c9418c5652ea2799b20c828f78b5e1b17b                                                0.1s
 => => exporting config sha256:e186d2797addaf35b0baba842de428b7ae348068b20cb132b01f0f23565e6a6a                                                  0.1s
 => => naming to docker.io/library/hello:latest                                                                                                  0.0s
Successfully built docker.io/library/hello:latest

real	0m53.773s
user	0m21.373s
sys	0m39.560s
img (aws:rean-gov-sd)(kc)$ time docker build -t hello .
Sending build context to Docker daemon  2.048kB
Step 1/4 : FROM registry.access.redhat.com/rhel7/rhel:latest AS base
 ---> fa6746839eb3
Step 2/4 : RUN echo hi > /tmp/hello
 ---> Running in d5c4ba2d55eb
Removing intermediate container d5c4ba2d55eb
 ---> 86d8fd396ed6
Step 3/4 : FROM registry.access.redhat.com/rhel7/rhel:latest
 ---> fa6746839eb3
Step 4/4 : COPY --from=base /tmp/hello /opt/
 ---> 6c64e61818d4
Successfully built 6c64e61818d4
Successfully tagged hello:latest

real	0m9.921s
user	0m0.078s
sys	0m0.018s
img (aws:rean-gov-sd)(kc)$ img ls
NAME												SIZE		CREATED AT	UPDATED AT	DIGEST
registry.access.redhat.com/rhel7/rhel:latest	126.9MiB	9 hours ago	7 hours ago	sha256:63e5dc4f05042ffc8e890445c6256a96d402704ee4ae5726d4648d9111040ac9
docker.io/library/hello:latest									126.9MiB	12 days ago	48 seconds ago	sha256:1df16f59ba2f9911984aa64f2ee0f5c9418c5652ea2799b20c828f78b5e1b17b
registry.hello.com/hello:latest									3.817MiB	10 days ago	6 days ago	sha256:1fa1344385f721eb881e0faea9b9db58343ca7a896cba8dc3c595a0df3e5990e
img (aws:rean-gov-sd)(kc)$ 

Dockerfile

FROM registry.access.redhat.com/rhel7/rhel:latest AS base
RUN echo hi > /tmp/hello

FROM registry.access.redhat.com/rhel7/rhel:latest
COPY --from=base /tmp/hello /opt/

From the stats img is over 6 times slower. ping @AkihiroSuda . Any pointers , coz buildkit is faster as per your comments above.

[AkihiroSuda]

@frezbo I guess you're using native snapshotter rather than overlayfs and that slowness seems different from my issue.

[frezbo]

@jessfraz @AkihiroSuda I really want to push engineers to use a tool that can build without root. Having docker and a user having permission to build/deploy in a CI system is basically superuser.
Does buidlkit supports running as non-root without having to unshare ?

[sneko]

Hi all,

I'm facing a speed issue when img has to build a Dockerfile which inherit from a 2.5GB parent, it's like taking forever even if the Dockerfile is just making a few COPY (times out after 30 minutes of building, but each tiny step can take 5 minutes or more).

I tested on Jenkins through a pipeline using an img container over a server having 14GB of RAM and 4 CPUs. I didn't find which is the cause... I didn't put any restrictions for the usage of resources.

Any advice to boost a bit performance?

Note that by running directly with docker CLI locally makes it finished within a few seconds (after caching the parent). I was not able to test directly Docker in my Jenkins since I removed any possibility to do so when I switched to img.

Dockerfile mentioned (but needs specific stuff to run fully, you may omit the RUN and COPY random directory):

FROM gcr.io/dataflow-templates-base/python3-template-launcher-base

ARG WORKDIR=/dataflow/template
RUN mkdir -p ${WORKDIR}
WORKDIR ${WORKDIR}

COPY requirements.txt .
COPY src ./src

ENV FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE="${WORKDIR}/requirements.txt"
ENV FLEX_TEMPLATE_PYTHON_PY_FILE="${WORKDIR}/src/XXXX.py"

RUN pip install -U -r ./requirements.txt

Thank you,

Note: I asked Googlers if they could reduce the image... it seems they think about it but I don't know when they will provide a slim image not embedding all third librairies not use in 90% of time by people

EDIT: I moved this specific CI/CD pipeline to Kaniko to build the final image... and it works fine. Within 2 minutes to build contrary to a timeout after 1h with img.

I will keep using img for all my other builds since the syntax is more clearer to me... but Kaniko may help for edge cases it seems 👍