let’s go build a minimal docker image

October 1, 2018 Lothar Schulz Cloud, Cluster, Container, DevOps, Docker, Dockerfile, Kubernetes, Technology 1

turtle hetching

How small can a docker image be?

I asked myself this question, when one of the teams I work with started a software project in golang recently. The target environment for this project is Kubernetes. The background of my question was kubernetes best practises that include small docker images.

We started with this first version of the Dockerfile:

FROM golang:1.11

RUN mkdir /app
WORKDIR /app
COPY hello.go	.

RUN go build -o hellogo .

RUN groupadd -g 99 appuser && useradd -r -u 99 -g appuser appuser 
USER appuser

CMD ["./hellogo"]

(source code)

Note: all source code in this post is sample code to show case the approaches we considered.

No more time? Jump straight to conclusion.

Why are smaller docker image preferred?

I see two main reasons to strive for smaller docker images:

  1. the security attack surface is often smaller
  2. pulling and pushing docker images from and to remote docker registries is faster

Why is the security attack surface often smaller?

The smaller docker images are in terms of number of layers as well as size of layers, the smaller is the probability to inherit CVEs (cybersecurity vulnerabilities and exposures e.g. listed in https://cve.mitre.org/cve/) being shipped with these layers. The number of CVEs per docker image also depends on the number of CVEs attached to specific layers.
I’d rather consider striving for smaller docker images due to security aspects a recommendation than a rule.
To underscore this recommendation, I did compare two docker images’ CVEs in CVE appendix.

Why is pulling and pushing of smaller docker images from and to remote docker registries faster?

Well, the fewer data are sent over the network, the faster pulling and pushing of docker images can be performed. In the appendix about docker image sizes and times, I do list sample docker pull and push times from circleCI to docker hub.

How to shrink a docker image?

Based on the first version of the Dockerfile, we tested multi stage docker builds that follow the builder pattern.

Builder pattern

Dockerfiles following the builder pattern are separated into two parts:

  1. part one creates a single binary that runs the application
  2. part two leverages the created binary file with a reduced docker image base layer

The above Dockerfile was changed to:

#part one
FROM golang:1.11 as builder

WORKDIR /go/src/github.com/lotharschulz/hellogodocker/

COPY hello.go	.

RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-w -extldflags "-static"' -o hellogodocker .

# part two
FROM alpine:latest
RUN apk --no-cache add ca-certificates && \
    addgroup -g 99 appuser && \
    adduser -D -u 99 -G appuser appuser

USER appuser

WORKDIR /app

COPY --from=builder /go/src/github.com/lotharschulz/hellogodocker/hellogodocker .

CMD ["./hellogodocker"]

(source code)

The first part contains the label builder. It is based on golang base docker image to create the binary file. The second part copies the binary from the builder. The resulting docker image is based on the smaller alpine docker base image and thus smaller overall.

The go build command includes
CGO_ENABLED=0 and -ldflags '-w -extldflags "-static"' that makes the binary artifact statically linked.

Alpine base image

Since the second part of the aforementioned Dockerfile is also based on alpine, we tested if a Dockerfile based on alpine base image only would help to reduce the docker image.
The Dockerfile was changed to:

FROM alpine:latest
RUN apk --no-cache add ca-certificates && \
    addgroup -g 99 appuser && \
    adduser -D -u 99 -G appuser appuser

ADD hellogo /
CMD ["/hellogo"]

(source code)

While this Dockerfile is smaller it does not show significantly reduced docker image sizes as well as not significantly reduced docker image build, pull and push times. Please refer to Appendix Docker base image sizes & push and pull times as of circleCi hellogodocker build 91 for details.

From scratch base image

We tried to further shrink the image using from scratch as base image:

FROM scratch
ADD ca-certificates.crt /etc/ssl/certs/
ADD hellogo /
CMD ["/hellogo"]

(source code)

This produces a rather minimal docker image. Actually it was the smallest we came up with.
However from time to time, we needed to debug the running app in a container.
We did experience issues in those situations with exec :

$ docker exec -it hellogodocker sh
OCI runtime exec failed: exec failed: container_linux.go:348:
starting container process caused
"exec: \"sh\": executable file not found in $PATH": unknown

Based on these findings, the team decided to rely on alpine based image as docker image.

Conclusion

With smaller docker images, you do reduce the security attack surface of docker images and you reduce the docker image round trip time (build, push & pull).

4.8 MB (2 MB compressed) is the smallest docker image size produced based on the hellogodocker sample repository. It is ~0.6% of the initial uncompressed docker image size.
I consider this one indicator that reducing docker image sizes opens up great potential.

While docker image build times did not reduce so much, docker image pull times were reduces from ~20 seconds to ~ 1 seconds. Container start time will benefit from reduced docker pull times.

The best solution for the team is to use a docker image based on alpine because preinstalled utilities like sh are useful for the team.

Below are the build, pull and push times as well as docker image sizes based on the hellogodocker sample repository with alpine docker image:

  • build time: 0.67 seconds
  • push time: 3.803 seconds
  • pull time 1.244 seconds
  • image size 11.1MB (compressed 5MB)

The team experiences similar data and considers those acceptable for software development cycle times.
The appendix about docker image sizes and times contains more details about times and image sizes.

Appendixes

Docker base image sizes & push and pull times as of circleCi hellogodocker build 91

CVEs for docker images lotharschulz/hellogo:build.docker-min-compress–0.2.91 and lotharschulz/hellogo:build.docker-cache–0.2.91 scanned with Klar and ClairOS at 2018 09 20.

How to scan docker images with Klar and ClairOS

  • install klar e.g. from source code 
    go get github.com/optiopay/klar
  • install minikube
  • git clone https://github.com/coreos/clair
  • cd clair
  • git checkout -b release-2.0 origin/release-2.0
  • adapt the clairOS config file in contrib/k8s/config.yaml as shown below: 
    # source: postgres://postgres:password@postgres:5432/postgres?sslmode=disable
source: host=postgres port=5432 user=postgres password=password sslmode=disable
  • minikube start
  • kubectl create secret generic clairsecret --from-file=./config.yaml
kubectl create -f clair-kubernetes.yaml
  • analyse docker images: 
    CLAIR_OUTPUT=High CLAIR_ADDR=http://192.168.99.100:30060 klar [docker file]

    e.g.

    CLAIR_OUTPUT=High CLAIR_ADDR=http://192.168.99.100:30060 klar lotharschulz/hellogo:build.docker-cache--0.2.91
    CLAIR_OUTPUT=High CLAIR_ADDR=http://192.168.99.100:30060 klar lotharschulz/hellogo:build.docker-min-compress--0.2.91

docker hub images sizes

docker images sizes
docker images sizes

Further reading

1 Comment

Leave a Reply

Your email address will not be published.


*


This site uses Akismet to reduce spam. Learn how your comment data is processed.