Veli Keçeci
Back to all posts
·5 min read

Speeding up Docker image builds: layer cache and multi-stage Dockerfiles

On many teams the slowest step in the CI pipeline is not the tests but building the Docker image. On every commit the dependencies are downloaded again, the compilation runs again, and ten minutes later the pipeline produces an image that is almost identical to the last one. Used properly, Docker's layer cache can skip most of that work.

In this post I explain how the cache works, how to arrange your Dockerfile around it, and how to keep the cache alive in CI. The examples use .NET and Node.js, but the logic is the same on any stack.

How the layer cache works

Every instruction in a Dockerfile produces a layer. Before running an instruction, Docker checks whether the same instruction has run before with the same inputs. If it has, the result is taken from the cache.

The critical rule: when one layer is invalidated, every layer after it is rebuilt too. So put things that change often near the end of the file and things that change rarely near the start. For COPY, the input is the content of the copied files. If a single source file changes, a COPY . . layer is invalidated.

Separate dependencies from source code

The most common mistake is copying the whole project and only then installing dependencies:

COPY . .
RUN npm ci

Here every code change re-runs npm ci. The right way is to copy only the dependency manifests first:

COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build

As long as package-lock.json is unchanged, npm ci comes from the cache. When code changes, only the last two steps run. In .NET the same pattern applies with project files:

COPY *.sln ./
COPY src/Api/Api.csproj src/Api/
COPY src/Domain/Domain.csproj src/Domain/
RUN dotnet restore
COPY . .
RUN dotnet publish src/Api -c Release -o /out --no-restore

With many projects, copying each csproj by hand is tedious, but it is what lets the restore step be cached.

Cut unnecessary input with .dockerignore

COPY . . sees everything in the build context. If .git, node_modules, bin, obj and local logs are part of the context, sending it gets slower and an unrelated file change can break the cache. A .dockerignore file fixes this:

.git
**/node_modules
**/bin
**/obj
*.log
.env*

The .env* line also matters for security: local secrets do not end up in the image by accident.

Shrink the image with multi-stage builds

Build tools are not needed at runtime. A multi-stage Dockerfile compiles in a heavy image and moves only the output into a lightweight one:

FROM mcr.microsoft.com/dotnet/sdk:9.0 AS build
WORKDIR /src
COPY *.sln ./
COPY src/Api/Api.csproj src/Api/
RUN dotnet restore
COPY . .
RUN dotnet publish src/Api -c Release -o /out --no-restore

FROM mcr.microsoft.com/dotnet/aspnet:9.0 AS runtime
WORKDIR /app
COPY --from=build /out .
ENTRYPOINT ["dotnet", "Api.dll"]

This has two benefits. The final image is small, so it is faster to pull and deploy, and its attack surface is smaller. The build stages are also cached independently. Change the tag versions to match the version your project uses; the pattern is what matters.

BuildKit cache mounts

The layer cache downloads all packages from scratch when the dependency file changes. BuildKit's cache mount keeps the package manager's cache directory between builds:

# syntax=docker/dockerfile:1
RUN --mount=type=cache,target=/root/.npm \
    npm ci

When you add one package, only the new package is downloaded and the rest comes from the cache directory. The cache mount content is not written into the image, so it does not bloat it. The same approach works for NuGet with /root/.nuget/packages and for pip with /root/.cache/pip.

Keep the cache alive in CI

Everything that works locally is wasted if CI runs on a clean machine, because the cache is empty on every run. There are two common solutions:

  • Registry cache: at the end of the build you push the cache layers to a registry and pull them in the next build. It is used like docker buildx build --cache-from type=registry,ref=... --cache-to type=registry,ref=...,mode=max.
  • The CI provider's cache storage: if your CI tool supports a local directory or its own cache type, use the matching buildx --cache-to/--cache-from type.

mode=max exports intermediate layers as well. This matters for multi-stage builds, because the default mode stores only the layers of the final image and the cache of the build stage is lost. The cache uses storage space, so set a retention policy that cleans up old entries.

Measure that the cache works

Do not optimize by guessing. In the build output, steps served from the cache are marked CACHED. Change some code, rebuild, and verify that the dependency step shows CACHED. In CI, track total build time on a dashboard and compare before and after the change.

When the cache breaks unexpectedly, look at these: files entering through COPY that change on every build (for example a file that writes the build time), values passed with ARG that differ every time (such as a commit SHA), and time-dependent commands like RUN apt-get update. If you can, use changing values like the commit SHA as late in the file as possible.

Closing

There is no magic setting for a fast Docker build, only a few simple habits. Copy dependency manifests first, use .dockerignore, build in multiple stages, add cache mounts for package caches, and keep the cache in CI. Measure the effect of each step from the CACHED lines and the build time. These changes fit into one afternoon and pay off on every commit.