BuildKit: Docker\'s Hidden Gem That Can Build Almost Anything

You've probably used BuildKit today without knowing it. Every time you run docker build, BuildKit is the engine behind it.

But reducing BuildKit to "the thing that builds Dockerfiles" is like calling LLVM "the thing that compiles C." It undersells the architecture by an order of magnitude.

BuildKit is a general-purpose, pluggable build framework. It can produce OCI images, tarballs, local directories, APK packages, RPMs, or anything else you can describe as a directed acyclic graph of filesystem operations.

The Dockerfile is just one frontend. You can write your own.

The Architecture

BuildKit's design is clean and surprisingly understandable once you see the layers. There are three key concepts.

LLB: The Intermediate Representation

At the heart of BuildKit is LLB (Low-Level Build definition). Think of it as the LLVM IR of build systems.

LLB is a binary protocol (protobuf) that describes a DAG of filesystem operations:

• Run a command
• Copy files
• Mount a filesystem

It's content-addressable, which means identical operations produce identical hashes, enabling aggressive caching.

When you write a Dockerfile, the Dockerfile frontend parses it and emits LLB. But nothing in BuildKit requires that the input be a Dockerfile. Any program that can produce valid LLB can drive BuildKit.

Frontends: Bring Your Own Syntax

A frontend is a container image that BuildKit runs to convert your build definition into LLB. The frontend receives:

• The build context
• The build file (through the BuildKit Gateway API)

And returns a serialized LLB graph.

The key insight: the build language is not baked into BuildKit. It's a pluggable layer.

You can write a frontend that reads:

• YAML specs
• TOML configs
• JSON schemas
• Custom DSLs

And BuildKit will execute it the same way it executes Dockerfiles.

You've actually seen this mechanism before:

# syntax=docker/dockerfile:1

This directive tells BuildKit which frontend image to use. # syntax=docker/dockerfile:1 is just the default. You can point it at any image.

Solver and Cache: Content-Addressable Execution

The solver takes the LLB graph and executes it. Each vertex in the DAG is content-addressed, so if you've already built a particular step with the same inputs, BuildKit skips it entirely.

This is why BuildKit is fast:

• It doesn't just cache layers linearly like the old Docker builder
• It caches at the operation level across the entire graph
• It can execute independent branches in parallel

The cache can be:

• Local - stored on your machine
• Inline - embedded in the image
• Remote - stored in a registry

This makes BuildKit builds reproducible and shareable across CI runners.

Not Just Images

BuildKit's --output flag is where things get practical. You can tell BuildKit to export the result as:

The type=local output is the most interesting for non-image use cases. Your build can produce:

• Compiled binaries
• Packages
• Documentation
• Any other artifacts

BuildKit dumps the result to disk. No container image required.

Real-World Examples

Projects built on BuildKit's LLB:

• Earthly - Dockerfile-like syntax for any language
• Dagger - Programmable CI/CD pipelines
• Depot - Fast hosted builds

It's a proven pattern at scale.

Building APK Packages: A Custom Frontend

To demonstrate this concretely, here's a custom BuildKit frontend that reads a YAML spec and produces Alpine APK packages—no Dockerfile involved.

The package YAML spec:

name: hello
version: "1.0.0"
epoch: "0"
url: https://example.com/hello
license: MIT
description: Minimal CMake APK demo

sources:
  app:
    context: {}

build:
  source_dir: hello

That's it. No Dockerfile. No shell scripts. BuildKit reads this spec through the custom frontend and produces a .apk file.

Running It

# Build the frontend image
docker build -t myorg/apkbuild -f Dockerfile .

# Use it to build an APK package
cd example
docker buildx build \
  -f spec.yml \
  --build-arg BUILDKIT_SYNTAX=myorg/apkbuild \
  --output type=local,dest=./out \
  .

BUILDKIT_SYNTAX tells BuildKit to use the custom frontend instead of the default Dockerfile parser. The --output type=local dumps the resulting .apk files to ./out.

No image is created. No registry is involved.

Why This Matters

BuildKit gives you a content-addressable, parallelized, cached build engine for free. You don't need to reinvent:

• Caching
• Parallelism
• Reproducibility

You write a frontend that translates your spec into LLB, and BuildKit handles the rest.

When to Use Custom Frontends

Getting Started

Enable BuildKit

# Docker 23.0+ enables BuildKit by default
# For older versions:
export DOCKER_BUILDKIT=1

# Or use buildx (recommended)
docker buildx install

Basic Multi-Platform Build

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t myapp:latest \
  --push \
  .

Export to Local Directory

docker buildx build \
  --output type=local,dest=./dist \
  .

Use a Custom Frontend

docker buildx build \
  -f build.yaml \
  --build-arg BUILDKIT_SYNTAX=myorg/custom-frontend \
  --output type=local,dest=./out \
  .

Key Takeaways

1. BuildKit ≠ Dockerfiles - It's a general-purpose build engine
2. LLB is the secret sauce - Content-addressable DAG of operations
3. Frontends are pluggable - Write your own build syntax
4. Output isn't limited to images - Export anything to local directories
5. Projects like Dagger prove it scales - Production-ready CI/CD

The Bottom Line

If you're building a tool that needs to:

• Compile code
• Produce artifacts
• Orchestrate multi-step builds

Consider BuildKit as your execution backend. The Dockerfile is just the default frontend. The real power is in the engine underneath.

Resources:

• BuildKit Documentation
• Dagger - Programmable CI/CD
• Earthly - Repeatable Builds
• LLB Definition Protocol