Docker in Production and CI/CD

Why the default driver doesn't scale to real production needs

The default json-file driver writes each container's logs to a local file on that host's disk, with no automatic rotation unless you explicitly set --log-opt max-size/max-file. Across a fleet of many hosts running many containers, this causes problems: logs are scattered across machines with no unified way to search them, a host being replaced (common in autoscaled or ephemeral infrastructure) takes its logs with it, and unbounded log growth can genuinely fill a host's disk if rotation isn't configured.

Setting a logging driver

docker run --log-driver=syslog --log-opt syslog-address=udp://loghost:514 myapp
docker run --log-driver=fluentd --log-opt fluentd-address=localhost:24224 myapp
docker run --log-driver=awslogs --log-opt awslogs-group=myapp --log-opt awslogs-region=us-east-1 myapp

Or set a default for the whole Docker daemon, instead of per-container:

// /etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Common production-oriented drivers

• syslog — forwards to a syslog server, a long-established standard for centralized Unix/Linux logging.
• journald — integrates with systemd's journal on hosts using systemd. Useful if the rest of the host's own logging already goes through journald.
• fluentd / gelf — forward to Fluentd or a GELF-compatible endpoint like Graylog. Common choices for feeding an Elasticsearch/Loki-based centralized logging stack — the same kind of architecture used for centralized logging in Kubernetes, except Docker's own driver feeds the stack directly instead of a separate log-shipping DaemonSet.
• awslogs — forwards directly to AWS CloudWatch Logs, a natural fit when already running on AWS.

The tradeoff: docker logs stops working locally

docker run --log-driver=awslogs myapp
docker logs myapp
# Error response from daemon: configured logging driver does not support reading

Once a non-default driver is configured, docker logs generally can no longer read the container's log output locally. Logs are only accessible through whatever external system the chosen driver forwards to. This can surprise teams used to reaching for docker logs directly during troubleshooting — the mental model needs to shift to "check the centralized logging system," not "SSH into the host and run docker logs."

Why this matters for anything beyond a single-host deployment

Centralized logging isn't optional once you're running more than a handful of containers across more than one host. Without it, diagnosing an issue that spans multiple services — a request that touches three different containers, possibly on three different hosts — means manually checking logs on each machine, which becomes impractical fast. A centralized logging driver feeding into a searchable system is what makes cross-service, cross-host troubleshooting actually tractable at scale, and it only needs configuring once at the daemon level for every container to benefit.

A representative pipeline sequence

# Simplified CI pipeline concept
steps:
  - name: Build image
    run: docker build -t myapp:${{ github.sha }} .

  - name: Run tests inside a container
    run: docker run --rm myapp:${{ github.sha }} npm test

  - name: Scan for vulnerabilities
    run: trivy image --exit-code 1 --severity CRITICAL myapp:${{ github.sha }}

  - name: Push to registry
    run: |
      docker tag myapp:${{ github.sha }} myregistry.example.com/myapp:${{ github.sha }}
      docker push myregistry.example.com/myapp:${{ github.sha }}

Why building the image is often the very first step, before tests even run

Building the actual production image first, then running tests inside a container built from that image, ensures the tests genuinely validate the same environment that will run in production. That's different from running tests directly on the CI runner's own environment, separately from the image build. This closes the exact "works on my machine/CI, breaks in production" gap containers exist to solve. Running tests against a different environment than what actually ships defeats much of the point of containerizing the application at all.

Tagging with the commit SHA for traceability

docker build -t myapp:${{ github.sha }} .

Tagging each CI-built image with the specific git commit SHA that produced it, rather than only a generic tag like latest or 1.0, gives an unambiguous, traceable link between a running image and the exact source code that built it. This is essential for debugging — it lets you answer exactly which commit is deployed right now — and it supports digest-pinning practices used when promoting images between environments.

Leveraging build cache across CI runs

- name: Build with registry cache
  run: |
    docker build \
      --cache-from myregistry.example.com/myapp:latest \
      -t myapp:${{ github.sha }} .

A fresh CI runner typically starts with no local build cache at all, unlike a developer's machine, which accumulates cache across many local builds. Without addressing this, every CI build is effectively a full, uncached rebuild, however well the Dockerfile itself is ordered for caching. Two techniques let CI builds benefit from layer caching despite starting from a clean runner each time: pulling a previous build's image as an explicit cache source with --cache-from, or using BuildKit's remote cache export/import capability.

Multi-stage builds work especially well in CI

FROM node:20 AS test
WORKDIR /app
COPY . .
RUN npm ci && npm test

FROM node:20-slim AS production
WORKDIR /app
COPY --from=test /app/dist ./dist
CMD ["node", "dist/server.js"]

A dedicated test stage can run the full test suite with all dev dependencies and test frameworks installed, while the final production stage only copies out the built artifacts. This combines "test in the real build environment" with "final shipped image stays minimal," both from multi-stage builds, in one Dockerfile.

Security scanning as a CI gate

Adding a vulnerability-scanning step that can fail the build on critical or high findings stops a genuinely dangerous image from ever reaching a registry or deployment target. It catches the issue as early in the pipeline as practical.

Why this scenario comes up at all

Many CI/CD systems run each job inside its own container, for isolation and reproducibility. But if that job's own work is to build a Docker image, a very common CI task, you end up needing to run Docker inside the container the CI job itself runs in. This is the scenario Docker-in-Docker addresses.

The DinD approach — a real, nested Docker daemon

docker run --privileged -d --name dind docker:24-dind
docker run --link dind:docker --env DOCKER_HOST=tcp://docker:2375 docker:24 docker build .

This runs an entirely separate Docker daemon inside a container, and a second container talks to that nested daemon to actually build images — genuinely "Docker inside Docker," not just talking to the host's existing daemon.

The real risks of this approach

• Requires --privileged mode. Running a nested Docker daemon generally requires disabling most container isolation for the outer container. --privileged grants nearly all capabilities and disables several security restrictions — a significant security relaxation, not a minor detail, and it directly undermines much of the isolation containers are meant to provide.
• Storage-driver complications. Running a container filesystem (an overlay or union filesystem) inside another container's own overlay filesystem has historically caused real compatibility and performance issues, since it layers the same kind of filesystem trickery on top of itself.
• Weaker isolation than the "in Docker" framing suggests. Despite feeling "extra isolated" since it's Docker inside Docker, the --privileged requirement actually means the outer container has less isolation from the host than an ordinary, non-privileged container would.

Alternative 1: mounting the host's Docker socket

docker run -v /var/run/docker.sock:/var/run/docker.sock docker:24 docker build .

This avoids running a nested daemon at all — the CI container talks directly to the host's own Docker daemon instead. It avoids DinD's storage-driver and --privileged concerns, but introduces its own well-documented, serious risk: socket access is functionally equivalent to host root. Any CI job with this mount effectively has host-level access, a serious concern for CI systems running untrusted or third-party pipeline code.

Alternative 2: purpose-built rootless image-building tools

# Kaniko, running inside a Kubernetes Pod with no special privileges,
# building an image without ever needing a Docker daemon (nested or host) at all

Tools like Kaniko (built by Google, common in Kubernetes-based CI) and Buildah can build OCI-compliant images without a Docker daemon at all. They implement the image-building logic directly in user space, with no need for privileged access or a socket to any daemon. This genuinely avoids both risks above, rather than just mitigating them, and is increasingly the preferred approach for CI systems, especially Kubernetes-based ones, that need to build images as an ordinary, unprivileged step in an otherwise-sandboxed pipeline.

Weighing the tradeoffs

Where DinD or socket-mounting genuinely can't be avoided — some legacy pipeline setups, specific tooling requirements — treat that CI runner as a fully trusted, high-privilege environment, and restrict what pipelines are allowed to run in it accordingly. Don't treat it as just another routine, low-stakes CI job.

Docker Swarm — Docker's own, simpler built-in orchestrator

docker swarm init                                    # initialize a Swarm on this node
docker service create --name web --replicas 3 -p 80:80 nginx   # deploy a replicated service across the Swarm

Swarm mode turns a group of Docker hosts into a cluster, using concepts like services, tasks, and overlay networks that closely mirror plain Docker's own CLI and mental model. This closeness is Swarm's biggest advantage — someone already comfortable with plain docker run/docker-compose concepts can pick up Swarm with relatively little extra learning. That's a much smaller learning curve than Kubernetes's larger, more distinct set of concepts: Pods, Deployments, Services, ConfigMaps, RBAC, and dozens more.

Kubernetes — the dominant, far more feature-rich orchestrator

Kubernetes provides sophisticated scheduling (affinity, taints and tolerations, priority and preemption), rich networking options (Ingress, NetworkPolicies, multiple CNI choices), and a vast ecosystem of extensions (CRDs, Operators, Helm charts for nearly any popular software). It's also what every major cloud provider offers a managed service for.

The key practical tradeoffs

Why this comparison matters for an interview, even though the answer leans clearly toward Kubernetes today

Recommending Swarm for a brand-new production system today, given the industry's clear consolidation around Kubernetes, would be an unusual choice needing strong justification — maybe a very small team, a very simple deployment need, and a strong preference for staying within familiar plain-Docker concepts rather than adopting Kubernetes's much larger surface area. A candidate should articulate this landscape honestly: acknowledge Swarm's genuine simplicity advantage, while recognizing the ecosystem has broadly moved on. Don't dismiss Swarm as having no merit, but don't recommend it without weighing that tradeoff against Kubernetes's now-dominant position.

When Swarm might still be a reasonable, deliberate choice

• A small team wanting basic multi-host orchestration — replicas, rolling updates, service discovery — without taking on Kubernetes's much larger learning curve and operational surface area.
• An organization already deeply invested in plain Docker/Compose workflows, looking for the smallest possible step up to multi-host capability, rather than a much bigger architectural leap to Kubernetes.

The anti-pattern: building a separate image per environment

# BAD: separate builds per environment, baking in environment-specific config
docker build -t myapp:staging --build-arg API_URL=https://staging-api.example.com .
docker build -t myapp:production --build-arg API_URL=https://api.example.com .

This means myapp:staging and myapp:production are, strictly speaking, different artifacts. Even if the only intended difference is a config value, nothing guarantees the build process produced byte-for-byte identical images apart from that one value. A subtle build-time issue — a flaky dependency resolution, a build tool behaving slightly differently — could introduce an unintended difference between what was tested in staging and what actually ships to production. That directly undermines the confidence that "what we tested is exactly what we're deploying."

The correct pattern: build once, configure at runtime

docker build -t myapp:1.0 .          # ONE build, used everywhere

docker run -e API_URL=https://staging-api.example.com myapp:1.0        # staging
docker run -e API_URL=https://api.example.com myapp:1.0                 # production

The exact same image, byte-for-byte, runs in every environment. Only the runtime configuration — environment variables, mounted config files, secrets — differs. This is the "build once, deploy many times, unchanged" principle. If something works correctly in staging, you have real, direct confidence that the identical artifact will behave the same way in production, since nothing about the image itself changed.

The twelve-factor app's "config" principle

This directly reflects Factor III — Config of the twelve-factor app methodology, which calls for a strict separation between an application's code (identical across environments) and its configuration (which legitimately varies by environment). Configuration belongs in the environment — environment variables or mounted files — never hardcoded into the build artifact.

# Kubernetes ConfigMaps/Secrets (see that stack), or Compose environment/.env
# files, or a cloud platform's own environment-variable configuration --
# all apply this same principle at whatever layer is actually deploying the container

This principle is exactly why Kubernetes ConfigMaps/Secrets and Compose's environment/env_file mechanisms exist as first-class concepts. They're the standard, orchestrator-level tools for injecting environment-specific configuration into an unchanged, promoted image, instead of requiring separate builds.

What this means for CI/CD pipeline design

1. Build the image ONCE, from a specific commit, tagged with that commit's SHA
2. Run tests against THAT SAME image
3. Push it to a registry
4. Deploy that SAME image (by digest, ideally -- see that question) to staging,
   with staging-specific configuration injected at deploy time
5. After validation, promote the SAME image (same digest) to production,
   with production-specific configuration injected at deploy time

This "build once, promote the same artifact through environments" pattern is a core CI/CD design principle. It eliminates an entire class of "it worked in staging but broke in production" bugs — the kind caused by staging and production actually running subtly different artifacts, rather than the same one with different configuration.