Configuration and Storage

Why decouple configuration from the image

Baking environment-specific configuration (database hostnames, feature flags, log levels) directly into a container image means building a separate image per environment — defeating the whole point of a container image being an immutable, promotable artifact that's identical from dev through production. A ConfigMap lets the same image run in every environment, with only the ConfigMap's contents differing.

Creating a ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  LOG_LEVEL: "info"
  MAX_CONNECTIONS: "100"
  app.properties: |
    feature.new_checkout=true
    cache.ttl_seconds=300

Consuming it as environment variables

spec:
  containers:
    - name: app
      image: myapp:1.0
      envFrom:
        - configMapRef:
            name: app-config    # every key becomes an env var: LOG_LEVEL, MAX_CONNECTIONS
      env:
        - name: LOG_LEVEL       # or reference just one specific key
          valueFrom:
            configMapKeyRef:
              name: app-config
              key: LOG_LEVEL

Consuming it as mounted files

spec:
  containers:
    - name: app
      image: myapp:1.0
      volumeMounts:
        - name: config-volume
          mountPath: /etc/app-config
  volumes:
    - name: config-volume
      configMap:
        name: app-config

Each key in the ConfigMap becomes a separate file inside /etc/app-config (e.g., /etc/app-config/app.properties), which is the natural approach for configuration formats applications expect to read as a whole file, rather than individual environment variables.

A key behavioral difference between the two consumption methods

Environment variables are only read once, at container start — updating the underlying ConfigMap has no effect on an already-running container's environment variables; the Pod must be restarted (e.g., via a rolling update) to pick up the change. Mounted ConfigMap volumes, by contrast, are updated automatically (after a propagation delay, typically up to a minute or so) without restarting the Pod — the kubelet periodically syncs the mounted files to match the ConfigMap's current content. This distinction matters when deciding how a config change should roll out: as environment variables, a change requires an explicit rollout to take effect (which some teams actually prefer, for predictability); as mounted files, the application needs its own file-watching logic to notice and react to the change live.

What ConfigMaps are not for

ConfigMaps are stored as plain, unencrypted data in etcd (readable by anyone with appropriate RBAC access to view the object) — they're explicitly not meant for passwords, API keys, or other sensitive values. That's what Secrets exist for (see that question), even though a Secret's data is only base64-encoded, not strongly encrypted, by default — the distinction between the two objects is more about signaling intent and enabling separate handling than about ConfigMaps being insecure and Secrets being inherently safe.

Creating and consuming a Secret

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  username: YWRtaW4=          # base64("admin")
  password: c3VwZXJzZWNyZXQ=  # base64("supersecret")

spec:
  containers:
    - name: app
      envFrom:
        - secretRef:
            name: db-credentials

Consumption (env vars or mounted volumes) works identically to ConfigMaps — the key practical difference is in how Kubernetes handles the data, not in the mechanics of using it in a Pod spec.

The critical misconception: base64 is not encryption

echo "c3VwZXJzZWNyZXQ=" | base64 -d
# supersecret

Base64 is a reversible encoding, not encryption — anyone with read access to the Secret object (via kubectl get secret db-credentials -o yaml, or direct etcd access) can trivially decode it back to plaintext. Base64 exists purely so arbitrary binary data (like a TLS private key) can be represented safely inside YAML/JSON text, not to provide any confidentiality. A Secret's actual security comes entirely from who is authorized to read it (RBAC) and whether the underlying storage is encrypted — not from the encoding itself.

What Kubernetes does provide, beyond plain ConfigMaps

• Excluded from some default output — kubectl describe pod doesn't print a mounted Secret's actual values (though kubectl get secret -o yaml still shows them to anyone with permission to run that command).
• tmpfs (memory-backed) storage on nodes — a mounted Secret volume is typically backed by tmpfs (RAM), not written to the node's actual disk, reducing the risk of leftover sensitive data on a decommissioned or compromised node's filesystem.
• Encryption at rest, if explicitly configured — by default, Secrets are stored in etcd with no encryption beyond base64 (i.e., effectively plaintext to anyone with etcd access); enabling encryption at rest (a control-plane configuration, not automatic out of the box) actually encrypts Secret data before it's written to etcd.

What Secrets still don't solve on their own

Even with encryption at rest enabled, anyone with sufficient RBAC permission to read the Secret object through the API server gets the plaintext value back — Kubernetes-native Secrets don't provide fine-grained audit trails of secret access the way a dedicated secrets manager typically does, and rotating a Secret's value still requires updating the object and getting consuming Pods to pick up the change (see the ConfigMap question's note on env vars vs. mounted volumes applying equally here). For production systems with genuinely sensitive data, many teams layer an external secrets manager (HashiCorp Vault, AWS Secrets Manager, paired with a tool like External Secrets Operator that syncs values into native Kubernetes Secrets, or injects them directly at runtime) on top of, or instead of, relying purely on native Kubernetes Secrets.

Treat a Kubernetes Secret as "slightly better protected than a ConfigMap, but not encrypted by default and not a substitute for tight RBAC" — always enable encryption at rest for any cluster handling real sensitive data, restrict who can read Secret objects via RBAC as tightly as possible (see the security topic's least-privilege question), and consider an external secrets manager for the most sensitive values, audit requirements, or rotation needs.

The problem Volumes solve

A container's own filesystem is ephemeral — anything written to it is lost the moment the container restarts, even within the same Pod (a crash and restart of a single container gets a fresh filesystem). This is fine for a purely stateless process, but breaks anything needing to persist data across a restart, or needing to share data between multiple containers in the same Pod.

The Volume abstraction: attached to the Pod, not the container

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: myapp:1.0
      volumeMounts:
        - name: cache-volume
          mountPath: /app/cache
    - name: sidecar
      image: sidecar:1.0
      volumeMounts:
        - name: cache-volume        # same volume, mounted into a second container
          mountPath: /shared/cache
  volumes:
    - name: cache-volume
      emptyDir: {}

Because the Volume is defined at the Pod level and mounted into whichever containers declare it, both containers in this Pod see the same underlying storage — and, depending on volume type, data written to it can survive an individual container within the Pod restarting, even though the Volume's own lifetime is still ultimately tied to the Pod (an emptyDir, specifically, is deleted when the Pod itself is deleted, regardless of how many times its containers individually restarted along the way).

Many volume "types," one consistent Pod-level interface

Kubernetes Volumes aren't one storage mechanism — the volumes field supports many distinct types, each backed by different underlying storage: emptyDir (ephemeral, node-local scratch space), hostPath (a path on the node's own filesystem), configMap/secret (presenting Kubernetes objects as files — see the ConfigMap question), and persistent types backed by a PersistentVolumeClaim (network/cloud-backed storage that can outlive the Pod entirely — see that question). All of these are mounted into containers using the exact same volumeMounts syntax, regardless of what's actually backing them.

How this differs from a plain Docker volume

Docker's own volume concept is scoped to a single container/host and doesn't have a native notion of "shared across a group of co-located containers with different lifecycle rules per volume type," nor does it have a built-in abstraction spanning many different network/cloud storage backends behind one consistent interface. Kubernetes's Volume model is a broader, Pod-centric abstraction specifically designed to unify wildly different storage backends (local ephemeral scratch space, cloud block storage, network file shares, Kubernetes-object-as-file) under one API, so a Pod spec doesn't need different mounting logic depending on what's actually providing the storage underneath.

Ephemeral vs. persistent, at a glance

Understanding this table is the key to answering "will my data survive X" questions correctly — the answer depends entirely on which volume type is in play, not on some single universal Kubernetes storage guarantee.

The separation of concerns: PV (supply) vs. PVC (demand)

# PersistentVolume: an actual piece of storage, typically created by an admin
# or dynamically by a StorageClass provisioner -- represents SUPPLY
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-database-1
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  awsElasticBlockStore:
    volumeID: vol-0abc123

# PersistentVolumeClaim: a request for storage, created by an application
# developer -- represents DEMAND, with no knowledge of the underlying implementation
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

# The Pod only references the PVC by name -- never the PV directly
spec:
  containers:
    - name: db
      volumeMounts:
        - name: data
          mountPath: /var/lib/data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: database-pvc

Kubernetes's PV controller binds the PVC to a PV that satisfies its requested size and access mode (in this example, pv-database-1 matches database-pvc's request exactly) — once bound, that specific PV is reserved exclusively for that PVC and can't be claimed by any other PVC.

Why this indirection is genuinely useful

An application developer writing a Deployment or StatefulSet manifest shouldn't need to know (or care) whether the underlying storage is an AWS EBS volume, a GCP persistent disk, or an on-prem NFS share — they just declare "I need 20Gi, ReadWriteOnce" via a PVC, and the actual storage implementation detail is handled entirely by whoever manages PVs (or, far more commonly today, by dynamic provisioning via a StorageClass — see that question). This mirrors the same "declare what you want, let something else figure out how" philosophy that runs through Kubernetes generally (see the reconciliation-loop question).

Static vs. dynamic provisioning

• Static provisioning: a cluster administrator manually creates PV objects ahead of time, representing real storage they've already set up, and PVCs bind to whichever pre-existing PV matches.
• Dynamic provisioning (the overwhelmingly more common approach today): a PVC references a StorageClass, and if no existing PV matches, the StorageClass's provisioner automatically creates a brand-new PV (and the underlying real storage — e.g., actually calling the cloud API to create an EBS volume) on demand, with no administrator needing to pre-provision anything.

The lifecycle and reclaim policy

When a PVC is deleted, what happens to its bound PV (and the real underlying storage) depends on the PV's reclaim policy: Delete (the PV and underlying storage are deleted too — common for dynamically-provisioned PVs) or Retain (the PV and its data survive, but move to a "released" state, no longer available to be bound automatically, requiring manual admin action to reuse or clean it up) — an important setting to check deliberately for any PV holding genuinely important data, since the default behavior for dynamically-provisioned volumes is frequently Delete, which can be a nasty surprise if a PVC is deleted accidentally.

A strong answer emphasizes the separation of concerns this design achieves — developers declare storage needs abstractly via PVCs, while the actual storage implementation (PVs, and typically dynamic provisioning via StorageClasses) is a cluster/infrastructure-level concern — rather than just describing PV and PVC as two similarly-named objects without explaining why the split exists.

Defining a StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: ebs.csi.aws.com     # which CSI driver actually creates the storage
parameters:
  type: gp3
  iopsPerGB: "50"
reclaimPolicy: Delete             # what happens to the PV when its PVC is deleted
volumeBindingMode: WaitForFirstConsumer

Requesting storage from it

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-pvc
spec:
  storageClassName: fast-ssd      # references the StorageClass above
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi

What happens when this PVC is created

1. The PVC controller checks for an existing, unbound PV matching the request — if none exists (the common case with dynamic provisioning, since PVs aren't pre-created), it proceeds to provisioning.
2. The fast-ssd StorageClass's provisioner (ebs.csi.aws.com, a CSI — Container Storage Interface — driver) is invoked, which calls the actual cloud provider API to create a real 100Gi gp3 EBS volume.
3. A new PersistentVolume object is automatically created, representing this newly-created real disk.
4. The PVC is bound to this new PV — the whole process happening automatically, with no administrator needing to have anticipated this specific request ahead of time.

The Container Storage Interface (CSI) — another standard plugin interface

Similar in spirit to CRI (for container runtimes) and CNI (for networking), CSI standardizes how Kubernetes talks to storage systems, so any storage vendor can write a CSI driver that Kubernetes can use without storage-vendor-specific code baked into Kubernetes core. This is what lets the same StorageClass mechanism work uniformly whether the underlying provisioner is AWS EBS, GCP Persistent Disk, Azure Disk, or an on-prem storage system's CSI driver.

volumeBindingMode — an important, easy-to-miss setting

WaitForFirstConsumer (increasingly the recommended default) delays actually provisioning the volume until a Pod that will use the PVC is scheduled — this matters because the volume might have topology constraints (e.g., an EBS volume can only be attached to a node in the same availability zone it was created in), and provisioning it before knowing which node/zone the Pod will actually run in could create a volume in the wrong zone entirely, causing the Pod to become unschedulable. The older default, Immediate, provisions the volume as soon as the PVC is created, without waiting to see where the consuming Pod lands — a real source of "PVC bound to a volume in the wrong availability zone" issues if not configured carefully.

Default StorageClass

A cluster can designate one StorageClass as the default (via an annotation), used automatically for any PVC that doesn't explicitly specify storageClassName — worth being aware of when a PVC's storage behavior seems to "just work" without an explicit StorageClass reference; it's still using one, just implicitly.

Cluster operators typically define a small number of StorageClasses representing meaningful tiers (e.g., fast-ssd for databases needing high IOPS, standard for general-purpose storage, cold-archive for infrequently accessed data), and application teams simply reference the appropriate one by name in their PVCs — keeping the "what kind of storage, from which provider, with which performance characteristics" decision centralized and consistent across the cluster.