How should Secrets be managed securely in a production cluster?

This builds directly on the earlier Secret question's core point: base64 encoding provides no confidentiality, so real security depends entirely on these additional layers.

1. Enable encryption at rest

By default, Secret data sits in etcd effectively as plaintext (base64 is trivially reversible) — anyone with direct etcd access, or an etcd backup file, can read every Secret in the cluster. Encryption at rest (a control-plane API server configuration, specifying an EncryptionConfiguration with a chosen provider — e.g., using a KMS-backed envelope encryption provider for production) ensures Secret data is actually encrypted before being written to etcd, closing this specific gap.

# Simplified EncryptionConfiguration concept (actual setup requires
# API server flags pointing at this config file)
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: ["secrets"]
    providers:
      - kms:
          name: my-kms-provider
          endpoint: unix:///var/run/kmsplugin/socket.sock
      - identity: {}   # fallback, unencrypted -- should generally come last, not first

Managed Kubernetes services often provide this as a simple toggle (e.g., integrating with the cloud provider's own KMS) rather than requiring you to hand-configure it.

2. Apply least privilege via RBAC

Restrict get/list/watch access on secrets as tightly as possible — most application code needs to consume a Secret (by having it injected as env vars/volumes at Pod creation, which the kubelet handles) without ever needing direct RBAC permission to read arbitrary Secret objects via the API itself. Broad get secrets access granted casually (e.g., to an overly permissive default ServiceAccount, or a CI/CD pipeline's ServiceAccount) is one of the most common real ways Secret data actually leaks in practice.

3. Never commit raw Secret manifests to version control

A plain Secret YAML file, even with base64-encoded values, is not safe to commit to git — anyone with repository access (and, for public repos, the entire internet, plus automated credential-scanning bots) can decode it instantly. Tools like Sealed Secrets (Bitnami) or SOPS encrypt Secret values with a key only the cluster (or a designated key holder) can decrypt, producing a genuinely safe-to-commit encrypted representation that's only decrypted back into a real Secret object inside the cluster itself.

# A "SealedSecret" -- safe to commit, since only the cluster's private key can decrypt it
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-credentials
spec:
  encryptedData:
    password: AgBy8hCi8vY2Kj...    # ciphertext, useless without the cluster's decryption key

4. Consider an external secrets manager for the most sensitive data

For genuinely sensitive values (production database credentials, payment processor API keys), many teams keep the actual secret value in a dedicated external secrets manager (HashiCorp Vault, AWS Secrets Manager) — which provides stronger audit trails, automated rotation, and centralized access control than native Kubernetes Secrets alone — and use a tool like the External Secrets Operator to sync those values into native Kubernetes Secrets automatically, or have applications fetch them directly at runtime via a sidecar/init pattern, never requiring the raw secret value to be manually authored as a Kubernetes manifest at all.

Practical guidance, layered

Not every cluster needs every layer here — a reasonable baseline for most production clusters is encryption at rest plus tight RBAC plus never committing raw Secrets to git; reaching for a full external secrets manager integration is justified once the sensitivity of the data, compliance requirements, or the scale/complexity of secret rotation needs genuinely warrant the additional operational complexity.