Environment Config Overlays for Kubernetes

Every Kubernetes setup eventually faces the same question: the manifests are almost identical across dev, staging, and production, but a handful of values genuinely differ. Replica counts, resource sizes, feature flags, replication factors. The tempting answer is to copy the manifests per environment and edit the differences. That is where Kubernetes environment config drift begins, and drift is what makes staging stop being a meaningful test of production.

The durable answer is base plus overlay: one shared base holding everything common, and a small per-environment patch that changes only what truly differs. It sounds like a minor tooling preference. It is actually the difference between environments that stay honest and environments that quietly diverge until a deploy that worked in staging breaks in production for reasons nobody can reconstruct.

Why per-environment copies fail

Start with the honest case for copying: it is simple, it is obvious, and every environment’s config is right there in one file you can read top to bottom. For a while it works fine.

The failure is not dramatic; it is gradual and silent. Someone fixes a production issue by adjusting a probe timeout in the production manifest. Staging’s copy still has the old value. Someone else adds a new environment variable to staging while testing, and it never reaches production. A third person adds a PodDisruptionBudget to production because a node drain took the service down, and dev and staging never get one.

After six months, the three files share a filename and not much else. The specific damage is that staging is no longer a test of production. The entire justification for having a pre-production environment is that it behaves like production, so a deploy that passes there will pass here. Once the manifests differ in ways nobody has tracked, staging tests staging, and production remains untested until users find the problem.

The second damage is that every shared change now costs N edits and N chances to forget one. Adding a resource limit to a service across four environments is four pull-request diffs that must agree, and the one you forget is discovered later, usually under load. This post is part of the Kubernetes operations series.

What is a config overlay in Kubernetes?

An overlay is a small patch applied on top of a shared base configuration. The base contains the full, valid manifest set with everything common to every environment. Each overlay contains only the deltas for one environment: this many replicas, these resource requests, this feature flag on.

The rendered result for any environment is base plus that environment’s overlay. Nothing is duplicated except the values that genuinely differ, and because the overlay file contains only differences, it doubles as documentation: reading production’s overlay tells you, exhaustively, how production differs from everywhere else. That is a question that is otherwise surprisingly hard to answer.

k8s/
  base/
    deployment.yaml        # the full manifest, common values
    service.yaml
    kustomization.yaml
  overlays/
    dev/
      kustomization.yaml   # replicas: 1
    staging/
      kustomization.yaml   # replicas: 2
    prod/
      kustomization.yaml   # replicas: 6, larger resources, PDB

The properties that fall out of this structure are the reason it wins:

• A shared change is one edit. Change the base, every environment gets it on next deploy. No fan-out, nothing to forget.
• Differences are explicit and reviewable. “Why does prod behave differently?” is answered by a short file, in version control, with history.
• Adding an environment is cheap. Copy the closest overlay, adjust a few values, done.
• Drift becomes visible. If an overlay is growing large, that is a signal that an environment is diverging, and you can see it happening instead of discovering it later.

Should the base config hold production or development values?

Put the smallest, safest value in the base and escalate upward in the production overlay. This is a small decision with an outsized safety consequence, and it is worth reasoning through rather than picking by feel.

Consider both directions of failure. If the base holds production values and an environment’s overlay is missing or misapplied, that small environment inherits production sizing: six replicas and large resource requests on a dev cluster that cannot schedule them. Best case it fails to schedule and you notice. Worst case it schedules and quietly consumes the cluster.

If the base holds the smallest values and an overlay goes missing, production comes up under-provisioned: fewer replicas than intended, smaller resources. That is also a problem, but it is a visible one, the service is obviously degraded, alerts fire, and you fix it. Failing small and loud beats failing large and silent.

This is precisely the discipline I hold for Kafka replication factor. The shared topic registry declares the smallest-tier value, because the smallest environment runs a single broker and physically cannot replicate. The production replication factor is applied through an environment overlay and never appears in the base registry at all. One place holds tier-specific values, the base is universally applicable, and there is no path by which a single-broker environment tries to create replicated topics or a production cluster silently creates unreplicated ones. The reasoning behind those specific values is covered in Kafka partitions, retention, and compaction.

Kustomize or Helm for environment configuration?

Both solve the problem; they solve it differently and suit different jobs.

	Kustomize	Helm
Mechanism	Patches plain YAML, no templating	Go templates rendered with values files
Manifests stay	Valid, readable YAML	Templates (not directly readable as manifests)
Best at	Your own manifests, small deltas per env	Packaging and distributing software
Parameterization	Deliberately limited	Very powerful (conditionals, loops, functions)
Learning curve	Low	Moderate
Release lifecycle	None (it just renders)	Install/upgrade/rollback, release history
Built into kubectl	Yes (kubectl apply -k)	No, separate tool

Kustomize is the better fit when you own the manifests and environments differ by a handful of values. Its defining virtue is that nothing is templated: every file is real YAML you can read, validate, and diff. That readability matters more than it sounds during an incident, when you want to know what is actually deployed and not mentally render a template.

Helm is the better fit when you are packaging software for others to install, or when parameterization is genuinely complex enough to need conditionals and loops. It also gives you a release lifecycle (versioned installs, upgrades, and rollbacks) which is real operational value that Kustomize does not attempt.

The common resolution, and the one I would default to: Helm for third-party charts you consume, Kustomize for manifests you author. You will install other people’s software as charts regardless, and your own services benefit from staying plain YAML. The two compose fine; Kustomize can even patch rendered Helm output when you need to adjust a third-party chart without forking it.

The deeper point is that this choice matters less than the discipline. A team using Helm values files per environment with one shared chart has the same correct structure as a team using Kustomize overlays. A team with four copied manifest directories has the wrong structure regardless of which tool they hold.

Why not just use environment variables or conditionals?

Two tempting alternatives are worth naming because both degrade badly.

One manifest with conditionals (“if environment is prod, then…”) puts every environment’s configuration in one file with branching logic. It avoids duplication, but the file becomes progressively unreadable, and you can no longer answer “what is deployed in staging?” without mentally evaluating the conditions. It also invites conditions to multiply until the template is a program, at which point reviewing a change means reasoning about branches rather than reading values.

Everything through environment variables pushes configuration out of manifests and into runtime env vars set by the deployment platform. Some of this is correct and healthy, application-level settings genuinely belong in env vars or ConfigMaps. But it fails for the values we are discussing here, because replica counts, resource requests, and PodDisruptionBudgets are Kubernetes object fields, not application settings. They cannot be an env var; they are structure. Trying to force them through that channel usually ends with a bespoke templating script, which is Helm with none of the maturity.

The clean split: object structure and sizing → overlays. Application behavior → ConfigMaps and env vars. Credentials → a secret manager.

Secrets do not belong in overlays

An overlay is a file in version control, which makes it exactly the wrong place for a credential. This is worth stating flatly because the shape of the problem invites the mistake: your production overlay holds the production-specific values, and the production database password is a production-specific value.

Keep secrets out. Overlays should reference secrets by name; the values live in a secret manager or are injected by an operator that syncs them into the cluster. The overlay says “this deployment uses the secret named db-credentials,” and what that secret contains never enters the repository. Kubernetes Secrets are base64-encoded, not encrypted, so committing one is committing a credential in a thin disguise.

If you want secrets in git for GitOps reasons, they must be encrypted in git, sealed secrets or an external-secrets operator pulling from a real manager. That is a legitimate pattern. Plain values in an overlay is not, and it is the kind of thing that stays undetected until a repository access review or a leaked clone. This is the same principle as the broader rule that credentials live in platform secrets, never in config files.

A worked example: what belongs where

Making the split concrete, for a typical service:

Setting	Base	Overlay	Why
Container image name	✅	—	Same everywhere; tag varies by deploy
Ports, probes, labels	✅	—	Structural, identical across environments
Readiness/liveness config	✅	—	Should be the same, or staging tests nothing
Replica count	✅ (1)	✅ prod: raise	Genuine per-tier difference
Resource requests/limits	✅ (small)	✅ prod: raise	Genuine per-tier difference
PodDisruptionBudget	✅	✅ prod tune	Prod may need a stricter budget
Ingress hostname	—	✅ each env	Inherently per-environment
Feature flags	✅ (off)	✅ where on	Safe default, explicit enablement
Database credentials	❌	❌	Secret manager only
Log level	✅ (info)	✅ dev: debug	Safe default, escalate for debugging

The pattern in that table is consistent: the base is a working, safe, minimal deployment, and each overlay states exactly how one environment departs from it. Anything identical across environments stays in the base where it can only be changed once. Notice especially that probes stay in the base, differing probe configuration between staging and production is a classic way for staging to fail to catch a readiness bug.

How do image tags fit into overlays?

The image tag is the one value that changes on every deploy rather than per environment, and it is worth handling deliberately because the obvious approaches both have failure modes.

Never use a floating tag like latest in any environment. It makes deploys non-reproducible: two pods from the same manifest can run different code depending on when they pulled, and a rollback has nothing specific to roll back to. Pin an immutable tag, ideally a digest or a commit SHA, so that a manifest fully determines what runs.

That leaves the question of where the tag lives. Putting it in the base means every environment moves together, which is wrong, the whole point of staging is that it runs a build before production does. Putting a hand-edited tag in each overlay works but invites the copy-paste drift you adopted overlays to avoid.

The pattern that holds up is that the tag is set by the deploy pipeline, into the environment’s overlay, as a mechanical edit. Kustomize has first-class support for this: the pipeline sets the image tag for the target overlay, commits it, and that commit is the deploy record. You get an auditable history of exactly what was deployed to each environment and when, and promotion between environments is the ordinary act of setting the tag that staging validated into production’s overlay.

This is also what makes rollback trivial: reverting the commit that changed the tag restores the previous deployment exactly, because everything else about the environment is unchanged and version-controlled.

Keeping environments honest over time

Structure alone does not prevent drift; it makes drift visible and cheap to fix, which is the most any tooling can do. Three habits keep it from creeping back.

Watch overlay size. A production overlay with forty patches is telling you production has diverged substantially, and each of those patches is something staging does not test. Periodically ask which of them should be promoted into the base so every environment gets them.

Render and diff. Both Kustomize and Helm can render the final manifests without applying them. Diffing rendered production against rendered staging answers “how do these actually differ?” concretely, which is far more reliable than reading overlays and assuming. Doing this in CI on every change catches unintended divergence at review time.

Reconcile actual state against declared state. The manifests describe what should be deployed; someone with cluster access can change what is deployed. GitOps tooling closes this by continuously reconciling the cluster to the repository and flagging or reverting manual changes. Even without full GitOps, periodically diffing live objects against rendered manifests catches the hand-edit that fixed an incident at 3 a.m. and was never written down, which is the most common source of “it works in prod and nowhere else.”

Namespaces are not a substitute for overlays

A related question that comes up: if environments live in separate namespaces, is that not already isolation? It is isolation of runtime, not of configuration. Two namespaces can happily run manifests that have drifted apart, because a namespace says nothing about where the YAML came from.

The two solve different problems and compose well. Namespaces (or separate clusters) give you runtime and policy isolation, the subject of Kubernetes namespace strategy. Overlays give you configuration lineage: a guarantee that staging and production are the same manifests plus a documented, reviewable set of differences. You want both, and having one does not excuse skipping the other.

A config overlay checklist

• One base holds everything common; environments never have full copies.
• The base contains the smallest/safest values; overlays escalate upward.
• The base alone is deployable (works on a laptop or CI cluster with no overlay).
• Overlays contain only genuine differences and read as documentation of them.
• Probes, labels, and structural config stay in the base so staging tests production behavior.
• No secrets in overlays: reference by name; store in a secret manager or encrypt in git.
• Tool choice is deliberate: Kustomize for your manifests, Helm for packaged software.
• CI renders and diffs environments to surface unintended divergence.
• Overlay size is reviewed periodically; recurring patches get promoted to the base.
• Live cluster state is reconciled against declared state to catch manual edits.

Validate rendered output, not just the source

One habit worth adopting early: run schema validation and policy checks against the rendered manifests for each environment, not against the base. A base that validates cleanly can still render into something invalid once an overlay patches it, and the environment where that happens is usually production, because production has the most patches.

Rendering every environment in CI and validating each output catches a whole class of problems at review time: a patch that targets a field that no longer exists, a resource request that exceeds a namespace quota, a manifest that violates a policy rule. Each of those is trivial to fix in a pull request and genuinely disruptive to discover during a deploy.

What I’d do differently

The mistake I would warn against most is letting the first environment split happen by copy, because it is genuinely faster in the moment and the cost is entirely deferred. Copying deployment.yaml to deployment-prod.yaml takes ten seconds and feels harmless. The bill arrives months later, as an incident whose root cause is “staging didn’t have that setting,” and by then unpicking two divergent files into a base and an overlay is a real project that nobody wants to schedule.

If I were setting up a cluster again, I would create the base-and-overlay structure at the moment the second environment appeared, before there was anything meaningful to reconcile. At that point the base is trivially the existing manifests and the first overlay is three lines. The structure costs nothing to adopt early and is tedious to retrofit, which is the classic signature of a decision worth making up front.

The second thing I would be stricter about is the direction of escalation. It is easy to write production’s numbers into the base because production is the environment you think about most, and it seems harmless until a small environment inherits them or a missing overlay goes unnoticed. Smallest in the base, escalate upward, always, so that the failure mode of a forgotten overlay is an obviously under-provisioned service rather than a silent, expensive, or unschedulable one.

Sources

• Kustomize documentation, bases and overlays: kubectl.docs.kubernetes.io/references/kustomize
• Kubernetes, declarative management with Kustomize: kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization
• Helm documentation, charts and values: helm.sh/docs/topics/charts
• Kubernetes, Secrets (encoding is not encryption): kubernetes.io/docs/concepts/configuration/secret