GitOps Best Practices for Production Kubernetes

GitOps is not "git push to deploy". It is a rule: nothing exists on a cluster that is not written in a repository first. This is how I build it for every Segla tenant. Steal it.

---

The GitOps practices that hold up in production are few. Put everything that runs on the cluster - the applications and the platform underneath them - in one repository that the team owns, and make the cluster reconcile from it. Promote between environments with directory overlays on one branch, never with branches. Keep secrets out of git and reference them from a store. Let the operator self-heal workloads, but only detect drift on the things you would rather look at first. Roll back with git revert. Everything below is the reasoning behind those five sentences, and the shape they take at Segla.

---

What GitOps is

Git is the single source of truth for declarative infrastructure and application configuration. An operator - Argo CD here - continuously reconciles the cluster against what the repository declares.

1. Declarative - the whole system is described, not scripted
2. Versioned and immutable - the desired state lives in git history
3. Pulled, not pushed - the agent in the cluster fetches the state; CI never holds a kubeconfig
4. Continuously reconciled - drift is corrected, or at least surfaced

The fourth point is what makes the rest worth doing. A repository that describes the cluster is documentation. A repository the cluster is held to is an operational model.

---

Repository structure

The three usual answers

Approach	Pros	Cons
Mono-repo	One PR for infra and app changes, easy to search	Permission boundaries are hard, CI builds everything
Multi-repo	Clear ownership, fine-grained access	Cross-repo changes need coordination
Hybrid	App manifests in the app repo, cluster resources in an infra repo	Two places to look, two review paths

I used to recommend the hybrid split: manifests next to the code in a deploy/ directory, cluster-level resources in a separate infra repo. It is reasonable advice for a team that already has a platform group, and an earlier version of this post said exactly that.

The answer Segla settled on: one platform repository that is the whole platform

At onboarding, Segla's GitHub or GitLab app creates one repository inside the customer's own organisation and configures it as the GitOps source for their clusters. Everything Segla puts on a cluster is committed there:

• the manifests for every app, generated from the repositories the team connected
• the platform stack itself - Argo CD, Istio, Prometheus, Grafana, cert-manager, ingress, the operators that hold it together

acme/platform/                      # in acme's own GitHub org
├── apps/
│   ├── shop/
│   │   ├── base/
│   │   └── overlays/
│   │       ├── staging/
│   │       └── production/
│   └── checkout/
├── platform/
│   ├── argocd/
│   ├── istio/
│   ├── prometheus/
│   ├── grafana/
│   └── cert-manager/
└── clusters/
    ├── eu-central/
    └── us-east/

Not the hybrid. Not a Segla-hosted repository with an export button. The repository is the contract: anything on a cluster must be reproducible from it on any Kubernetes, with minimal effort.

Why it lives in the customer's org

Two reasons, and neither is convenience.

Every change is readable. Segla commits as its own app identity. When a node is replaced, an ingress rewired or a version bumped, the diff is in the team's git history, reviewable like any other change. There is no "the platform did something" - there is a commit.

The exit door is built in. A team that leaves Segla keeps the whole platform, not a cluster they cannot rebuild. Clone the repository, point Argo CD at it on any Kubernetes, and the same apps come up on the same stack. This is the honest answer to the hosted-PaaS question "what if we want to run it ourselves later" - and it is a discipline for us: if a feature cannot be expressed in the repository, it is not a platform feature. That single rule is most of what Segla runs for you.

---

Environment promotion

Branch-per-environment - do not

Each branch is an environment; promotion is a merge or a cherry-pick. Branches diverge within weeks and merge conflicts become the deploy process.

Directory-per-environment with Kustomize - yes

One branch, main. One directory per environment, each an overlay on a shared base. Promotion is a change to the image tag in the target overlay - a commit, reviewed like any other.

# apps/shop/overlays/staging/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
images:
  - name: shop
    newTag: v1.2.3-rc.1

# apps/shop/overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
images:
  - name: shop
    newTag: v1.2.2  # promoted after staging held

In the platform repository this is what a release is. Push to the branch, Segla builds the image and commits the new tag into the overlay; the commit is the deploy, and the previous commit is the rollback. A cluster in a fleet gets the release when its overlay does, which is how a canary to one region is just a commit that touches one directory.

Preview environments - when you want them

Argo CD's ApplicationSet with the pull-request generator creates an environment per open PR and deletes it on merge. Useful, and the same repository model carries it: the generated Applications land in the platform repository too.

---

Secrets

Never in git, even encrypted. The repository is the whole platform and every engineer can read it - that is the point - so a secret in it is a secret shared with everyone who will ever have the clone.

Tool	How it works	Best for
External Secrets Operator	Syncs from AWS Secrets Manager, Vault and others into Kubernetes Secrets	Production, more than one provider
Sealed Secrets	Encrypts client-side; only the cluster can decrypt	Single cluster, simple setups
SOPS + age	Encrypts YAML values in place, decrypted on apply	Small teams

The platform repository holds references, and External Secrets Operator resolves them in the cluster:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: shop-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: shop-secrets
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: production/shop/database-url

The repository says which secret an app needs. It never says what it is.

---

Reconciliation: self-heal where it is cheap, detect where it is not

Workloads: self-heal and prune

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  syncPolicy:
    automated:
      selfHeal: true      # put drift back
      prune: true         # remove what the repository no longer declares
    retry:
      limit: 3
      backoff:
        duration: 5s
        maxDuration: 3m

A hand-edited Deployment is reverted to what the repository says within the sync interval. That is the behaviour you want for applications: the repository wins, always.

The platform stack: detect first

syncPolicy:
  automated:
    selfHeal: false   # report the drift, do not undo it yet

For cert-manager, the ingress controller or Istio, a silent self-heal can undo the very thing someone did at 3 a.m. to stop an incident. Detect-only means the drift shows up as an out-of-sync Application and a notification, and a person decides whether the repository or the cluster is right - then commits the answer, so the two agree again.

Rollback is git revert

git revert HEAD
git push
# Argo CD reconciles to the reverted state

No rollback command, no snapshot to find. The previous state is the previous commit, and the audit trail of every rollback is the git log. In Segla a release is a version you can return to for exactly this reason: returning is a revert of the commit that promoted it.

---

What to take from this

1. One platform repository per organisation, in the organisation's own git - the apps and the stack together; the repository is the contract and the exit door
2. Promote with directory overlays on one branch, never with branches
3. Secrets are references - External Secrets Operator or Sealed Secrets, never values in git
4. Self-heal workloads, detect-only the platform layer - the repository wins for apps; a person decides for infrastructure
5. Every change is a PR, every rollback is a revert - the log is the audit trail
6. Validate manifests in CI - kustomize build, kubeconform, conftest before merge

The hybrid split is still a defensible answer for a team with a platform group to own the infra repository. Segla's customers are the teams that do not have one, and for them one repository they own beats two they have to coordinate.

---

This is the model Segla builds for every tenant: connect a repository, and the platform repository in your own org is where every deploy and every platform change lands - readable, and yours to keep.