GitOps Best Practices for Production Kubernetes

GitOps is more than "git push to deploy". It's an operational model where the entire desired state of your system is declared in Git. This guide covers real-world patterns we use at Segla.

---

What is GitOps?

GitOps uses Git as the single source of truth for declarative infrastructure and application configuration. An operator (like ArgoCD) continuously reconciles the cluster state with what's declared in the repository.

Core principles:

1. Declarative - the entire system is described declaratively 2. Versioned and immutable - the desired state is stored in Git 3. Pulled automatically - agents pull the desired state, not push 4. Continuously reconciled - drift is automatically corrected

---

Repository Structure

Mono-repo vs multi-repo

Approach	Pros	Cons
Mono-repo	Single PR for infra + app changes, easier search	Permission boundaries harder, CI builds everything
Multi-repo	Clear ownership, fine-grained access	Cross-repo changes need coordination
Hybrid	App configs in app repo, infra in dedicated repo	Best of both, slightly more complex

Our recommendation: Hybrid approach. Keep application manifests close to the code (same repo, deploy/ directory), but maintain a separate infra repo for cluster-level resources.

# Application repo
app-repo/
├── src/
├── Dockerfile
└── deploy/
    ├── base/
    │   ├── deployment.yaml
    │   ├── service.yaml
    │   └── kustomization.yaml
    └── overlays/
        ├── staging/
        └── production/
# Infrastructure repo
infra-repo/
├── clusters/
│   ├── staging/
│   └── production/
├── namespaces/
├── rbac/
└── monitoring/

---

Environment Promotion

Strategy 1: Branch-per-environment (simple, not recommended)

Each branch represents an environment. Promotions happen via merge/cherry-pick.

Problem: Branches diverge over time. Merge conflicts become a constant headache.

Strategy 2: Directory-per-environment with Kustomize (recommended)

A single branch (main) with directory overlays per environment. Promote by copying or updating image tags in the target overlay.

# overlays/staging/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
images:
  - name: app
    newTag: v1.2.3-rc.1
# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
images:
  - name: app
    newTag: v1.2.2  # promoted after staging validation

Strategy 3: ApplicationSet with pull request generator (advanced)

Use ArgoCD ApplicationSet to automatically create preview environments from pull requests.

---

Secrets Management

Never store secrets in Git, even encrypted. Use one of these approaches:

Tool	How it works	Best for
External Secrets Operator	Syncs secrets from AWS SM, Vault, etc. into K8s Secrets	Production, multi-cloud
Sealed Secrets	Encrypts secrets client-side, only the cluster can decrypt	Simple setups, single cluster
SOPS + age	Encrypts YAML values in-place, decrypted during apply	Small teams, straightforward

External Secrets Operator example

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: app-secrets
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: production/app/database-url

---

Rollback Patterns

Automatic rollback with ArgoCD

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  syncPolicy:
    automated:
      selfHeal: true      # auto-fix drift
      prune: true          # remove orphaned resources
    retry:
      limit: 3
      backoff:
        duration: 5s
        maxDuration: 3m

Manual rollback

# Revert the last commit
git revert HEAD
git push
# ArgoCD will automatically sync to the reverted state

The beauty of GitOps: every rollback is just a git revert. The full audit trail lives in Git history.

---

Drift Detection

Configure ArgoCD to alert on drift without auto-fixing (useful for critical namespaces):

syncPolicy:
  automated:
    selfHeal: false   # detect but don't auto-fix

Set up notifications to Slack/email when drift is detected. Investigate manual changes before they become invisible.

---

Key Takeaways

1. Use the hybrid repo approach - app configs with app code, infra configs in a dedicated repo 2. Promote with directory overlays, not branches 3. Never put secrets in Git - use External Secrets Operator or Sealed Secrets 4. Enable self-heal for non-critical workloads, detect-only for critical ones 5. Every change goes through PR - this is your audit trail and approval gate 6. Test manifests in CI - kustomize build, kubeval, conftest before merge

---

At Segla, GitOps is a first-class citizen. Every cluster gets an ArgoCD instance pre-configured with these patterns. Get started free →