A realistic EKS GitOps reference implementation - not a drop-in production platform, but a working end-to-end example of how one could be built.
It covers the full lifecycle from infrastructure provisioning to GitOps-managed deployments, observability, autoscaling, backup/restore, policy enforcement, and centralized logging. Deliberate demo trade-offs are documented so the boundary between learning environment and production hardening is explicit.
Who this is for: engineers who want a concrete, opinionated starting point for EKS + GitOps - something closer to real-world platform complexity than a tutorial, while still making the cost and operational shortcuts clear so you know exactly what to harden before taking it further.
| Component | What it does |
|---|---|
| SimpleTimeService | Minimal Python microservice - returns timestamp + caller IP as JSON |
| Terraform | Provisions AWS VPC, EKS cluster, IRSA roles, Karpenter, S3 buckets for Thanos, Loki, and Velero, and Secrets Manager secrets |
| ArgoCD (App of Apps) | GitOps engine - self-managed via Helm, all platform components reconcile from this repo |
| AWS Load Balancer Controller | Provisions ALB Ingress for services and ArgoCD |
| EBS CSI Driver | Provisions EBS volumes for stateful workloads (Prometheus, Thanos); creates gp3 as the default StorageClass |
| External DNS | Creates Route53 DNS records from Ingress/Service annotations |
| Cluster Autoscaler | Scales the managed node group based on pending pods |
| Karpenter | Provisions workload nodes on-demand (t3a.medium / c6a.large) |
| External Secrets Operator | Syncs AWS Secrets Manager secrets into Kubernetes Secrets |
| Reloader | Rolls Deployments automatically when referenced Secrets or ConfigMaps change |
| Prometheus + Grafana | Metrics collection, pre-built dashboard, Slack alerting. Grafana is publicly accessible via ALB ingress; Prometheus and Alertmanager have no ingress and are accessible via kubectl port-forward only. |
| Prometheus CRDs | Prometheus Operator CRDs managed as a separate ArgoCD app (sync-wave 0) to allow safe CRD upgrades |
| Prometheus Adapter | Exposes Prometheus metrics via the Kubernetes custom metrics API, enabling custom-metric HPA |
| Thanos | Long-term metric storage - Prometheus sidecar ships data to S3; Query, Compactor, and StoreGateway provide durable retention |
| HPA + metrics-server | Horizontal pod autoscaling based on CPU utilization |
| Loki + Fluent Bit | Centralized log aggregation backed by S3 object storage, queryable in Grafana |
| OTel Collector (agent + gateway) | Two-tier telemetry pipeline: a DaemonSet agent on every node receives spans from local pods; a central Deployment gateway batches and forwards to Tempo |
| Grafana Tempo | Distributed trace storage - receives spans from the OTel gateway via OTLP gRPC, queryable in Grafana with trace-to-log correlation through Loki; backed by S3 |
| Velero | Cluster backup and restore - backs up Kubernetes resources and EBS volume snapshots to S3 |
| Kyverno | Admission controller - evaluates ClusterPolicy rules on every pod admission and writes PolicyReport objects; all policies run in Audit mode by default |
| Policy Reporter | Web UI for Kyverno PolicyReport and ClusterPolicyReport objects - accessible via kubectl port-forward |
Name mapping:
SimpleTimeService= source inapp/= Helm releasesimple-time-service= manifest ink8s/microservice.yaml. All the same thing.
This platform implements IRSA for every workload, secrets via AWS Secrets Manager + External Secrets Operator, pod security contexts (non-root, read-only filesystem, all capabilities dropped), Kyverno admission policies, Trivy and Checkov scanning in CI, and ArgoCD AppProjects that restrict workload deployments to their own namespaces.
See docs/security.md for a full inventory of implemented controls and the production hardening roadmap.
Running this stack continuously costs roughly $310–330/month at idle (on-demand, ap-south-1). Under active load expect $400–500/month. Tear down the cluster when not in use (
./scripts/cleanup.shfromterraform/) - at rest only S3, Route 53, and Secrets Manager accrue charges (under $10/month).
See docs/cost.md for a full per-component breakdown, cost-control tips, and safe teardown steps.
| .github/CI.md | CI workflows - triggers, steps, required secrets |
| .github/branch-protection.md | Branch protection rules and required status checks |
| app/README.md | Docker image, CI pipeline, endpoints, container security |
| terraform/README.md | Infrastructure provisioning, bootstrap module, remote state |
| charts/namespaces/README.md | Namespace management chart - creates namespaces and ResourceQuotas from a values list |
| charts/simple-time-service/README.md | Helm chart values, install/upgrade, examples |
| charts/raw/README.md | Generic chart for deploying arbitrary K8s resources via ApplicationSets |
| k8s/README.md | Raw Kubernetes manifest (quick-start, no Helm) |
| scripts/README.md | Load testing with Python and k6 |
| GitOps | |
| gitops/README.md | ArgoCD install, bootstrap, sync policy, application inventory |
| gitops/argocd/README.md | AppProjects, least-privilege design, sync wave ordering |
| gitops/auto-scaling/README.md | Cluster Autoscaler, Karpenter, NodePool config, metrics-server, HPA |
| gitops/networking/README.md | AWS Load Balancer Controller, ExternalDNS, Ingress annotations |
| gitops/secrets/README.md | External Secrets Operator, ClusterSecretStore, ExternalSecret mappings, Reloader |
| gitops/storage/README.md | EBS CSI Driver, gp3 StorageClass, PVC inventory |
| gitops/backup/README.md | Velero, S3 backup location, EBS snapshots, backup and restore commands |
| gitops/monitoring/README.md | Prometheus, Grafana, Thanos, Prometheus Adapter, ServiceMonitor, EBS persistence |
| gitops/alerts/README.md | PrometheusRules, Slack alerting, silencing, grouping |
| gitops/logs/README.md | Loki S3 backend, Fluent Bit, log querying in Grafana |
| gitops/tracing/README.md | OTel Collector agent/gateway pipeline, Tempo S3 backend, trace querying and log correlation in Grafana |
| gitops/security/README.md | Kyverno policies, Policy Reporter UI, inspecting policy reports |
| Design | |
| docs/design-decisions.md | Why each architectural choice was made - tool selection, configuration defaults, and demo trade-offs |
| docs/security.md | Implemented security controls and production hardening roadmap |
| docs/cost.md | Monthly cost estimate, cost-control tips, and teardown guidance |
| Runbooks | |
| docs/runbooks/README.md | 13 operational runbooks - compute, networking, GitOps, observability, secrets, security, backup |
| Tool | Version | Purpose |
|---|---|---|
| Terraform | ~> 1.14 |
Provision AWS infrastructure |
| AWS CLI | v2 | Authenticate to AWS |
| Docker | any | Build and run the container locally |
| kubectl | any | Interact with the cluster |
| Helm | >= 3 |
Deploy the chart (bootstrap path only) |
httpd (htpasswd) |
any | Generate bcrypt password hashes for bootstrap/upgrade scripts (brew install httpd on macOS) |
The ALB Ingress for ArgoCD, Grafana, and SimpleTimeService uses HTTPS with HTTP → HTTPS redirect. Prometheus and Alertmanager have no ingress - they are accessed via kubectl port-forward. The ALB listener looks up a certificate by domain match, so an ACM certificate covering your domain must exist and be validated before running bootstrap.sh.
A wildcard certificate is the simplest option - one cert covers all subdomains:
# Request a wildcard cert for *.platform.<your-domain>
aws acm request-certificate \
--domain-name "*.platform.<your-domain>" \
--validation-method DNS \
--region ap-south-1
# The command returns a CertificateArn. Add the CNAME validation record to Route53:
aws acm describe-certificate \
--certificate-arn <CertificateArn> \
--region ap-south-1 \
--query "Certificate.DomainValidationOptions[0].ResourceRecord"Add the returned CNAME to your Route53 hosted zone and wait for the certificate status to reach ISSUED before proceeding:
aws acm wait certificate-validated \
--certificate-arn <CertificateArn> \
--region ap-south-1The ALB controller discovers the certificate automatically by matching the Ingress hostname against ACM-issued certificates in the same region - no ARN needs to be set in the Ingress annotations.
Two paths to a running service - pick one.
One terraform apply provisions the VPC, EKS cluster, and the Helm release. The service is immediately reachable on a public NLB - no ArgoCD required.
cd terraform/app-bootstrap && terraform init && terraform apply
aws eks update-kubeconfig --region ap-south-1 --name simple-eks
curl $(terraform output -raw application_url)Terraform provisions infrastructure only. ArgoCD takes over everything else - the service, monitoring, autoscaling, and logging all reconcile from this repo.
# 1. Set required env vars (Slack webhook required; passwords are auto-generated if omitted)
export TF_VAR_alertmanager_slack_webhook_url="https://hooks.slack.com/..."
# export ARGOCD_ADMIN_PASSWORD="<your-password>"
# export TF_VAR_grafana_admin_password="<your-password>"
# 2. Run the bootstrap script - provisions infra, installs ArgoCD via Helm, applies projects + root app
bash terraform/scripts/bootstrap.sh
# Once complete, services are reachable at:
# ArgoCD: https://argocd.platform.<your-domain>
# Service: https://simple-time-service.platform.<your-domain>
# Grafana: https://grafana.platform.<your-domain>
# Prometheus: kubectl port-forward svc/prometheus-kube-prometheus-prometheus -n monitoring 9090:9090
# Alertmanager: kubectl port-forward svc/prometheus-kube-prometheus-alertmanager -n monitoring 9093:9093
# Policy Reporter: kubectl port-forward svc/policy-reporter-ui -n security 8080:8080
# 3. Watch ArgoCD reconcile (optional, takes up to 3 minutes for first sync)
kubectl get applications -n argocd -wUse upgrade.sh whenever you need to re-apply Terraform changes or rotate secrets on an existing cluster. Unlike bootstrap.sh it does not reinstall ArgoCD or re-apply the root app - it just runs Terraform, refreshes the ArgoCD cluster secret with the latest outputs, and force-annotates all ExternalSecrets so they re-pull from Secrets Manager.
When to use it:
- You changed a Terraform resource and want to apply it
- You want to rotate the ArgoCD admin, Grafana admin, or Alertmanager Slack webhook secret
- The ArgoCD cluster secret is stale (e.g. after a Karpenter instance profile or Thanos, Loki, or Velero bucket name change)
# Slack webhook is always required
export TF_VAR_alertmanager_slack_webhook_url="https://hooks.slack.com/..."
# Passwords are auto-generated if not set; set them to rotate or pin a specific value
# export ARGOCD_ADMIN_PASSWORD="<plaintext>" # script hashes it with bcrypt
# export TF_VAR_argocd_admin_password_hash='$2b$...' # supply a pre-computed bcrypt hash instead
# export TF_VAR_grafana_admin_password="<plaintext>"
bash terraform/scripts/upgrade.shWhat it does:
- Validates required tools (
terraform,aws,kubectl,openssl,git,htpasswd) - Generates or accepts ArgoCD bcrypt hash and Grafana password
- Runs
terraform init→terraform validate→terraform apply - Updates kubeconfig
- Re-applies the ArgoCD cluster secret with the latest Terraform outputs (account ID, VPC ID, domain, Karpenter instance profile, Thanos/Loki/Velero/Tempo bucket names)
- Force-annotates
argocd-admin,grafana-admin, andalertmanager-webhookExternalSecrets to trigger an immediate re-sync from Secrets Manager
| # | What to verify | How |
|---|---|---|
| 1 | Cluster is up | kubectl get nodes - all Ready |
| 2 | ArgoCD synced (GitOps path) | argocd app get root-app - Synced / Healthy |
| 3 | Service responds | Bootstrap: curl http://$(terraform -chdir=terraform/app-bootstrap output -raw application_url)/ GitOps: curl https://simple-time-service.platform.<your-domain>/ - returns timestamp + ip JSON |
| 4 | Prometheus scraping | kubectl port-forward svc/prometheus-kube-prometheus-prometheus -n monitoring 9090:9090 → open http://localhost:9090/targets - simple-time-service shows UP |
| 5 | Grafana dashboard live | Open https://grafana.platform.<your-domain> - SimpleTimeService dashboard has data |
| 6 | HPA reacts to load | kubectl get hpa -n simple-time-service -w while running python3 scripts/load_test.py |
| 7 | Slack alert fires | Fire test alert (see gitops/alerts/README.md) - appears in #alerts-test within 30s |
| 8 | Loki API reachable | kubectl port-forward svc/loki-gateway -n logging 3100:80 → curl 'http://localhost:3100/loki/api/v1/labels' |
| 9 | Logs in Grafana | Explore → Loki datasource → {namespace="simple-time-service"} - log lines include trace_id / span_id fields in JSON |
| 9a | OTel pipeline healthy | kubectl get pods -n tracing - otel-gateway (Deployment) and otel-agent (DaemonSet) all Running |
| 9b | Traces in Grafana | Explore → Tempo datasource → Search → service simple-time-service - spans visible; clicking a trace shows correlated Loki logs |
| 10 | Thanos Query healthy | kubectl get pods -n monitoring -l app.kubernetes.io/name=thanos-query - Running; add Thanos datasource in Grafana pointing to thanos-query.monitoring.svc:9090 |
| 11 | EBS volumes provisioned | kubectl get pv - PVs for Prometheus (20Gi), Alertmanager (2Gi), Thanos Compactor (10Gi), StoreGateway (5Gi) all Bound |
| 12 | Velero running | kubectl get pods -n velero - Running; kubectl get backupstoragelocation -n velero - Available |
| 13 | Kyverno policies synced | kubectl get clusterpolicy - four policies listed; kubectl get policyreport -A - reports generated after pod activity |
| 14 | Policy Reporter UI reachable | kubectl port-forward svc/policy-reporter-ui -n security 8080:8080 → open http://localhost:8080 |
Screenshots captured from a live deployment.
.
├── .github/
│ ├── CI.md # CI workflows - triggers, steps, secrets
│ ├── CODEOWNERS # Automatic review assignment by path
│ ├── branch-protection.md # Branch protection rules and required status checks
│ └── workflows/
│ ├── app-image.yaml # CI - build, scan, and push Docker image to Docker Hub
│ ├── terraform-ci.yaml # CI - format, validate, lint, and security scan Terraform
│ ├── gitops-ci.yaml # CI - helm lint, kubeconform, and yamllint for GitOps manifests
│ ├── actions-check.yaml # CI - lint all GitHub Actions workflow files with actionlint
│ ├── codeowners-check.yaml # CI - validate CODEOWNERS syntax and owner existence
│ └── pr-title-check.yaml # CI - enforce Conventional Commits format on PR titles
├── .yamllint.yaml # yamllint config - rules for gitops-ci YAML linting
├── compose.yaml # Docker Compose for local development
├── app/
│ ├── Dockerfile
│ ├── requirements.txt
│ ├── .dockerignore
│ └── src/
│ └── app.py
├── k8s/
│ └── microservice.yaml # Deployment + ClusterIP Service (quick-start, no Helm)
├── charts/ # READMEs auto-generated by helm-docs (`helm-docs --chart-search-root charts/`)
│ ├── namespaces/ # Namespace management chart - creates namespaces + ResourceQuotas
│ │ ├── Chart.yaml
│ │ ├── values.yaml # Annotated with # -- comments for helm-docs
│ │ ├── README.md # Auto-generated - do not edit directly
│ │ ├── README.md.gotmpl # helm-docs template - edit this for structural changes
│ │ └── templates/
│ │ ├── namespace.yaml
│ │ └── resourcequota.yaml
│ ├── raw/ # Generic chart - renders any K8s resource via .Values.resources
│ │ ├── Chart.yaml
│ │ ├── values.yaml # Annotated with # -- comments for helm-docs
│ │ ├── README.md # Auto-generated - do not edit directly
│ │ ├── README.md.gotmpl # helm-docs template - edit this for structural changes
│ │ └── templates/
│ │ └── resources.yaml
│ └── simple-time-service/ # Helm chart for the microservice
│ ├── Chart.yaml
│ ├── values.yaml # Annotated with # -- comments for helm-docs
│ ├── README.md # Auto-generated - do not edit directly
│ ├── README.md.gotmpl # helm-docs template - edit this for structural changes
│ └── templates/
│ ├── deployment.yaml
│ ├── service.yaml
│ ├── serviceaccount.yaml
│ ├── ingress.yaml
│ ├── hpa.yaml
│ ├── pdb.yaml
│ ├── networkpolicy.yaml
│ ├── servicemonitor.yaml
│ ├── _helpers.tpl
│ └── NOTES.txt
├── gitops/
│ ├── bootstrap/
│ │ └── root-app.yaml # ArgoCD root Application - bootstraps everything below
│ ├── argocd/
│ │ ├── projects/
│ │ │ ├── bootstrap-project.yaml # AppProject - root-app only, minimal scope
│ │ │ ├── namespaces-project.yaml # AppProject - cluster namespace lifecycle
│ │ │ ├── platform-project.yaml # AppProject - ArgoCD, infra, networking, secrets
│ │ │ ├── observability-project.yaml # AppProject - monitoring, logging, alerting
│ │ │ ├── security-project.yaml # AppProject - Kyverno admission controller and policy enforcement
│ │ │ └── workloads-project.yaml # AppProject - application workloads
│ │ ├── argocd.yaml # Application - ArgoCD self-managed via Helm
│ │ ├── argocd-admin-secret.yaml # ApplicationSet - ExternalSecret for ArgoCD admin password
│ │ └── argocd-ingress.yaml # ApplicationSet - ALB Ingress at argocd.platform.<domain>
│ ├── namespaces/
│ │ └── cluster-namespaces.yaml # ApplicationSet - creates all cluster namespaces (wave -1)
│ ├── app/
│ │ └── simple-time-service/
│ │ └── simple-time-service.yaml # ApplicationSet - Helm deploy with ALB Ingress
│ ├── auto-scaling/
│ │ ├── cluster-autoscaler/
│ │ │ └── cluster-autoscaler.yaml # ApplicationSet - Cluster Autoscaler
│ │ ├── karpenter/
│ │ │ ├── karpenter.yaml # ApplicationSet - Karpenter controller (sync-wave 2)
│ │ │ └── karpenter-nodepools.yaml # ApplicationSet - EC2NodeClass + NodePool (sync-wave 3)
│ │ └── metrics-server/
│ │ └── metrics-server.yaml # ApplicationSet - metrics-server (HPA prerequisite)
│ ├── networking/
│ │ ├── ingress-controller/
│ │ │ └── aws-load-balancer-controller.yaml # ApplicationSet - AWS Load Balancer Controller
│ │ └── external-dns/
│ │ └── external-dns.yaml # ApplicationSet - ExternalDNS for Route53
│ ├── storage/
│ │ ├── README.md
│ │ └── ebs-csi-driver/
│ │ └── ebs-csi-driver.yaml # ApplicationSet - AWS EBS CSI Driver (sync-wave 1)
│ ├── backup/
│ │ ├── README.md
│ │ └── velero/
│ │ ├── velero.yaml # ApplicationSet - Velero backup and restore (sync-wave 2)
│ │ └── velero-schedule.yaml # ApplicationSet - daily full-cluster backup Schedule (sync-wave 3)
│ ├── secrets/
│ │ ├── external-secrets/
│ │ │ ├── external-secret-operator.yaml # ApplicationSet - External Secrets Operator
│ │ │ └── cluster-secret-store.yaml # ApplicationSet - ClusterSecretStore (sync-wave 2)
│ │ └── reloader/
│ │ └── reloader.yaml # ApplicationSet - Stakater Reloader
│ ├── monitoring/
│ │ ├── prometheus/
│ │ │ ├── prometheus-crds.yaml # ApplicationSet - Prometheus Operator CRDs (sync-wave 0)
│ │ │ ├── prometheus.yaml # ApplicationSet - kube-prometheus-stack (sync-wave 4)
│ │ │ └── prometheus-adapter.yaml # ApplicationSet - custom metrics API bridge (sync-wave 5)
│ │ ├── thanos/
│ │ │ ├── thanos-objstore-secret.yaml # ApplicationSet - S3 objstore Secret via raw chart (sync-wave 3)
│ │ │ └── thanos.yaml # ApplicationSet - Thanos query/compactor/storegateway (sync-wave 6)
│ │ └── grafana/
│ │ ├── grafana-admin-secret.yaml # ApplicationSet - ExternalSecret for Grafana credentials
│ │ └── simple-time-service-dashboard.yaml # ApplicationSet - Grafana dashboard ConfigMap
│ ├── alerts/
│ │ ├── simple-time-service-alerts.yaml # ApplicationSet - PrometheusRule (alert expressions)
│ │ ├── alertmanager-webhook-secret.yaml # ApplicationSet - ExternalSecret for Slack webhook
│ │ └── alertmanager-slack.yaml # ApplicationSet - AlertmanagerConfig (Slack routing)
│ ├── logs/
│ │ ├── loki/
│ │ │ ├── loki.yaml # ApplicationSet - Loki single-binary log store (sync-wave 3)
│ │ │ └── grafana-loki-datasource.yaml # ApplicationSet - Loki datasource ConfigMap (sync-wave 4)
│ │ └── fluent-bit/
│ │ └── fluent-bit.yaml # ApplicationSet - Fluent Bit DaemonSet (sync-wave 4)
│ ├── tracing/
│ │ ├── README.md # OTel Collector pipeline, Tempo S3 backend, trace querying and log correlation
│ │ ├── tempo/
│ │ │ ├── tempo.yaml # ApplicationSet - Tempo trace store S3 backend (sync-wave 6)
│ │ │ └── grafana-tempo-datasource.yaml # ApplicationSet - Tempo datasource ConfigMap + Loki correlation (sync-wave 8)
│ │ └── otel-collector/
│ │ ├── otel-collector-gateway.yaml # ApplicationSet - OTel Collector Deployment gateway (sync-wave 6)
│ │ └── otel-collector-agent.yaml # ApplicationSet - OTel Collector DaemonSet agent (sync-wave 7)
│ └── security/
│ └── kyverno/
│ ├── kyverno.yaml # ApplicationSet - Kyverno admission controller (sync-wave 3)
│ ├── kyverno-policies.yaml # ApplicationSet - ClusterPolicy rules (sync-wave 5)
│ └── policy-reporter.yaml # ApplicationSet - Policy Reporter UI (sync-wave 6)
├── scripts/
│ ├── load_test.py # Python load generator (no dependencies)
│ └── k6-staged.js # k6 staged ramping-arrival-rate scenario
├── docs/
│ ├── design-decisions.md # Architectural choices and trade-offs
│ ├── security.md # Implemented security controls and production hardening roadmap
│ ├── cost.md # Monthly cost estimate, cost-control tips, and teardown guidance
│ ├── runbooks/ # 13 operational runbooks (incidents, DR, debugging)
│ └── images/ # Screenshots referenced in sub-READMEs
└── terraform/
├── main.tf # Root module - wires VPC and EKS modules
├── backend.tf # S3 remote state
├── variables.tf
├── terraform.tfvars
├── outputs.tf
├── providers.tf
├── versions.tf
├── karpenter.tf # Karpenter IAM + EKS Pod Identity association
├── cluster-autoscaler-irsa.tf # IRSA for Cluster Autoscaler
├── aws-load-balancer-controller-irsa.tf # IRSA for AWS Load Balancer Controller
├── external-dns-irsa.tf # IRSA for ExternalDNS
├── external-secrets-irsa.tf # IRSA for External Secrets Operator
├── ebs-csi-driver-irsa.tf # IRSA for EBS CSI Driver controller
├── thanos.tf # S3 bucket + IRSA roles for Thanos Prometheus sidecar, Compactor, StoreGateway
├── loki.tf # S3 bucket + IRSA role for Loki object storage
├── velero.tf # S3 bucket + IRSA role for Velero backups
├── tempo.tf # S3 bucket + IRSA role for Tempo trace storage
├── secrets-manager.tf # AWS Secrets Manager secrets (ArgoCD, Grafana, Slack)
├── .tflint.hcl # TFLint AWS ruleset plugin config
├── .checkov.yaml # Checkov global skip rules (documented false positives)
├── scripts/ # Shell scripts for full GitOps lifecycle
│ ├── bootstrap.sh # End-to-end: terraform + ArgoCD Helm + projects + root-app
│ ├── upgrade.sh # Re-apply terraform + refresh cluster secret + force ExternalSecret sync
│ └── cleanup.sh # Tear down ArgoCD apps and destroy infrastructure
├── app-bootstrap/ # One-step Terraform module - infra + Helm release
│ ├── main.tf
│ ├── backend.tf
│ ├── variables.tf
│ ├── terraform.tfvars
│ ├── outputs.tf
│ ├── providers.tf
│ └── versions.tf
└── modules/
├── vpc/ # VPC - 2 public + 2 private subnets across 2 AZs
│ ├── main.tf
│ ├── variables.tf
│ └── outputs.tf
└── eks/ # EKS 1.34 - managed node group (tainted app=core:NoSchedule)
├── main.tf
├── variables.tf
└── outputs.tf
- Design Decisions - architectural choices, tool selections, configuration defaults, and demo trade-offs
- Operational Runbooks - 13 runbooks covering incidents, DR, and debugging across all platform components
# GitOps path - the cleanup script removes all ArgoCD apps then destroys infrastructure
bash terraform/scripts/cleanup.sh
# App-bootstrap path
cd terraform/app-bootstrap && terraform destroyThe cleanup script runs six steps in order:
- Kubernetes resources - deletes the root app, all ArgoCD Applications and ApplicationSets, Ingresses, LoadBalancer Services, Karpenter NodePools and NodeClaims, PersistentVolumes, and all namespaces labelled
app.kubernetes.io/part-of=eks-gitops-platform(includingsecurity) (every namespace created by thecluster-namespaceschart) - AWS Load Balancers - deletes any leftover ALB/NLB/Classic ELBs tagged for the cluster that were not removed by step 1
- EBS volumes and snapshots - deletes EBS volumes tagged
kubernetes.io/cluster/<cluster>=owned(CSI Driver PVCs with Retain policy), then deletes EBS snapshots tagged for the cluster or with avelero.io/backuptag - S3 buckets - empties all versioned S3 buckets in the same region whose name contains the cluster name (Thanos, Loki, Velero, Tempo), draining all object versions and delete markers in batches so that Terraform can delete the buckets cleanly
- Kyverno CRDs - deletes all CRDs in the
kyverno.ioandwgpolicyk8s.ioAPI groups; this is handled automatically bycleanup.shusing a label selector - Terraform destroy - removes all AWS resources provisioned by Terraform
The Terraform state bucket is not managed by Terraform itself - delete it manually when no longer needed:
aws s3 rb s3://<your-state-bucket> --force
Setup issues
| Symptom | Fix |
|---|---|
kubectl get nodes - Unauthorized |
Re-run aws eks update-kubeconfig as the same IAM identity that ran terraform apply. |
| ArgoCD apps missing after bootstrap | argocd app sync root-app or click Sync on the root-app tile in the UI. |
| Service unreachable from localhost | Service type is ClusterIP - use kubectl port-forward svc/simple-time-service -n simple-time-service 8080:80. |
Operational issues - see docs/runbooks/ for full diagnosis and fix steps:
| Symptom | Runbook |
|---|---|
kubectl top pods - Metrics API not available / HPA shows <unknown> |
hpa-not-scaling.md |
| HPA not scaling under load | hpa-not-scaling.md |
| ServiceMonitor missing from Prometheus targets | prometheus-target-down.md |
| Prometheus Adapter not serving custom metrics | prometheus-target-down.md |
| Grafana shows "No data" across all dashboards | thanos-no-metrics.md |
Thanos pods in CrashLoopBackOff |
thanos-no-metrics.md |
| Slack alerts not arriving | alertmanager-not-firing.md |
alertmanager-slack app degraded in ArgoCD |
alertmanager-not-firing.md · external-secrets-not-syncing.md |
Kyverno pods crash-looping / kubectl get clusterpolicy returns nothing |
kyverno-policy-violation.md |
| Policy Reporter UI not loading | kyverno-policy-violation.md |




