ApplicationSets Deep Dive

Overview

ApplicationSets are ArgoCD's solution to the "application explosion" problem. Instead of manually creating dozens of Application manifests, you create one ApplicationSet that auto-generates Applications based on patterns and conventions.

---

The Problem

Without ApplicationSets, deploying 6 services to 2 environments requires 12 Application manifests:

apps/api-staging.yaml
apps/api-production.yaml
apps/web-staging.yaml
apps/web-production.yaml
apps/pm-staging.yaml
apps/pm-production.yaml
... (6 more files)

Maintenance nightmare: - Add new service? Create 2 new files - Change common config? Update 12 files - Easy to make mistakes and create drift

---

The Solution: ApplicationSets

One ApplicationSet can generate all 12 Applications automatically:

# applicationsets/syrf.yaml (ONE file)
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: syrf-applications
spec:
  generators:
    - matrix:
        generators:
          - git: # Discover environments
          - git: # Discover services
          - git: # Discover configs
  template:
    # Template for generated Applications

Benefits: - ✅ Add new service: Just create directory + config file - ✅ Change common config: Update template once - ✅ Consistent: All Applications use same pattern - ✅ Scalable: 6 services or 60 services, same effort

---

How SyRF ApplicationSet Works

High-Level Flow

1. Git Generators discover:
   ├─ Environments (staging, production)
   ├─ Service lists (which services in each env)
   └─ Service configs (chart paths, repos)

2. Matrix Generator creates combinations:
   ├─ api × staging = api-staging
   ├─ api × production = api-production
   ├─ web × staging = web-staging
   └─ (etc. for all services × all environments)

3. Template generates Applications:
   └─ Each combination → Complete Application manifest

Detailed Walkthrough

Step 1: Discover Environments

- git:
    repoURL: https://github.com/camaradesuk/cluster-gitops
    revision: main
    files:
      - path: "environments/*/namespace.yaml"

Discovers:

environments/staging/namespace.yaml
  ↓
  environment:
    name: staging
    namespace: syrf-staging
    project: syrf-staging

environments/production/namespace.yaml
  ↓
  environment:
    name: production
    namespace: syrf-production
    project: syrf-production

Result: 2 environments discovered

Step 2: Discover Services (Updated 2025-11-13)

New structure: Service configs are now self-contained files

- git:
    repoURL: https://github.com/camaradesuk/cluster-gitops
    revision: main
    files:
      - path: "environments/{{.environment.name}}/services/*.yaml"

For each environment, discovers:

environments/staging/services/api.yaml
  ↓
  service:
    name: api
    enabled: true
    chartTag: api-v8.21.1
    chartRepo: https://github.com/camaradesuk/syrf
    chartPath: src/services/api/.chart

environments/staging/services/web.yaml
  ↓
  service:
    name: web
    enabled: true
    chartTag: web-v5.0.2
    chartRepo: https://github.com/camaradesuk/syrf
    chartPath: src/services/web/.chart

environments/production/services/api.yaml
  ↓
  service:
    name: api
    enabled: true
    chartTag: api-v8.21.0
    chartRepo: https://github.com/camaradesuk/syrf
    chartPath: src/services/api/.chart

Result: Self-contained service configs per environment (includes chartTag, chartRepo, chartPath)

Note: Disabled services are excluded by not having a file in that environment's services/ directory

Step 2a: Matrix Combines Everything

Matrix generator creates cartesian product:

Staging × API:
  environment.name = staging
  environment.namespace = syrf-staging
  environment.project = syrf-staging
  service.name = api
  service.chartTag = api-v8.21.1
  service.chartPath = src/services/api/.chart
  service.chartRepo = https://github.com/camaradesuk/syrf

Production × API:
  environment.name = production
  environment.namespace = syrf-production
  environment.project = syrf-production
  service.name = api
  service.chartTag = api-v8.21.0
  service.chartPath = src/services/api/.chart
  service.chartRepo = https://github.com/camaradesuk/syrf

(etc. for all combinations)

Step 3: Template Generates Applications

For each combination, template generates:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-staging  # ← From template: {{.service.name}}-{{.environment.name}}
  namespace: argocd
  labels:
    app: api
    environment: staging
    project: syrf-staging
spec:
  project: syrf-staging  # ← From environment config

  sources:
    - repoURL: https://github.com/camaradesuk/syrf.git  # ← From service config
      targetRevision: api-v8.21.1  # ← From service list
      path: src/services/api/charts/api  # ← From service config
      helm:
        valueFiles:
          - $values/global/values.yaml
          - $values/environments/staging/shared-values.yaml
          - $values/syrf/api/values.yaml
          - $values/syrf/api/values-staging.yaml
        parameters:
          - name: image.tag
            value: "8.21.1"  # ← Derived from chartTag

    - repoURL: https://github.com/camaradesuk/cluster-gitops
      targetRevision: main
      ref: values

  destination:
    server: https://kubernetes.default.svc
    namespace: syrf-staging  # ← From environment config

  syncPolicy: # ← From environment config
    automated:
      prune: true
      selfHeal: true

Result: Complete Application manifest, auto-generated from discovered data

---

Custom Charts ApplicationSet

Purpose

Automatically deploy custom Helm charts from charts/ directory.

How It Works

generators:
  - matrix:
      generators:
        # Discover charts using directory structure
        - git:
            directories:
              - path: "charts/*"

        # Deploy to both environments
        - list:
            elements:
              - env: staging
              - env: production

Example:

You create:

charts/extra-secrets/
  ├── Chart.yaml
  ├── values.yaml
  ├── values-staging.yaml      # Environment-specific values
  ├── values-production.yaml
  └── templates/
      └── external-secrets.yaml

ApplicationSet automatically creates:

extra-secrets-staging Application
extra-secrets-production Application

Values are loaded from:

global/values.yaml (all charts)
charts/extra-secrets/values-staging.yaml (co-located with chart)
charts/extra-secrets/values-production.yaml (co-located with chart)

Why This is Useful

Before ApplicationSets: 1. Create chart in charts/extra-secrets/ 2. Manually create apps/extra-secrets-staging.yaml 3. Manually create apps/extra-secrets-production.yaml 4. Create separate values files in infrastructure/ 5. Update values paths in both Application files 6. Commit 5+ files

With ApplicationSets: 1. Create chart in charts/extra-secrets/ with Chart.yaml, templates/, values files 2. Push to git 3. Done! Applications auto-created, values co-located with chart

---

Infrastructure ApplicationSet

Purpose

Deploy infrastructure components (cert-manager, ingress-nginx, rabbitmq, etc.) using external Helm charts.

How It Works

generators:
  - git:
      files:
        - path: "infrastructure/*/config.yaml"

Discovers:

infrastructure/cert-manager/config.yaml
  ↓
  infra:
    name: cert-manager
    repoURL: https://charts.jetstack.io
    chart: cert-manager
    version: v1.15.0
    namespace: cert-manager

infrastructure/rabbitmq/config.yaml
  ↓
  infra:
    name: rabbitmq
    repoURL: https://charts.bitnami.com/bitnami
    chart: rabbitmq
    version: 14.6.6
    namespace: syrf-staging

Generates Applications:

cert-manager → Uses Jetstack chart v1.15.0
rabbitmq → Uses Bitnami chart 14.6.6
(etc.)

---

Advanced Features

Selective Deployment with Filters

Problem: Not all services should be deployed to all environments

Solution: Filter in service list + ApplicationSet selector

# environments/staging/syrf.yaml
services:
  - name: quartz
    enabled: false  # ← Not deployed to staging

# ApplicationSet selector
selector:
  matchExpressions:
    - key: enabled
      operator: In
      values: ["true"]

Result: Quartz is skipped in staging, only deployed to production

Image Tag Derivation

Problem: Don't want to duplicate image tag in multiple places

Solution: Derive image tag from chartTag

# environments/staging/syrf.yaml
services:
  - name: api
    chartTag: api-v8.21.1  # ← Only place version is defined

# ApplicationSet template
parameters:
  - name: image.tag
    value: '{{trimPrefix (printf "%s-v" .service.name) .service.chartTag}}'
    # api-v8.21.1 → 8.21.1

Result: Single source of truth for versions

Value Hierarchy

ApplicationSet injects values in order:

helm:
  valueFiles:
    - $values/global/values.yaml                    # 1. Global
    - $values/environments/staging/shared-values.yaml  # 2. Environment
    - $values/syrf/api/values.yaml                  # 3. Service defaults
    - $values/syrf/api/values-staging.yaml          # 4. Service + env
  ignoreMissingValueFiles: true  # Files 3 & 4 optional

Helm merges them (last wins): 1. Chart defaults (in chart repo) 2. Global values (all services, all envs) 3. Environment shared (all services in this env) 4. Service defaults (this service, all envs) 5. Service + environment (this service, this env)

---

Troubleshooting

ApplicationSet Not Creating Applications

Check ArgoCD ApplicationSet controller logs:

kubectl logs -n argocd -l app.kubernetes.io/component=applicationset-controller

Common issues: - Git repository not accessible - Malformed YAML in discovered files - Selector filtering out all services - Template syntax errors

Applications Not Syncing

Check individual Application status:

kubectl get application -n argocd api-staging -o yaml

Common issues: - AppProject restrictions (source repos, destinations) - Invalid helm values - Missing secrets/configmaps - Sync policy misconfiguration

Image Tag Not Updating

Check: 1. Is chartTag updated in environments/*/syrf.yaml? 2. Is ApplicationSet regenerating Application? 3. Is ArgoCD syncing the Application? 4. Is image pullPolicy correct?

Debug:

# Check ApplicationSet status
kubectl get applicationset -n argocd syrf-applications -o yaml

# Check generated Application
kubectl get application -n argocd api-staging -o yaml | grep image.tag

Application Degraded - "path does not exist"

Symptom:

Application status: Degraded, Unknown sync
Error: plugins/local/my-chart-production/resources: app path does not exist

Root Cause: Multi-source Applications require all paths to exist, even if empty. When an ApplicationSet template references multiple sources (e.g., chart + values + resources), Git generators will fail if any referenced directory doesn't exist in the repository.

Solution: Create empty directories with .gitkeep files to preserve them in Git:

# Create missing directories
mkdir -p plugins/local/my-chart-production/resources
mkdir -p plugins/local/my-chart-staging/resources

# Add .gitkeep to preserve empty directories
touch plugins/local/my-chart-production/resources/.gitkeep
touch plugins/local/my-chart-staging/resources/.gitkeep

# Commit and push
git add plugins/local/my-chart-*/resources/
git commit -m "fix: add missing resources directories for ApplicationSet"
git push

Prevention: When creating new ApplicationSet entries that use multiple sources: 1. Review the ApplicationSet template to identify all required paths 2. Create all directories upfront, even if initially empty 3. Add .gitkeep files to empty directories 4. Test by checking Application sync status after pushing

Example: The extra-secrets ApplicationSet uses three sources: - charts/extra-secrets/ - Helm chart - plugins/helm/external-secrets-operator/ - ESO values - plugins/local/extra-secrets-{env}/resources/ - Local resources

All three paths must exist for each environment, or the Application will show Degraded status.

---

Best Practices

1. Keep Templates Simple

Bad:

template:
  spec:
    {{- if eq .environment.name "production" }}
    syncPolicy:
      automated:
        selfHeal: false
    {{- else }}
    syncPolicy:
      automated:
        selfHeal: true
    {{- end }}

Good:

template:
  spec:
    syncPolicy: {{.environment.syncPolicy | toJson}}

# Store policy in environment config
# environments/production/namespace.yaml:
#   syncPolicy:
#     automated:
#       selfHeal: false

Why: Configuration belongs in data files, not templates

2. Use Filters Liberally

Example:

selector:
  matchExpressions:
    - key: enabled
      operator: In
      values: ["true"]

Why: Explicit control over what gets deployed

3. Validate Before Pushing

Test template rendering:

# Dry-run to see generated manifests
argocd appset generate syrf-applications --dry-run

Why: Catch errors before they reach the cluster

4. Document Your Generators

Add comments explaining logic:

generators:
  # Matrix: environment × services
  # Creates all combinations (api-staging, api-production, etc.)
  - matrix:
      generators:
        # Discover environments (staging, production, preview)
        - git: ...

Why: Future maintainers will thank you

---

Summary

Feature	Benefit
Auto-discovery	Add service → Push to git → Auto-deployed
DRY templates	Change once, applies to all generated Applications
Scalability	6 services or 60 services, same effort
Consistency	All Applications follow same pattern
Filtering	Selective deployment per environment
Type safety	Go templates catch errors early

Bottom line: ApplicationSets are essential for managing applications at scale with GitOps.

---

Further Reading

• Official ArgoCD ApplicationSet Docs
• Git Generator Docs
• Matrix Generator Docs