Skip to content

SyRF ApplicationSet Architecture

Overview

The syrf-applications ApplicationSet automatically generates ArgoCD Applications for all SyRF services across all environments. Instead of manually creating 12 separate Application manifests (6 services × 2 environments), it uses generators to dynamically create them from configuration files.

Location: cluster-gitops/argocd/applicationsets/syrf.yaml

Generator Structure

The ApplicationSet uses a merge generator containing a matrix generator:

┌─────────────────────────────────────────────────────────────────┐
│                      MERGE GENERATOR                             │
│  Combines results based on keys: [serviceName, envName]          │
│                                                                  │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │              MATRIX GENERATOR (Base)                     │    │
│  │  Creates Cartesian product of:                           │    │
│  │                                                          │    │
│  │  namespace.yaml files    ×    service/config.yaml files  │    │
│  │  ─────────────────────       ─────────────────────────   │    │
│  │  • envName: staging          • serviceName: api          │    │
│  │  • envName: production       • serviceName: docs         │    │
│  │  • environment:              • serviceName: web          │    │
│  │      project: syrf-staging   • service:                  │    │
│  │      namespace: syrf-staging     chartPath: src/...      │    │
│  │      syncPolicy: {...}           chartRepo: github/syrf  │    │
│  │                                                          │    │
│  │  Result: 2 envs × 6 services = 12 combinations           │    │
│  │  Each has: envName + environment.* + serviceName + service.* │    │
│  └─────────────────────────────────────────────────────────┘    │
│                              +                                   │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │           OVERRIDE GENERATOR                             │    │
│  │  Reads: syrf/environments/*/*/config.yaml                │    │
│  │  (e.g., staging/api/config.yaml)                         │    │
│  │                                                          │    │
│  │  Contains:                                               │    │
│  │  • serviceName: api                                      │    │
│  │  • envName: staging                                      │    │
│  │  • service.enabled: true                                 │    │
│  │  • service.chartTag: api-v9.12.0  ← CI/CD updates this   │    │
│  │  • service.imageTag: "9.12.0"                            │    │
│  │                                                          │    │
│  │  ⚠️  Does NOT have: environment.project, environment.syncPolicy │    │
│  └─────────────────────────────────────────────────────────┘    │
│                                                                  │
│  MERGE: Match on [serviceName, envName], combine all fields      │
└─────────────────────────────────────────────────────────────────┘

Configuration Files

Environment Namespace Files

Path: syrf/environments/*/namespace.yaml

Contains environment-wide settings shared by all services in that environment:

# staging/namespace.yaml
envName: staging
environment:
  namespace: syrf-staging
  project: syrf-staging
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false
    syncOptions:
      - CreateNamespace=true
      - PrunePropagationPolicy=foreground
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

Base Service Config Files

Path: syrf/services/*/config.yaml

Contains service defaults (chart location, name):

# services/api/config.yaml
serviceName: api
service:
  chartPath: src/services/api/.chart
  chartRepo: https://github.com/camaradesuk/syrf

Environment-Specific Service Config Files

Path: syrf/environments/*/*/config.yaml

Contains version and deployment info (updated by CI/CD):

# environments/staging/api/config.yaml
serviceName: api
envName: staging
service:
  enabled: true
  chartTag: api-v9.12.0   # Updated by CI/CD
  imageTag: "9.12.0"      # Updated by CI/CD
gitVersion:
  version: 9.12.0
  sha: "3167ff4b17fd01b2014cfa558e13fb2f447e20b3"

How the Merge Works

  1. Matrix generator creates 12 combinations (2 envs × 6 services)

  2. Each result has: envName, environment.*, serviceName, service.*

  3. Override generator reads 12 environment-specific configs

  4. Each result has: envName, serviceName, service.enabled, service.chartTag, etc.

  5. Does NOT have: environment.* fields

  6. Merge combines on matching [serviceName, envName] keys

  7. Base values from matrix + override values from environment-specific config
  8. Final result has all fields needed for template

Template Evaluation Behavior

The Challenge: Intermediate State Evaluation

ArgoCD doesn't just evaluate templates on final merged results - it also evaluates them on intermediate generator results during processing.

Step 1: Matrix generator produces 12 results
        Each has: envName, environment.*, serviceName, service.*
        ✅ Template can render: {{.serviceName}}-{{.envName}} works
        ✅ Template can render: {{.environment.project}} works

Step 2: Override generator produces 12 results
        Each has: envName, serviceName, service.enabled, service.chartTag
        ❌ Missing: environment.project, environment.syncPolicy

Step 3: ArgoCD tries to evaluate template on EACH result
        Template: project: '{{.environment.project}}'

        On override result → {{.environment.project}} = "<no value>"

Step 4: ArgoCD validation fails on intermediate results
        ❌ ERROR: "application references project <no value>"

Step 5: Merge happens, final 12 results have all fields
        ✅ Actual applications created correctly

Why Applications Work Despite Errors

The final merged results have all required fields, so actual Applications are created correctly. The error is about ArgoCD's intermediate validation of incomplete data, not the final output.

Think of it like a compiler warning vs. a runtime error - the code runs fine, but the compiler complains about something during processing.

Design Solutions

1. No Strict Template Mode

spec:
  goTemplate: true
  # Note: Removed goTemplateOptions: ["missingkey=error"] because it causes
  # RenderTemplateParamsError with merge+matrix generators. ArgoCD evaluates
  # templates before the merge completes, so intermediate results lack keys.

The strict missingkey=error option causes errors on intermediate states. Removing it allows Go's default behavior (render <no value> for missing keys).

2. Conditional templatePatch

templatePatch: |
  {{- if .environment.syncPolicy }}
  spec:
    syncPolicy:
      automated:
        prune: {{.environment.syncPolicy.automated.prune}}
        selfHeal: {{.environment.syncPolicy.automated.selfHeal}}
        allowEmpty: {{.environment.syncPolicy.automated.allowEmpty}}
      syncOptions: {{toJson .environment.syncPolicy.syncOptions}}
      retry: {{toJson .environment.syncPolicy.retry}}
  {{- end }}

The conditional ensures the patch only renders for fully-merged results. Empty templatePatch = no patch applied, which is safe for intermediate states.

3. Selector to Filter Incomplete Results

selector:
  matchExpressions:
    - key: environment.project
      operator: Exists          # Filter out results without this key
    - key: service.enabled
      operator: NotIn
      values: ['false']
    - key: service.chartTag
      operator: NotIn
      values: ['']

The environment.project: Exists selector filters out intermediate override-only results before template evaluation, preventing validation errors.

Visual Summary

WITHOUT SAFEGUARDS:
───────────────────
Matrix Results (12)     Override Results (12)
     ✅                        ❌ (missing environment.*)
       \                      /
        \                    /
         ↘                  ↙
          Template Evaluation
                 ↓
         ❌ Errors on override results
                 ↓
          Merge happens
                 ↓
         ✅ 12 healthy apps (but error status shown)


WITH SAFEGUARDS:
────────────────
Matrix Results (12)     Override Results (12)
     ✅                        ❌ filtered out by selector
       \
        \
         ↘
          Template Evaluation (only complete results)
                 ↓
         ✅ No errors
                 ↓
          Merge happens
                 ↓
         ✅ 12 healthy apps (clean status)

Troubleshooting

Common Errors

"map has no entry for key X"

  • Cause: goTemplateOptions: ["missingkey=error"] with merge generator
  • Fix: Remove the strict option or add default values

"application references project \<no value>"

  • Cause: Template evaluated on intermediate results missing environment.project
  • Fix: Add environment.project: Exists to selector

"json: cannot unmarshal string into Go struct field"

  • Cause: <no value> placeholder in JSON boolean field
  • Fix: Wrap templatePatch in conditional {{- if .environment.syncPolicy }}

Checking ApplicationSet Status

# View conditions
kubectl get applicationset syrf-applications -n argocd -o jsonpath='{.status.conditions}' | jq '.'

# View generated applications
kubectl get applicationset syrf-applications -n argocd -o jsonpath='{.status.resources[*].name}'

# Count applications
kubectl get applicationset syrf-applications -n argocd -o jsonpath='{.status.resourcesCount}'