How To: Manage DNS Records with External-DNS¶

Overview¶

This guide explains how DNS records are managed in the SyRF cluster using external-dns and what to do if you need to work with DNS records.

How External-DNS Works¶

External-DNS automatically creates and manages DNS records for Kubernetes Ingresses and LoadBalancer Services.

Key Concepts¶

TXT Registry: External-DNS uses TXT records to track ownership of DNS records
For each A record, it creates corresponding TXT records
Example: argocd.camarades.net A record has argocd.camarades.net and a-argocd.camarades.net TXT records
Upsert-Only Policy: Configured to avoid deleting existing records
Will create new records
Will update records it owns (with TXT ownership tracking)
Will NOT touch records without TXT ownership records
Managed Zones:
syrf-org-uk-zone (syrf.org.uk) - Service ingresses for staging/production
camarades-net-zone (camarades.net) - Infrastructure services (ArgoCD, etc.)

Configuration¶

External-DNS configuration: cluster-gitops/plugins/helm/external-dns/values.yaml

provider: google
policy: upsert-only  # Won't delete records without ownership
domainFilters:
  - syrf.org.uk
  - camarades.net
sources:
  - service
  - ingress

Common Tasks¶

Verify DNS Records¶

# List all records in a zone
gcloud dns record-sets list --zone=camarades-net-zone

# Check specific record
gcloud dns record-sets list --zone=camarades-net-zone | grep argocd

# Check DNS resolution
nslookup argocd.camarades.net

Check External-DNS Logs¶

# View recent logs
kubectl logs -n external-dns -l app.kubernetes.io/name=external-dns --tail=50

# Check for errors
kubectl logs -n external-dns -l app.kubernetes.io/name=external-dns | grep -i error

# Watch for DNS record creation
kubectl logs -n external-dns -l app.kubernetes.io/name=external-dns -f | grep "Add records"

Trigger Manual Sync¶

External-DNS syncs every 60 seconds by default. To force an immediate sync:

# Restart external-dns pod
kubectl rollout restart deployment external-dns -n external-dns

Troubleshooting¶

External-DNS Not Creating Records¶

Symptoms: Ingress exists but no DNS records created

Checks:

Verify Ingress has a LoadBalancer IP assigned:
```
kubectl get ingress -A
```

Check external-dns logs for errors:

kubectl logs -n external-dns -l app.kubernetes.io/name=external-dns --tail=100

Verify the hostname matches domain filter:
Must end with .syrf.org.uk or .camarades.net
Check Ingress annotations (none required for basic functionality)

Records Exist But Not Updating¶

Symptoms: DNS record points to old IP, external-dns logs show no changes

Most Likely Cause: Record created manually without TXT ownership tracking

Solution:

# 1. Delete the manually-created record
gcloud dns record-sets delete <hostname> \
  --zone=<zone-name> \
  --type=A

# 2. Wait ~60 seconds for next external-dns sync
sleep 60

# 3. Verify external-dns recreated with ownership
gcloud dns record-sets list --zone=<zone-name> | grep <hostname>
# Should see both A and TXT records

DNS Resolution Not Working¶

Symptoms: nslookup returns NXDOMAIN

Checks:

Verify record exists in Cloud DNS:

gcloud dns record-sets list --zone=<zone-name> | grep <hostname>

Check DNS propagation (use public DNS):
```
nslookup <hostname> 8.8.8.8
```
Check local DNS cache:
Local resolvers may cache NXDOMAIN responses for 5-15 minutes
Try using a public DNS server (8.8.8.8) to bypass cache

CRITICAL WARNINGS¶

⚠️ NEVER Manually Create DNS Records¶

DON'T DO THIS:

# ❌ WRONG - Creates record without ownership tracking
gcloud dns record-sets update argocd.camarades.net. \
  --zone=camarades-net-zone \
  --type=A \
  --rrdatas=34.13.63.21

Why this is bad: - External-DNS won't recognize the record as owned - Record won't be updated automatically if LoadBalancer IP changes - Creates management inconsistency

What to do instead: - Create/update the Kubernetes Ingress or Service - Let external-dns manage the DNS records automatically

⚠️ Deleting External-DNS Managed Records¶

If you must delete a DNS record managed by external-dns:

Delete the source (Ingress/Service) first
External-DNS will automatically clean up DNS records
If you delete DNS records directly, external-dns will recreate them

Examples¶

Example 1: New Ingress Creates DNS¶

# 1. Create ingress
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app
  namespace: my-namespace
spec:
  ingressClassName: nginx
  rules:
  - host: my-app.syrf.org.uk
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-app
            port:
              number: 80
EOF

# 2. Wait for LoadBalancer IP assignment
kubectl get ingress my-app -n my-namespace -w

# 3. Wait ~60 seconds for external-dns sync

# 4. Verify DNS record created
gcloud dns record-sets list --zone=syrf-org-uk-zone | grep my-app
# Should show both A and TXT records

Example 2: Fix Manually-Created Record¶

# Problem: argocd.camarades.net exists but no TXT ownership

# 1. Delete manual record
gcloud dns record-sets delete argocd.camarades.net. \
  --zone=camarades-net-zone \
  --type=A

# 2. Watch external-dns logs
kubectl logs -n external-dns -l app.kubernetes.io/name=external-dns -f

# Within ~60 seconds, you should see:
# "Add records: argocd.camarades.net. A [34.13.63.21] 300"
# "Add records: argocd.camarades.net. TXT [\"heritage=external-dns...\"] 300"

# 3. Verify records created with ownership
gcloud dns record-sets list --zone=camarades-net-zone | grep argocd

References¶

External-DNS Documentation
Google Cloud DNS Provider
Configuration: cluster-gitops/plugins/helm/external-dns/values.yaml

History¶

2025-11-16: Initial guide created after troubleshooting manual DNS record issue