Verifying PostSync Hook Execution¶

This guide explains how to verify that ArgoCD PostSync hooks are running and creating deployment notifications in GitHub.

Overview¶

PostSync hooks are Kubernetes Jobs that run after ArgoCD successfully syncs an application. In our setup, these hooks:

Update GitHub deployment status to success (ensures deployment status reflects actual cluster state)
Create GitHub commit statuses on the source repository (backward compatibility)
(Production only) Create GitHub releases for deployed versions

Key Design Principle: Deployment success is determined by ArgoCD's actual sync status, not just PR merges. The PostSync hook only runs when ArgoCD successfully deploys to the cluster, making the GitHub deployment status accurately reflect reality.

How It Works¶

Deployment Flow¶

CI/CD Workflow (.github/workflows/ci-cd.yml):
Creates GitHub deployment with status in_progress
Creates PR to cluster-gitops with deployment ID in config.yaml
PR auto-merges
Workflow completes (doesn't wait for ArgoCD)
ArgoCD (cluster):
Detects config change in cluster-gitops
Syncs application to cluster
If sync succeeds, runs PostSync hook
PostSync Hook (Kubernetes Job):
Reads deployment ID from config.yaml
Updates GitHub deployment status to success
Creates commit status (backward compatibility)
(Production only) Creates GitHub release

Why this is better: The deployment status accurately reflects whether ArgoCD successfully deployed to the cluster, not just whether a PR merged. A merged PR doesn't guarantee a successful deployment - sync could fail due to invalid manifests, resource constraints, etc.

Where to Look for Evidence¶

1. GitHub Deployments (Primary)¶

Location: Repository → Deployments tab

Navigate to: https://github.com/camaradesuk/syrf/deployments
Or click on a commit → "Deployments" section on the right

What to expect:

Deployment to staging environment
Status: ✅ Success (updated by PostSync hook)
Environment URL: https://staging.syrf.org.uk (or service-specific URL)
Timeline showing: in_progress → success

If deployment failed:

Status remains in_progress (ArgoCD sync failed, PostSync never ran)
Or status shows failure (if explicitly set)

2. GitHub Commit Statuses (Legacy)¶

Location: Go to the commit that was deployed

Navigate to: https://github.com/camaradesuk/syrf/commits/main
Click on a specific commit SHA
Look for status checks at the bottom of the commit page

What to expect:

✅ Green check mark with context: argocd/deploy-staging
✅ Green check mark with context: argocd/deploy-production
Description: "Deployed to staging environment" or similar

2. ArgoCD UI¶

Location: https://argocd.camarades.net

Steps:

Login to ArgoCD UI
Navigate to the application (e.g., syrf-staging-api)
Click on the "Resources" tab
Look for Jobs with the hook: PostSync annotation

What to expect:

Job named like: api-deployment-notification-{timestamp}
Job status: ✅ Succeeded
Pod logs showing notification success

3. Kubernetes Jobs¶

Using kubectl:

# List all PostSync jobs in staging namespace
kubectl get jobs -n staging -l argocd.argoproj.io/hook=PostSync

# Check specific job details
kubectl describe job api-deployment-notification-12345 -n staging

# View job logs
kubectl logs -n staging job/api-deployment-notification-12345

Expected output:

NAME                              COMPLETIONS   DURATION   AGE
api-deployment-notification-123   1/1           5s         2m
web-deployment-notification-456   1/1           4s         2m

4. GitHub Actions Workflow¶

Location: In the CI/CD workflow run

What to look for:

The "Wait for PR merge and update deployment status" step
Deployment status updates in the workflow summary
GitHub Deployments tab on the repository

5. ArgoCD Application Events¶

In ArgoCD UI:

Navigate to the application
Click on "Events" tab
Look for events like:
"PostSync hook started"
"PostSync hook completed"

6. Kubernetes Events¶

# Check events for the namespace
kubectl get events -n staging --sort-by='.lastTimestamp' | grep -i hook

# Expected output:
# Job/api-deployment-notification created
# Job/api-deployment-notification succeeded

Troubleshooting Missing PostSync Hooks¶

Check 1: Verify Hook Configuration¶

# Check if the service has deployment notifications enabled
kubectl get configmap api-staging-values -n staging -o yaml | grep -A2 deploymentNotification

Expected:

deploymentNotification:
  enabled: true

Check 2: Verify Helm Template¶

# Test helm template generation locally
cd /home/chris/workspace/syrf
helm template api src/services/api/charts/api \
  --values cluster-gitops/syrf/environments/staging/api/values.yaml \
  --set deploymentNotification.enabled=true \
  | grep -A10 "hook:"

Check 3: Check Job TTL¶

PostSync jobs are configured with ttlSecondsAfterFinished: 3600 (1 hour). After this time, completed jobs are automatically cleaned up. If you're checking later, the job may have been deleted.

# Check for recently deleted jobs
kubectl get events -n staging | grep "Deleted job"

Check 4: Verify GitHub App Credentials¶

# Check if the GitHub App secret exists
kubectl get secret github-app-credentials -n staging

# Verify the secret has required keys
kubectl get secret github-app-credentials -n staging -o jsonpath='{.data}' | jq 'keys'
# Expected: ["app-id", "app-installation-id", "app-private-key"]

Common Issues and Solutions¶

Issue 1: No PostSync Jobs Created¶

Symptom: No jobs appear after deployment

Solution:

Verify deploymentNotification.enabled: true in values.yaml
Check ArgoCD sync policy includes hooks
Ensure the Helm chart includes the hook template

Issue 2: Jobs Fail¶

Symptom: Jobs created but show Failed status

Common causes:

GitHub App credentials expired or invalid
Network connectivity issues
Incorrect GitHub repository or commit SHA

Debug steps:

# Get job logs
kubectl logs -n staging job/api-deployment-notification-12345

# Common error messages:
# "401 Unauthorized" - Check GitHub App credentials
# "404 Not Found" - Check repository name and commit SHA
# "Connection refused" - Network/firewall issues

Issue 3: No GitHub Commit Status¶

Symptom: Job succeeds but no commit status appears

Check:

GitHub App has correct permissions (commit statuses: write)
The commit SHA exists in the repository
The GitHub App is installed on the repository

Monitoring PostSync Hooks¶

Set up alerts for failed hooks¶

# Watch for failed PostSync jobs
kubectl get jobs -n staging -l argocd.argoproj.io/hook=PostSync \
  --field-selector status.successful=0

View recent hook executions¶

# Last 10 PostSync job executions
kubectl get jobs -n staging -l argocd.argoproj.io/hook=PostSync \
  --sort-by='.metadata.creationTimestamp' | tail -10

Expected Behavior Summary¶

When everything is working correctly:

After Staging Deployment:
PostSync job creates commit status: argocd/deploy-staging ✅
Job completes successfully and is cleaned up after 1 hour
Notification appears in GitHub commit history
After Production Deployment:
PostSync job creates commit status: argocd/deploy-production ✅
GitHub Release created with version tag
Job completes successfully and is cleaned up after 1 hour
In ArgoCD:
Application shows "Synced" and "Healthy"
PostSync hook resources show as completed
No error events in application events

Verifying PostSync Hook Execution¶

Overview¶

How It Works¶

Deployment Flow¶

Where to Look for Evidence¶

1. GitHub Deployments (Primary)¶

2. GitHub Commit Statuses (Legacy)¶

2. ArgoCD UI¶

3. Kubernetes Jobs¶

4. GitHub Actions Workflow¶

5. ArgoCD Application Events¶

6. Kubernetes Events¶

Troubleshooting Missing PostSync Hooks¶

Check 1: Verify Hook Configuration¶

Check 2: Verify Helm Template¶

Check 3: Check Job TTL¶

Check 4: Verify GitHub App Credentials¶

Common Issues and Solutions¶

Issue 1: No PostSync Jobs Created¶

Issue 2: Jobs Fail¶

Issue 3: No GitHub Commit Status¶

Monitoring PostSync Hooks¶

Set up alerts for failed hooks¶

View recent hook executions¶

Expected Behavior Summary¶

Related Documentation¶