Lambda GitOps Integration¶
Executive Summary¶
This document consolidates the Lambda GitOps strategy for SyRF, providing a clear roadmap for integrating the S3 Notifier Lambda with our GitOps deployment workflow. It supersedes three earlier proposals that explored different approaches.
Key Decisions Made:
- Keep Terraform/AWS CLI deployment - ACK migration deferred (complexity outweighs benefits)
- Version tracking in cluster-gitops - Lambda versions tracked alongside K8s services
- Staged implementation - Tier 1 (version tracking) → Tier 2 (staging isolation) → Tier 3 (deferred)
Current State¶
What Exists¶
| Component | Status | Notes |
|---|---|---|
| Production Lambda | ✅ Exists | syrfAppUploadS3Notifier |
| Preview Lambdas | ✅ Exists | syrfAppUploadS3Notifier-pr-{N} (ephemeral) |
| Staging Lambda | ❌ Missing | No isolated staging environment |
| S3 Bucket | ✅ Shared | syrfappuploads with prefix isolation |
| Version Tracking | ✅ Fixed | s3NotifierVersion in api/values.yaml |
Architecture Diagram¶
┌─────────────────────────────────────────────────────────────────────────────┐
│ CURRENT LAMBDA ARCHITECTURE │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ S3 Bucket: syrfappuploads │
│ ├── Projects/ → syrfAppUploadS3Notifier (PRODUCTION) │
│ ├── staging/ → (NO LAMBDA - gap to fix) │
│ └── preview/pr-{N}/ → syrfAppUploadS3Notifier-pr-{N} (PREVIEW) │
│ │
│ Version Tracking (cluster-gitops): │
│ ├── staging/api/values.yaml → s3NotifierVersion: "X.Y.Z" │
│ ├── production/api/values.yaml → s3NotifierVersion: "X.Y.Z" │
│ └── preview/pr-{N}/services/api.values.yaml → s3NotifierVersion │
│ │
│ CI/CD Flow: │
│ 1. Build Lambda package (.zip) │
│ 2. Deploy to AWS Lambda (aws lambda update-function-code) │
│ 3. Update cluster-gitops api/values.yaml with version │
│ 4. ArgoCD syncs → API pod gets new env var → Version dialog shows version │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Implementation Tiers¶
Tier 1: Version Tracking (✅ COMPLETED - PR #2284)¶
Goal: Track Lambda versions in GitOps values files for visibility
What was done:
- Fixed
pr-preview-lambda.ymlto writes3NotifierVersiontoapi.values.yaml - Fixed
ci-cd.ymlstaging promotion to write toapi/values.yaml - Fixed
ci-cd.ymlproduction promotion to read/write fromapi/values.yaml - API exposes version via
ApplicationInfoendpoint - Web Version Info dialog displays S3 Notifier version
Files changed:
.github/workflows/pr-preview-lambda.yml.github/workflows/ci-cd.ymlsrc/services/api/SyRF.API.Endpoint/Controllers/ApplicationController.cssrc/charts/syrf-common/env-mapping.yaml
Tier 2: Staging Lambda Isolation (PLANNED)¶
Goal: Create isolated staging Lambda for safe testing before production
Problem Statement:
Currently, there is no staging Lambda. Changes to the Lambda code go directly from preview environments to production. This creates risk:
- No integration testing with staging K8s services
- No verification with staging MongoDB
- Production is the first "real" environment
Proposed Solution:
┌─────────────────────────────────────────────────────────────────────────────┐
│ TIER 2: STAGING LAMBDA ISOLATION │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ New Lambda: syrfAppUploadS3Notifier-staging │
│ │
│ S3 Bucket: syrfappuploads (unchanged) │
│ ├── Projects/ → syrfAppUploadS3Notifier (production) │
│ ├── staging/ → syrfAppUploadS3Notifier-staging (NEW) │
│ └── preview/pr-{N}/ → syrfAppUploadS3Notifier-pr-{N} │
│ │
│ S3 Notification Configuration: │
│ - Production: prefix=Projects/ → production Lambda │
│ - Staging: prefix=staging/ → staging Lambda (NEW) │
│ - Preview: prefix=preview/pr-{N}/ → preview Lambda │
│ │
│ Environment Variables (staging Lambda): │
│ - RABBITMQ_HOST: amqp://rabbitmq.syrf-staging.svc.cluster.local:5672 │
│ - ENVIRONMENT: staging │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Implementation Tasks:
Phase 1: Infrastructure Setup¶
- T2.1 Create staging Lambda function in AWS
aws lambda create-function \
--function-name syrfAppUploadS3Notifier-staging \
--runtime dotnet8 \
--handler "SyRF.S3FileSavedNotifier.Endpoint::SyRF.S3FileSavedNotifier.Endpoint.Function::FunctionHandler" \
--role arn:aws:iam::ACCOUNT_ID:role/lambda-s3-notifier-execution-role \
--environment "Variables={RABBITMQ_HOST=amqp://rabbitmq.syrf-staging.svc.cluster.local:5672,ENVIRONMENT=staging}"
- T2.2 Add S3 notification for staging prefix
aws s3api put-bucket-notification-configuration \
--bucket syrfappuploads \
--notification-configuration '{
"LambdaFunctionConfigurations": [
{
"LambdaFunctionArn": "arn:aws:lambda:eu-west-1:ACCOUNT_ID:function:syrfAppUploadS3Notifier",
"Events": ["s3:ObjectCreated:*"],
"Filter": {"Key": {"FilterRules": [{"Name": "prefix", "Value": "Projects/"}]}}
},
{
"LambdaFunctionArn": "arn:aws:lambda:eu-west-1:ACCOUNT_ID:function:syrfAppUploadS3Notifier-staging",
"Events": ["s3:ObjectCreated:*"],
"Filter": {"Key": {"FilterRules": [{"Name": "prefix", "Value": "staging/"}]}}
}
]
}'
- T2.3 Add Lambda permission for S3 to invoke staging Lambda
aws lambda add-permission \
--function-name syrfAppUploadS3Notifier-staging \
--statement-id AllowS3Invoke \
--action lambda:InvokeFunction \
--principal s3.amazonaws.com \
--source-arn arn:aws:s3:::syrfappuploads
Phase 2: CI/CD Updates¶
- T2.4 Update
ci-cd.ymlto deploy to staging Lambda - Add deploy step after Lambda build
- Deploy to
syrfAppUploadS3Notifier-stagingon merge to main -
Update staging api/values.yaml with version
-
T2.5 Update promotion workflow
- Production promotion deploys to
syrfAppUploadS3Notifier(existing) - Copies version from staging to production in cluster-gitops
Phase 3: Validation¶
- T2.6 Test staging Lambda end-to-end
- Upload file to staging environment
- Verify Lambda triggers
-
Verify RabbitMQ message received by staging services
-
T2.7 Document staging Lambda in CLAUDE.md
Estimated Effort: 1-2 days
Dependencies:
- AWS IAM permissions (existing role should work)
- S3 bucket notification update (one-time)
Tier 3: ACK Migration (PLANNED)¶
Goal: Full Kubernetes-native Lambda management via AWS Controllers for Kubernetes
Status: ✅ APPROVED FOR FUTURE IMPLEMENTATION (2026-02-06)
The ACK approach has been re-evaluated and a detailed technical plan has been validated against the codebase. Separate S3 buckets per environment aligns with the existing full isolation pattern (MongoDB, namespaces). The PostSync Job pattern solves the Lambda Permission CRD gap and secret management.
See Lambda ACK GitOps Technical Plan for full implementation details.
Summary of approach:
| Phase | Goal | Duration |
|---|---|---|
| Phase 0 | PoC: GKE → AWS OIDC cross-cloud auth | 1-2 days |
| Phase 1 | Install ACK S3 + Lambda controllers | 1 day |
| Phase 2 | Helm chart with Bucket + Function CRDs + PostSync Job | 1-2 days |
| Phase 3 | Deploy new staging bucket + Lambda via ACK | 1-2 days |
| Phase 4 | Production cutover (adopt existing resources) | 1 day + soak |
| Phase 5 | Preview integration + Terraform cleanup | 2-3 days |
Key architecture decisions:
- Separate S3 buckets per environment — aligns with existing MongoDB isolation
- PostSync Job for credentials + permissions — solves ACK's lack of SecretKeyRef and Permission CRD
- ACK adoption for production — existing bucket + Lambda adopted without recreation
- GKE → AWS OIDC federation — validated as Phase 0 PoC (highest risk, done first)
Reference Documentation:
- Technical Plan (validated 2026-02-06)
- Migration Runbook (to be updated before Phase 4)
Alternatives Considered¶
Option 1: GitOps-Triggered Terraform (Analyzed, Not Chosen)¶
Approach: ArgoCD detects cluster-gitops change → triggers GitHub Actions → runs Terraform
Pros:
- Version in Git as source of truth
- Familiar Terraform workflow
Cons:
- Adds latency (ArgoCD → webhook → GHA → Terraform)
- Complex orchestration
- Terraform state management overhead
Decision: Not worth the complexity for a rarely-changing Lambda
Option 2: AWS Controllers for Kubernetes (Analyzed, Deferred)¶
Approach: Lambda and S3 as Kubernetes CRDs managed by ACK controllers
Pros:
- Unified K8s-native management
- ArgoCD visibility
- Continuous reconciliation
Cons:
- High setup cost (cross-cloud IAM)
- Missing CRDs (Lambda Permission)
- Overkill for stable component
Decision: Deferred - complexity outweighs benefits (see Tier 3 above)
Option 3: Lambda Versioning with Aliases (Analyzed, Not Chosen)¶
Approach: Use Lambda aliases ($LATEST, staging, production) with weighted routing
Pros:
- Native AWS pattern
- Easy rollback
Cons:
- Doesn't solve environment isolation
- Complex alias management
- Less GitOps alignment
Decision: Doesn't address the core problem (staging isolation)
Option 4: Separate S3 Buckets Per Environment (Analyzed, Deferred)¶
Approach: syrfappuploads-staging, syrfappuploads-production instead of prefix isolation
Pros:
- Complete isolation
- Simpler notification config (1:1 bucket:lambda)
Cons:
- Data migration needed
- Application code changes required
- Preview environments would need buckets too
Decision: Not worth the migration effort; prefix isolation is sufficient
Version Tracking Architecture¶
How It Works¶
┌─────────────────────────────────────────────────────────────────────────────┐
│ S3 NOTIFIER VERSION TRACKING FLOW │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ 1. CI/CD builds Lambda package │
│ └── GitVersion calculates: s3-notifier-v1.2.3 │
│ │
│ 2. CI/CD deploys Lambda to AWS (ONE function, shared by all envs) │
│ └── aws lambda update-function-code --function-name ... │
│ │
│ 3. CI/CD updates cluster-gitops SHARED SERVICE VALUES │
│ └── syrf/services/api/values.yaml (applies to staging + production) │
│ s3NotifierVersion: "1.2.3" │
│ │
│ 4. ArgoCD syncs API deployment (both environments) │
│ └── Pod gets env var: SYRF__S3NotifierVersion=1.2.3 │
│ │
│ 5. API exposes via /api/application endpoint │
│ └── { "s3NotifierVersion": "1.2.3", ... } │
│ │
│ 6. Web displays in Version Info dialog │
│ └── "S3 Notifier: v1.2.3" │
│ │
│ NOTE: Preview environments have per-PR Lambdas, so they use per-PR values │
│ files: syrf/environments/preview/pr-{N}/services/api.values.yaml │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Files Involved¶
| File | Purpose |
|---|---|
src/charts/syrf-common/env-mapping.yaml |
Defines s3NotifierVersion → SYRF__S3NotifierVersion |
src/services/api/.chart/values.yaml |
Default value ("CHANGEME" for local dev) |
cluster-gitops/syrf/services/api/values.yaml |
Shared version (staging + production) |
cluster-gitops/syrf/environments/preview/pr-{N}/services/api.values.yaml |
Per-PR preview version |
.github/workflows/ci-cd.yml |
Writes version to shared service values on Lambda deploy |
.github/workflows/pr-preview-lambda.yml |
Writes version for preview environments |
Key Design Decision: Shared vs Per-Environment Values¶
Current state (ONE Lambda):
- Version is written to
syrf/services/api/values.yaml(shared) - Both staging and production inherit this automatically
- No "promotion" step needed for s3NotifierVersion
Future state (Tier 2 - Staging Lambda):
- Version will be written to per-environment files
syrf/environments/staging/api/values.yamlfor staging Lambda versionsyrf/environments/production/api/values.yamlfor production Lambda version- Production promotion will copy version from staging to production
Success Metrics¶
Tier 1 (Completed)¶
- Version Info dialog shows S3 Notifier version in all environments
- Production promotion correctly propagates version
Tier 2 (Planned)¶
- Staging Lambda exists and processes files
- CI/CD automatically deploys to staging on merge
- End-to-end file upload works in staging environment
- No production deployments without staging validation
References¶
- env-mapping.yaml - Environment variable mapping
- ci-cd.yml - Main CI/CD workflow
- pr-preview-lambda.yml (archived) - Preview Lambda workflow (replaced by ACK in pr-preview.yml)
- ADR-003: Cluster Architecture - Overall GitOps architecture