Skip to content

Lambda Environment Isolation & Version Tracking

SUPERSEDED: This document has been superseded by Lambda GitOps Integration. The staging Lambda isolation work is planned as Tier 2 in the consolidated document. This document is preserved for historical reference only.

Summary

Add dedicated staging Lambda function with isolated S3 prefix, and enable querying deployed Lambda version without function invocation via AWS tags and GitOps values files.

Problem Statement

Currently, staging and production share the same Lambda function (syrfAppUploadS3Notifier), meaning:

  1. No staging validation - Lambda changes go directly to production
  2. No version tracking - Cannot query which Lambda version is deployed to an environment
  3. Shared S3 prefix - Both staging and production use Projects/ prefix

Preview environments are properly isolated with per-PR Lambda functions and preview/pr-{n}/ S3 prefixes.

Current Architecture

Environment S3 Prefix Lambda Function Issue
Production Projects/ syrfAppUploadS3Notifier OK
Staging Projects/ syrfAppUploadS3Notifier Shares production Lambda
Preview preview/pr-{n}/ syrfAppUploadS3Notifier-pr-{n} OK

Proposed Architecture

Environment S3 Prefix Lambda Function Trigger
Production Projects/ syrfAppUploadS3Notifier Projects/
Staging staging/ syrfAppUploadS3Notifier-staging staging/
Preview preview/pr-{n}/ syrfAppUploadS3Notifier-pr-{n} preview/pr-{n}/

Promotion strategy: Lambda follows same pattern as K8s services:

  • Staging: Auto-deploy on merge to main
  • Production: Manual PR approval (consistent with K8s workflow)

Implementation Plan

Phase 1: S3 Path Prefix Logic (SyRF Monorepo)

File: src/libs/kernel/SyRF.SharedKernel/Settings/SyrfSettings.cs

public class SyrfSettings
{
    public string? PrNumber { get; set; }
    public string? RuntimeEnvironment { get; set; }  // NEW: "staging", "production", etc.

    public bool IsPreviewEnvironment => !string.IsNullOrEmpty(PrNumber);
    public bool IsStagingEnvironment =>
        RuntimeEnvironment?.Equals("staging", StringComparison.OrdinalIgnoreCase) == true;

    public string GetS3PathPrefix()
    {
        // Preview takes precedence
        if (!string.IsNullOrEmpty(PrNumber))
            return $"preview/pr-{PrNumber}/";

        // Staging gets its own prefix
        if (IsStagingEnvironment)
            return "staging/";

        // Production uses no prefix (backward compatible with existing Projects/ paths)
        return string.Empty;
    }
}

File: src/charts/syrf-common/env-mapping.yaml

Add mapping for RuntimeEnvironment to SyrfSettings:

syrfSettings:
  services: [api, project-management, quartz]
  displayName: "SyRF Settings"
  envVars:
    - name: SYRF__RuntimeEnvironment
      valuePath: environment.name
      default: "local"

Phase 2: Staging Lambda Infrastructure (Terraform)

File: camarades-infrastructure/terraform/lambda/variables.tf

Add staging variables:

variable "staging_version" {
  description = "Semantic version for staging Lambda deployment"
  type        = string
  default     = ""
}

variable "staging_commit_sha" {
  description = "Commit SHA for staging Lambda deployment"
  type        = string
  default     = ""
}

variable "staging_source_code_hash" {
  description = "Base64-encoded SHA256 hash of staging Lambda zip"
  type        = string
  default     = ""
}

File: camarades-infrastructure/terraform/lambda/main.tf

Add staging Lambda resource:

resource "aws_lambda_function" "s3_notifier_staging" {
  function_name = "syrfAppUploadS3Notifier-staging"
  runtime       = "dotnet8"  # TODO: Update to dotnet10 when available
  handler       = "SyRF.S3FileSavedNotifier.Endpoint::SyRF.S3FileSavedNotifier.Endpoint.S3FileReceivedHandler::HandleEvent"
  role          = aws_iam_role.staging_lambda_role.arn
  timeout       = var.lambda_timeout
  memory_size   = var.lambda_memory_size

  s3_bucket         = var.lambda_packages_bucket
  s3_key            = "lambda-packages/staging.zip"
  source_code_hash  = var.staging_source_code_hash

  environment {
    variables = {
      RabbitMqHost     = var.rabbitmq_host
      RabbitMqUsername = var.rabbitmq_username
      RabbitMqPassword = var.rabbitmq_password
      S3Region         = "eu-west-1"
    }
  }

  tags = {
    Name        = "syrfAppUploadS3Notifier-staging"
    Environment = "staging"
    Version     = var.staging_version
    CommitSha   = var.staging_commit_sha
    Purpose     = "S3 upload notification for staging"
  }
}

Update S3 bucket notification to include staging:

resource "aws_s3_bucket_notification" "uploads" {
  bucket = var.s3_bucket_name

  # Production notification - triggers on Projects/ prefix
  lambda_function {
    lambda_function_arn = aws_lambda_function.s3_notifier_production.arn
    events              = ["s3:ObjectCreated:*"]
    filter_prefix       = "Projects/"
  }

  # Staging notification - triggers on staging/ prefix
  lambda_function {
    lambda_function_arn = aws_lambda_function.s3_notifier_staging.arn
    events              = ["s3:ObjectCreated:*"]
    filter_prefix       = "staging/"
  }

  # Preview notifications - dynamic per PR (existing)
  dynamic "lambda_function" {
    for_each = var.preview_prs
    content {
      lambda_function_arn = aws_lambda_function.s3_notifier_preview[lambda_function.key].arn
      events              = ["s3:ObjectCreated:*"]
      filter_prefix       = "preview/pr-${lambda_function.key}/"
    }
  }
}

Phase 3: CI/CD Workflow Updates

File: .github/workflows/ci-cd.yml

3a. Rename and split Lambda deployment

Rename deploy-lambda → deploy-lambda-staging (deploys only to staging):

deploy-lambda-staging:
  name: Deploy Lambda to Staging
  needs: [detect-changes, version]
  if: needs.detect-changes.outputs.s3_notifier_changed == 'true'
  runs-on: ubuntu-latest
  concurrency:
    group: staging-lambda-terraform
    cancel-in-progress: false

  steps:
    - name: Build Lambda package
      id: build-lambda
      run: |
        cd src/services/s3-notifier/SyRF.S3FileSavedNotifier.Endpoint
        dotnet publish -c Release -r linux-x64 --self-contained true -o publish
        cd publish
        zip -r /tmp/lambda.zip .
        HASH=$(openssl dgst -sha256 -binary /tmp/lambda.zip | openssl base64)
        echo "source_code_hash=$HASH" >> "$GITHUB_OUTPUT"

    - name: Upload Lambda package to S3 (staging)
      run: |
        aws s3 cp /tmp/lambda.zip s3://camarades-terraform-state-aws/lambda-packages/staging.zip

    - name: Terraform Apply (staging only)
      env:
        TF_VAR_staging_version: ${{ needs.version.outputs.s3_notifier_version }}
        TF_VAR_staging_commit_sha: ${{ github.sha }}
        TF_VAR_staging_source_code_hash: ${{ steps.build-lambda.outputs.source_code_hash }}
      run: |
        terraform apply -auto-approve -target=aws_lambda_function.s3_notifier_staging tfplan

3b. Update staging GitOps values

In promote-to-staging job, include s3NotifierVersion:

- name: Update staging s3NotifierVersion
  if: needs.deploy-lambda-staging.result == 'success'
  run: |
    VERSION="${{ needs.version.outputs.s3_notifier_version }}"
    SHA="${{ github.sha }}"

    yq -i '.s3Notifier.version = "'$VERSION'"' syrf/environments/staging/staging.values.yaml
    yq -i '.s3Notifier.commitSha = "'$SHA'"' syrf/environments/staging/staging.values.yaml

3c. Production Lambda deployment (manual promotion)

Add new job deploy-lambda-production that runs after promote-to-production PR is merged:

deploy-lambda-production:
  name: Deploy Lambda to Production
  needs: [version, promote-to-production]
  if: needs.promote-to-production.result == 'success'
  runs-on: ubuntu-latest
  concurrency:
    group: production-lambda-terraform
    cancel-in-progress: false

  steps:
    - name: Download staging Lambda package
      run: |
        # Reuse the same package that was deployed to staging
        aws s3 cp s3://camarades-terraform-state-aws/lambda-packages/staging.zip /tmp/lambda.zip
        aws s3 cp /tmp/lambda.zip s3://camarades-terraform-state-aws/lambda-packages/production.zip
        HASH=$(openssl dgst -sha256 -binary /tmp/lambda.zip | openssl base64)
        echo "source_code_hash=$HASH" >> "$GITHUB_OUTPUT"

    - name: Terraform Apply (production)
      env:
        TF_VAR_production_version: ${{ needs.version.outputs.s3_notifier_version }}
        TF_VAR_production_commit_sha: ${{ github.sha }}
        TF_VAR_production_source_code_hash: ${{ steps.download.outputs.source_code_hash }}
      run: |
        terraform apply -auto-approve -target=aws_lambda_function.s3_notifier_production tfplan

3d. Update production GitOps values

In promote-to-production job, include s3NotifierVersion in the PR:

- name: Update production s3NotifierVersion
  if: needs.deploy-lambda-staging.result == 'success'
  run: |
    VERSION="${{ needs.version.outputs.s3_notifier_version }}"
    SHA="${{ github.sha }}"

    yq -i '.s3Notifier.version = "'$VERSION'"' syrf/environments/production/production.values.yaml
    yq -i '.s3Notifier.commitSha = "'$SHA'"' syrf/environments/production/production.values.yaml

Phase 4: GitOps Values Schema

File: cluster-gitops/syrf/environments/staging/staging.values.yaml

Add s3Notifier section:

# S3 Notifier Lambda version tracking
# Updated by CI/CD when Lambda is deployed
s3Notifier:
  version: ""        # Semantic version (e.g., "0.1.5")
  commitSha: ""      # Full commit SHA for traceability

File: cluster-gitops/syrf/environments/production/production.values.yaml

Same structure:

s3Notifier:
  version: ""
  commitSha: ""

File: cluster-gitops/syrf/environments/preview/preview.values.yaml

Default for all previews (overridden per-PR):

s3Notifier:
  version: ""
  commitSha: ""

Phase 5: Application Configuration

File: src/charts/syrf-common/env-mapping.yaml

Add s3NotifierVersion to env mapping:

s3NotifierVersion:
  services: [api]
  displayName: "S3 Notifier Version"
  envVars:
    - name: SYRF__S3NotifierVersion
      valuePath: s3Notifier.version
      default: "Unknown"

This allows the API's /api/application/info endpoint to report the Lambda version.

Files to Modify

Repository File Change
syrf src/libs/kernel/SyRF.SharedKernel/Settings/SyrfSettings.cs Add RuntimeEnvironment, update GetS3PathPrefix()
syrf src/charts/syrf-common/env-mapping.yaml Add RuntimeEnvironment and s3NotifierVersion mappings
syrf .github/workflows/ci-cd.yml Deploy staging Lambda, update GitOps values
camarades-infrastructure terraform/lambda/variables.tf Add staging variables
camarades-infrastructure terraform/lambda/main.tf Add staging Lambda resource and S3 notification
cluster-gitops syrf/environments/staging/staging.values.yaml Add s3Notifier section
cluster-gitops syrf/environments/production/production.values.yaml Add s3Notifier section
cluster-gitops syrf/environments/preview/preview.values.yaml Add s3Notifier section

Querying Lambda Version

After implementation, version can be queried via:

1. AWS Lambda Tags (Direct)

aws lambda list-tags \
  --resource arn:aws:lambda:eu-west-1:318789018510:function:syrfAppUploadS3Notifier-staging \
  --query 'Tags.Version' --output text

2. GitOps Repository (Source of Truth)

yq '.s3Notifier.version' cluster-gitops/syrf/environments/staging/staging.values.yaml

3. API Endpoint (Runtime)

curl https://api.staging.syrf.org.uk/api/application/info | jq '.s3NotifierVersion'

Verification

  1. S3 Prefix Isolation:
  2. Upload file in staging → verify it lands at staging/Projects/...
  3. Upload file in production → verify it lands at Projects/...

  4. Upload file in preview → verify it lands at preview/pr-{n}/Projects/...

  5. Lambda Invocation:

  6. Staging upload triggers only syrfAppUploadS3Notifier-staging

  7. Production upload triggers only syrfAppUploadS3Notifier

  8. Version Tracking:

  9. Check Lambda tags show correct version
  10. Check GitOps values updated after deployment

  11. Check API endpoint returns correct s3NotifierVersion

  12. RabbitMQ Routing:

  13. Staging Lambda publishes to syrf-staging vhost
  14. Production Lambda publishes to syrf-production vhost

Future Consideration: Production Prefix

To add production/ prefix for new production uploads:

  1. Update GetS3PathPrefix() to return "production/" for production
  2. Update production Lambda to trigger on BOTH Projects/ (legacy) AND production/ (new)
  3. Gradually migrate - new uploads go to production/, existing data stays in Projects/

This is optional and can be done later without breaking changes.

Scope Summary

This feature spans 3 repositories:

  • syrf (monorepo) - App code + CI/CD workflow
  • camarades-infrastructure - Terraform Lambda config
  • cluster-gitops - Environment values

Estimated changes: ~15 files across 3 repos