Technical Plan: MongoDB Testing Infrastructure¶

Overview¶

This document provides detailed implementation guidance for the MongoDB testing and isolation strategy outlined in the Feature Brief.

Related Documents:

Atlas Manual Setup - Production/staging user creation

Atlas Operator Setup - PR preview automation

Security Model - Defense-in-depth architecture

Future DB Rename Runbook - Deferred migration plan

Part 1: TestContainers Integration¶

1.1 NuGet Package Additions¶

Add to each test project that needs MongoDB integration tests:

<!-- Example: SyRF.ProjectManagement.Endpoint.Tests.csproj -->
<ItemGroup>
  <PackageReference Include="Testcontainers.MongoDb" Version="3.10.0" />
  <PackageReference Include="FluentAssertions" Version="6.12.0" />
</ItemGroup>

1.2 Base Test Fixture¶

Create a shared test fixture in a new test utilities library:

File: src/libs/testing/SyRF.Testing.Common/Fixtures/MongoDbTestFixture.cs

using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using MongoDB.Driver;
using SyRF.Mongo.Common;
using SyRF.SharedKernel.Infrastructure.ConfigModel;
using Testcontainers.MongoDb;
using Xunit;

namespace SyRF.Testing.Common.Fixtures;

/// <summary>
/// Shared MongoDB test fixture using TestContainers.
/// Provides an isolated MongoDB instance for integration tests.
/// </summary>
/// <remarks>
/// CRITICAL: This fixture ensures CSUUID (C# Legacy GUID) serialization is configured,
/// matching the production database format where all IDs use BinData subtype 3.
/// </remarks>
public class MongoDbTestFixture : IAsyncLifetime
{
    private readonly MongoDbContainer _container;

    public MongoClient Client { get; private set; } = null!;
    public IMongoDatabase Database { get; private set; } = null!;
    public string ConnectionString => _container.GetConnectionString();
    public string DatabaseName { get; }

    /// <summary>
    /// Creates a new MongoDB test fixture with an auto-generated database name.
    /// xUnit collection fixtures require a single public parameterless constructor.
    /// </summary>
    public MongoDbTestFixture()
    {
        DatabaseName = $"integration_test_{Guid.NewGuid():N}";
        _container = new MongoDbBuilder()
            .WithImage("mongo:7.0")
            .WithName($"mongo-test-{Guid.NewGuid():N}")
            .Build();

        // CRITICAL: Configure CSUUID (C# Legacy GUID) serialization
        // SyRF stores all GUIDs as BinData subtype 3 (CSUUID), not subtype 4 (UUID)
        // This must be done before any MongoDB operations
        SyRF.Mongo.Common.MongoUtils.EnsureLegacyGuidSerializer();
    }

    public async Task InitializeAsync()
    {
        await _container.StartAsync();
        Client = new MongoClient(_container.GetConnectionString());
        Database = Client.GetDatabase(DatabaseName);
    }

    public async Task DisposeAsync()
    {
        await _container.DisposeAsync();
    }

    /// <summary>
    /// Get a clean collection, dropping any existing data.
    /// </summary>
    public IMongoCollection<T> GetCleanCollection<T>(string name)
    {
        Database.DropCollection(name);
        return Database.GetCollection<T>(name);
    }

    /// <summary>
    /// Create a MongoContext configured for this test database.
    /// </summary>
    /// <param name="resumeRepo">Optional resume point repository (defaults to InMemoryResumePointRepository)</param>
    public MongoContext CreateContext(IResumePointRepository? resumeRepo = null)
    {
        // TestContainers connection string format: mongodb://mongo:mongo@127.0.0.1:49152
        // Parse host:port from after the @ symbol (if credentials present)
        var connectionString = _container.GetConnectionString();
        var afterProtocol = connectionString
            .Replace("mongodb://", "")
            .Split('/')[0];

        // Handle credentials format: user:password@host:port
        var hostPort = afterProtocol.Contains('@')
            ? afterProtocol.Split('@')[1]
            : afterProtocol;

        // Extract credentials if present (format: user:password@host:port)
        // TestContainers may provide unauthenticated MongoDB instances
        var hasCredentials = afterProtocol.Contains('@');
        string? username = null;
        string? password = null;

        if (hasCredentials)
        {
            var credentialsPart = afterProtocol.Split('@')[0];
            var colonIndex = credentialsPart.IndexOf(':');
            if (colonIndex > 0)
            {
                username = credentialsPart.Substring(0, colonIndex);
                password = credentialsPart.Substring(colonIndex + 1);
            }
        }

        var settings = new MongoConnectionSettings
        {
            ClusterAddress = "", // Empty to use ClusterAddresses instead
            ClusterAddresses = new List<ServerAddress>
            {
                new()
                {
                    Host = hostPort.Split(':')[0],
                    Port = hostPort.Split(':')[1]
                }
            },
            DatabaseName = DatabaseName,
            UseSSL = false,
            // Only set credentials if present in connection string
            // TestContainers may run with or without authentication
            Username = username,
            Password = password,
            AuthDb = hasCredentials ? "admin" : null
        };

        // Use existing InMemoryResumePointRepository for tests
        // (tokens don't need to persist between test runs)
        var logger = NullLoggerFactory.Instance.CreateLogger<MongoContext>();
        return new MongoContext(
            settings,
            logger,
            resumeRepo ?? new InMemoryResumePointRepository());
    }

    /// <summary>
    /// Create a fresh database with a unique name for isolated test scenarios.
    /// </summary>
    public IMongoDatabase CreateIsolatedDatabase(string? prefix = null)
    {
        var dbName = $"{prefix ?? "isolated"}_{Guid.NewGuid():N}";
        return Client.GetDatabase(dbName);
    }
}

1.3 Collection Fixture for xUnit¶

File: src/libs/testing/SyRF.Testing.Common/Fixtures/MongoDbCollection.cs

using Xunit;

namespace SyRF.Testing.Common.Fixtures;

/// <summary>
/// xUnit collection definition for MongoDB integration tests.
/// Tests in the same collection share the same MongoDbTestFixture instance,
/// avoiding container startup overhead while maintaining test isolation through
/// unique database names or collection cleanup.
/// </summary>
/// <remarks>
/// Usage:
/// <code>
/// [Collection(MongoDbCollection.Name)]
/// public class MyRepositoryTests(MongoDbTestFixture fixture)
/// {
///     [Fact]
///     public async Task MyTest()
///     {
///         var collection = fixture.GetCleanCollection&lt;MyEntity&gt;("myCollection");
///         // ... test code
///     }
/// }
/// </code>
/// </remarks>
[CollectionDefinition(Name)]
public class MongoDbCollection : ICollectionFixture<MongoDbTestFixture>
{
    /// <summary>
    /// The collection name used in [Collection] attributes.
    /// </summary>
    public const string Name = "MongoDB";
}

1.4 Test Data Builders¶

File: src/libs/testing/SyRF.Testing.Common/Builders/ProjectBuilder.cs

using SyRF.ProjectManagement.Core.Aggregates;

namespace SyRF.Testing.Common.Builders;

public class ProjectBuilder
{
    private readonly Project _project;

    public ProjectBuilder()
    {
        _project = new Project
        {
            Id = Guid.NewGuid(),
            Name = $"Test Project {Guid.NewGuid():N}",
            Keywords = new List<string> { "test" },
            IsPublic = false,
            Audit = new Audit
            {
                Version = 1,
                SchemaVersion = 1,
                CreatedDate = DateTime.UtcNow,
                CreatedBy = Guid.NewGuid()
            }
        };
    }

    public ProjectBuilder WithId(Guid id) { _project.Id = id; return this; }
    public ProjectBuilder WithName(string name) { _project.Name = name; return this; }
    public ProjectBuilder WithKeywords(params string[] keywords) { _project.Keywords = keywords.ToList(); return this; }
    public ProjectBuilder AsPublic() { _project.IsPublic = true; return this; }

    public ProjectBuilder WithOwner(Guid investigatorId, string email = "owner@test.com")
    {
        _project.Memberships.Add(new ProjectMembership
        {
            InvestigatorId = investigatorId,
            Email = email,
            Role = ProjectRole.Owner,
            JoinedDate = DateTime.UtcNow
        });
        return this;
    }

    public ProjectBuilder WithStage(string name, StageType type = StageType.Screening)
    {
        _project.Stages.Add(new Stage
        {
            Id = Guid.NewGuid(),
            Name = name,
            Type = type,
            Order = _project.Stages.Count
        });
        return this;
    }

    public Project Build() => _project;

    public async Task<Project> BuildAndSaveAsync(IProjectRepository repository)
    {
        await repository.SaveAsync(_project);
        return _project;
    }
}

1.5 Example Integration Test¶

File: src/services/project-management/SyRF.ProjectManagement.Endpoint.Tests/Integration/ProjectRepositoryTests.cs

using FluentAssertions;
using SyRF.Testing.Common;
using SyRF.Testing.Common.Builders;

namespace SyRF.ProjectManagement.Endpoint.Tests.Integration;

[Collection("MongoDB")]
[Trait("Category", "Integration")]
public class ProjectRepositoryTests : IAsyncLifetime
{
    private readonly MongoDbTestFixture _fixture;
    private readonly MongoContext _context;
    private readonly ProjectRepository _repository;

    public ProjectRepositoryTests(MongoDbTestFixture fixture)
    {
        _fixture = fixture;
        _context = fixture.CreateContext();
        _repository = new ProjectRepository(_context);
    }

    public async Task InitializeAsync()
    {
        _repository.CreateMappings();
        await _repository.InitialiseIndexes();
    }

    public Task DisposeAsync() => Task.CompletedTask;

    [Fact]
    public async Task SaveAsync_NewProject_PersistsCorrectly()
    {
        var project = new ProjectBuilder()
            .WithName("Integration Test Project")
            .WithOwner(Guid.NewGuid())
            .Build();

        await _repository.SaveAsync(project);

        var retrieved = await _repository.GetAsync(project.Id);
        retrieved.Should().NotBeNull();
        retrieved!.Name.Should().Be("Integration Test Project");
        retrieved.Memberships.Should().HaveCount(1);
    }
}

Part 2: Configuration for Database Isolation¶

2.1 Helm Values Structure¶

File: src/services/api/.chart/values.yaml

mongoDb:
  authSecretName: mongo-db
  clusterAddress: cluster0.siwfo.mongodb.net/admin?retryWrites=true&w=majority
  databaseName: ""  # MUST be set per environment
  authDb: admin
  ssl: true

2.2 Environment-Specific Values¶

See cluster-gitops environment values for current configuration:

Environment	Database	File
Production	`syrftest`	`production/production.values.yaml`
Staging	`syrf_staging`	`staging/staging.values.yaml`
PR Preview	`syrf_pr_{number}`	Set via ApplicationSet

Part 3: CI Pipeline Integration Tests¶

3.1 Integration Test Job¶

Add to .github/workflows/ci-cd.yml:

test-integration:
  name: Integration Tests
  runs-on: ubuntu-latest
  needs: [detect-changes]
  if: |
    needs.detect-changes.outputs.api_changed == 'true' ||
    needs.detect-changes.outputs.project_management_changed == 'true'

  services:
    mongodb:
      image: mongo:7.0
      ports:
        - 27017:27017

  steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-dotnet@v4
      with:
        dotnet-version: '8.0.x'
    - run: dotnet restore syrf.sln
    - run: dotnet build syrf.sln --no-restore
    - name: Run Integration Tests
      run: |
        dotnet test syrf.sln \
          --no-build \
          --filter "Category=Integration" \
          --logger "trx;LogFileName=integration-results.trx"
      env:
        MONGO_CONNECTION_STRING: mongodb://localhost:27017

3.2 Test Categories¶

[Trait("Category", "Unit")]       // No external dependencies
[Trait("Category", "Integration")] // Require MongoDB
[Trait("Category", "E2E")]        // Full system tests

Part 4: Migration Framework¶

DEFERRED: This phase is deferred to a follow-up PR. Implement Phases 1-3 and 5 first, then evaluate whether the migration framework is needed based on actual schema change requirements.

Part 5: Seed Data for PR Previews¶

5.1 Seed Data Definition¶

public static class PreviewSeedData
{
    public static readonly Guid TestUserId = Guid.Parse("00000000-0000-0000-0000-000000000001");
    public static readonly Guid SampleProjectId = Guid.Parse("00000000-0000-0000-0000-000000000002");

    public static async Task SeedAsync(IPmUnitOfWork uow)
    {
        var testUser = new Investigator
        {
            Id = TestUserId,
            Email = "preview-test@syrf.org.uk",
            FirstName = "Preview",
            LastName = "Tester"
        };
        await uow.Investigators.SaveAsync(testUser);

        var project = new Project
        {
            Id = SampleProjectId,
            Name = "Sample Systematic Review",
            Keywords = new[] { "test", "preview", "sample" }
        };
        project.AddMember(testUser.Id, ProjectRole.Owner);
        project.AddStage(new Stage { Name = "Title/Abstract Screening", Type = StageType.Screening });
        project.AddStage(new Stage { Name = "Data Extraction", Type = StageType.Annotation });
        await uow.Projects.SaveAsync(project);

        // 10 sample studies
        for (int i = 1; i <= 10; i++)
        {
            await uow.Studies.SaveAsync(new Study
            {
                ProjectId = project.Id,
                Title = $"Sample Study {i}: Effects of Treatment on Outcome",
                Authors = new[] { "Smith, J.", "Jones, A." },
                Year = 2020 + (i % 5)
            });
        }
    }
}

Implementation Checklist¶

Phase 1: TestContainers Setup ✅¶

Create SyRF.Testing.Common library project
Add TestContainers.MongoDB package
Implement MongoDbTestFixture
Implement MongoDbCollection for xUnit
Create test data builders
Write example integration tests
Document local development setup

Phase 2: Environment Isolation¶

Update Helm values structure
Update cluster-gitops with environment-specific database names
Create MongoDB users - See Atlas Manual Setup
Create ExternalSecrets for credentials
Staging: extra-secrets-staging/values.yaml syncs syrf-staging-mongodb → mongo-db secret
Production: extra-secrets-production/values.yaml syncs syrf-prod-mongodb → mongo-db secret
Test staging deployment with new database
Verified 2025-12-18: ExternalSecret SecretSynced, username syrf_staging_app
API deployment env var confirms DatabaseName: syrf_staging

Phase 3: PR Preview Isolation¶

Install Atlas Operator - See Atlas Operator Setup
Installed via GitOps in mongodb-atlas-system namespace
ExternalSecret configured for API credentials
ConfigMap with Atlas Project ID
Deploy policy enforcement - See Security Model
Kyverno ClusterPolicy blocks production/staging access
All 5 security tests passed (production, staging, admin, broad roles, valid PR)
Update PR preview workflow for per-PR databases
Workflow creates mongodb-user.yaml with AtlasDatabaseUser CR
ApplicationSet includes mongodb-user.yaml in namespace app
Services read from syrfdb-cluster0-syrf-pr-{n}-app secret (Atlas Operator naming convention)
Add database cleanup on PR close
User cleanup: Automatic via operator when AtlasDatabaseUser CR deleted
Database: Empty databases remain (no cost) - optional periodic cleanup via CronJob
Test with actual PR
PR-2234 verified working (2025-12-18)
API, PM, Quartz services all using PR-specific credentials
Change streams confirmed working with per-PR MongoDB user
Create seed data mechanism
Implemented DatabaseSeeder (IRunAtInit) for staging/preview environments
Creates Investigator, Project (with stages), SystematicSearch, and 8 sample Studies
Controlled via SYRF_SEED_DATA_ENABLED environment variable
Unit tests in DatabaseSeederTests.cs (12 passing tests)

Phase 4: Migration Framework (DEFERRED)¶

Deferred to follow-up PR.

Phase 5: CI Integration¶

Add test-integration job to ci-cd.yml
New test-dotnet-integration job with --filter "Category=Integration"
Runs when any .NET service changes
15-minute timeout for container startup overhead
Configure MongoDB service container
Uses TestContainers (Docker available on ubuntu-latest)
No separate Service Container needed - TestContainers works in GitHub Actions
Add test result upload as artifacts
Test results: dotnet-integration-test-results-{sha}
Coverage: dotnet-integration-coverage-{sha}
Create E2E test suite for staging

Data Strategy¶

Overview¶

Environment	Database	Auth0	Data Strategy
Production	`syrftest`	`syrf` tenant	Real user data
Staging	`syrf_staging`	`syrf` tenant (shared)	Minimal seed
PR Preview	`syrf_pr_{n}`	`syrf` tenant (shared)	Shared minimal seed
CI Tests	TestContainers	N/A	In-memory builders

Auth0 Configuration¶

All non-production environments share the production Auth0 tenant:

Real user accounts work in staging/preview
Database isolation (not auth) is the critical concern
Users see their account but NOT their production data

Add visual indicator to prevent confusion:

@Component({...})
export class AppComponent {
  environment = inject(ConfigService).runtimeEnvironment;

  get showBanner(): boolean {
    return this.environment !== 'production';
  }

  get bannerText(): string {
    if (this.environment?.startsWith('pr-')) {
      return `⚠️ PR PREVIEW (${this.environment}) - Test data only`;
    }
    return `⚠️ ${this.environment?.toUpperCase()} - Not production`;
  }
}

Local Development Setup¶

Prerequisites¶

Docker Desktop (or Docker Engine on Linux)
.NET 8 SDK

Running Integration Tests¶

# Run all tests
dotnet test src/libs/testing/SyRF.Testing.Common.Tests/

# Run with verbose output
dotnet test src/libs/testing/SyRF.Testing.Common.Tests/ --logger "console;verbosity=detailed"

# Run specific test
dotnet test --filter "FullyQualifiedName~Fixture_ShouldProvideWorkingMongoDbConnection"

Using Test Data Builders¶

var investigator = InvestigatorBuilder.Default()
    .WithName("John", "Doe")
    .WithEmail("john.doe@example.com")
    .Build();

var project = ProjectBuilder.Default()
    .WithName("My Test Project")
    .WithCreatorId(investigatorId)
    .Build();

Notes¶

TestContainers vs Service Containers¶

Approach	Description	Best For
TestContainers	Library that starts Docker containers programmatically from test code. Container lifecycle controlled by test fixtures.	Local development, consistent behavior across environments
Service Containers	GitHub Actions spins up containers before job starts (YAML config). Accessible at `localhost:port`.	CI-only optimization when local parity isn't needed

SyRF uses TestContainers because:

Works identically on developer machines and in CI
No CI-specific configuration needed
Each test collection gets its own isolated container
Developers can run the same tests locally that CI runs

The optional Service Container config in section 3.1 is for potential CI optimization, but TestContainers works well in GitHub Actions too

CSUUID (C# Legacy GUID) Format¶

SyRF stores all document IDs as CSUUID (BinData subtype 3). See MongoDB Reference.

// ✅ CORRECT - CSUUID format
db.pmStudy.find({ _id: CSUUID("550e8400-e29b-41d4-a716-446655440000") })

Production Database Naming¶

Environment	Database
Production	`syrftest` (existing - see future runbook)
Staging	`syrf_staging` (new)
PR Preview	`syrf_pr_{number}` (ephemeral)

Technical Plan: MongoDB Testing Infrastructure¶

Overview¶

Part 1: TestContainers Integration¶

1.1 NuGet Package Additions¶

1.2 Base Test Fixture¶

1.3 Collection Fixture for xUnit¶

1.4 Test Data Builders¶

1.5 Example Integration Test¶

Part 2: Configuration for Database Isolation¶

2.1 Helm Values Structure¶

2.2 Environment-Specific Values¶

Part 3: CI Pipeline Integration Tests¶

3.1 Integration Test Job¶

3.2 Test Categories¶

Part 4: Migration Framework¶

Part 5: Seed Data for PR Previews¶

5.1 Seed Data Definition¶

Implementation Checklist¶

Phase 1: TestContainers Setup ✅¶

Phase 2: Environment Isolation¶

Phase 3: PR Preview Isolation¶

Phase 4: Migration Framework (DEFERRED)¶

Phase 5: CI Integration¶

Data Strategy¶

Overview¶

Auth0 Configuration¶

Environment Banner¶

Local Development Setup¶

Prerequisites¶

Running Integration Tests¶

Using Test Data Builders¶

Notes¶

TestContainers vs Service Containers¶

CSUUID (C# Legacy GUID) Format¶

Production Database Naming¶

References¶