Skip to content

Multi-Repository Documentation Publishing

Problem Statement

The SyRF documentation is aggregated from 3 repositories:

  • syrf/ - Application code and architecture
  • cluster-gitops/ - Deployment operations
  • camarades-infrastructure/ - Infrastructure operations

Challenge: Changes to cluster-gitops/docs/ or camarades-infrastructure/docs/ need to trigger a rebuild and redeployment of the unified documentation site at docs.syrf.org.uk.

Current State

What Works

  • ✅ Documentation structure established across all 3 repos
  • ✅ MkDocs monorepo plugin configured in syrf/mkdocs.yml
  • ✅ YAML front matter standardized across all repos
  • ✅ Local development works (if repos are cloned as subfolders)

What Doesn't Work

  • ❌ Changes to cluster-gitops/ docs don't trigger docs rebuild
  • ❌ Changes to camarades-infrastructure/ docs don't trigger docs rebuild
  • ❌ Dockerfile only includes files from syrf/ repo
  • ❌ CI/CD only monitors syrf/ repo for docs changes

Solution Options

How it works: The docs Docker image clones all 3 repos during build time and aggregates them.

Dockerfile Changes:

# Stage 1: Build the MkDocs site
FROM python:3.10-alpine AS builder

# Install git and build dependencies
RUN apk add --no-cache git build-base

WORKDIR /workspace

# Clone all repositories
ARG GITHUB_TOKEN
RUN git clone https://github.com/camaradesuk/syrf.git && \
    git clone https://github.com/camaradesuk/cluster-gitops.git && \
    git clone https://github.com/camaradesuk/camarades-infrastructure.git

# Install dependencies
WORKDIR /workspace/syrf
RUN pip install --no-cache-dir -r requirements.txt

# Build the unified site
RUN mkdocs build --strict --site-dir _docs_build

# Stage 2: Serve with NGINX
FROM nginx:alpine
COPY --from=builder /workspace/syrf/_docs_build /usr/share/nginx/html

CI/CD Changes:

  1. Add GitHub token as build arg for private repos (if needed)
  2. Trigger docs rebuild on:
  3. Push to syrf/master with docs changes
  4. Push to cluster-gitops/master with docs changes
  5. Push to camarades-infrastructure/master with docs changes

Pros:

  • ✅ Simple - standard Docker build
  • ✅ Always uses latest docs from all repos
  • ✅ No manual coordination needed
  • ✅ Works with public repos (no auth needed)

Cons:

  • ❌ Slower builds (clones repos every time)
  • ❌ Requires webhook/dispatch from other repos

Option 2: GitHub Actions Repository Dispatch

How it works: Each repo triggers a rebuild in the syrf repo when docs change.

cluster-gitops/.github/workflows/docs-trigger.yml:

name: Trigger Docs Rebuild

on:
  push:
    branches: [master]
    paths:
      - 'docs/**'

jobs:
  trigger:
    runs-on: ubuntu-latest
    steps:
      - name: Trigger syrf docs rebuild
        uses: peter-evans/repository-dispatch@v2
        with:
          token: ${{ secrets.DOCS_TRIGGER_TOKEN }}
          repository: camaradesuk/syrf
          event-type: docs-update
          client-payload: '{"source": "cluster-gitops"}'

syrf/.github/workflows/docs-rebuild.yml:

name: Rebuild Docs on External Change

on:
  repository_dispatch:
    types: [docs-update]

jobs:
  build-docs:
    # ... rebuild docs image

Pros:

  • ✅ Automatic triggers from any repo
  • ✅ Centralized build in syrf repo
  • ✅ Fine-grained control over when to rebuild

Cons:

  • ❌ Requires PAT token with repo scope
  • ❌ More complex CI/CD setup
  • ❌ Three separate workflows to maintain

Option 3: Scheduled Rebuild

How it works: Rebuild docs on a schedule (e.g., every hour or nightly).

syrf/.github/workflows/docs-scheduled.yml:

name: Scheduled Docs Rebuild

on:
  schedule:
    - cron: '0 * * * *'  # Every hour
  workflow_dispatch:      # Manual trigger

jobs:
  build-docs:
    # ... rebuild docs image with Option 1 Dockerfile

Pros:

  • ✅ Very simple - no cross-repo coordination
  • ✅ No secrets/tokens needed
  • ✅ Predictable rebuild times

Cons:

  • ❌ Docs can be stale for up to 1 hour
  • ❌ Wastes CI time if no changes
  • ❌ Not immediate feedback on doc changes

Option 4: Git Submodules

How it works: Include cluster-gitops and camarades-infrastructure as git submodules in syrf.

Setup:

cd /home/chris/workspace/syrf
git submodule add https://github.com/camaradesuk/cluster-gitops.git
git submodule add https://github.com/camaradesuk/camarades-infrastructure.git

Pros:

  • ✅ Standard git feature
  • ✅ Version pinning (specific commits)
  • ✅ Works with existing Dockerfile

Cons:

  • ❌ Manual submodule updates required
  • ❌ Developers must remember git submodule update
  • ❌ Not truly automatic
  • ❌ Submodule hell (notoriously painful)

Recommendation

Use Option 1 (Monorepo-Style Build) + Option 2 (Repository Dispatch)

Phase 1: Quick Solution (Option 1 Only)

Update the Dockerfile to clone all 3 repos during build. This works immediately for public repos.

Phase 2: Add Automation (Option 2)

Set up repository dispatch to trigger docs rebuilds when any repo's docs change.

Why This Combination?

  1. Option 1 ensures the build always has the latest docs
  2. Option 2 triggers rebuilds only when needed (efficient)
  3. Simple fallback: Manual workflow_dispatch if automation fails

Implementation Plan

Step 1: Update Dockerfile (5 minutes)

# Edit docs/Dockerfile to clone all repos
# Test locally: docker build -t syrf-docs:test -f docs/Dockerfile .

Step 2: Test Multi-Repo Build (10 minutes)

# Verify all 3 repos are included
docker run --rm syrf-docs:test ls /usr/share/nginx/html

Step 3: Add Repository Dispatch (30 minutes)

  • Create workflows in cluster-gitops and camarades-infrastructure
  • Create PAT token with repo scope
  • Add token as secret in both repos
  • Test dispatch triggers

Step 4: Update Documentation (10 minutes)

  • Document the build process
  • Add troubleshooting guide
  • Update CLAUDE.md

Security Considerations

Public Repos (Current State)

  • No authentication needed
  • Dockerfile can clone via HTTPS without tokens
  • Safe to use in GitHub Actions

Private Repos (Future)

  • Requires GitHub token as build arg
  • Token needs repo scope for cloning
  • Should use GitHub Actions secrets
  • Consider using GitHub App tokens (more secure)

Testing Strategy

  1. Local Test: Clone all 3 repos manually, run mkdocs build
  2. Docker Test: Build Docker image, verify all docs included
  3. CI/CD Test: Push doc change to cluster-gitops, verify rebuild
  4. End-to-End Test: Deploy to staging, verify docs.syrf.org.uk updates

Next Steps

  1. Decide on Option 1 vs Option 1+2
  2. Update Dockerfile
  3. Test multi-repo build
  4. Set up repository dispatch (if Option 2)
  5. Update documentation
  6. Deploy and verify