Skip to content

Terminology: Candidate vs Reconciliation Sessions

Purpose: Correct the terminology and behaviour around annotation reconciliation vs screening tie-breaker and ensure our advanced screening design aligns with current SyRF behaviour.

Correction Summary

  • Reconciliation queue ≠ screening tie-breaker. It refers to annotation reconciliation only and is a manual process performed by a designated reconciler via special reconciliation annotation sessions.
  • Reconciled annotations (from reconciliation sessions) are the settled truth used later for annotation-based filtering (post-MVP). Candidate session responses are not authoritative for filtering.
  • Stage Study Pool (defined by FilterSet) is reviewer-agnostic. Selection Subsets are reviewer-aware, derived at selection time; reconciliation sessions are explicitly assigned/entered, not randomly sampled from the pool.

Canonical Terms

Stage Study Pool

The reviewer-agnostic set defined by the stage's FilterSet.

Selection Subset

Reviewer-aware subset used by the next-item selector. Applies:

  • HideExcludedStudiesFromReviewers
  • Per-reviewer suppression
  • MaxInProgress limit
  • Mode-specific eligibility rules

Candidate Annotation Session

An annotation session that counts toward the stage's SessionCountTarget. These are the ordinary annotation sessions (Reconciliation = false).

Reconciliation Annotation Session

A special session created for designated reconcilers to resolve conflicts across candidate responses for a study. Does not count toward SessionCountTarget (Reconciliation = true).

Reconciled Annotations

The final, authoritative annotation values produced only by reconciliation sessions. These become the source of truth for annotation-based filtering (Phase-2).

Candidate Sessions — Current Behaviour

Definition

Candidate sessions are annotation sessions that count toward a stage's SessionCountTarget. They are non-reconciliation sessions for a study in a specific stage.

Storage

  • Stored in ExtractionInfo.Sessions with Reconciliation = false
  • Aggregated per stage in ExtractionInfo.SessionTallies as:
  • NumberOfCandidateSessions
  • NumberOfCompletedCandidateSessions

Creation Rules

Created when a reviewer starts annotating a study for a stage if:

  • The study is not excluded by the stage's active settings; and
  • That reviewer has not already started a candidate session for that study in that stage

Status Values

  • Incomplete (open) or Completed (finalized)
  • For a given reviewer and study, maps to AnnotationStatus:
  • InProgress — an incomplete session exists
  • Completed — completed session exists and no incomplete

Query Semantics

Query Description
InsufficientlyAllocatedAndNotStartedForAnnotationInStage(stage, reviewer) Studies where NumberOfCandidateSessions < SessionCountTarget and the reviewer has not started a candidate session
SufficientlyAllocated(stage) Studies where NumberOfCandidateSessions ≥ SessionCountTarget
ReviewerStageSessionInProgress / ReviewerStageSessionCompleted Check the reviewer's candidate sessions in that stage

Note: Not to be confused with Reconciliation sessions (Reconciliation = true), which are separate and not counted toward SessionCountTarget.

Reconciliation Sessions — Annotation

Purpose

Consolidate multiple candidate responses into a single, authoritative set of annotation values.

Who Performs

Performed by a designated Reconciler role (or admins), not by general reviewers.

Lifecycle

  1. Created explicitly (assignment-based or entered via reconciliation mode) — not by random selection
  2. Produces Reconciled Annotations stored alongside the study's extraction info
  3. Candidate responses remain as provenance but are not used for filtering once reconciled values exist

Storage

  • Stored in ExtractionInfo.Sessions with Reconciliation = true
  • Not included in SessionCountTarget
  • Optionally track NumberOfReconciliationSessions separately for admin dashboards

Selection vs Pool

Pool (Reviewer-Agnostic)

GET /api/projects/{projectId}/studies?stageId=<stageId>

Returns the Stage Study Pool based on FilterSet only. No per-reviewer suppression applied.

Selection (Reviewer-Aware)

POST /api/projects/{projectId}/stages/{stageId}/select_next

Returns a next study for the calling user by applying selection policies on top of the pool:

  • HideExcludedStudiesFromReviewers (bool)
  • Per-reviewer suppression (don't re-serve items the reviewer personally excluded for the active definition)
  • MaxInProgress guard / resume saved sessions
  • StudySelectionMode ∈ {Screening, Annotation, ScreeningAndAnnotation}

Reconciliation Mode

Not part of random selection; started/continued via explicit reconciliation routes. The reconciler sees studies from their reviewer-aware subset that:

  • Have sufficient completed candidate sessions (≥ SessionCountTarget)
  • Don't include a candidate session by the reconciler (if SelfReconciliation = false)

Selection Decision Flow

  1. If user is in Reconciliation Mode → serve randomly from eligible reconciliation subset
  2. Else if restrictToSaved || reachedMax → resume incomplete candidate sessions (respect HideExcluded…)
  3. Else choose based on StudySelectionMode:
  4. Screening → sample from screening-eligible Selection Subset
  5. Annotation → sample from InsufficientlyAllocatedAndNotStartedForAnnotationInStage
  6. ScreeningAndAnnotation → union of the above
  7. If none available but incomplete exist → resume saved
  8. Else → Complete (nothing new available)

Filtering with Annotations (Phase-2)

  • Only Reconciled Annotations are used as filterable values
  • Candidate session answers may be shown as context but are not evaluated by filter rules

Confirmed Defaults

  • Filtering uses only Reconciled Annotations for future annotation-based filtering (post-MVP)
  • Reconciliation selection is random from the reconciler-aware Selection Subset
  • Per-reviewer suppression is a selection-time rule (never part of the Stage Study Pool)
  • Default HideExcludedStudiesFromReviewers = ON (selection-time visibility policy)
  • StudySelectionMode has no global default; required field when creating a stage
  • SessionCountTarget is a stage property (no TA/FT defaults baked in)
  • SelfReconciliation = OFF by default
  • Session creation rule: create a candidate session only if the study is not excluded by stage settings and the reviewer has not already started one in that stage
  • Exports/reporting: candidate annotation exports remain available; when reconciled values exist, they become the authoritative values for filtering

Open Question

Should HideExcludedStudiesFromReviewers also apply in Reconciliation Mode? (Decide when reconciliation is fully implemented in the frontend.)