Validation (validation.*)

One of the poq.toml spec sections.

What validators interact with and how their answers are scored: task UI, evidence, rubric, dimension-level consensus outcomes, ground truth, and onboarding. Who does the reviewing — counts, pay, classes, routing, claim, and qualification gates — lives in the Validators section.

• validation.evidence (required) — What validators see on the task page.
• validation.rubric (required) — Questions validators answer.
• validation.answer_key (optional) — Hidden correct-answer fields by rubric dimension.
• validation.reported_label (optional) — Provided values by rubric dimension, shown to validators or used as a hidden quality-distance reference.
• validation.instructions (optional) — Onboarding slides. Currently not rendered.

Values of ingestion_field = "..." are always keys from [ingestion.fields] unless this doc marks them as literal UI copy. Keys on source_excerpt / source_link blocks (repository, path, commit_sha, …) also hold [ingestion.fields] key names.

validation.evidence

Each [[validation.evidence]] block is one thing validators see on the task page (an image, markdown text, a source link, a fact table, etc.). Blocks resolve per-row values from [ingestion.fields] only — do not put static markdown or catalogue text in evidence (use [[validation.instructions]] for fixed validator guidance).

When a validator opens a task, each evidence block says which fields to show and how.

Every block starts with type. After that, keys on the block either map renderer inputs from [ingestion.fields] or provide literal renderer config. Values on the right of ingestion_field are always [ingestion.fields] key names, not file paths or raw JSON properties; parameterized evidence sections below mark which sibling keys are field keys versus literal/config values.

Optional title on any evidence block sets the panel heading above that widget. The value is literal display text (not an [ingestion.fields] key). Omit title to render no heading — there are no default headings per type.

Pick the evidence blocks that match your data. Order in the file is display order.

Worked example: one task page, every evidence type

This example builds on the JSON findings walkthrough. First map the columns you need in [ingestion.fields]:

[ingestion.fields]
description       = "findings.description"
image_path        = "images.path"
finding_json      = "findings.payload"
repository        = "findings.repository"
source_path       = "findings.sourcePath"
commit_sha        = "findings.commitSha"
line_number       = "findings.lineNumber"
finding_title     = "findings.title"
proposed_severity = "findings.proposedSeverity"

Add one [[validation.evidence]] block per widget. Order in the file is display order (markdown and images tend to appear first).

[[validation.evidence]]
type = "markdown"
title = "Finding"
ingestion_field = "description"

[[validation.evidence]]
type = "image"
title = "Dermoscopy image"
ingestion_field = "image_path"

[[validation.evidence]]
type = "json_finding"
ingestion_field = "finding_json"

[[validation.evidence]]
type = "source_excerpt"
repository = "repository"
path = "source_path"
commit_sha = "commit_sha"

[[validation.evidence]]
type = "source_link"
repository = "repository"
path = "source_path"
commit_sha = "commit_sha"
line_number = "line_number"
label = "Open in GitHub"

[[validation.evidence]]
type = "datapoint_facts"
fields = [
  { label = "Title", field = "finding_title" },
  { label = "Severity", field = "proposed_severity" },
]

You rarely need every type on one project — pick the blocks that match your data. A melanoma project might only declare the image block; an audit might use markdown, source_excerpt, and datapoint_facts.

How to read a block: repository = "repository" means “read the repository column from this task’s row.” The string on the right is a [ingestion.fields] key, not a GitHub URL literal.

Evidence types

Shared on all types: optional title (literal panel heading; omit for no heading).

The first three (markdown, image, json_finding) bind a single column with ingestion_field. The rest are structured renderers that bind several named keys directly on the block (and reject the single-ingestion_field form); each is documented below.

Single-column evidence (image, markdown, json_finding)

One column from the row — set type and ingestion_field:

[[validation.evidence]]
type = "markdown"
ingestion_field = "description"

Parameterized rich evidence

These renderer types use sibling keys on the evidence block instead of ingestion_field. Values marked as field key must name a key from [ingestion.fields]; values marked as literal are copied as UI/config text.

[[validation.evidence]]
type = "image_comparison"
left_image = "before_image_url"
right_image = "after_image_url"
left_label = "Before"
right_label = "After"
mode = "side_by_side"

[[validation.evidence]]
type = "bounding_box_overlay"
image = "image_url"
boxes = [
  { label = "lesion", x = "box_x", y = "box_y", w = "box_w", h = "box_h", score = "box_score" },
]

[[validation.evidence]]
type = "video_clip"
video_url = "video_url"
poster = "poster_url"
start_seconds = 5
end_seconds = 20

[[validation.evidence]]
type = "prompt_response_pair"
prompt = "prompt_text"
response = "model_response"
alt_response = "reference_response"
model = "candidate-v1"

For image_comparison, mode must be one of side_by_side, slider, or diff. For bounding_box_overlay, boxes must contain at least one row; label, x, y, w, and h are required on each row. boxes[].kind may be either literal text or an [ingestion.fields] key.

source_excerpt

Fetches and displays source from a public GitHub repo at a pinned commit (in-panel viewer):

[[validation.evidence]]
type = "source_excerpt"
repository = "repository"
path = "source_path"
commit_sha = "commit_sha"

Use path or paths, not both.

source_link

Compact link to a pinned GitHub file (opens in a new tab). Verifies the path exists at the commit before linking.

[[validation.evidence]]
type = "source_link"
repository = "repository"
path = "source_path"
commit_sha = "commit_sha"
line_number = "line_number"
label = "Open in GitHub"

datapoint_facts

Structured label/value rows (severity, id, found-by, etc.). Declare fields as an inline array on the block — each element has label (UI copy) and field (an [ingestion.fields] key):

[[validation.evidence]]
type = "datapoint_facts"
title = "Finding facts"
fields = [
  { label = "Title", field = "finding_title" },
  { label = "Severity", field = "proposed_severity" },
]

image_comparison

Two images shown together (side-by-side, slider, or diff). left_image / right_image are [ingestion.fields] keys; mode is a literal or a per-row field.

[[validation.evidence]]
type = "image_comparison"
left_image = "before_url"
right_image = "after_url"
left_label = "Before"
right_label = "After"
mode = "slider"

bounding_box_overlay

An image with one or more labelled boxes drawn over it. image is a [ingestion.fields] key; declare each box as a [[…boxes]] table whose coordinate keys (x, y, w, h, optional score) are [ingestion.fields] keys.

[[validation.evidence]]
type = "bounding_box_overlay"
image = "frame_url"
[[validation.evidence.boxes]]
label = "Detection"
x = "box_x"
y = "box_y"
w = "box_w"
h = "box_h"
score = "box_score"

video_clip

Inline video player, optionally trimmed to a span. video_url / poster are [ingestion.fields] keys; trim bounds are numeric literals.

[[validation.evidence]]
type = "video_clip"
video_url = "clip_url"
poster = "thumbnail_url"
start_seconds = 12
end_seconds = 30

prompt_response_pair

A prompt with one or two model responses — for RLHF / preference tasks. prompt, response, alt_response are [ingestion.fields] keys.

[[validation.evidence]]
type = "prompt_response_pair"
prompt = "user_prompt"
response = "response_a"
alt_response = "response_b"
model = "Model A vs B"

validation.rubric

Each [[validation.rubric]] block is one question validators answer on the task page — its label, answer buttons (scale.), and how that single question is scored.

This section is only about questions. Overall finding rollup is fixed in code as strict AND over required dimensions.

[[validation.rubric]]
id = "severity"
label = "Severity"
prompt = "How severe is this finding?"
scale.type = "ordinal"
scale.labels = ["info", "low", "medium", "high", "critical"]
consensus_weight = 1.0

Add one [[validation.rubric]] block per question. Reference dimensions elsewhere by id (for example route match keys, consensus_skip.match applicability rules, or ground-truth wiring).

Rubric dimension fields

Each [[validation.rubric]] block sets question copy, scale (scale.), optional consensus tuning (consensus_strength_required, consensus_skip.match), and consensus weight on the same dimension.

Do not rename consensus_weight to importance. Removed keys verdict.type, verdict.contested_rule.*, verdict.match, and verdict.verified_threshold on rubric rows fail parse.

The scale block (scale.)

There is no separate top-level [scale] section. On each [[validation.rubric]] block, dotted keys (scale.type, scale.size, …) define the answer buttons validators see and the 0–100 number stored when they pick one.

At compile time the server turns every scale into a list of (label, value) pairs. Validators click labels; consensus and storage always use the numeric values. Different dimensions in the same project can use different scale types (e.g. severity as ordinal, confidence as numeric, diagnosis as 7-point Likert). These values are also the dimension's quality direction — the label mapped to 100 reads as "top quality" — see Quality Rating vs. scale.values below.

scale.type — which answer template

Required. Picks one of three templates:

For ordinal and likert types, the compiler auto-generates evenly spaced 0–100 values from the label count or scale.size. For numeric, you declare the stored scores in scale.values.

AI validator classes: When any [[validators.classes]] row has type = "ai", every rubric scale must resolve to integer-only anchors (the AI worker constrains model output to integers per dimension). Practical rules: likert with scale.size = 5 is fine; scale.size = 7 produces fractional steps and is rejected. ordinal label counts must satisfy (count - 1) divides 100 (e.g. five labels). numeric scales must use integer scale.values. See AI agent classes for class fields.

Examples by scale type

Ordinal (multi-bucket) — labels required; values auto-spaced:

[[validation.rubric]]
id = "severity"
label = "Severity"
scale.type = "ordinal"
scale.labels = ["info", "low", "medium", "high", "critical"]

Ordinal (binary yes/no) — still scale.type = "ordinal", but exactly two labels:

[[validation.rubric]]
id = "is_valid"
label = "Is this finding valid?"
scale.type = "ordinal"
scale.labels = ["no", "yes"]

Conditional dimension — grey out when the validator's answer on another dimension matches — use consensus_skip.match on the dependent dimension:

[[validation.rubric]]
id = "validity"
label = "Validity"
prompt = "Is this finding a real issue?"
scale.type = "ordinal"
scale.labels = ["False positive", "Unlikely valid", "Unclear", "Likely valid", "Clearly valid"]

[[validation.rubric]]
id = "fix_soundness"
label = "Fix soundness"
prompt = "How sound is the proposed fix?"
scale.type = "ordinal"
scale.labels = ["Unsound", "Weak", "Partial", "Mostly sound", "Fully sound"]
consensus_skip.match = { validity = ["False positive"] }

When a validator selects False positive on Validity, Fix soundness greys out in the UI and becomes optional. If they leave it unanswered, the server backfills the bottom-of-scale filler; that vote still counts toward consensus. The engine assigns consensus_reached or consensus_not_reached on every dimension — consensus_skip.match does not bypass consensus — see Dimension-level verdict.

Likert — size required; labels optional:

[[validation.rubric]]
id = "diagnosis_match"
label = "Does the diagnosis match?"
scale.type = "likert"
scale.size = 7

Numeric — values required; labels optional:

[[validation.rubric]]
id = "certainty"
label = "How certain are you?"
scale.type = "numeric"
scale.values = [0, 25, 50, 75, 100]
scale.labels = ["none", "low", "medium", "high", "certain"]

[[validation.rubric]]
id = "confidence"
label = "Confidence Level"
prompt = "How confident are you in this assessment?"
scale.type = "numeric"
scale.values = [0, 20, 80, 100]
scale.labels = ["Guessing", "Low", "High", "Certain"]

Dimension-level verdict

Each [[validation.rubric]] block can carry optional consensus_strength_required and consensus_skip.match. After validators submit, the engine assigns one of two outcomes per dimension, then rolls those up into a single finding outcome with a fixed strict AND over every required dimension (consensus_weight > 0).

Dimension outcomes (runtime)

These are computed — not TOML keys you set. The engine produces exactly two:

There is no consensus_skipped engine outcome. consensus_skip.match affects only the validator UI (greyed-out, optional dimension); filler votes from self-skipping validators count toward consensus like any other vote.

How each dimension is classified

1. If there are no valid votes or no resolvable consensus label on this dimension → consensus_not_reached.
2. If consensus strength meets the dimension's consensus_strength_required (default 0.70) → consensus_reached.
3. Otherwise → consensus_not_reached.

Consensus strength is the strongest distance-weighted support for any configured scale label — not a raw vote share. Votes at a candidate label contribute 1.0, votes at the farthest label contribute 0.0, and nearer labels contribute partial support by distance; the consensus label is the candidate with the highest weighted support. Ties prefer the label nearest the weighted mean, then the higher scale value.

Finding rollup is fixed in code as strict AND: a finding is verified only when every required dimension (consensus_weight > 0) reaches consensus_reached. Dimensions at consensus_weight = 0 are advisory and never gate the finding. There is no [validation.verdict] table — per-dimension consensus_strength_required is the only tuning knob.

Optional dimension consensus keys

Removed keys (parse error if present): verdict.type, verdict.contested_rule.*, verdict.match, verdict.verified_threshold on rubric dimensions, and the [validation.verdict] / [stage.verdict] tables.

Quality Rating vs. scale.values. Consensus Strength above measures agreement (how tightly the panel clustered) and is direction-agnostic. The Quality Rating is the other reported pillar: a directional 0–100 score, the consensus-weight-weighted mean of the scale values the panel actually picked. It answers where on the scale the answer landed, so the label you map to 100 reads as "top quality" and the one at 0 as "low quality." That direction is set by scale.values:

• numeric: the direction is exactly the order of your scale.values. To make the first label the high-quality end, give it the higher value — e.g. scale.values = [100, 0] with scale.labels = ["A", "B"] makes A top quality; [0, 100] makes B top quality.
• ordinal / likert: values are auto-generated evenly across 0–100 by position, so the last label is 100 (top) and the first is 0. Order the labels low→high, or switch to numeric if you need an order-independent mapping.

Reversing this direction does not change which label wins or whether a dimension reaches consensus (both key off agreement/dispersion) — only the directional Quality Rating flips. To score a dimension by distance to a reference value instead of its raw value, use [[validation.reported_label]] with quality = true. See Model-Prediction Agreement for a worked spec.

validation.answer_key

Use [[validation.answer_key]] when a dataset includes hidden correct answers for one or more rubric dimensions. Each row maps a rubric dimension id to an [ingestion.fields] key. Answer-key fields are stripped before validator-facing canonical fields are returned, including AI validator payloads.

Skip when the project has no answer key (for example open-ended preference data).

[[validation.answer_key]]
dimension = "diagnosis"
field = "correct_label"

[[validation.answer_key]]
dimension = "severity"
field = "expert_severity"

validation.reported_label

Use [[validation.reported_label]] when a dataset includes a value validators should see next to a rubric dimension, such as a model prediction or source-provided value. The validator UI labels this as Provided value. By default it never auto-submits a rubric answer, but the flags below can pre-select an option or repurpose the value as a hidden quality reference.

[[validation.reported_label]]
dimension = "diagnosis"
field = "model_prediction"
default_selection = true            # pre-select the mapped scale option
map = { M = "malignant", B = "benign" }   # stored code -> scale label

# Quality-reference form: the reported value scores the dimension's Quality
# Rating by distance, and is stripped from everything validators can see.
[[validation.reported_label]]
dimension = "severity"
field = "severity_in_report"
quality = true

How the Quality Rating is scored. With quality = true, each validator's vote is first converted to its closeness to the reference, and those closeness values are combined by consensus weight:

closeness = 100 * (1 - min(1, |vote - reference| / range))

range is the scale span, max(scale.values) - min(scale.values). The dimension's Quality Rating is the consensus-weight-weighted mean of the per-vote closeness, so a validator landing on the reference scores 100 and one at the farthest end scores 0. For example, on a [0, 25, 50, 75, 100] scale with the reference at 100 and panel votes of 100, 100, 75, closeness is 100, 100, 75 and the Quality Rating is 91.7. Move the reference to 0 with the same votes and it drops to 8.3, since the panel now sits far from the provided value. Consensus Strength is unaffected either way; only the directional Quality Rating changes.

The reference is resolved against the dimension's scale: the field value is matched through map, then to a scale label (case-insensitive), then to an exact scale.values anchor. A value that lands on no anchor (for example 33 on a 0/25/50/75/100 scale) does not error; the dimension quietly reverts to raw-value quality. See Model-Prediction Agreement for a full spec.

quality = true is a hidden reference, like [[validation.answer_key]]: both strip the field before validators see it, but they answer different questions. An answer key marks the correct label for accuracy scoring, while a quality reference scores how close the panel lands to a provided value on the Quality Rating pillar.

validation.instructions

[[validation.instructions]] holds onboarding slides for validators. Currently not rendered — the onboarding modal uses hardcoded copy.

Use TOML literal strings ('''…''', triple single quotes) for markdown content and any other verbatim user or policy text. Literal strings do not process backslash escapes, so Markdown escapes like \<NONE\> in tables parse correctly. Do not use basic strings ("""…""") for slide bodies — invalid escapes (e.g. \<) fail TOML parse with invalid escaped character. Reserve basic strings for short values that need escapes (e.g. delimiter = "\n\n---\n" under attestation).

[[validation.instructions]]
type = "markdown"
content = '''
Rate each finding using the OWASP severity definitions.
'''

[[validation.instructions]]
type = "image"
src = "instructions/severity.png"
caption = "Severity ladder"

The same '''…''' literal rule applies to prompt on [[validators.classes]] when type = "ai" — prompts often embed sample rows or policy text with backslashes.