THE VALIDATION PLAN
Define the claim before measuring it.
1 / Intended use
Prioritise scientific answer review. The model predicts a binary reference label, not scientific truth or biological risk. This app is not a clinical decision tool.
2 / Reference evidence
Every case contains a question, answer and reference. Supplied labels may be overridden by reviewer A. Verify sources and use qualified, independent reviewers before making claims about real models.
3 / Group separation
All variants of a question stay in one partition. Seeded group shuffling divides groups into roughly 60% training, 20% validation and 20% held-out test. Preprocessing is fitted on training records only.
4 / Report both errors
False negatives miss flawed answers. False positives send supported answers for unnecessary review. Sensitivity and specificity include 95% Wilson intervals.
Biosurveillance study and reproducibility
The starter pack contains 24 constructed challenge answers across 12 benign questions, grounded in short paraphrases of CDC, WHO and Public Health Agency of Canada material. It is an evaluation workflow demonstration, not a measured benchmark of a deployed LLM. Draft labels have not been independently adjudicated. Sources can overlap across question groups, so this split is not source-disjoint.
Read the protocol and source register ↗ · Download starter dataset · Download command-line evaluator
The export includes source metadata, model provenance, claim reviews, split membership and a SHA-256 fingerprint. Run the same exported records with the included evaluator to reproduce scores. No genomics or sequence analysis is implemented.
Model implementation
Binary logistic regression in JavaScript, 800 full-batch gradient-descent iterations, learning rate 0.12, L2 = 0.03 and unpenalised intercept. Features: one minus answer/reference unique-token overlap, a capped count of absolute-certainty terms, and log answer/reference word-count ratio. No reference labels or reviewer notes enter the features. The model is trained locally; no LLM or embedding API is called.
Evidence limitations
The demo contains constructed correct and flawed variants, including repeated reference text. It has lexical shortcuts and is small. Its scores do not support conclusions about any real AI provider. Use independently collected responses and expert-reviewed references for substantive research.
Grouping by question reduces direct variant leakage, but related concepts may remain across partitions. Repeated test-set exposure, label changes informed by predictions, correlated questions and label uncertainty can bias results. Blind review is on by default, but it is not enforced across sessions.
Import format
Upload a JSON array with id, group, question, reference, answer, label. Label is 0 supported, 1 flawed, or null unlabelled. Give variants of the same question the same group. Optional fields: model, topic, source_title, source_url. Use at least six distinct groups with enough labelled examples of both classes in each partition to train.
The app never fetches source links or sends your data to a model. Data and reviews remain in tab memory; reloading clears them. Exported datasets preserve annotations. No account, cloud database or long-term audit log is configured. AI-assisted portfolio development.
Reference: preventing data leakage ↗