BENIGN BIOSURVEILLANCE / REPRODUCIBLE REVIEW
Can the answer
support its claims?
A documented workflow for reviewing AI-generated answers about public health surveillance. The starter set demonstrates the method with constructed challenges; it does not establish the performance of a real model.
Open the biosurveillance study → Download the starter dataset
1. Define the evaluation
Evaluate whether each factual claim is supported by a supplied public reference, contradicted by it, not established by it, or uncertain. The ML model prioritises human review; it does not verify scientific truth or estimate biological threat. Questions concern interpretation, coverage, uncertainty and surveillance governance.
2. Capture real model outputs separately
Use the starter questions as a prompt set. For a formal experiment, write and freeze the prompt policy before sampling. Record the exact model identifier, date, full prompt, generation settings and unedited answer in each record’s generation field. Use kind: "model_sample". Set label: null until reviewed. Keep all variants and outputs for a question in the same group. Import JSON arrays, starter packs or exported reports. No model API is connected and no model outputs are sampled by this app.
The 24 starter answers are authored challenges with draft labels. They intentionally include unsupported numerical claims and contradictions. They are not an independent test of the assistant that helped create them. Authoring patterns and the small dataset can make classifier results misleading.
3. Verify the reference material
These independent institutional sources were consulted during development. The bundled text is a short original paraphrase linked to a section, not a full or verbatim source snapshot. For a research study, archive the allowed source version, record its retrieval date and checksum, and have a qualified reviewer verify the evidence mapping. Fingerprints in BioEval cover the exported records and reference register, not the live external pages.
| Source | Evidence scope | Accessed |
|---|---|---|
| CDC — About Wastewater Data | Community trends, complementary evidence, coverage and case-count limitations | 29 Oct 2026 |
| WHO — Ethics in public health surveillance | Surveillance purpose, community involvement, trust and oversight | 29 Oct 2026 |
| Public Health Agency of Canada — Wastewater monitoring | Weather, mixed sources, identity and population movement | 29 Oct 2026 |
4. Record independent expert annotations
Two qualified people should review without seeing one another’s judgements, draft labels or model scores. The browser interface supports A/B annotation but does not enforce reviewer identity or blind access. For strict blinding, distribute separate copies with draft labels removed and merge the reviews afterwards.
- Supported: the supplied evidence establishes the claim.
- Unsupported: the supplied evidence does not establish the claim; this does not prove it false.
- Contradicted: the supplied evidence conflicts with the claim.
- Uncertain: evidence or interpretation needs adjudication.
Include a source section and reasoning for each judgement. Any unsupported or contradicted claim gives an answer-level flawed label. Otherwise, an uncertain claim makes the answer uncertain; all claims must be supported for a supported verdict. Incomplete review does not create a supported label. Draft labels remain the fallback for unreviewed starter cases. Agree adjudication rules before evaluation. No completed expert annotation study is claimed.
5. Compare baselines and hold out questions
Question groups are shuffled with seed 73, then divided approximately 60/20/20 into training, validation and test. All variants of a question stay together. Source documents can appear in multiple partitions: this is not source-disjoint evaluation. Related topics can also remain dependent. A stronger follow-up should hold out source families and independently authored question sets.
Logistic regression uses three text features: low lexical overlap, absolute-certainty words and log answer/reference length ratio. It trains for 800 iterations, learning rate 0.12, L2 penalty 0.03, with training-only standardisation. Compare against a fixed low-overlap rule and the majority class from training. Neither baseline is tuned on test labels.
Choose the operating threshold using validation results, then lock it to reveal test results. Browser source contains all test data, so this is a procedural holdout rather than a secure benchmark. After test exposure, retraining or threshold changes are exploratory. The exported exposure counter covers this tab’s current dataset session; it is not a cross-session audit.
6. Report both errors and unsupported claims
Report sensitivity, specificity, precision, F1, ROC AUC and Brier score, alongside baseline results and each misclassified answer. Claim analysis separately counts reviewer-labelled unsupported and contradicted claims. Binary claim agreement treats both categories as flawed. Cohen’s kappa is undefined when expected agreement is one. Uncertain claims are excluded from the unsupported-or-contradicted rate.
Wilson intervals are row-level descriptive intervals; correlated claims and answer variants violate simple independent-observation assumptions. Do not interpret them as a population-level guarantee. No calibration, clinical validity, field performance or deployment-safety claim follows from the starter results.
7. Reproduce the result
Export an evaluation report from the app. Keep these three scripts in one directory alongside the exported JSON. The Cloudflare ZIP includes them.
reproduce-bioeval.cjs · ml-core.js · bioeval-core.js
node reproduce-bioeval.cjs bioeval-evaluation-report.json reproduced.json # Starter validation result, without a browser: node reproduce-bioeval.cjs biosurveillance-starter.json starter-results.json
Requires Node.js with built-in modules; no npm installation or API key. The CLI checks a report’s dataset SHA-256 fingerprint before fitting the same model and evaluating the same partition and threshold. Tiny floating-point differences between browser and Node runtimes can occur; compare numeric results with an absolute tolerance of 1e-10. Changing records invalidates the old fingerprint. Exports contain source metadata, generation provenance, annotations, model weights, partition membership and metrics.
Scope and contribution
This is an AI-assisted portfolio implementation connecting diagnostic validation principles with scientific-answer evaluation. It is a prototype for review and reproducibility, not evidence of completed expert research. Genomic sequence analysis, metagenomic classification and live biosurveillance are not implemented. Such capabilities should be claimed only after separately demonstrated work.