Known Limitations

Scope and Validation Status

These are genuine constraints; we’re not being modest. Understanding them is important for appropriate use of the system.

We have conducted our own validation so far

The test suite was designed by the same people who built the methodology. The expected verdicts and detection criteria were set by people who understand the decision trees intimately. While the test suite expanded substantially across three phases - eventually covering genuinely contested ground truth, non-English sources, real-world disinformation patterns, and AI-generated content - external validation where neither the claims nor the expected results are designed by the methodology’s authors would provide stronger evidence.

We have explicitly invited external parties to submit claims they believe Veridi will handle incorrectly. We look forward to these challenges.

Human volunteer testing is next

The validation was conducted by AI following the methodology’s documented procedures. This confirms that the methodology produces correct results when followed as intended, but it does not test whether human volunteers - the intended users - can follow it correctly either as a manual process (rather than AI-enabled) or in terms of providing claims in a way the system will correctly parse and handle.

Usability testing with real volunteers is a necessary and separate validation step. Questions to answer: Can a careless, vague, or self-contradictory claim input be processed reliably? Can a non-expert follow the decision trees? Do the source hierarchy definitions lead to consistent classification? Are the gaming countermeasures actionable for someone without training in disinformation analysis?

Scale testing has not yet been conducted

The methodology has been validated on 97 claims in a controlled testing environment. It has not been used in continuous production at scale. Real-world conditions introduce variables that testing cannot fully replicate: claims that don’t fit clean categories, rapidly evolving evidence, claims in languages or domains not yet tested, and adversarial techniques not yet documented.

Confidence calibration is in data-collection stage

The methodology includes a Brier score tracking framework for measuring whether confidence ratings match real-world outcomes over time. As of v2.5, outcomes are defined as correspondence to external ground truth (election results, court rulings, scientific replications) rather than verdict persistence. As of v1.2.1 (Praxis and Pragma) the framework operates as Brier-lite drift detection in production: per-cell N≥50 minimum before any flag fires; thresholds locked at ±0.10 absolute Brier deviation; methodology files are never auto-modified and flags surface for maintainer review. The calibration dataset contains 50+ entries, though most are from known test sets. Statistically meaningful calibration requires production claims where outcomes are not known at verification time. Until sufficient production data accumulates, the confidence ratings should be understood as structured estimates based on source quality, not empirically validated probabilities.

Field coefficients: 4 empirically grounded, 9 labeled as expert estimates

Of the thirteen field reliability coefficients in the confidence calibration framework, only four have strong empirical grounding from peer-reviewed replication studies. The remainder are labeled “expert estimate - needs empirical validation.” This is disclosed explicitly in the framework - presenting unsourced numbers as authoritative would be calibration theater - but it means the coefficients are hypotheses, not established facts.

Non-English coverage is limited

The methodology has been tested with sources in Japanese, Turkish, Chinese, and Hindi. It has not been tested in Arabic, Russian, Portuguese, Korean, or many other languages where disinformation operates at scale. The AI implementation can process most written languages, but the methodology’s domain-specific frameworks (particularly the IRI) are primarily calibrated for English-language institutional contexts.

The IRI covers limited institutions

The Institutional Reliability Index currently covers US and Canadian federal agencies (Statistics Canada, IRCC, Health Canada, ECCC, Bank of Canada were added in v2.5) and a small number of international institutions (TurkStat, China NBS). It does not cover most other national government agencies, international organizations, academic institutions, or private-sector entities whose reliability may be in question.

Original reporting is beyond scope

Veridi evaluates existing evidence; it does not generate new evidence. It cannot contact sources, request documents, conduct interviews, file FOIA requests, or perform investigative journalism. Claims that require original reporting to resolve will receive lower confidence or an UNVERIFIABLE verdict.

Adversarial claims were mostly constructed

Four of the 24 adversarial test claims were based on documented real-world disinformation patterns, but even these were adapted and formalized for testing. The methodology should eventually be tested against raw, unedited disinformation as it actually appears on social media, news sites, and political communications.

Validation surface expanded in 2026; metrics not yet reported

Phase 5 Wave 1 (April 2026) and Wave 2 Week 1 (May 2026) added new evaluation frameworks: a StrongREJECT capability-aware judge readiness ladder (4 readiness gates), a Multitrait-Multimethod (MTMM) protocol-doc with trait × method matrix and four pre-registered Campbell-Fiske decision rules across all three methodologies, and an Inspect AI specification (Veridi). The frameworks are documented and ready to run; the metrics they produce are not yet reported. MTMM expert panels are not running this quarter. Until these frameworks have been executed end-to-end and their results published, treat the methodology’s validation status as anchored to the v2.6 / v2.7 controlled-environment results plus the operational Brier-lite drift detection in production.

Some sites block our declared bot user-agent

As of v2.8, every web request from Veridi carries the declared user-agent Veridi Fact-Checker/1.0 with a contact URL. The bot honours robots.txt. Some sites with anti-bot defenses (notably some major news outlets and AI-company sites) return HTTP 403 to the declared user-agent, which means the fact-checker can see the URL but cannot retrieve the body. When this happens, fact-checks involving those sources route to UNVERIFIABLE or INSUFFICIENT EVIDENCE with a LIMITATIONS note documenting the block. We do not work around the block by spoofing a browser user-agent; integrity of the system depends on it identifying as itself. See the bot information page for what the bot does and how operators can allowlist it.

The AI layer has its own constraints

Veridi is implemented as a prompt system for Claude (Anthropic), adjustable for other models. This means:

• The AI’s training data has a knowledge cutoff. Claims about very recent events may lack sufficient background knowledge. (This is mitigated by explicit search-first guidance.)
• AI systems can produce confident-sounding errors. The methodology’s structural checks (source hierarchy, gaming countermeasures, confidence ceilings) are designed to constrain this, but they cannot eliminate it.
• Different AI models or versions may produce different results following the same methodology. The methodology has been validated on one implementation.

Our set of known disinformation vectors is not exhaustive

Twelve documented attack vectors cover the most common and well-understood disinformation techniques, but they are not a complete catalog. New techniques emerge, and some existing techniques may not yet be documented in the methodology. The gaming countermeasures should be treated as a strong baseline, not a comprehensive defense.

As of v2.5, Standard+ assessments surface the top three claim-relevant vectors with explicit assessments. The remaining vectors are still checked but reported as clear rather than individually detailed. Full and Forensic tiers continue to display all twelve vectors.

What we’re doing about it

If you can identify additional limitations we’ve missed, we want to know.