Known Limitations

Scope and Validation Status

These are genuine constraints; we’re not being modest. Understanding them is important for appropriate use of the system.

We have conducted our own validation so far

The test suite was designed by the same people who built the methodology. The expected verdicts and detection criteria were set by people who understand the decision trees intimately. While the test suite expanded substantially across three phases - eventually covering genuinely contested ground truth, non-English sources, real-world disinformation patterns, and AI-generated content - external validation where neither the claims nor the expected results are designed by the methodology’s authors would provide stronger evidence.

We have explicitly invited external parties to submit claims they believe Veridi will handle incorrectly. We look forward to these challenges.

Human volunteer testing is next

The validation was conducted by AI following the methodology’s documented procedures. This confirms that the methodology produces correct results when followed as intended, but it does not test whether human volunteers - the intended users - can follow it correctly either as a manual process (rather than AI-enabled) or in terms of providing claims in a way the system will correctly parse and handle.

Usability testing with real volunteers is a necessary and separate validation step. Questions to answer: Can a careless, vague, or self-contradictory claim input be processed reliably? Can a non-expert follow the decision trees? Do the source hierarchy definitions lead to consistent classification? Are the gaming countermeasures actionable for someone without training in disinformation analysis?

Scale testing has not yet been conducted

The methodology has been validated on 97 claims in a controlled testing environment. It has not been used in continuous production at scale. Real-world conditions introduce variables that testing cannot fully replicate: claims that don’t fit clean categories, rapidly evolving evidence, claims in languages or domains not yet tested, and adversarial techniques not yet documented.

Confidence calibration is in data-collection stage

The methodology includes a Brier score tracking framework for measuring whether confidence ratings match real-world outcomes over time. This framework is designed but has not accumulated sufficient data points for statistical analysis. Until it does, the confidence ratings should be understood as structured estimates based on source quality, not empirically validated probabilities.

Field coefficients: 4 empirically grounded, 9 labeled as expert estimates

Of the thirteen field reliability coefficients in the confidence calibration framework, only four have strong empirical grounding from peer-reviewed replication studies. The remainder are labeled “expert estimate - needs empirical validation.” This is disclosed explicitly in the framework - presenting unsourced numbers as authoritative would be calibration theater - but it means the coefficients are hypotheses, not established facts.

Non-English coverage is limited

The methodology has been tested with sources in Japanese, Turkish, Chinese, and Hindi. It has not been tested in Arabic, Russian, Portuguese, Korean, or many other languages where disinformation operates at scale. The AI implementation can process most written languages, but the methodology’s domain-specific frameworks (particularly the IRI) are primarily calibrated for English-language institutional contexts.

The IRI covers limited institutions

The Institutional Reliability Index currently covers US federal agencies and a small number of international institutions (TurkStat, China NBS). It does not cover most non-US government agencies, international organizations, academic institutions, or private-sector entities whose reliability may be in question.

Original reporting is beyond scope

Veridi evaluates existing evidence; it does not generate new evidence. It cannot contact sources, request documents, conduct interviews, file FOIA requests, or perform investigative journalism. Claims that require original reporting to resolve will receive lower confidence or an UNVERIFIABLE verdict.

Adversarial claims were mostly constructed

Four of the 24 adversarial test claims were based on documented real-world disinformation patterns, but even these were adapted and formalized for testing. The methodology should eventually be tested against raw, unedited disinformation as it actually appears on social media, news sites, and political communications.

The AI layer has its own constraints

Veridi is implemented as a prompt system for Claude (Anthropic), adjustable for other models. This means:

• The AI’s training data has a knowledge cutoff. Claims about very recent events may lack sufficient background knowledge. (This is mitigated by explicit search-first guidance.)
• AI systems can produce confident-sounding errors. The methodology’s structural checks (source hierarchy, gaming countermeasures, confidence ceilings) are designed to constrain this, but they cannot eliminate it.
• Different AI models or versions may produce different results following the same methodology. The methodology has been validated on one implementation.

Our set of known disinformation vectors is not exhaustive

Eleven documented attack vectors cover the most common and well-understood disinformation techniques, but they are not a complete catalog. New techniques emerge, and some existing techniques may not yet be documented in the methodology. The gaming countermeasures should be treated as a strong baseline, not a comprehensive defense.

What we’re doing about it

If you can identify additional limitations we’ve missed, we want to know.