How Pragma Handles Contested Questions

The core problem

Most policy questions worth asking involve genuine disagreement. Minimum wage, carbon pricing, drug policy, housing regulation, immigration - the evidence on these topics is not settled, and where it is settled, the conclusions depend on which outcomes you prioritize.

Traditional policy analysis handles this in one of two ways, both bad. Option one: pick a side and marshal evidence for it. This is advocacy, not analysis. Option two: present “both sides” as if they’re equally supported, regardless of evidence weight. This is false balance, not analysis.

Pragma takes a third approach: assess the evidence rigorously, identify where the disagreement is empirical (the data disagrees) versus where it’s normative (the values disagree), and present both dimensions explicitly.

Empirical disputes vs. value disputes

This distinction matters more than almost anything else in policy analysis.

Empirical disputes are about facts. Does minimum wage increase unemployment? Does carbon pricing reduce emissions? Does Housing First reduce chronic homelessness? These questions have answers, even if the current evidence is incomplete or contested. Better evidence can, in principle, resolve them.

Value disputes are about priorities. Should we prioritize employment levels or wage adequacy? Should we prioritize economic growth or emissions reduction? Should we prioritize individual liberty or collective welfare? These questions do not have empirical answers. No amount of evidence resolves a disagreement about what matters more.

Most real policy questions involve both. “Should we implement a carbon tax?” has an empirical component (does it reduce emissions? what are the economic effects?) and a value component (how much economic cost is acceptable for what level of emissions reduction? whose costs matter most?).

Pragma’s job is to separate these cleanly. When it issues a CONTESTED assessment, it specifies: contested on empirical grounds, contested on normative grounds, or both.

The Contested Value Map

When a policy question has both efficiency and equity dimensions - which is most of the time - Pragma assesses each separately.

Take carbon pricing. The efficiency evidence is relatively strong: carbon taxes reduce emissions, and the economic costs are manageable at moderate price levels. The equity evidence tells a different story: carbon taxes are regressive, imposing higher costs relative to income on lower-income households. Revenue recycling can mitigate this, but the degree of mitigation depends on design choices that themselves involve value trade-offs.

Pragma doesn’t average these into a single recommendation. It presents the efficiency assessment (with its confidence level) and the equity assessment (with its confidence level) side by side, flags the value dependency explicitly, and states: “The recommendation depends on the relative weight assigned to efficiency vs. equity. Under efficiency-prioritized weighting, the evidence supports implementation. Under equity-prioritized weighting, the evidence supports implementation only with specific redistributive design features.”

The reader sees both assessments. The value choice - how much to weight equity against efficiency - remains where it belongs: with the decision-maker.

NOT ASSESSABLE

When the evidence is genuinely insufficient to support any recommendation, Pragma says so. NOT ASSESSABLE is not a failure; it’s a necessary answer to a question the evidence cannot yet address.

This matters because the pressure to say something is enormous. Policymakers need answers. Journalists need quotes. Advocates need ammunition. Saying “the evidence doesn’t support a recommendation either way” is unsatisfying but sometimes true.

Pragma documents what evidence would be needed to move beyond NOT ASSESSABLE. This turns an absence into something actionable: not “we don’t know” but “we don’t know, and here is what would need to be studied to find out.”

Gaming countermeasures for contested questions

Contested policy questions are especially vulnerable to manipulation. Pragma checks for 13 specific attack vectors, several of which are particularly relevant when evidence is contested:

Value laundering - concealing normative choices inside empirical language. “The evidence shows we should prioritize X” - when “prioritize X” is a value choice the evidence cannot resolve. This is the most dangerous vector because it presents advocacy as analysis.

Mechanism laundering - treating an observed correlation as a usable causal mechanism for intervention. “Countries with more vacation time are happier, therefore mandating more vacation time will increase happiness.” The correlation may reflect selection rather than causation.

Transferability theater - acknowledging that evidence from one context might not apply to another, then proceeding as if it does. The acknowledgment provides rhetorical cover while the recommendation ignores the constraint.

Trade-off burial - mentioning trade-offs briefly then treating them as negligible. “There may be some short-term costs, but the long-term benefits are clear.” Clear to whom? Assessed how?

Counterfactual suppression - evaluating a policy’s outcomes without specifying what it’s being compared to. “Universal healthcare reduced mortality” - compared to what? The pre-implementation baseline? The best available alternative? Doing nothing? Counterfactual choice dramatically affects apparent effectiveness.

Status quo bias exploitation - using uncertainty about an intervention to defend inaction while ignoring that the status quo also has costs and its own uncertainty. The evidential standard must be applied symmetrically.

Each vector has a named detection procedure. When a flag fires, the assessment discloses what was detected and how it affects the recommendation.

What this looks like in practice

A carbon tax assessment from Pragma does not say “implement a carbon tax” or “don’t implement a carbon tax.” It says:

The efficiency evidence supports carbon pricing at moderate levels (Moderate-High confidence). The equity evidence shows regressive impact without revenue recycling (Moderate confidence). With progressive revenue recycling, equity concerns are substantially mitigated (Low-Moderate confidence - less studied). The recommendation depends on the relative weight assigned to efficiency vs. equity and on the specific revenue recycling design. Transferability from jurisdictions with existing carbon taxes to the target jurisdiction is Moderate (institutional match varies, scale match is Weak for small-jurisdiction evidence applied to large jurisdictions). Implementation gap is Moderate (political economy constraints are significant; administrative infrastructure exists).

That’s not a soundbite. It’s a complete answer to a hard question.