What AI Is, What It Is Not, and What Remains Uncertain

Series: Community-Scale AI Governance — A Research Perspective on the Village Platform (Article 1 of 5) Author: My Digital Sovereignty Ltd Date: March 2026 Licence: CC BY 4.0 International

Statistical Prediction at Scale

The core mechanism of contemporary large language models (LLMs) is next-token prediction. Given a sequence of tokens, the model generates a probability distribution over possible continuations, informed by patterns extracted from a training corpus of considerable scale — typically billions of documents spanning multiple domains, languages, and registers.

This mechanism produces outputs that are frequently useful: coherent prose, competent summarisation, plausible answers to factual queries, and functional code. The practical utility is not in dispute.

What is in dispute — and what matters for governance — is the nature of the process that produces these outputs and, consequently, how much trust can be placed in them.

The Reasoning Question: An Open Empirical Problem

Early characterisations of LLMs as "stochastic parrots" — systems that reproduce statistical regularities without any form of understanding — captured something important about the technology's foundations. However, as model scale has increased, behaviours have emerged that resist simple characterisation.

Large models demonstrate capacity for multi-step logical inference, analogical reasoning across domains, and performance on novel problems structurally dissimilar to training examples. Some researchers describe these as emergent capabilities — properties that arise at scale without being explicitly engineered. Others argue that apparent reasoning is a sophisticated form of pattern interpolation that merely resembles reasoning when evaluated by human observers predisposed to attribute understanding.

The empirical evidence is, at present, insufficient to resolve this question. Several observations complicate any confident position:

• Models solve problems that require compositional generalisation, suggesting something beyond simple retrieval.
• Models also exhibit failures — confident generation of false statements, brittleness under adversarial perturbation, sensitivity to surface features of prompts — that are inconsistent with robust reasoning.
• The internal representations of large models are not well understood. Mechanistic interpretability research has identified circuit-like structures that correlate with specific capabilities, but the field is in its early stages.
• The question of whether the distinction between "genuine reasoning" and "reasoning-like behaviour" is empirically meaningful, or whether it reduces to a philosophical commitment, remains unresolved.

For governance purposes, the pragmatic implication is this: one cannot safely assume that an LLM will reason correctly, nor can one dismiss its outputs as unreliable. The system occupies an uncomfortable middle ground where outputs are often useful, sometimes wrong, and not reliably distinguishable from one another without external verification.

Novelty and Synthesis

A related question concerns whether LLMs can produce genuinely novel outputs. The strong claim — that models generate only recombinations of training data — is narrowly correct and broadly misleading.

Consider a model that has absorbed texts on polycentric governance theory, organisational behaviour, and community informatics as separate bodies of work. When prompted appropriately, it may synthesise connections across these domains that no individual researcher has made, because no individual researcher has the same breadth of exposure. The constituent ideas are not new. The synthesis, however, may be new to any given reader — and may identify genuine structural parallels that warrant investigation.

This is not equivalent to the novelty of primary research. The model has no access to empirical data it was not trained on, no capacity for experimental design, and no ability to evaluate whether its synthesised connections hold under scrutiny. The synthesis is a hypothesis generator, not a hypothesis validator. But hypothesis generation has value, provided one does not conflate it with hypothesis confirmation.

For researchers evaluating AI systems, the implication is that LLM outputs may be useful as a starting point for literature review, cross-domain exploration, and identification of structural analogies — but require the same critical scrutiny one would apply to any unverified source.

Training Data as Worldview

Every LLM inherits the statistical distribution of its training corpus. This is not a correctable bias — it is a structural property of the technology.

A model trained predominantly on English-language, commercially-oriented, Western internet content will produce outputs that reflect the assumptions, framing, and priorities of that corpus. When asked to address topics where the training data is sparse — indigenous governance traditions, liturgical language, oral culture, small-community decision-making — the model defaults to statistically dominant patterns rather than acknowledging the gap.

This has direct implications for any deployment in a specific community context. A model asked to generate content for a research group studying communal governance will default to the language of corporate management — not because it has evaluated the alternatives, but because corporate management language predominates in its training data. The substitution is silent: the model does not flag that it is operating outside its domain of competence.

This phenomenon — which might be termed distributional drift in a governance context — is well-documented but not well-solved. Techniques such as fine-tuning, retrieval-augmented generation (RAG), and system prompting can mitigate the effect but do not eliminate it. The residual bias of the base model persists, particularly under novel or complex queries where the fine-tuning signal is weaker than the base distribution.

Implications for Governance Research

The characteristics described above — useful but unreliable outputs, silent distributional bias, uncertain reasoning capacity — collectively define the governance challenge.

An AI system that is occasionally wrong is a quality-assurance problem. An AI system that is occasionally wrong in ways that silently substitute one value framework for another is a governance problem. The distinction matters because the first can be addressed by error-checking, while the second requires structural mechanisms that detect value-level drift, not merely factual error.

This is the problem the Tractatus framework is designed to address. Whether it succeeds is an empirical question examined in subsequent articles. What can be stated here is that the problem is real, well-characterised, and not adequately addressed by the policy-based approaches that currently dominate AI governance discourse.

What This Article Does Not Claim

This article does not claim that LLMs are incapable of reasoning — the evidence is insufficient for that conclusion. It does not claim that LLMs can reason — the evidence is equally insufficient. It does not claim that distributional bias is unsolvable — only that current mitigation techniques are partial. And it does not claim that AI governance is impossible — only that the governance challenge is more structural than is commonly acknowledged.

The next article examines the specific structural differences between commercial AI platforms and community-governed AI systems, and analyses the trade-offs involved.

This is Article 1 of 5 in the "Community-Scale AI Governance" series. For the full technical architecture, visit Village AI — Agentic Governance.

Next: Platform AI vs. Community-Governed AI — A Structural Analysis