Foundation Models vs. Domain-Specialised Inference — A Structural Analysis

Series: Architectural AI Governance at Community Scale — A Technical Examination of Village AI (Article 2 of 5) Author: My Digital Sovereignty Ltd Date: March 2026 Licence: CC BY 4.0 International

The Distributional Prior Problem

Frontier language models are trained on corpora that reflect the statistical distribution of text on the internet. This is a well-understood observation, but its implications for deployed systems operating in specialised domains are underexamined in the alignment literature.

The training distribution of a typical large language model overrepresents:

• English-language content (and within English, American English orthography and idiom)
• Commercial, marketing, and corporate communication
• Individualistic framing of social and moral questions
• Secular therapeutic language for emotional and ethical contexts
• Technical and professional discourse
• Content from the last two decades, with limited historical depth

It correspondingly underrepresents:

• Liturgical, devotional, and theological language
• Communal decision-making traditions
• Non-Western moral and philosophical frameworks
• Oral traditions and narrative cultures
• The operational language of small, rooted communities
• Domain-specific corpora not well represented on the open web

The consequence is not a random distribution of errors. It is a systematic distributional bias: under uncertainty, the model reverts to the statistical centre of its training distribution. For communities whose norms, vocabulary, and values are underrepresented in that distribution, this reversion is silent and consistent.

Why Prompting and RLHF Do Not Resolve Distributional Priors at the Tail

Three common approaches to adapting foundation model behaviour are system prompts, RLHF, and constitutional AI. Each has documented limitations when operating at the tail of the training distribution.

System prompts provide context at inference time that conditions the model's output. They are effective when the requested behaviour is well-represented in the training distribution — the prompt activates existing patterns. They are substantially less effective when the requested behaviour conflicts with strong distributional priors. The model will follow the prompt where it can, but where the prompt specifies behaviour that is statistically rare in the training data, the prior reasserts itself. This is not a failure of prompt engineering; it is a consequence of how conditional distributions work.

RLHF learns a reward model from human preference data and uses it to fine-tune the base model. The reward model is itself a learned function, subject to the same distributional limitations as the base model. If the preference data does not adequately cover the domain in question — if, for instance, the annotators ranking outputs do not have domain expertise in Anglican liturgy or Maori tikanga — the reward model will not penalise domain-inappropriate outputs. The fine-tuned model may appear aligned on the evaluation set while remaining misaligned in precisely the domains where alignment matters most.

Constitutional AI defines principles and uses the model itself to evaluate its outputs against those principles. This is an elegant approach, but it inherits a circularity: the model's ability to evaluate domain-specific compliance is bounded by the same distributional priors that cause the misalignment. A model that substitutes therapeutic language for theological language because its training distribution favours the former will not reliably detect that substitution when asked to self-evaluate — because it lacks the distributional basis to recognise the substitution as a substitution.

None of these approaches are without value. They improve average-case behaviour substantially. The claim is narrower: they do not resolve the tail distribution problem for communities whose norms are underrepresented in the training corpus. For those communities, the failure mode is not occasional poor outputs; it is systematic, silent reversion to distributional priors.

The Specialised Layer Strategy

Village AI takes a different approach. Rather than attempting to align a frontier model through training-time interventions, it operates a smaller base model (8B parameters) with domain-specific fine-tuning layers.

The architecture has three components:

The base model: villageai-8b-corrected-v4. This is the foundation layer, trained on platform operational knowledge — how the Village system works, what features exist, how to navigate the interface. Every tenant shares this layer. It provides competent general-purpose language generation within a constrained domain.

Specialised layers: per-product-type fine-tuning. On top of the base model, domain-specific fine-tuned variants are trained for each product type. The first production specialisation is villageai-8b-episcopal-v2, trained on Episcopal/Anglican liturgical, pastoral, and governance content. The fine-tuning objective is to shift the model's distributional priors within the target domain — not to add knowledge that can be retrieved via RAG, but to reshape the model's default language, framing, and normative assumptions.

Tenant-specific content via RAG. Individual community content — their bulletins, stories, governance documents — is not baked into the model. It is retrieved at inference time via vector search (Qdrant) and provided as context. This separates the model's linguistic behaviour (shaped by fine-tuning) from the factual content it references (retrieved from the community's own corpus).

The distinction between fine-tuning and RAG is architecturally significant. Fine-tuning shifts the model's priors — its default vocabulary, its framing assumptions, its normative register. RAG provides factual grounding. The two mechanisms address different failure modes: fine-tuning addresses distributional bias (the model uses the wrong register), while RAG addresses hallucination (the model invents facts).

The Trade-Off: Capability vs. Verifiability

This approach involves a deliberate trade-off that should be stated plainly.

An 8B parameter model cannot match a 700B+ frontier model on general capability. It produces less fluent prose on out-of-domain topics. It has a smaller effective context window. It is less capable at complex multi-step reasoning. It cannot generate images, write code in obscure languages, or engage in wide-ranging philosophical discourse with the facility of a frontier system.

What it offers in exchange:

Verifiability. The model's outputs can be checked against a bounded source corpus. When the Guardian Agent architecture (described in Article 3) computes cosine similarity between the model's response embedding and the embeddings of source documents, the corpus is small enough that verification is tractable. Verifying a frontier model's response against "the internet" is not tractable.

Auditability. The model's fine-tuning data is known and controlled. The distributional priors introduced by fine-tuning are, in principle, auditable — one can examine the training corpus and understand why the model defaults to particular language. This is substantially harder with a model trained on trillions of tokens of web data.

Inference control. The model runs on controlled hardware — an AMD RX 7900 XTX GPU accessed via WireGuard VPN, with CPU fallback on a 3B degraded model for availability. No inference request leaves the operator's infrastructure. No prompt or response is logged by a third-party API provider.

Domain fidelity. Within its target domain, the specialised model's distributional priors are closer to the community's actual norms than a frontier model's priors would be. The Episcopal specialisation uses resurrection language, not therapeutic language, as its default register for bereavement contexts — because that is what its fine-tuning data contains.

The sacrifice is real. The gain is real. Whether the trade-off is appropriate depends on the deployment context. For a community that needs a general-purpose intellectual companion, this system is inadequate. For a community that needs a domain-faithful assistant whose outputs can be verified against its own records, the trade-off may be favourable.

The Embedding Pipeline

The vector search component deserves brief technical description.

Community content is processed through an embedding pipeline (EmbeddingService) that generates vector representations stored in Qdrant. At query time, the user's input is embedded using the same model, and cosine similarity search retrieves the most relevant source documents from the community's corpus.

These retrieved documents serve two purposes: they provide factual context for the model's response (standard RAG), and they provide the reference corpus against which the Guardian Agents verify the response (described in Article 3). The same embedding space is used for both retrieval and verification, which creates a coherence advantage — the verification operates in the same representational space as the retrieval — but also a potential vulnerability: systematic biases in the embedding model would affect both retrieval and verification simultaneously.

We note this as an unresolved limitation. The embedding model is a shared dependency, and its failure modes could be correlated with the generation model's failure modes in ways that are difficult to detect from within the system.

This is Article 2 of 5 in the "Architectural AI Governance at Community Scale" series. For the full Guardian Agents architecture, visit Village AI on Agentic Governance.

Previous: What AI Is — And Where the Alignment Problem Actually Sits Next: Why Training-Time Governance Fails — Architectural Constraints as an Alternative