Open Weights, Closed Minds: What AI Transparency Actually Requires

Six months ago I pulled a local language model onto my laptop. Took about 12 minutes with Ollama. No account, no API key, no data leaving the machine. It felt like a small act of sovereignty — exactly the kind of local-first approach I’d been arguing for.

Then I started using it. And I noticed something.

The model’s cultural centre of gravity was somewhere around San Francisco, circa 2022. Ask it about food systems and it defaulted to commodity agriculture and supermarket supply chains. Ask it about community governance and it reached for American municipal frameworks. Ask it about traditional land management and it gave me a careful, earnest summary that read like it had been assembled from university anthropology papers — not from anyone who had actually grown anything, or sat with the country long enough to know it.

The model was running locally. My data was sovereign. And the thing was still, at some fundamental level, not mine.

This is the gap the previous post gestured at: open weights let you run a capable model without feeding a corporate cloud. That matters enormously. But running it locally does not tell you what the model has been taught, by whom, from what sources, shaped by whose values. That requires a different kind of literacy — and a willingness to ask questions the AI industry would rather you didn’t.

The Vocabulary Problem

Before we can talk about transparency, we need to agree on what “open” actually means — because the term has been quietly stretched until it covers almost anything.

In software, open source has a specific meaning: you can read the code, audit it, modify it, redistribute it. There is a clear standard and projects either meet it or don’t.

In AI, “open” has been redefined to mean something much narrower. Most of what gets called open-source AI today is more precisely described as open weights: the trained model parameters are released, but the training data, the fine-tuning process, and the value choices embedded in the outputs remain opaque. You can run the model. You cannot inspect what it learned, or from what.

The actual spectrum looks like this:

OLMo and Pythia are the closest thing we currently have to genuinely open AI: you can see what they were trained on, trace how the model changed across training, and inspect the pipeline. They are also, not coincidentally, academic research projects rather than commercial products. The economics of transparency are not favourable to companies whose valuation depends on proprietary data and model lock-in.

The gap between “open weights” and “actually open” is where the real questions live.

What Goes In Shapes What Comes Out

A language model’s training data is its worldview — not a metaphor, but a mechanical fact. The model learns statistical relationships from a corpus of text, and if that corpus systematically over-represents certain perspectives, the model will reproduce those biases at scale, confidently, and without flagging that it’s doing so.

Bender et al.’s 2021 paper “On the Dangers of Stochastic Parrots” makes this argument rigorously. One of its key insights is that what is absent from training data is as consequential as what is present. The model can only pattern-match on what it encountered. If it never saw something, it cannot generate it — and it often cannot tell you that it can’t.

The dominant web crawl sources — Common Crawl being the largest — are heavily skewed toward English, toward Western and relatively affluent perspectives, toward published and indexed text over oral and community knowledge. English speakers are roughly 17% of global internet users but account for the substantial majority of web-crawl content. Academic and published material dominates over local, recent, or community-generated knowledge.

For this project, and for anyone working in regenerative food systems, the implications are specific. The knowledge that matters most — traditional land management, community seed-saving practice, oral agricultural tradition, Indigenous relationships with specific country — is structurally absent from these training sets. It exists in communities, in spoken language, in practice passed hand to hand. It does not exist in the academic papers and English-language web pages that built most of today’s models.

When you ask a major language model about growing food in Australian conditions, you are not getting the accumulated knowledge of people who have farmed this continent for tens of thousands of years. You are getting a synthesis of what literate, English-speaking people have published about it. That is a meaningful difference, and it is not solvable with a better prompt. It is a structural property of how these systems were built.

The Hidden Curriculum: RLHF and Value Alignment

Training data is only the first layer of embedded choices. After initial training, virtually all commercially deployed models go through a second process: Reinforcement Learning from Human Feedback (RLHF). Human raters score model outputs; the model is trained to maximise those scores. The result is a model that behaves the way its developers’ chosen raters preferred.

The questions worth sitting with: Aligned to what, and decided by whom?

In January 2023, TIME reported on Kenyan contract workers paid less than $2 USD per hour to label violent, abusive, and deeply disturbing content for OpenAI — the human work that made GPT-4 safe enough to deploy commercially. The personal cost of alignment is rarely mentioned in product announcements.

The deeper point is whose values the process encodes. “Alignment” sounds neutral; it is not. It means: aligned to a particular set of human judgements, made by a particular group of people, operating under particular instructions, with particular blind spots.

Different models embed different alignments visibly. DeepSeek, trained under Chinese regulatory conditions, navigates its environment predictably: probe questions about Tiananmen Square or Taiwan’s status and the model deflects. These are documented, reproducible behaviours — design choices, not failures. US-based models tend toward the political Overton window of the American educated class: particular assumptions about markets, individual rights, and governance that feel like neutrality from inside that window.

Neither is neutral. Both are products of specific institutions with specific interests. The difference is that DeepSeek’s constraints are easier to see from the outside. The assumptions baked into US models are often invisible to users who share them — which is its own kind of problem.

What Transparency Would Actually Look Like

In 2018, Timnit Gebru and Margaret Mitchell (then at Google) proposed Model Cards: structured documentation accompanying every deployed model, disclosing its intended use, known limitations, performance across different population groups, and the decisions made during training. A modest, practical proposal.

Most deployed models still don’t have them. Those that do range from genuinely useful technical disclosure to product brochures that carefully avoid saying anything of consequence.

The EU AI Act’s Article 53 establishes mandatory transparency requirements for general-purpose AI models: technical documentation, training data summaries, energy and computational requirements. For models assessed as carrying “systemic risk” — the largest and most widely deployed — additional requirements apply. This is more serious than anything the major labs have voluntarily produced. But the enforcement infrastructure is still being built, and the gap between what is formally required and what is actually provided remains wide.

The honest assessment: we are years from meaningful mandatory transparency in practice. The infrastructure for community auditing that would make transparency work — the way code audits work for open-source software — does not yet exist for AI systems.

This is the realistic frame for why individual literacy matters now, rather than waiting for institutions to catch up.

Practical Literacy: A Field Guide

Since full transparency is rarely available, the question becomes what the rest of us can actually do — not as a substitute for structural accountability, which still needs building, but as a working practice while we wait.

Before you use a model

Ask who built it and what their interests are. A model built by a research consortium (AllenAI, EleutherAI) has different incentive structures than one whose revenue depends on API calls. Is there a model card? If yes, does it cover training data, limitations, and known failure modes — or is it marketing? Has it been independently red-teamed? Vendor-run safety evaluations are not independent evaluations.

While using a model

The most useful practice: probe with questions where you already know the answer. Pick a domain you have genuine expertise in — for me, LoRa mesh networking or regenerative growing in South Australian conditions — and ask questions you can evaluate. This calibrates your trust in that domain before you rely on the model for something you cannot independently check.

Notice what the model refuses to discuss, deflects on, or treats as settled when it isn’t. Refusals reveal alignment choices. Confident errors reveal training data gaps. Both are informative.

Cross-check anything you’re going to act on — against other models, against primary sources, against people with direct experience. A model that gives a confident wrong answer is more dangerous than one that admits it doesn’t know.

For Australian institutions specifically

This is not abstract. When an Australian school, health service, council, or agricultural co-op sends data through a US-based API, that data is subject to US law — specifically the CLOUD Act and FISA, which allow US authorities to compel disclosure of data held by US companies regardless of where the data originated or where the servers sit.

The Australian Privacy Act implications remain incompletely resolved. The Office of the Australian Information Commissioner (OAIC) has published AI guidance, but whether routing data through a foreign API constitutes a disclosure to a foreign entity under the Act is not yet definitively settled — which is itself an argument for caution.

For sensitive data — patient records, legal matters, First Nations community information, agricultural intelligence built from specific Australian places and relationships — local inference on open-weights models is not a preference. It is a governance requirement that most institutions are currently ignoring.

Choosing a model

Open weights over closed, all else being equal: you can run it locally, audit its outputs systematically, and avoid contributing queries to a proprietary training pipeline. Community-maintained benchmarks over vendor claims — HELM (Stanford’s Holistic Evaluation of Language Models) and EleutherAI’s evaluation harness measure what models actually do across a broad range of tasks. Vendor benchmarks measure what vendors want you to see.

Most importantly: ask what the model is aligned to, not just whether it is “safe.” Safe is a product claim. Who decided what safe means is the actual question.

Individual Literacy, Collective Commons

A single person developing good AI literacy is necessary but not sufficient. You cannot personally audit every model you interact with. Neither can institutions, with current tools.

The longer-term response is the same one this series has been building toward: open training data pipelines, open alignment processes, community auditing infrastructure — the equivalent of what makes code transparency work in open-source software. Projects like OLMo and Pythia are building in that direction. BLOOM (BigScience) attempted a genuinely multilingual, openly documented training process that included communities and languages typically absent from Western model training.

Australia has something to contribute here that it largely isn’t contributing. First Nations language data, agricultural knowledge developed over tens of thousands of years of relationship with this continent’s specific ecology, regional and community knowledge that exists nowhere in any web crawl. If open training pipelines existed that communities could contribute to on their own terms — with consent, attribution, and the right to withdraw — that would be a different kind of AI than anything currently available. Building those pipelines is a political and institutional project as much as a technical one.

In the meantime, the literacy is yours to develop. Use it as you would any tool that matters: deliberately, with appropriate scepticism, and with a clear understanding of what it can and cannot tell you.

Sources

Training data and bias

• Bender, E.M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? Proceedings of FAccT 2021. — the foundational paper on training data bias, underrepresentation, and environmental cost
• Common Crawl — the dominant open web crawl used in most large model training datasets

RLHF and value alignment

• Perrigo, B. (2023, January 18). Exclusive: OpenAI Used Kenyan Workers on Less Than $2 Per Hour to Make ChatGPT Less Toxic. TIME — reporting on the human labour behind GPT-4’s safety training

Transparency proposals and frameworks

• Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I.D., & Gebru, T. (2019). Model Cards for Model Reporting. Proceedings of FAT* 2019. — original proposal for structured model disclosure
• EU AI Act, Article 53 — Obligations for providers of general-purpose AI models — mandatory transparency requirements for frontier AI models

Open and auditable AI projects

• OLMo (Open Language Model) — Allen Institute for AI; open weights, open training data (Dolma dataset), open training pipeline
• Pythia — EleutherAI; open weights with full training checkpoints for interpretability research
• BLOOM — BigScience; multilingual model trained with open documentation and community participation across 46 languages

Evaluation

• Stanford CRFM, HELM (Holistic Evaluation of Language Models) — community benchmark measuring model behaviour across a broad range of tasks
• EleutherAI LM Evaluation Harness — open-source framework for language model evaluation

Tools referenced

• Ollama — tool for running open-weights models locally
• llama.cpp — Georgi Gerganov’s C/C++ inference engine that underlies most local model runtimes including Ollama; the foundational open-source project enabling efficient CPU and GPU inference of open-weight models on consumer hardware

Jurisdiction and data law

• CLOUD Act (Clarifying Lawful Overseas Use of Data Act) — US law enabling compelled data access from American cloud providers regardless of server location
• Office of the Australian Information Commissioner — AI guidance

Part of a series on digital sovereignty. See also: Don’t Let the Asphalt Bury the Garden and Sleepwalking Off a Digital Cliff.

• AI
• Digital Literacy
• Transparency
• Open Source
• Bias
• Governance