What Regulators Get Wrong About Open-Source AI

Policymakers are racing to regulate AI. The EU AI Act, U.S. executive orders, and various national frameworks draw lines between “high-risk” systems, banned uses, and transparency duties. But much of the debate—and many of the rules—are built around a picture of AI that doesn’t match open-source reality. Regulators often assume centralised deployment, clear commercial actors, and controllable distribution. Open-source models blow that frame apart. What they get wrong about open-source AI isn’t just a technical detail; it shapes who gets burdened, who gets exempted, and whether the rules can work at all.

The Default Assumption: Big Vendors and Centralised Control

Most regulatory frameworks implicitly assume that AI systems are supplied by a countable set of companies. You license a model or use an API; the provider is responsible for compliance, documentation, and risk. That works when the “product” is a service with a known operator. It breaks when the artifact is a set of weights and code that anyone can download, run, modify, and redistribute. Open-source releases—from Llama and Mistral to smaller community models—are not “placed on the market” in the classic sense. They’re published. Once they’re out, there is no single entity that can patch them, recall them, or enforce terms of use. Regulators who treat open-source like a slower version of commercial AI are missing the structural difference.

Downstream Use Is Unknowable

Laws that impose obligations on “providers” or “deployers” assume you can identify who those are. With open-source, the original developer might release a model under a permissive licence. Thousands of individuals and firms then run it on their own hardware, fine-tune it, or embed it in products. There is no registry of “deployers.” No one has a complete picture of where the model runs or how it’s used. Requiring documentation, risk assessments, or human oversight for “providers” of open-source models pushes responsibility onto researchers and nonprofits who have no visibility into downstream use—and no leverage to control it. Meanwhile, bad actors who use the same model don’t identify themselves or comply. The burden falls on the wrong people; the intended targets slip through.

Some proposals try to draw a line between “open” and “closed” models and impose lighter rules on the former. That can help, but the definitions are messy. Is a model “open” if the weights are public but the training data isn’t? If it’s released under a licence that restricts commercial use? If it’s hosted by a single company that can revoke access? Regulators end up creating loopholes or punishing the wrong parties based on labels that don’t track how models actually move through the ecosystem.

Compute and Thresholds Don’t Map Onto Open Source

Some rules use compute (e.g. training FLOPs) or model size as a trigger for stricter obligations. The idea is to catch the most capable systems. But open-source models are trained once and then copied. The “training” event might have been done by a lab with huge resources; the person running the model might be on a single GPU. Regulating by training compute doesn’t cleanly map onto who is running what today. A model that required a large cluster to train can be fine-tuned and deployed by many small actors. So you either impose duties on the original trainer (who may have no ongoing relationship to the model) or you try to regulate every deployer (who are often unknown and unreachable). Neither fits the open-source lifecycle.

Worse, thresholds create perverse incentives. If “above X FLOPs” means heavy regulation, labs may cap public releases and keep the largest models closed—reducing transparency instead of increasing it. Or they may split releases to stay under the line. The result can be less visibility into the most capable systems, not more.

Transparency and Documentation at Scale

Transparency and model cards are good practices. But requiring detailed documentation, intended-use statements, and risk assessments for every open-source release creates a chilling effect. Many open-source contributors are academics, hobbyists, or small teams. They don’t have legal or compliance departments. Heavy obligations will push them to stop publishing or to publish in jurisdictions that don’t impose them—so the same models stay available, but with less accountability from the original authors. Meanwhile, malicious users don’t fill out forms. Again, the rule lands on the wrong group.

Licensing is another area where regulation and open-source clash. Some proposals would require licences to restrict certain uses or to include safety commitments. That might sound reasonable for “high-risk” systems, but open-source culture has long relied on permissive licences (MIT, Apache, etc.) that minimise friction. Forcing restrictive terms onto model releases could fragment the ecosystem and push developers toward closed releases to avoid legal ambiguity. The goal of “responsible” release is laudable; the mechanism has to fit how open-source actually works.

What Would Help Instead

Regulators could focus on use cases and sectors (e.g. hiring, critical infrastructure, law enforcement) rather than on “AI” as a monolithic product. That would apply to open-source and proprietary systems alike when they’re used in regulated contexts. They could exempt general-purpose model release from the heaviest duties and reserve strict obligations for deployers in high-stakes applications. And they could support transparency and safety research without requiring every open-source release to carry the same compliance load as a commercial product.

Another approach is to target the points of leverage that do exist: app stores, cloud providers, and large platforms that host or distribute models. They can enforce terms of use and report abuse without requiring every open-source developer to police the world. That doesn’t solve the “someone ran it on their own server” case, but it covers a lot of real-world deployment. Getting open-source right doesn’t mean leaving it unregulated; it means designing rules that don’t assume a single vendor and a controllable supply chain. Until that shift happens, regulators will keep getting open-source AI wrong—and the real risks and benefits will stay misaligned with the law.