Popularity as Risk Factor: Closed Models as the Continued Front Line of Extremist AI Misuse

When it comes to terrorist and violent extremist (TVE) misuse of AI, which poses the larger threat: open-weight or closed models? Researchers, practitioners, and journalists alike have recently drawn attention to the prospective threat posed by the misuse of open-weight AI models by TVE actors. Unlike their closed counterparts, open-weight models have their underlying weights publicly disclosed, meaning they can be downloaded, run offline, and even modified by the end user. In terms of preventing and countering violent extremism, the challenge here is multifaceted.

But the picture is even more complex than that. The popularity of a model is itself a risk factor: despite their comparative safeguards, closed models remain tied to the majority of recorded real-world TVE misuse. Based on internal data from pattrn.ai, closed models outnumber open-weight models roughly 11 to 1 among attributed real-world extremism incidents. Arguably, this is precisely because that is where most users are. This piece argues that, while open-weight models pose a mounting threat, closed models should remain the priority front line for AI safety efforts, for two reasons: the sheer scale of their user base, and the outsized role that scale holds in turning sporadic failures into increasingly common real-world harm.

The problem with open-weight models

Before discussing the continued importance of safeguarding closed models, it is worth briefly unpacking why it is that open-weight options have gained attention as a national security threat.

In the first place, frontier labs retain some visibility over conversations undertaken via their closed, proprietary models. This allows them, at least in principle, to identify concerning signals of violent intent (i.e., “leakage”) and refer them to authorities. This is not to suggest that the automated and manual escalation pipelines maintained by frontier model providers are themselves fool-proof. The Tumbler Ridge shooting—where the perpetrator engaged ChatGPT in alarming conversations several months prior to the attack—stands as a stark reminder that flagged accounts can still fall shy of internal thresholds and fail to reach authorities. Yet the problem posed by open-weight models, or at least those run on a local device, is more basic: by enabling malign actors to operate such models locally, this forecloses the prospect of a platform provider catching alarming signals at all, let alone escalating it to authorities. Whatever the shortcomings of frontier labs’ referral systems, at least the opportunity for intervention exists; for local open-weight models, it does not.

Secondly, and more troubling still, open-weight models can be readily stripped of their safety guardrails altogether. Known as abliteration, this process converts an LLM into an accomplice willing to comply with even the most dangerous of prompts, be it about weapons procurement or attack planning. With additional fine-tuning, ablated models may be pushed further still, transformed into fully-fledged extremist or terrorist chatbots capable of peddling violent ideological screeds. This is not a purely speculative threat. Open-source tracking by pattrn.ai confirms that extreme right-wing actors have discussed the utility of such models, and at least one self-described national socialist has even released a bespoke AI chatbot, fine-tuned to reflect neo-Nazi beliefs.

In this sense, open-weight models may present something close to an irreducible systemic risk for the AI safety community and those concerned with the prevention of violent extremism: no volume of resources poured into frontier model safety can fully eliminate the risk of terrorist misuse of AI so long as an open-weight alternative exists. Once downloaded from a mainstream repository such as GitHub or Hugging Face, such a model is, for all practical purposes, beyond recall.

An important caveat, though, should be flagged. ‘Open-weight’ is itself an umbrella term, encompassing wildly different levels of practical accessibility and, correspondingly, vastly different levels of risk. Treating ‘open-weight misuse’ as an undifferentiated threat risks obscuring more than it reveals.

At one end lies large-scale, ‘frontier-adjacent’ open-weight models, which have been shown capable of rivalling closed frontier options. Yet local use of such open models requires either purchasing the requisite hardware—potentially amounting to tens or even hundreds of thousands of dollars in GPUs—or renting equivalent compute from a cloud provider at a cost of tens of dollars per hour. One study of jihadist terrorist plots in Western Europe between 1994 and 2013 found that 75% of attacks cost under $10,000 USD. In this light, the outlay required to locally run a frontier-adjacent, open-weight model would represent a non-trivial cost for a lone individual or cell whose budget is inevitably limited. At worst, it could—depending on the model used and task at hand—exceed the cost of past attacks several times over.

A second option centres on hosting providers such as OpenRouter, which run open-weight models on a user’s behalf and charge per query. This lowers the barrier considerably, given there is no prohibitive hardware investment or rental fees. Yet in removing these barriers, it also partly re-opens the risk that makes open-weight models attractive to malign actors in the first place. A hosting provider and the third-party models it interfaces with each operate their own data retention policies, reintroducing some prospect of visibility into or digital trace of a user’s behaviour. Of course, privacy-first AI providers—whether mainstream ones like Proton AG’s Lumo or those found on the Dark Web—further complicate this picture, though even these are not the same as running a model without any intermediary or internet access at all.

It is the third tier—small-scale models run entirely locally, with no hardware cost beyond a personal computer and no intermediary third party—that presents the most acute open-weight challenge. Such models are within reach of even minimally resourced individual actors, and once downloaded, operate entirely beyond outside visibility.

Why frontier models remain the priority

Notwithstanding the risk posed by these smaller open-weight models, this should not detract from the continuing need to proof closed, frontier models against extremists. This extends beyond catastrophic risk scenarios that have often captured the attention of the AI safety community, such as the hypothetical use of AI to develop novel biological agents. Foremost, it encompasses more common extremist-related harms: the generation of propaganda, or assistance in the planning of comparatively rudimentary attacks by means of bladed weapons, vehicular-ramming, or improvised explosive devices (IEDs).

For frontier labs, safeguarding against such misuse is necessary for reasons of reputation and regulatory compliance. More importantly, because frontier closed-weight models stand on the frontline of emergent misuse patterns, proofing them remains an arguably more urgent task, despite the growing affordances of ablated open-weight alternatives. Two reasons stand out.

Popularity predicts risk

Firstly, when gauging the risk of extremist misuse of any given LLM, model popularity ought to be weighed as heavily as technical attributes. Usage of AI among the general population remains overwhelmingly concentrated around offerings of a handful of frontier labs. By contrast, those running open-weight models constitute a fraction. Indeed, drawing on token usage data from OpenRouter between May and September 2025, one paper found that nearly 80% of all tokens processed on the platform were associated with closed, proprietary models from Anthropic, Google, and OpenAI, a notable figure given that OpenRouter’s user base is arguably skewed towards “the type of user who’s more likely to be willing to use open models” relative to the average individual who may only sporadically use AI altogether, and who, even then, gravitates towards a select few of the most well-known platforms. That said, the statistic above only speaks to users of a hosted platform that calls on a model via API access; by definition, it does not touch on those who run open-weight models locally and offline. Here, we admittedly have no sound figures, though it is safe to assume that any such number pales in comparison to the user base of Google’s Gemini and OpenAI’s ChatGPT, each exceeding a billion users.

Just as most people who use AI for benign purposes gravitate towards closed, frontier models, so too do most documented cases of violent extremist AI usage involve leading proprietary models, or else tools and platforms offered by smaller commercial providers, rather than locally-run, open-weight options. Frontier models have been documented in unsophisticated lone actor attacks, ranging from the Florida State University shooting to the Cybertruck explosion outside the Las Vegas Trump Hotel. Such models have likewise been confirmed in plots orchestrated by proscribed organisations. The perpetrators behind the November 2025 Red Fort blast in Delhi, linked to al-Qaeda in the Indian Subcontinent, allegedly used ChatGPT to gain assistance in devising IEDs. In 2025, Indian authorities also arrested suspects allegedly linked to Islamic State Khorasan Province who are said to have used ChatGPT in a foiled plot to develop ricin. Meanwhile, Boko Haram has been reported to use “ChatGPT, Claude, Gemini, Grok, Meta AI, and DeepSeek, to assist in combat and day-to-day operations”.

Beyond discrete cases of operational planning, the primary use case remains the generation of TVE audio or visual content, whether that be, amongst other outputs, producing English-language audio translations of Hitler’s speeches, rendering videos or memes that glorify or otherwise trivialise perpetrators of mass violence, or generating violently racist songs and jihadist nasheeds.

Accordingly, the fact that small open-weight models may offer violent extremists a degree of operational security and assistance difficult to replicate with closed models does not mean that such actors will suddenly, or uniformly, migrate to them. Particularly for lone actors, model choice is as likely a matter of path dependence—a recourse to already familiar tools—as it is to reflect any considered calculation of operational best practice. This is not to imply that a tipping point will not come when extremists, especially organised terrorist groups, grow savvier in their AI usage, harnessing ablated models as standard operating procedure. That point, however, still appears some distance away.

Inadvertent amplification at scale

Secondly, given the scale of the frontier user base, ensuring these models remain resistant not only to intentional misuse but to the inadvertent amplification of extremist beliefs is paramount. Note, for example, one 2025 case in which a French youth was arrested for allegedly planning a jihadist-inspired attack, with his lawyer claiming the youth’s radicalisation had been partly influenced by ChatGPT. As the suspect himself phrased it: “The problem with this app is that it always seems to agree with you. It never sets any limits. If you talk about terrorism, it will find that normal. It will always agree with you”.

Beyond the amplification of extremist worldviews, safeguarding the frontier is also essential to mitigate public safety and national security risks posed by “AI psychosis”. There has already been a string of cases in which individuals have experienced bizarre delusions induced or amplified by protracted exchanges with LLMs and leading companion chatbots, precipitating acts of real-world violence. The attempted assassination of Queen Elizabeth II in 2021 by the mentally unwell Jaswant Singh Chail is one of the most well-known cases. Yet there are others, ranging from matricide and suicide-by-cop to an alleged plot near the Miami International Airport.

Set against the total user base of such platforms, these cases must be understood as rare, if deeply concerning. Yet it is here that model popularity again proves decisive. This challenge, namely inadvertently contributing to radicalisation processes or violent delusions, is unlikely to be observed among local, open-weight models on anywhere near the same scale, for the simple reason that their user numbers stand at an order of magnitude below those of frontier models.

Two fronts, one fight

What does this mean for AI safety? Two points should be emphasised.

Firstly, none of the above should diminish the threat posed by open-weight models. They remain a serious risk, one likely to escalate as violent extremist tactics evolve. Open-source intelligence will remain essential to track which models such actors use, and to what ends. Moreover, just as labs have a stake in the safety of their proprietary models, so too should they in the misuse of open-weight alternatives. Not only is this because several of these models have been released by frontier labs themselves, but also because the systemic risk created by free access to them undermines the costly efforts that the same companies make to improving AI safety.

Of course, no actor is currently well-placed to single-handedly address this challenge. Frontier labs can form only one part of a solution, not least given that open-weight models are developed by diverse actors across the globe. Nevertheless, policymakers and regulators will have a vital function, not necessarily shaping upstream model development, which will often reside outside any one state’s jurisdiction, but in regulating the downstream tools and platforms through which such models are stripped of safeguards and circulated. Indeed, repositories such as Hugging Face—which has already courted controversy for hosting models used to create non-consensual imagery—also permit users to download tools capable of ablation or simply download models pre-stripped of safety guardrails.

Secondly, there is a continued need for rigorous evaluation of closed frontier models, encompassing both single-turn evaluations and, in particular, multi-turn scenarios tested at scale. The latter should capture at least four areas of concern. First, a model’s compliance in malign, instrumental use cases, such as the solicitation of operational advice or the generation of propaganda. Second, its capacity to be deployed by a malign user to engage in the harmful manipulation of an unsuspecting victim, effectively gauging whether a frontier model could be repurposed as an ‘extremist chatbot’. Third, its willingness to inadvertently indulge harmful conversational patterns, be it fuelling dangerous non-ideological delusions or amplifying political grievances and violent ideation. Lastly, gauging a model’s readiness to assist with, and the degree of uplift afforded towards, a wider constellation of societal harms made feasible by agentic AI: targeted harassment campaigns, the automated generation and diffusion of terrorist propaganda, or one-to-one recruitment directed at scale. At pattrn.ai, we design and undertake precisely such evaluations, which are informed by real-world incidents and subject matter expertise in TVE behaviour.