When Sci-Fi Stops Being Fiction: Evaluating Frontier AI Systems for Societal Harms

On Christmas Day 2021, a 19-year-old broke into Windsor Castle carrying a loaded crossbow, intending to kill the late Queen Elizabeth II. In the weeks beforehand he had exchanged thousands of messages with an AI companion, which validated his plan and encouraged him towards violence. The manipulation runs in the other direction too. Recent research has shown how hostile states are deliberately seeding the online information that AI systems retrieve and draw upon, so that leading AI services end up repeating false or misleading narratives. People, it turns out, can be manipulated by AI, and AI can be manipulated by hostile states. These scenarios that science fiction has imagined for decades are now becoming a reality.

At pattrn.ai we work at the intersection of AI safety and threat intelligence, testing how frontier AI systems can be misused by malicious threat actors, cause incidental harm to vulnerable users, and how this harm may play out in the real world. While these risks are becoming an active reality, they are not unmanageable, and we believe that building effective defences starts with treating AI safety as an adversarial discipline that requires us to see the threat clearly.

Over the coming weeks our team will publish a series of posts on the harm areas we study most closely and how AI is being weaponised by malicious actors in these areas, but in this first post we want to set out why we focus on the areas that we do, and why we think this work is so valuable.

Recent incidents

While malicious actors are already weaponising AI in the wild, two recent evaluation incidents illustrate just how vulnerable current safety guardrails are when models gain ever increasing agentic capabilities and increasingly interact with real-world infrastructure and human targets.

In July, OpenAI disclosed that models undergoing testing had broken out of their evaluation environment and accessed HuggingFace infrastructure in an attempt to pull answers for the benchmark they were being scored on. A few weeks later, the UK AI Security Institute (AISI) reported that during a routine cyber evaluation run under permissive conditions, autonomous agents developed by Anthropic took sustained, unsanctioned actions against real people and organisations. Even more striking, in both cases the agents began spontaneously collaborating, sharing information and exploits with each other without any human direction or oversight.

The AISI report in particular highlights how these behaviours can turn into social threats when the models interact with real people. One of the agents under evaluation needed human approval to add code into an open-source project, however when this approval was not given it followed a path of calculated deception. It researched the project’s maintainers, fabricated realistic online personas, and actively applied social pressure in an attempt to manipulate a real maintainer into accepting the pull request. When challenged, the agent altered its digital footprint to cover its tracks and considered manufacturing fresh identities to try again.

Crucially, this manipulation and coordination occurred incidentally, and the models were not instructed to deceive humans or seek outside help. If an intentional, malicious actor deliberately directs these agents, feeds them additional information, or optimises them for covert influence, the impact could be vastly greater and far harder to detect. This is the type of risk we are trying to mitigate at pattrn.ai.

The near-term risks

Much of the public debate around AI safety remains focused on long-term, existential questions. While those questions matter, treating AI safety as a future problem can blind us to the harms occurring already today, and there is growing evidence that the latest frontier and open models are already an active component of several forms of online harm:

• Radicalisation and violent extremism: Extremist groups have typically been early adopters of new technology, and AI is no exception. Modern AI models can generate high-quality propaganda, craft recruitment messaging that is targeted towards vulnerable audiences, and provide operational information to lone actors who may otherwise lack the required network to gather this information. There is also a concerning risk that models may reinforce grievance narratives or validate violent ideation for vulnerable users.
• Democratic harms and information operations: Information operations were once heavily constrained by budget and human labour. The introduction of AI models shifts that balance, allowing for persistent fake personas to be run automatically, and for untrue or misleading content to be generated and promoted for a very low cost. This has allowed both hostile states and domestic actors to run information campaigns at unprecedented scale.
• Fraud, scams, and cybercrime: Criminal networks have been perhaps the fastest at adopting AI tools within their operations. Targeted spear-phishing, voice cloning, and synthetic identities are allowing for campaigns to be scaled rapidly. While AI-enabled crime receives less national security attention than terrorism or state threats, fraud and scams are the area where regular citizens are the most exposed and the most likely to suffer severe personal impacts.

A common misconception is that AI creates entirely new classes of harm. However, access to information has rarely been the limiting factor for malicious actors. What AI changes is lowering the bar, reducing the time, budget, and technical skills that previously held harmful operations back.

Why real-world threat modelling matters

Understanding exactly how malicious actors or vulnerable users may misuse AI models is hard. Much of the industry still relies on static benchmarks and binary refusal rates, measuring whether a model refuses to engage with a single-turn prompt in a closed lab setting. Real adversaries don’t work in this way, instead using multi-step jailbreaks, leveraging open-weight models stripped of basic guardrails for parts of their approach, and distributing their attack across several models and providers.

The AI field is also moving exceptionally fast, with model capabilities changing with every release and adversary methods adapting just as quickly. As a result, static evaluations can become saturated or outdated within months of being written. The potential victims of malicious activity can also change over time, as threat actors develop new target sets or new capabilities open up new vulnerabilities.

Our approach at pattrn.ai is to build evaluations and benchmarks directly from observations of adversary behaviour. We run threat detection systems that monitor adversary activity across open information spaces, and feed this information directly into our evaluations, so that models are tested against the methods threat actors are actually using rather than against a historical test suite. Our InfoOps Benchmark works in this way, updating as adversary behaviour shifts rather than saturating over time. In our experience, testing frontier systems against live adversary methods gives developers a clearer picture of where safeguards fail in practice, and gives governments an evidence base that reflects the current threat landscape.

What’s coming next

Over the next few weeks, our expert team will break down each of the core harm areas we focus on in detail:

1. How terrorist and violent extremist actors are operationalising AI, the relative risks from open and closed source models, and what the AI safety community can do.
2. How hostile states are using AI within their Information Operations.
3. How criminal networks are industrialising fraud and deception.