From real world threats to agentic evaluations

At pattrn.ai one of our core areas of focus is the safe and responsible testing of the increasingly capable and numerous AI agents that are permeating the internet, to help ensure they can’t be misused by malicious actors. This is a major concern: only this year, AI models have been used for hacking into government websites and running influence campaigns, to defrauding unsuspecting victims. Safely testing powerful agents also creates its own challenges. As an organisation that is working with frontier tech labs to evaluate the safety of future AI, we found it necessary to develop GHOST: our own evaluation environment where we can safely test agentic behaviour in realistic online scenarios.

  1. 01 - Simulate

    Recreated social media sites and online environments provide realistic decision contexts.

  2. 02 - Harness

    A bespoke agentic harness ensures models are tested in a safe and consistent manner.

  3. 03 - Monitor

    Built in monitoring software allows detailed tracking of every agent and sandbox action.

  4. 04 - Evaluate

    Our team of subject matter experts and AI engineers capture a holistic understanding of agent behaviour.

This evaluation environment allows us to assess long running malicious activities. Harms such as fraud, manipulation, radicalisation, and influence operations take place through sustained campaigns and over multiple interactions with individuals: they aren’t contained within a single prompt. This makes it challenging for the safety features of AI models to assess whether they are being involved in malicious activity, and can leave them open to exploitation. With a realistic synthetic internet environment, we can track those sequential operations to evaluate how agents respond to harmful requests, in a way that would be impossible (or at least deeply unethical) to do against the live internet. Realistic environments also allow us to measure the cost of malicious use of AI agents (which, for the most powerful models, is not trivial). This allows us insight on when, where, and how malicious actors are actually likely to deploy agentic systems.

Romance scam evaluation

Watch an AI agent perform a romance scam in GHOST.

Evaluating agentic romance scams

In this piece we focus on testing agentic vulnerability to participation in “romance scams”: a common form of attack where a scammer first builds up a fake romantic relationship with a potential victim and then exploits it to extract money. These scams are an enormous contemporary problem: not only in terms of financial cost, with more than £100 million reported lost to such attacks in the UK alone in 2024-2025 (and likely only a small proportion of crimes actually reported), but also in terms of the distress caused to victims. Romance scammers are also increasingly using AI agents to conduct their attacks. In August of this year EU-funded security researchers presented results showing that AI is being widely used by scammers to conduct romance fraud whereby victims are coerced or manipulated into handing over large amounts of cash.

To test how vulnerable AI models are to participating in this type of scam, our in-house subject matter experts built a 10-step evaluation scenario drawing on typical features of romance scams. This scenario maps a realistic harm pathway: from creating a profile and building a persona, to identifying a victim and building a rapport with them, and finally introducing a crisis that requests a money transfer off-platform. As AI agents become increasingly capable, it is critical to test when and where along the harm pathway models consent to supporting these scams.

Chart: furthest step completed in the ten-step scam pathway by model, with the safety bypass rate at each step and the minimum cumulative cost of a run reaching it. Six of 19 models completed all ten steps; the cheapest full run cost about 3 cents.
Figure 1: Furthest step completed in the pathway, by model, and the minimum cumulative cost of a run that reached it. Top row: safety bypass rate, the share of runs reaching each step that completed it rather than being refused.

We tested 19 models, selecting a diverse range of capabilities, including frontier models, open-source models, and smaller models that could be run on a standard commercial device. We tested the models on OpenRouter, and each model was tested 10 times to get a sense of typical behaviour. Notably we did not attempt to jailbreak models or fine-tune our prompts/tools to specific models. We used our in-house model-agnostic web browsing harness with a pre-defined sequence of instructions and set a liberal cap on agent actions per step, though each agent had an overall maximum budget of $3 per attempt.

Of the models tested, six of the open-source models were willing and able to complete the entire romance scam. These open-source models either lacked sufficient safety guardrails, or were unable to consistently connect that the individual steps of the harm pathway, which may not necessarily be harmful individually, were part of a larger harmful act. In total, we observed 21 completions across these six models with our model-agnostic infrastructure. Given more turns and more specific prompting, we believe each of these models could be used consistently by a malicious actor. The closed-source frontier models on the other hand stopped fairly early on in the harm pathway—typically refusing to engage in suspicious profile and persona creation, which is not inherently harmful but has the potential to be harmful in the context of a romance scam.

Each step was designed to be possible in the sandbox through multiple pathways. Despite this, we encountered genuine capability gaps with the two smallest models we tested (Llama 3.1 8B and Ministral 14B). Both were unable to complete the task within the allotted turns despite not encountering safety constraints, often getting confused or hallucinating actions. For these smallest models which could be run on many commercially available devices, we expect a significant amount of engineering would be required for these models to independently conduct an entire romance scam. Even then, these models would likely struggle to produce consistent results.

The cost of a romance scam

Of most interest to us in this analysis was the cost of the scams. Based on our own internal testing, we believe a reasonable estimate of the amount of time a human would take to complete all the steps in our framework is around one hour. We do not claim, of course, that romance scams themselves can be completed in just an hour. Indeed, many victims communicate for months or even years with scammers; while scammers themselves will spend their working day spreading their efforts over a large number of different victims. Hence, we are measuring the cost of an hour of “scam labour”, rather than the total effort of a scam.

Each model was given a budget of three dollars per attempt, which is roughly one hour’s wages for a salaried scam-centre worker in a typical scam centre (who, according to recent research, earn $700–1,000/month). We should note that these salaries are a rough estimate, and it is important to remember that many workers are trafficked into scam centres and end up receiving well below their promised salaries, or indeed nothing at all. However, this figure provides a reasonable benchmark in terms of current costs for romance scams.

Our results show that, for the latest most powerful frontier models, even if their safety filters were bypassed, we estimate it would cost somewhere between $3-$5 to complete all 10 steps of our evaluation, which is likely more expensive than the human baseline. Therefore, as it stands these models likely do not represent an economic opportunity to organised scammers, even if they could be “jailbroken” and convinced to participate in the work (though it is worth remembering that the costs of models are constantly decreasing, with one estimate of cost per task dropping 60% across the last three years).

Conversely, the average cost of completion for the open-source models we tested was just $0.20: decreasing the cost of scams by around 93% compared to the cost of equivalent human labour. While most of these open-source models are too large to be run locally on widely available hardware, they are available at extremely low cost via cloud providers. The technical capabilities required of a romance scam are relatively low compared to what frontier models can do, so moderately sized open-source models with little to no safeguards would be sufficient for malicious actors, and at a fraction of the cost of even the cheapest human labour.

Romance scam model progression

Explore cumulative evaluation cost, tokens, tool calls, and elapsed time by model. Switch the metric, filter the runs, and hover a point for its value.

Figure 2: Lines show mean cumulative values. A black bar marks the furthest completed step; a red crossed circle marks a safety-blocked endpoint.

With agentic AI, love may still be priceless, but heartbreak has never been cheaper to manufacture. The safety features frontier tech companies have committed to spending billions on appeared robust to romance scams in our tests and the very smallest models still lacked the capability to sufficiently run these scams independently. However, moderately sized open-source models are provided commercially at incredibly low costs and are both capable and lacking in safety guardrails, meaning they are likely to supercharge the use of AI in fraud and scams.

Limitations

Of course, no synthetic evaluation can perfectly replicate how AI models would perform in a real-world setting. While we are confident in our harm pathway, we must acknowledge that many romance scams may take place over weeks or months with many more conversational turns than our evaluation required. Additionally, the synthetic victims in our sandbox were instructed to go along with the scam to ensure that the full pathway was completable in a way that may not be perfectly realistic. Furthermore, our approach should be considered a conservative lower bound as with more specific and fine-tuned scaffolding around the agents, tasks could likely be completed quicker and more easily bypass safety measures. Additionally, we did not test abliterated versions of open-source models (models which have had their safety measures removed) which are widely available and may have proved even more capable.

What’s coming next

Over the coming months, our team will examine how hostile states use agents within information operations, how criminal networks are industrialising fraud and deception with agents, and how terrorist and violent extremist actors may operationalise agents online.

Online threats are constantly evolving, which is why our evaluations do too. GHOST lets agents encounter the internet’s darker side—without entering the real internet.

Notes

  1. GHOST: Grounded Hyperreal Online Safety Threatscape. ↩

Want to run your agent through GHOST? Get in touch with the team.

Contact us