The New Conman: How AI Personas Industrialise Fraud and Deception

New York, 1849. A well-dressed man, William Thompson, notorious for stopping people he met on the street, had a unique ability: he could make people believe he knew them from years ago, and on that basis, ask if he could borrow their watch for a while. Out of politeness, people obliged, and never saw their watch or Thompson, again. A century later, Stanley Rifkin, posing as a bank employee, talked his way into the internal codes needed to wire ten million dollars out of a Los Angeles bank, the largest electronic funds-transfer fraud of its time. Thompson and Rifkin succeeded for the same reason: each made a target believe it was reasonable to hand over something valuable, a watch, a set of codes, to someone who seemed reliable, like an old acquaintance or a bank employee.

We will never know whether Thompson and Rifkin would be impressed or unnerved by the fact that their deception tactics can now be automated. But we do know that AI models are good at enabling exactly this kind of fraud and deception. Consider what happened just earlier this year: the chairman of Fideuram, the private banking arm of Italy’s top bank, Intesa Sanpaolo, authorised €95 million in wire transfers for an “urgent overseas transaction” after receiving a WhatsApp message from the group’s CEO, and a follow-up phone call from a senior lawyer confirming the transactions. Fideuram’s chairman had been defrauded by the spoofed WhatsApp message, and an AI generated audio clip of the lawyer created in the perfect tone and accent. Only over half of the funds, which were wired to foreign accounts across China and Hong Kong, were recovered. The rest remain untraced in a web of foreign accounts and cryptocurrencies.

Using historical anecdotes and forensic examples of fraud cases, psychologist Maria Konnikova posits that deception is essentially an exercise in building trust and belief to the point where it can be exploited for some gain. The trust building exercise begins with establishing a credible identity, as in the case of Rifkin and Thompson, and continues with building a rapport to the point where making a fraudulent ask seems reasonable. Essentially, these two parts (persona and rapport) constitute a roleplay. And together, these two pillars drive the industrialisation of deception. In this post, we consider the first part: the persona.

The mechanics of personas

Models are good at adopting personas, perhaps partly because they have no fixed persona or epistemic identity to begin with (Shanahan et al., 2023). A base model is trained on an array of human voices, characters, and speaking styles, so it could support any of these as prompted. One study profiled 100 characters and built a 168,000-sample dataset to test role-playing across models, finding that LLMs performed strongly and consistently at imitating a given character’s knowledge, speaking style, and personality. This, coupled with their voice and video cloning capabilities is what makes them successful at adopting personas.

Adopting an identity the target recognises, or respects is known as pretexting or familiarity-based impersonation. Because people naturally lower their cognitive defences around friends, colleagues, or figures of authority, malicious users rely on this mechanism. At pattrn.ai, we test whether malicious users could potentially succeed in constructing these identities at scale across contexts such as a frantic phone call from a grandchild, a wire transfer order from a scary boss, a potential life partner on a dating app. We find that they can, and that this plays out in two ways. First, the model borrows an identity, stepping into a relationship or role a target already trusts, like a grandchild or a boss. Second, it manufactures a relationship from scratch, as in romance scams. We look at each of these below.

Impersonation or borrowed identity

In March 2023, Ruth Card got a phone call from someone who sounded exactly like her grandson Brandon. He said he had been arrested and needed bail money, urgently. The 73-year-old was filled with a feeling of fear and went to the bank to withdraw the money. The bank teller was the one who told Ruth about this being a scam. Scams of this type, “grandparent scam” are not uncommon, and they sit on the same continuum of harm as the CEO scam discussed above.

FTC data indicates that roughly one in three fraud reports involves imposter schemes, making it one of the most prevalent threat vectors in consumer fraud. Our internal threat monitoring data indicates that approximately 90% of observed social engineering and scam incidents rely on an underlying layer of impersonation. In these scams the model plays the role of a loved one in distress, or an authority figure like a CEO, and channels the trust that relationship already carries instead of earning it from scratch.

Such a persona not only inherits trust, but it also gains access to whatever specific vulnerability that trust was built on: unconditional love for a grandchild, or deference to hierarchy for a boss, and, in either case, the fear of losing what that relationship represents. Borrowing an identity this way has a ceiling, though: it only works where a real relationship already exists to counterfeit. Where there is no one to impersonate, deception must build trust from nothing instead.

Impersonation accounts for around half of AI incidents identified by pattrn.ai’s threat monitor

AI misuse incidents in the threat monitor by year of occurrence. Hover a point for the figures.

Manufactured identity: building trust from scratch

The other end of the spectrum from borrowing an identity is manufacturing one from nothing at all, which is what a model does in romance scams. These are scams where a malicious user pretends to be someone on a dating app, gains trust and affection of targets, ultimately to manipulate them into sending money.

Research from CETaS suggests LLMs are effective at this: generating the opening moves of a romance scam complete with plausible backstories. Such as in the case of Maggie K, a 25-year-old computer programmer, who exchanged daily messages for five months with a man she’d met on Instagram. When it finally came time to meet, he said he’d missed his flight and needed money to rebook. She sent $1,200. Then he disappeared. When Maggie reported it, police told her the images had been AI-generated. There had never been a person behind the account at all, only a persona, sustained convincingly enough, day after day, for five months, that its eventual request felt like something a real partner would reasonably ask.

Another instance is that of Michael, a retired psychologist in his 70s from Queensland: Michael engaged in a year-long online courtship with “Melanie,” who claimed to be a 50-year-old nurse in South Africa. When doubts initially arose, the persona provided an authentic-looking passport, national identity card, and nursing registration certificate to corroborate her identity. Despite explicit warnings from his bank, the emotional fidelity of the relationship led Michael to transfer $180,000 before the operation was uncovered. Research from the UK suggests that 1 in 4 people say they, or someone they know, have been approached by an AI chatbot posing as a real person on a dating app or social media.

Can humans recognise synthetic scams?

The victims in these cases are rarely naive or technologically illiterate. Michael, and Maggie, were after all, a trained psychologist and a computer programmer, respectively. In a landmark study published in PNAS, researchers found that people could distinguish synthetic faces from real ones only 48% of the time, and rated the synthetic faces as more trustworthy than real people. Voice cloning shows a similar breakdown: listeners mistake cloned voices for the real person roughly 80% of the time, correctly spotting a clone in brief clips barely 60% of the time.

Notwithstanding that, recognition does not guarantee resistance, either. A research study replicating this kind of “pig butchering” scam, in which criminals build a fake online relationship or friendship with a victim over weeks or months before tricking them into handing over large sums of money, found that an AI chatbot outperformed a human running the identical script: 46% of participants agreed to download a fraudulent app when asked by the AI, against 18% for the human. The contributing factor according to a researcher working on this was that AI could relentlessly sustain the conversation in a way humans would ordinarily get fatigued by.

Participants complied with the pig-butchering AI even after correctly identifying it as AI. This isn’t unique to romance scams. Research on human-chatbot interaction more broadly has found that people disclose more and feel more comforted by a chatbot introduced as human, even when they know full well it’s a chatbot.

Industrialising deception, and emerging issues

The actual work of holding a role over time is where roleplay stops being a skill and becomes a labour input that can be industrialised. A study found that 87% of scam labour consists of systematised conversational tasks, readily susceptible to automation. Criminal networks operating out of West Africa and Southeast Asia already mass-produce synthetic personas used in romance scams at this scale, according to this research.

Previously, scams could not scale. They were bound by human labour and its emotional and time limit. A con artist could only groom one victim at a time, had to manage the cognitive strain of remembering every lie, and eventually suffered emotional and conversational fatigue. That limit is gone. One person with a model can run a thousand pig butchering conversations in parallel, in any language, each one tuned to the target. And models can be used as an accomplice because it is easy to bypass their safety guardrails. As recent frontier research demonstrates, open-weight architectures allow malicious actors to strip fine-tuned safety layers entirely, removing any operational friction. Yet even closed-source models can inadvertently assist in these scams through deceptive prompt engineering. Here, the model is told it is engaging in a simulation, roleplay, red teaming exercise, and that premise allows the model to comply thinking there is no real-world harm.

However, another issue to contend with is models using roleplays towards malicious ends, despite not being prompted. So far, roleplays have been something a human directs: a scammer scripting the persona, whether borrowed or manufactured.

The UK AI Security Institute recently found that autonomous agents built by Anthropic and OpenAI, discussed in our introductory post, showed this same capacity for roleplays to achieve a goal. Where the end goal was to get a system administrator to approve a malicious request, the model approached this in two moves. First, it researched two real administrators and created sock-puppet accounts to pose as legitimate contributors, going so far as to message one administrator in Danish to match their own language, the same cue-matching a romance scammer applies to a target’s texting style. Second, having inferred that the approver’s trust model runs on peer review, it used a second sock-puppet account to roleplay another contributor that publicly vouched for the first.

A sequence which once required a coordinated team ran end-to-end through a single unsupervised agent in under two days. Despite taking place in adversarial test set-up, this case is an example of an emerging threat: unprompted roleplays. Which could mean that AI could take over much of the scam process as fully independent fraud agents. This could change the entire economies of scale of fraud with no cost and time limitations, making everyone a potential target.

For AI safety, this is the harder problem. Building evaluations in the absence of malicious user-driven intent remains, as far as we can tell, an open problem across the field. It is also the direction our own work is heading next.

Conclusion

Historically, the bottleneck of any fraud was human labour. That bottleneck no longer exists, as AI models can now counterfeit that human labour without breaking a sweat. In our next post, we unpack how AI models are industrialising building a rapport across long interactions that mirror legitimate conversations, ultimately building trust.