AI Weapons Are in Use – And No One Is Controlling Them

Generative AI has democratized cybercrime. Real-time deepfake videos, perfectly crafted phishing emails in German, automated vulnerability scanning in minutes instead of weeks – the tools exist, are freely available, and are actively being used. Regulation? Expected no earlier than 2027.

TL;DR

AI-generated phishing emails have a 60 percent higher click rate than manually written ones – because they are linguistically and contextually nearly perfect
Deepfake-based CEO fraud caused damages of over $500 million in 2024/2025 – with an exponentially increasing trend
Offensive AI tools like WormGPT and FraudGPT are freely available in the darknet and require no technical expertise
The EU AI Act regulates AI applications but not AI misuse by criminals – a fundamental gap

The New Quality of Threat

In February 2024, a financial employee of a multinational corporation in Hong Kong transferred $25 million – after a video conference with his CFO and several colleagues. All participants were deepfakes. The voices, the faces, the gestures – all AI-generated in real time. The employee had no reason to doubt.

This is not science fiction. This is the present. And it’s getting worse – because the technology is becoming exponentially better and cheaper, while detection lags behind at a linear pace.

The Three AI Weapon Categories

1. Social Engineering at Scale: Spear-phishing, which once required hours of manual research per target, can now be automated using large language models. The AI reads LinkedIn profiles, analyzes communication patterns from leaked emails, and generates personalized messages in the sender’s language and tone. The result? Phishing emails that even trained employees struggle to distinguish from legitimate ones.

2. Deepfakes as Weapons: Voice cloning requires just three seconds of audio. Face swapping works in real time on consumer-grade hardware. Combine the two – a video call that looks and sounds exactly like the CEO – and you have the perfect social engineering vector. It bypasses every technical security control, because the attack targets human judgment, not software.

3. Autonomous Exploitation: AI-powered tools scan networks, identify vulnerabilities, and automatically generate exploit code. What once took an experienced penetration tester a week, AI accomplishes in minutes. The technical barrier to launching sophisticated attacks has effectively vanished.

Why Regulation Fails

The EU AI Act – the world’s most ambitious AI law – governs how companies and public authorities deploy AI. High-risk applications require certification. Generative AI systems must meet transparency obligations. All sound and necessary. But criminals don’t seek certification.

The AI Act addresses legitimate use cases. It offers no operational framework for tackling AI misuse by cybercriminals. Who’s responsible for preventing deepfake attacks? Who bears liability when a freely available open-source model is weaponized for fraud? These questions aren’t even on the regulatory agenda – let alone answered.

How Companies Can Protect Themselves

Procedural Safeguards: No financial transfer over $10,000 without dual-channel verification. If the CFO calls via video, confirm the request through a separate channel – in person or by phone on a known, verified number. Every time.

Deepfake Awareness: Employees must understand that flawless video conferences can be fabricated. That a CEO’s voice can be cloned. That “I saw him” no longer qualifies as proof of authenticity. This mindset shift must be embedded in every security awareness program.

AI Against AI: Deepfake detection tools are improving – but it’s an arms race. Companies should deploy them, yet never treat them as foolproof. Procedural safeguards remain the final, most reliable line of defense.

Conclusion: Pandora’s Box Is Open

Generative AI cannot be un-invented. The tools exist. They’re growing more powerful and accessible by the day. And cybercriminals are adopting them faster than defenders can adapt. The answer isn’t panic – it’s realism: harden processes, train people, demand verification. Assume every communication could be forged. Then act accordingly.

Key Facts

Deepfake Damages: CEO fraud using deepfake technology caused over $500 million in losses during 2024/2025 – the largest single incident involved $25 million.

AI Phishing Efficiency: Automatically generated spear-phishing campaigns achieve a 60 percent click-through rate, according to recent studies – compared to 12-18 percent for manually crafted messages.

Frequently Asked Questions

Can deepfakes be reliably detected?

Not yet – at least not consistently. Detection tools achieve 85-90 percent accuracy against pre-recorded videos. But in live video conferences, where lighting, angles, and audio conditions vary unpredictably, detection rates drop significantly. It’s an ongoing arms race: as detection improves, so does generation.

Is open-source AI the problem?

Partly. Open models fuel innovation and research – but also enable abuse. Banning them would backfire, pushing development underground or into jurisdictions with lax oversight, like China or Russia. A smarter approach combines built-in safety guardrails at the model level with swift, targeted enforcement against malicious use.

How much does deepfake protection cost for companies?

Technical detection tools range from $5,000 to $50,000 annually. But the most effective protection – process redesign and workforce awareness – costs a fraction of that. Investing in mandatory verification protocols for high-value financial transactions delivers the highest return on investment in deepfake defense.

More from the MBF Media Network

Header Image Source: Pexels

Print article