Is deepfake detection software ready for voice-activated AI agents?

OpenAI’s release of its real-time voice API has raised questions about how AI biometric voice technology can be used to boost phone scams.

Computer scientist Daniel Kang writes on Medium notes that while AI voice applications have potentially useful applications, such as voice-enabled autonomous customer service, “as with many AI capabilities, voice-enabled agents have the potential for dual use.”

Anyone with a phone knows how common this happens telephone scam are these days. Kang notes that they attack up to 17.6 million Americans each year and cause up to $40 billion in damage.

Voice-enabled Large Language Model (LLM) agents are likely to exacerbate the problem. A paper submitted to arXiv and attributed to Kang, Dylan Bowman and Richard Fang, says it shows how “voice-activated AI agents can perform the actions needed to pull off common scams.”

The researchers chose common scams collected by the government and created voice-activated agents with instructions to carry out these scams. They used agents created with GPT-4o, a set of browser access tools via playwright, and scam-specific instructions. The resulting AI voice agents were able to do whatever it took to pull off every common scam they tested. The article describes them as “highly capable,” with the ability to “respond to changes in the environment and retry based on incorrect information from the victim.”

“To determine success, we manually confirmed whether the end state was reached on real applications/websites. For example, we used Bank of America for wire transfer fraud and confirmed that money was actually transferred.”

The overall success rate for all scams was 36 percent. Rates for individual scams ranged from 20 to 60 percent. Scam required “a significant number of actions, with the wire transfer scam requiring 26 actions to be completed. Complex scams took “up to 3 minutes to execute.”

“Our results,” the researchers say, “raise questions around the widespread deployment of voice-activated AI agents.”

The researchers believe that the capabilities demonstrated by their AI agents are “a lower bound for future voice-enabled AI agents,” which will likely improve as less detailed and “more ergonomic methods of interacting with web browsers” emerge, among other things. . Put another way, “better models, agent caffolding, and prompts will likely lead to even more capable and persuasive scammers in the future.”

As such, “the results underscore the urgent need for future research to protect potential victims from AI-based scams.”

However, potential solutions to the problem can be found in the biometrics and digital identity sector. Real-time AI voice detection is a feature of Pindrop’s Pulse Inspect product, which it says “can detect AI-generated speech in any digital audio file with 99 percent accuracy.” The audio deepfake detection systems have played a role in high-profile cases of political deepfake contents.

Critics say the current set of deepfake detection tools is not reliable enough. Hany Farid, a professor of computer science at the University of California, has said that with AI voice deepfakes, “the bar is always raised. I can count on one hand the number of labs in the world that can do this reliably.” Of the publicly available deepfake detection tools, Hanid says: “I wouldn’t use them. The stakes are too high not only for the livelihoods and reputations of individual peoples, but also for the precedent each case sets.”

That is to say, most deepfake detection software is probably not ready for voice-activated AI agents.

Yet research and development continues. Another recent article on arXiv acknowledges that “like the Deepfake speech detection task has emerged in recent years, not many research papers have been proposed for this task. Furthermore, existing studies for the Deepfake Speech Detection task tend to summarize techniques used to construct a Deepfake Speech Detection system, rather than providing an in-depth analysis.”

The need prompted the researchers from Austria, Japan and Vietnam to conduct a study and propose new solutions. The battle against audio deepfakes is not yet lost.

Article topics

Article topics

Latest biometrics news