Next time, when someone's "super-enhanced" AI proxy tries to call to negotiate a cable TV bill, it might not be software that answers the phone and bargains with customer service, but a real person.
The reason is that the AI agents, which are driven by cutting-edge foundational models developed by companies such as Meta and OpenAI, are still unable to perform actions reliably across the entire network, especially when it comes to making phone calls. The former CEO of Yahoo, who now oversees the agency startup Dazzle AI, Marissa Mayer, stated that the current top-of-the-line AI models perform excellently in filtering online home improvement services for consumers, but they may fail when it comes to booking and arranging roof upgrades.
"We rely on cutting-edge models to provide good recommendations and a list of candidates, but we will double-check to ensure that what we deliver is indeed a service for which we can take responsibility," said Mayer, referring to the fact that Dazzle will hand over some tasks to human contractors.
After companies such as Meta, OpenAI and others promoted this technology as capable of performing tasks like helping people find shoes online, AI-powered digital assistants are once again gaining attention.
The Muse of Meta, the Dots of OpenAI, and a series of other proxies can perform various web browsing tasks, but this technology is still in its early stages and often encounters numerical obstacles. Meghan Joyce, the CEO of the digital proxy startup Duckbill, stated that these obstacles typically arise in tasks that require the proxy to make calls, book appointments on behalf of others, or reserve local cleaning services such as sofa cleaning.
Joyce says that for tasks that require real-world judgment, humans still outperform AI. Companies that create digital assistants for consumers need to integrate real people to take over these tasks. This is also the reason why Duckbill recently launched an application programming interface (API) API, which is designed to help companies that build personal agents connect with human contractors.
"Our approach isn't much different from Uber; we break down tasks into smaller, atomic, and modular steps," said Joyce, who previously worked at this ride-hailing company.
Companies that develop proxies can use API of Duckbill to hand over tasks that the proxy might mess up. Duckbill will break these tasks down into smaller pieces that ordinary adults can complete stably and reliably.
“Then we followed up on the results,” said Joyce.
When talking about "confused agents," Mayer mentioned that the AI model sometimes still lacks common sense. She said that a heavy user of Dazzle AI once tried to use this technology to handle a series of home improvement projects simultaneously, including replacing the roof, fixing broken windows, and purchasing a glass desktop.
But Mayer said that this AI agent confused the window glass with the desktop glass, and as a result, this digital assistant “tried to figure out whether it was possible to put the desktop inside the window.”
Mayer said that this mistake did not last long, as Dazzle's technology identified the task and outsourced it to a real person to complete.
“In just a few seconds, the real person sorted out the situation. While the agent would have burned a large amount of token,” said Mayer, “or they might never find a good solution at all.”
The individual AI, who acts as the CEO for the startup company Wajo AI, is also introducing human employees to handle phone calls, and even to manage those CAPTCHA tests aimed at preventing robots from accessing the website.
“I believe that combining AI with human collaboration and AI is the true way to accomplish things effectively,” said Poddar, emphasizing that if agents are unable to complete tasks, people will stop using this technology.
The interaction between chatbots and proxies can be extremely private, and users have every reason to worry about human intervention, especially when the company does not explain how they operate.
Previously, an employee from Meta raised concerns about privacy regarding an internal company test: in this test, human contractors were used to answer calls on behalf of Muse AI. A spokesperson from Meta stated at the time that such tests were "a core part of the product development process."
Meta currently does not use real people to fulfill users' requests for Muse.
Joyce indicates that her startup will conduct 'comprehensive background checks' on the 1099 contract workers they employ, and compare them with the drivers from Uber and DoorDash, 'who live in the United States and are looking to earn an additional source of income.' Duckbill also uses contract workers from companies that typically provide call center labor for businesses.
Mayer indicates that Dazzle will conduct background checks and look for contractors who possess characteristics similar to those of human personal assistants.
"We usually select those people who we consider to be extremely cautious, have good taste, and strong judgment. Whether it's the way they interact with users or their ability to forget about things afterwards and process information categorically," said Mayer. "I know this implies a certain level of privilege, but I've been hiring personal assistants for some time now. I know what I want them to be like, and I also know how to guide them."
Mayer says that although the basic models are still constantly being improved, the market is still in a "transition period" until technology can perform more complex web tasks and handle phone calls as efficiently as humans.
"We don't even intend to hide that; we believe that having someone in the loop is an advantage in itself," said Mayer.
