Today’s AI models are smart. But they're far from infallible.

Hallucination has been an issue plaguing AI researchers since the tech’s inception. It’s something even the most powerful and intelligent AI models on the market fall victim to, with the problem worsening as tasks become more complex. The answer to this might not be bigger, more intelligent models, but rather more reliable ones, Dan Klein, CTO and co-founder of AI neolab Scaled Cognition and computer science professor at UC Berkeley, told The Deep View.

“I think a prerequisite for AI to be its best self is having systems that are reliable and trustworthy and controllable,” Klein said. “A system that's just throwing tokens at you without even itself knowing what is true and what is not, it feels like a recipe for systems that you can't ship because they just aren't reliable.”

It’s natural that the industry believes scale will close the accuracy gap, said Klein, because it was true for a long time. Instances of hallucination vastly decreased as models became larger and smarter.

• But because frontier language models have used up most of the available data at their disposal, that gap isn’t going to get much smaller. “We've kind of consumed the most important bits.”
• Additionally, language models are not “truth machines,” Klein noted. They’ve simply been optimized to create outputs that are “indistinguishable from the truth.”

For enterprise, the dangerous hallucinations aren’t the ones that are obvious, like telling users to put rocks on their pizza, said Klein. Rather, it’s the ones that can go easily unnoticed that present the highest risks. For instance, if a customer asks a chatbot for their bank balance and pulls up an incorrect number, if it’s off by one digit but resembles an accurate amount, that could lead to a customer making bad transactions.

The problem comes down to how these models are architected, Klein said. And it’s something that Scaled Cognition is dedicated to solving. Rather than doing “next token prediction” and retrofitting for reliability, as conventional LLMs are built to do, Scaled Cognition’s models treat the information prompted for as “first-order objects” that are prioritized, giving the enterprise more control over what the model will and won’t do.

And though Scaled Cognition’s model, APT-1, is smaller than the massive models that frontier labs continue to build, its size makes it far more efficient, and “its correctness properties are in the structure of the model itself,” Klein said.

“The problem is not that the models aren't intelligent enough. It's reliability,” said Klein. “I feel like that's just not a consensus by any means. Superintelligence is going to have its place. But I think for the vast majority of things we want to do with AI, super-reliability is more important than superintelligence.”

Our Deeper View

The crisis of scale versus accuracy may only be amplified by the desire for speed. The AI industry is obsessed with doing things quickly: Faster outputs mean faster results, and faster results (in theory) mean faster returns. In the eyes of shareholders and major tech firms, bigger and faster will always be better, whether or not that kind of intelligence is necessary for all of the tasks most people will want AI to handle. Meanwhile, enterprises are under pressure to embed AI more deeply into their processes, offloading an increasing number of tasks to agents while reducing human oversight. To Klein’s point, stealthy hallucinations that mimic the truth may slip through the cracks, and risk widening the disconnect between frontier labs' priorities and those of enterprise customers.