enerative AI might need to grow out, not up.

Frontier model firms continue to scale their LLMs to new heights in the pursuit of artificial general intelligence that can do it all. But another one of the field’s leading scholars is questioning whether bigger is actually better.

On a podcast with Dwarkesh Patel last week, Ilya Sustkever, OpenAI co-founder and founder of Safe Superintelligence, called into question the validity of scaling laws, or the idea that developing larger and more powerful models inherently makes them better. Sutskever noted that in 2020, we moved from the “age of research” to the “age of scaling,” with the goal shifting from discovering new AI models to pure growth. 

Though the bigger equals better mentality is a “very low-risk way” of investing resources, Sustkever said, some are starting to realize that scale isn’t everything. “It’s back to the age of research again, just with big computers,” he told Patel.

Sustkever isn’t the only one challenging the current frenzy around scaling large language models.

• Yann Lecun, Meta AI’s former chief scientist and one of the so-called godfathers of AI, said on the Big Technology podcast in May that large language models won’t be the way we achieve “human-level AI.”
• Benjamin Riley, founder of Cognitive Resonance, wrote in an essay published in The Verge last week that human thinking and language are two distinct things. “We use language to think, but that does not make language the same as thought,” Riley wrote. 

The increased skepticism is coinciding with a growing interest in AI that better understands the world around us. In November, Dr. Fei-Fei Li’s World Labs released Marble, its first commercial world model project focused on “spatial intelligence.” Researchers at Mohamed bin Zayed University of Artificial Intelligence released their next-generation world model, PAN, last month. And robotics firm Physical Intelligence last week raised $600 million, valuing the startup at $5.6 billion. 

The recent hype around robotics and world models signals that researchers and investors alike are looking beyond language models for the next advances in AI.

Our Deeper View

Though industry leaders are butting heads on the path to AI that can do it all, the question remains: If we ever achieve AGI, what is the point of it? How can enterprises, governments, or individuals leverage a model that can do it all? What is the purpose of that kind of power? Research has shown that small models have made incredible strides at handling niche tasks within enterprises, and may even outperform large models at certain tasks. Plus, small models tend to be more affordable and less resource-intensive. The world's current massive data center buildout is based on the premise that scaling laws will remain universally true for years to come. Now that scaling is being called into question, it raises serious questions about whether it's worth the trillion-dollar price tag.