Sutton批评单纯算力提升AI

Richard Sutton, a Turing Award winner, recently stated that raw compute alone isn't enough for intelligence and AI science needs clever new methods, being far from finished. His post sparked wide discussion in the AI community.

In his 2019 essay "The Bitter Lesson," Sutton argued that general methods scaling with compute eventually beat sophisticated human-engineered approaches, emphasizing ideas that grow with compute over human assumptions.

He criticized simply scaling large language models and called synthetic data a "big mistake" due to the real world's infinite complexity.

He called for agents with world models that continuously learn from experience, as the world is vastly more complex than what internet text can capture, and current language models don't keep learning after training.

AI researcher Jerry Tworek, who left OpenAI, contextualized Sutton's statement, noting that while the bitter lesson still holds, the more compute is used, the more the algorithm matters.

Tworek wrote that temporary success with transformer scaling, combined with a misreading of the bitter lesson, has "reward-hacked the brains of a whole generation of scientists.

AI researcher Jayden Teoh added that the bitter lesson has often been misread, causing promising ideas to be dismissed for straying too far from its spirit, though he argues this is not Sutton's fault.

Teoh also noted that ideas not following the bitter lesson were sometimes replaced by cleaner, more general approaches, enabling progress at the time.

The tension between Sutton's call for new learning methods and Tworek's criticism of blind transformer scaling is evident in a growing wave of startups betting on these gaps, such as Core Automation run by Tworek.

These founders, including Sutton with Oak Lab, believe in more capable AI but doubt today's recipe alone will get there.

Some industry leaders, like OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei, still bet on LLM scaling, with Altman pointing to an OpenAI model that disproved a long-standing mathematical conjecture and Amodei calling scaling progress a rainbow with no end.