Are AI Scaling Laws Hitting A Wall? That’s The Wrong Question To Ask
In recent weeks there have been multiple reports that AI scaling laws may be hitting a wall. Many of these articles cite former OpenAI employees that have said that OpenAI’s forthcoming Orion model (what would be GPT 5) has not shown the level of improvement over GPT 4 as GPT 4 did over GPT 3. OpenAI co-founder and CEO Sam Altman has said that the scaling laws still apply, specifically, that “there is no wall”. Anthropic co-founder and CEO Dario Amodei has said that there are no limits to the scaling laws below human intelligence.
However, these questions around AI scaling assume the requisite capital will be there to create ever larger models. If the current gen frontier model is a $1 billion model, the next gen frontier model will be a $10 billion model (late 2025?) and the subsequent gen model will be a $100 billion model (2026 or 2027?). Where is that capital going to come from? It seems clear to me that capital will become a constraint to AI scaling.
Training smaller language models on data from specific domains (Health; Materials Science; Insurance Underwriting; Online and Physical Retail; Video Consumption; Mobile Comms), including proprietary data, may prove more cost effective than building ever larger frontier models.