Why Your AI Is Getting More Expensive — and Smarter

Why Your AI Is Getting More Expensive — and Smarter

The compute war has moved. Pre-training was just the warm-up.

---

In 2020, scaling laws told us one thing: more parameters, more data, more compute at training time, and the model gets better. Simple. Brutal. True.

By 2025, every major lab had absorbed this lesson. GPT-4 class models require nine-figure training budgets. The top labs are reportedly spending $1B+ on a single training run. Energy consumption for a single big model equals a small town's annual usage.

And yet — the models keep improving. Not just from bigger pre-training runs.

Something else is happening. Something more interesting.

The New Frontier Lives at Inference Time

The arms race has shifted. The battle now is fought in the microseconds between your prompt hitting the API and the response arriving. This is test-time compute — the idea that you can make a model think harder when it needs to, rather than baking all intelligence into weights at training time.

Think of it like a math student. You can memorize 10,000 solved equations during a semester (pre-training). Or you can learn how to think through a new problem during the exam (test-time compute). Same student, same knowledge base, wildly different outputs depending on how much "thinking" budget you give them.

OpenAI's o1 and o3 models are the most visible examples. When you ask o1 a hard problem, it doesn't just generate a response. It thinks. For seconds. Sometimes minutes. It runs a long internal chain of thought, exploring multiple reasoning paths, self-correcting, backtracking. The tokens it burns during this process aren't free — each additional "thinking" token costs money and latency.

But the results are categorically different. o3 scored 87.5% on ARC-AGI, a benchmark designed to resist rote memorization. That's not a marginal improvement — it's a qualitative leap that previous models couldn't touch.

The Chinese Labs Are Paying Attention

While the Western narrative fixates on OpenAI, Chinese labs are running their own race with striking results.

DeepSeek-R1 demonstrated that reasoning capability can be distilled and transferred — their "thinking" approach spread rapidly across the open-source ecosystem. Kimi's K2.5 model pushed further on token efficiency: the ability to get the same reasoning quality with fewer thinking tokens. This matters enormously at scale. If a million users are running reasoning-heavy tasks, halving the average token consumption halves your inference bill.

The insight from K2.5's research is particularly sharp: token efficiency isn't just a cost optimization trick. It's an architectural constraint. When compute at test-time is expensive, forcing the model to be concise actually improves its reasoning quality. Compressed thinking appears to be clearer thinking.

This is the opposite of the "more parameters, more better" mental model. At test-time, smarter compute allocation beats brute-force compute allocation.

What This Means for the Industry

Test-time compute scaling introduces a new set of tradeoffs that didn't exist before:

Latency vs. Quality. A model that "thinks" for 30 seconds before answering is unusable for real-time applications. Labs are now engineering dynamic budget allocation — give the model more compute only when the problem warrants it, route simple queries to fast paths.

Cost structures are shifting. Pre-training is a one-time capital expense. Inference is a recurring operational cost. As reasoning models scale to millions of daily users, inference costs become the dominant budget line. This is why token efficiency research is suddenly a first-class concern, not an afterthought.

The moat is different now. In the pre-training era, the moat was data and compute. In the test-time era, the moat is knowing how to allocate compute intelligently. This is partly algorithmic, partly architectural, partly a question of inference infrastructure.

The Honest Caveat

We don't fully understand why test-time scaling works as well as it does. The theory lags behind the practice. Researchers can measure that giving models more "thinking budget" improves outputs on hard tasks, but the mechanism — how exactly the internal reasoning chain produces better final answers — remains partially opaque.

There are also hard ceilings. Giving a model a million thinking tokens doesn't make it AGI. At some point, the model runs out of useful thoughts to think. Understanding where that ceiling sits — and whether it can be raised with better architectures or training methods — is the central open question.

But the direction is clear. The next chapter of AI progress will be written not just in petabytes of training data, but in the clever distribution of compute between training and inference. The models that win won't just be the biggest. They'll be the ones that know when to think.