NVIDIA keeps hammering the same point in every keynote and investor call: the metric that matters for AI isn’t FLOPS, isn’t parameter count, isn’t benchmark scores. It’s cost per token.

I agree. And once you run the numbers for a real enterprise workload, it’s the only metric that makes any sense.

Why FLOPS and Benchmarks Stopped Being Useful

Raw compute numbers were fine when AI was a research exercise. Now it’s a production workload running billions of tokens a day across agents, copilots, summarisation jobs, and retrieval pipelines.

In my experience, the question a CIO actually asks isn’t “how fast is this GPU.” It’s “what does it cost me every time an employee asks Copilot a question, and can I afford to scale that to the whole business.”

FLOPS don’t answer that. Benchmark wins don’t either. Cost per token does.

The Math Most Teams Skip

Here’s the calculation I run with architects who want to size an AI workload properly.

Take a single knowledge worker using an AI assistant. A realistic estimate is 50 prompts a day, averaging 2,000 input tokens and 800 output tokens. That’s 140,000 tokens per user per day.

Multiply by 250 working days, and you get 35 million tokens per user per year. For an organisation of 5,000 employees, that’s 175 billion tokens a year. Just for one assistant use case.

Now apply pricing. At $3 per million input tokens and $15 per million output tokens (roughly where frontier models sit in late 2025 and into 2026), you’re looking at roughly $1.5–$2 million a year in inference spend for that one workload.

If the model you pick is 30% cheaper per token, you’ve just saved $500,000 a year without changing a single thing about your architecture.

Why NVIDIA Is Right About the Metric

NVIDIA’s argument is that every generation of their hardware — Hopper, Blackwell, now Rubin — drives the cost per token down by an order of magnitude when you factor in throughput, memory bandwidth, and inference optimisations.

They’re not wrong. Blackwell delivered roughly a 25x improvement in cost per token for frontier model inference versus Hopper, depending on the workload. Rubin is promising similar jumps again.

From a buyer’s perspective, that’s the only number that compounds. A model that costs 40% less per token means you can either run 40% more workloads for the same budget, or you can run the same workloads and redirect that spend to data, governance, or security.

Where the Metric Breaks Down

Cost per token is necessary. It’s not sufficient.

The trap I see enterprises fall into is chasing the cheapest tokens without thinking about the quality of those tokens. A model that’s half the price but needs two attempts to get the right answer isn’t actually cheaper. Neither is one that forces you to stuff 10,000 tokens of context into every prompt to compensate for weaker reasoning.

The real metric is cost per successful task. Cost per token is the foundation you build that on, but you have to layer in accuracy, retry rates, and context window efficiency before you can compare models fairly.

What This Means for Architecture Decisions

Once I started treating cost per token as the anchor metric, a few things changed in how I design AI systems.

Routing became a first-class problem. Not every prompt needs the most expensive model. Simple classification, extraction, and summarisation tasks can run on a mid-tier model at a fraction of the cost. Only the hard reasoning tasks get routed to the frontier.

Caching became non-negotiable. Prompt caching and semantic caching can cut token spend by 30–50% on workloads with repetitive context. If you’re not caching, you’re overpaying.

Context engineering started mattering more than prompt engineering. Every unnecessary token in the context window is money you’re setting on fire at enterprise scale.

The Uncomfortable Conclusion

If cost per token is the metric that matters, then a lot of the AI architecture decisions being made right now are wrong.

Teams pick models based on benchmark leaderboards. They design agents that burn tokens on verbose reasoning chains. They deploy RAG systems that stuff entire documents into context instead of retrieving the right chunks.

Every one of those decisions looks fine on a whiteboard. At 175 billion tokens a year, they add up to real money.

NVIDIA’s framing is useful because it forces the conversation back to economics. AI is a utility now. You measure utilities by unit cost. The sooner enterprise architects start thinking that way, the sooner they stop being surprised by their cloud bill.