If inference gets cheaper, do we spend less on it, or find more work for it to do?
Both can happen. A cheaper request can reduce the bill for a fixed workload. It can also make a previously impractical task worth attempting. More attempts, longer runs and new applications can change the total in the other direction.
This is the broader question behind the serving experiments. I want to understand what gets cheaper, what becomes useful and which constraint appears next.
Unit cost and total demand are different quantities
Start with a deliberately small model: total spending equals tasks multiplied by attempts per task multiplied by price per attempt. It leaves out quality, adoption and how an attempt’s size changes, but it prevents one easy confusion.
If price halves and work stays fixed, spending halves. If attempted work doubles as price halves, spending stays constant. If the additional work is larger still, total spending rises. None of those scenarios follows from a price reduction alone.
What changes when unit price falls?
These sliders are assumptions, not fitted demand curves or a prediction. They do not infer energy use from dollar spending, and they do not say whether the additional attempts produce anything useful.
Where does Moore’s law fit?
I find hardware progress a useful lens, but transistor density is not a complete price model for an agent request. Memory bandwidth, capacity, networking, power, software and the model’s structure also matter.
The prefill/decode article gives a concrete example: more arithmetic throughput may not help a workload that is waiting for data. A successful optimization can move the bottleneck rather than end the need for engineering.
That is why I want to connect broad hardware claims to the operations actually on the path. “Faster chips” needs a workload before it explains a user’s wait or the cost of finishing a task.
Where does Jevons fit?
Jevons’ question is useful when an efficiency improvement makes more uses economical. It is not a guarantee that every cheaper AI service creates unlimited demand or greater total energy use.
For agents, the extra uses might include attempting a task that previously exceeded a budget, checking more alternatives, or running a longer workflow. The product still needs enough reliability and value for people to want those attempts. A reduction in price per token is not automatically a reduction in cost per successful task.
Epoch AI’s inference-capacity essay is useful here because it makes its model/workload assumptions and uncertainty explicit. I read such estimates as arguments to examine, not settled future demand. SemiAnalysis’ computation and data-movement account supplies another serving-wide perspective. Publicly available portions inform the reading; inaccessible paid material is not assumed.
Measurement changes the argument
“What gets measured gets fixed” is a useful working principle, but choosing a metric also chooses what becomes visible.
Optimizing current-input cached percentage can encourage a different decision from preserving the full eligible prior prefix. Maximizing aggregate tokens per second can hurt the interval between tokens for an interactive user. Reducing price per call can hide more retries per completed task.
The observability article therefore starts with the question each metric should answer. I want the measurements to expose the tradeoff, not merely make a dashboard look healthier.
What about a change in the frame itself?
Kuhn’s theory of scientific change is a lens for asking when an old way of describing a problem becomes insufficient. I would not use it as evidence that any particular AI capability or business outcome is inevitable.
The smaller engineering version is already useful: a product built around isolated chat requests may need a different accounting model when its requests become steps in a long-running tool loop. The unit being optimized changes. Cached prefixes, durable sessions and recoverable work become important parts of the product rather than invisible implementation detail.
The manure-and-cars analogy from Silicon Valley raises a related question: will a new technology remove a feared bottleneck, or move the costs somewhere we were not looking? I am using that as an analogy, not quoting unverified dialogue or treating the story as historical proof. A convincing analogy still needs a mechanism and evidence for the situation it is applied to.
What would change my view?
I would look for useful-task demand after holding quality and workload definitions as steady as possible. Are users adopting new tasks, simply spending less on existing ones, or retrying more because the system is unreliable? Which constraints remain: model capability, latency, price, integration effort or trust?
The personal runtime cannot establish global AI demand. It can make one small workload inspectable and show how its cost changes when we alter a cache or context policy. Public research supplies broader hypotheses; the experiment supplies a bounded observation.
That is the connection I want these posts to make. Start with the work performed for one request, then ask what happens when that work becomes cheaper or more useful to repeat.