When useful AI gets smaller

There is an interesting possibility hiding behind the race to build larger AI systems: a meaningful share of useful intelligence may become cheap enough to run close to the person using it.

A laptop is a less photogenic infrastructure announcement than a field of data centres. It is still worth asking what happens if the laptop can do more of the work.

There are already technical reasons to take that question seriously. Apple's 2025 foundation model report describes both an on-device model and a server model. The on-device model uses architectural and compression techniques designed for constrained hardware. The two approaches exist together, which is a useful starting point for thinking about the market.

Microsoft's Phi-3 research similarly explored how a relatively small model could deliver useful capabilities while fitting on a phone. Benchmark results have limits, and a small model that performs well on selected tests may struggle with an unfamiliar business task. Even so, the direction matters. Useful capability does not always require the largest available system.

Most work also contains more than one kind of difficulty. Sorting a document, extracting a date and deciding whether a transaction looks unusual are different tasks. Giving each step the same expensive engine is convenient for the developer. It may be an unnecessarily generous arrangement for the compute bill.

A modular system could use a small local model for routine work, conventional software for deterministic checks and a larger remote model when the task genuinely needs it. Research such as RouteLLM investigates routing between stronger and weaker models to balance cost and response quality. That research does not prove that an entire business process can safely be broken into interchangeable parts. It does show that choosing the model is itself an engineering problem with economic consequences.

Imagine a document workflow in which most pages are familiar and a few require judgement. If the familiar pages can be handled locally, the remote service becomes an escalation path. The customer still buys capability, but fewer steps necessarily generate revenue for a central model provider.

The implications could be quite substantial. More of the value might accrue to the application, the orchestration layer or the person who understands the workflow. A company could build something useful without financing a frontier training programme. Hardware already purchased for other purposes might carry part of the workload.

There would still be costs. Local deployment requires distribution, updates, security and support across a varied collection of devices. A fleet of laptops is rarely as uniform as the procurement spreadsheet suggests. Model quality must be tested against the actual task, and errors can easily cost more than the inference they saved.

Large shared systems may also retain important advantages in capability, utilisation and operational simplicity. Training smaller models can depend on expensive research and larger models upstream. Moving inference to a device changes where some expenditure occurs; it does not make the underlying work disappear.

There is another complication for anyone hoping to draw a straight line from efficiency to falling infrastructure demand. Cheaper intelligence could encourage much more use. Processes that were uneconomic to automate might become viable. A reduction in compute per task could coexist with growth in total compute consumption.

For me, the useful distinction is between demand for intelligence and the amount of central infrastructure revenue attached to each unit of it. Those two things need not grow at the same rate.

A sensible business should therefore remain curious about several architectures. Measure the cost of a successfully completed task, including review and maintenance. Test where smaller models are sufficient. Keep the expensive capability available where it earns its place.

The interesting future may contain both extraordinary data centres and rather capable ordinary devices. If that happens, the economics will depend on how work travels between them. That is a less dramatic story than a single winner taking everything. It may also be a much more useful one to prepare for.