Cloud or on-premise AI: an honest comparison
Cloud AI is more capable, cheaper to start, and requires nothing of you. On-premise AI works where there is no route out, costs the same whether ten people use it or three hundred, and keeps the documents in the building. Those are different advantages, and which one matters is decided by your network and your contracts rather than by a feature comparison.
We sell an on-premise appliance, so treat everything below accordingly. What follows is the comparison we would want to read if we were on the other side of it, including the parts where we lose.
What cloud is genuinely better at
Raw capability
A frontier hosted model is far more capable at open-ended reasoning than anything that runs on one machine in a plant room. This is not close and it is not going to stop being true. If your question is "help me think through this failure mode from first principles", the hosted model is a better colleague.
Nothing to own
No hardware to specify, rack, patch or replace. No conversation with procurement about a capital item. For a plant with one IT person who is already fully occupied, this is a real advantage and it is routinely underweighted by people selling appliances.
It improves while you sleep
Models change faster than hardware does. A hosted service gets better without anybody doing anything. Any local system has to have a story for how it keeps up, and if it does not, it is the version you bought.
Starting is nearly free
A pilot with ten users costs almost nothing and can be running this afternoon. Nothing on-premise competes with that, and a plant that has not yet established whether this is useful at all should probably start there.
What on-premise is genuinely better at
Working where the work is
OT networks running production equipment are commonly segmented from IT and from the internet on purpose. A tool that needs an outbound call to answer cannot run there. This is not a preference and it is usually the end of the discussion rather than the start of one.
The cost model at plant scale
Per-token pricing is attractive for a pilot and punitive for a plant. The moment it is genuinely useful, usage grows, and with it the bill. A fixed cost has a marginal cost of roughly zero, which inverts the incentive: you want the second shift asking more questions, not fewer.
This shows up as a management problem before it shows up as a finance one. Someone starts telling people to use the tool less.
Latency is a floor, not an average
A technician standing at a stopped machine will wait a few seconds. They will not wait through a round trip over a congested plant WAN, and the average is irrelevant to them because they only experience the instance they are in. Local inference takes the network out of the critical path.
The boundary does not depend on behaviour
A private tenancy is a contractual boundary. On-premise is a network boundary. The first is a promise about what a company will do, the second is a property of the wiring, and they fail differently: promises get renegotiated, acquired, or forgotten by someone under time pressure at three in the morning.
The numbers, on both sides
Our own measurements, on the hardware and corpus named in the benchmark write-up, so you can argue with them:
| Hosted service | One local appliance | |
|---|---|---|
| Open-ended reasoning | Substantially better | Weaker, and honestly so |
| Answering from your documents | Good, if it may read them | Comparable, because the work is retrieval |
| Cost at ten users | Lower | Higher |
| Cost at three hundred users | Grows with use | Unchanged |
| Simultaneous questions | Provider's problem | Yours, because generation queues |
| On a segmented OT network | Cannot run | Runs |
| Model improvements | Automatic | Needs a release process |
The row worth dwelling on is concurrency, because it is where a local system actually hurts. On a single local instance, four simultaneous questions returned in 8.1, 11.7, 14.4 and 17.2 seconds. The queue is the constraint, not the model.
The fix is routing rather than hardware. A question whose answer already exists as a sentence in a manual should be found and quoted rather than generated, which is faster, cheaper and less able to go wrong. That path returns in 850 milliseconds and consumes no inference capacity at all, and it covers roughly half of what people actually ask.
Three questions that settle it
In the order that saves the most time:
- Are the documents bound by anything? Export control, a customer confidentiality clause naming permitted geographies, a regulated quality system. This is a contracts question, it is usually already answered in writing, and it settles the matter before any product is evaluated. We went through it in detail in the piece on putting SOPs into a hosted assistant.
- Does the network the work happens on have a route out? If not, the question is answered.
- How many people will use it once it is good? Not the pilot. The steady state. This decides which cost model you want to be in.
If all three come back permissive, use a hosted service. It will be cheaper and more capable, and we would rather tell you that than sell you something you did not need.
The split most plants end up with
Not one or the other. Unrestricted work goes to whichever tool is most capable, and the restricted document set stays on something with no route out.
The important part is that the split is by document rather than by instruction. A rule that depends on people remembering which drawing is export-controlled will hold for a while and then not, and the failure will be quiet. A rule where the restricted material lives on a system that cannot send it anywhere does not depend on anybody's memory.