On-premise AI for manufacturing: what it means and when you need it
On-premise AI runs the entire system (the language model, the search index and the documents) on hardware you control, with no request leaving your network during use. It differs from a private cloud tenancy, where data still travels to a third party under a contractual promise rather than a network boundary.
Most manufacturers investigating AI for maintenance and quality reach the same wall in the same week. The pilot works. Then someone from IT or legal asks where the documents go, and the answer (a vendor's cloud, in another jurisdiction, under terms that reserve the right to change) ends the conversation.
This isn't caution for its own sake. A plant's document set is a detailed description of how it makes things: process parameters, tolerances, supplier part numbers, failure history. It's among the most commercially sensitive material the company holds, and it is exactly what a document assistant needs to read.
Why manufacturers choose on-premise
Four reasons come up repeatedly, and they stack.
1. The network has no route out
OT networks running production equipment are commonly segmented from IT and from the internet, following a Purdue-model separation or similar. This isn't a policy that can be relaxed for a pilot. Any tool that needs an API call to answer a question cannot be deployed where the work happens.
2. The documents are export-controlled or contractually restricted
Defence, aerospace and medical-device suppliers hold documents under ITAR, EAR or customer-specific confidentiality terms that name permitted geographies and processors. Sending them to a general-purpose AI service is not a risk judgement: it's a breach.
3. The cost model inverts at volume
Per-token pricing is attractive for a pilot with ten users and punitive for a plant where three hundred people ask questions all shift. On-premise inference has a fixed hardware cost and a marginal cost of roughly zero, which reverses the incentive: you want people using it more, not less.
4. Latency is a floor, not an average
A technician standing at a stopped machine will wait a few seconds. They will not wait through a round trip over a congested plant WAN. Local inference removes the network from the critical path.
What actually changes in the design
"Run it locally" sounds like a deployment decision. It's an architectural one, because it removes the largest available model from the design and forces every other part to be better.
| Concern | Cloud assumption | On-premise reality |
|---|---|---|
| Model size | Frontier model, hundreds of billions of parameters | 3B–14B open-weight model that fits in available memory |
| Reasoning headroom | Model can recover from mediocre retrieval | Retrieval must be right; the model won't paper over it |
| Cost per question | Metered per token | Fixed hardware, negligible marginal cost |
| Concurrency | Provider's problem | Yours, because runtimes serialise generation by default |
| Model updates | Silent, continuous, sometimes regressive | Deliberate, versioned, testable before rollout |
| Failure mode | Service outage | Nothing external to fail |
That fourth row is where projects underestimate the work. Language model runtimes answer one request at a time unless configured otherwise. Measured on a single local instance, four simultaneous questions returned in 8.1, 11.7, 14.4 and 17.2 seconds. The queue, not the model, is the constraint.
The compensating advantage nobody expects
A small local model with correct retrieval beats a large cloud model with poor retrieval. For document question-answering, the hard part is finding the right paragraph, not composing the sentence around it.
Once you accept that the model is small, you stop asking it to do things it shouldn't. A question like "what is the torque spec for the spindle bolt" has an answer that already exists as a sentence in a manual. The correct behaviour is to find that sentence and quote it: no generation, no paraphrase, no opportunity to alter a number.
In our own system that path returns in about 850 milliseconds and never invokes the model at all. It is also the path where correctness matters most, because it's the one producing numbers a technician will act on.
The design consequence: classify the question before answering it. Lookups get quoted verbatim from the source. Only genuine synthesis ("why does this fault keep recurring") reaches the model. This removes roughly half the traffic from the slowest and least trustworthy component in the system.
What on-premise does not give you
It doesn't make the AI truthful. A local model invents torque specifications as readily as a hosted one; it simply does so on your hardware. Grounding, which is retrieving real passages and refusing when they don't exist, is a separate piece of engineering, and it's the one that determines whether the system is safe to put in front of a maintenance team.
It also doesn't remove the operational burden. Someone maintains the machine, applies model updates, watches the disk. That's a real cost, and it's the honest counterweight to the four advantages above.
Frequently asked questions
On-premise AI runs the language model, the search index and the documents on hardware the organisation controls, with no request leaving the network during use. A private cloud tenancy is not the same thing: the data still travels to a third party, protected by contract rather than by a network boundary.
Yes. Open-weight models such as Qwen, Llama and Mistral are downloaded once and run locally through a runtime like Ollama or llama.cpp. After that download there is no further network dependency, so the system operates normally on a network with no route to the internet.
Not for a single user. A 3-billion-parameter model answers in about 1.4 seconds on a 16 GB bench machine with no discrete GPU. A GPU becomes necessary for concurrency rather than corpus size, because generation serialises by default. See our hardware measurements.
The model is smaller and weaker at open-ended reasoning. For answering questions from a fixed document set this matters less than expected, because retrieval quality dominates. The gap shows up in tasks like long-form drafting, not in "what is the spindle taper on this machine".
The cost is hardware plus the person who maintains it. A single-plant deployment serving fifty people fits on one workstation-class machine with a 24 GB GPU. There is no per-question cost, which changes usage patterns: people ask more, and that is the desired outcome.
Inplant AI is an on-premise knowledge assistant for manufacturing. It indexes your manuals, SOPs, shift notes, photographs and voice memos, and answers with the document and page number. Book a walkthrough or read how we stop it inventing numbers.