What hardware do you need to run AI on-premise?
For one user, a 16 GB laptop. For fifty users, one workstation with a 24 GB GPU. The sizing driver is concurrency, not corpus size. Retrieval over a hundred thousand document sections costs milliseconds, while four simultaneous questions to one model instance take seventeen seconds because generation queues.
Hardware advice for local AI is usually written by people selling hardware. What follows is measured on a working document-QA system: a five-manual corpus of public machine-tool documentation, 374 pages, 983 indexed sections, answering real maintenance questions with citations.
The baseline: 16 GB, no graphics card
Everything below ran in our testing lab on a machine with 16 GB of unified memory and an integrated GPU. No discrete graphics card, no server. It is deliberately modest: the appliance we ship has four times the memory, and the point of the numbers is that they hold up well below it.
| Operation | Time | Notes |
|---|---|---|
| Embed the question | 32.6 ms | nomic-embed-text, 768 dimensions |
| Vector search | 12.7 ms | 983 sections, exhaustive scan, no index |
| Lookup answer | 850 ms | Quoted from the source; no model call |
| Reasoned answer, median | 1.4 s | Qwen 2.5 3B, 4-bit quantised |
| Reasoned answer, p90 | 2.2 s | Same |
| Ingest throughput | 19 pages/min | 4 CPU cores, includes OCR and layout parsing |
Two things stand out. Retrieval is free: 45 milliseconds for embedding and search combined, against a budget where anything under five seconds is acceptable. And the fastest answers involve no model at all.
Why "no graphics card" is not the problem it sounds like
There is a GPU; it is integrated, and it is why the numbers are as good as they are. All 37 layers of the model load onto it. What makes this class of machine well-suited is unified memory: a 16 GB discrete graphics card gives you 16 GB for models and not a byte more, whereas an integrated GPU sharing one pool addresses system memory directly. That is the architecture the appliance uses too, with 64 GB in the pool rather than 16.
The ceiling is real, though. On the bench machine the driver caps GPU allocation at roughly 10–11 GB of the 16, so a 7B model fits comfortably and a 14B is tight. That constraint, not benchmark scores, is what made a 3B model the right default there. It is worth being clear that the ceiling belongs to the bench and not to the product: the appliance has 64 GB of unified memory, and choosing what runs on it is a question we answer by measuring rather than by inheriting a laptop's limit.
Where it actually breaks: concurrency
Local model runtimes answer one request at a time unless told otherwise. Four questions asked simultaneously against one instance:
| Request | Returned after |
|---|---|
| First to finish | 8.1 s |
| Second | 11.7 s |
| Third | 14.4 s |
| Fourth | 17.2 s |
A single question takes about four seconds. Four take seventeen. They queued. Extrapolate to fifty people asking at once and the last one waits minutes, which, for someone standing at a stopped machine, is the same as broken.
This is the sizing rule. You are not buying a GPU to hold a bigger index. You are buying parallel inference slots. A 24 GB card holds the model weights once plus a KV cache per slot, giving roughly 8 to 16 concurrent generations.
Sizing for fifty users
| Component | Specification | Reason |
|---|---|---|
| GPU | 24 GB: L4, A10 or RTX 4090 | 8–16 parallel slots; 3–5× the token rate |
| CPU | 16+ cores | Document ingest, which is CPU-bound |
| Memory | 32–64 GB | Parsing, OCR and the index working set |
| Storage | NVMe, 1 TB | ~7 GB per million sections, plus models and originals |
Note what is absent: nothing scales with corpus size. At 6.5 KB per section, a hundred thousand sections is 650 MB, a rounding error on any modern disk. Exhaustive search would slow to about a second at that size, which is the point at which you add an approximate index and it returns to milliseconds.
The lever that beats hardware
Roughly half of maintenance questions never need a language model at all. "What is the spindle taper" has an answer that already exists as a sentence in a manual: find it, quote it, cite it. That path returns in 850 ms and consumes no inference capacity.
Every question routed away from the model is one that doesn't queue. Before sizing a GPU, measure your own ratio of lookups to genuine synthesis: it determines the answer more than user count does.
Ingest is a separate, slower problem
Answering is fast. Reading the documents in the first place is not. At 19 pages per minute on four CPU cores, a ten-thousand-page plant library takes about nine hours. That is a batch job to schedule, not an interactive step, and it parallelises across cores easily. Layout and OCR models can use a GPU too, which is the second reason to have one.
Frequently asked questions
One user: a 16 GB laptop, answering in about 1.4 seconds with a 3B model. Fifty users at one site: a workstation with a 24 GB GPU (L4, A10 or RTX 4090), 16+ CPU cores and 32–64 GB of system memory.
Barely. Search over 983 sections takes 12.7 ms exhaustively, and storage is 6.5 KB per section. A hundred thousand sections is roughly 650 MB and stays fast once you add an approximate index.
One at a time by default. Four simultaneous questions measured 8.1, 11.7, 14.4 and 17.2 seconds. Concurrency requires raising the runtime's parallelism setting and having GPU memory for each slot's KV cache.
For answering from retrieved passages, yes. The work is finding the right paragraph; the model summarises what it is given. We compared five configurations and chose a 4-bit 3B model on measured latency and citation accuracy, not on general benchmark scores.
Inplant AI runs entirely on your hardware and cites the document and page for every answer. Book a walkthrough, or read why manufacturers deploy on-premise.