Private Local AI for Engineering Work: What I Built, and Why Public Chatbots Were Never Going to Do This Job

Try feeding 5,000 pages of EPC specs, grid codes, and plant manuals into an online chatbot. It doesn't work.

Anyone in power generation, utilities, or consultancy knows how much of the job comes down to digging through massive document sets. A single evaluation can rest on thousands of pages: standards, regulatory determinations, OEM manuals, bid packages, shift logs, test reports. Finding the one clause that decides a question is the work. The rest is just reading around it.

The shortcut too many teams are taking

Right now, too many engineers and teams take a risky shortcut: uploading confidential bid evaluations, contract clauses, or plant records into public tools like ChatGPT and Claude. Putting proprietary client data onto third-party servers is a serious compliance risk, and most engineers who do it know it. They do it anyway because the alternative — searching by hand — is slow.

But beyond privacy, that approach hits a hard wall. You cannot use an online model for a truly large dataset. You cannot feed a multi-gigabyte library of manuals, plant logs, and tariff filings into a web chat window. Chatbots have upload limits, their context windows fill up fast, and past a few documents they skip clauses or guess. In engineering, a "mostly right" answer is useless. You need the exact clause, the exact table, and the exact page number, because that is what a client, a regulator, or a tribunal will ask for.

That bottleneck pushed me to build a private, local AI setup on my own workstation. No data leaves my network. Nothing is uploaded anywhere. The whole archive is searchable offline, and every answer comes back with page citations from verified files.

What the setup is made of

Four pieces, all running locally, all open source:

Ollama runs the language models on my own machine. It is the engine: it loads a model, takes a question plus retrieved passages, and writes the answer. No account, no API key, no call to an outside server.

Qdrant is the vector database. It does not generate answers. It stores and searches. Every passage of every document is stored as a vector — a long list of numbers that captures what the passage means — together with a payload that records which document it came from and which page. When a question arrives, Qdrant returns the passages closest in meaning to the question, even when they share none of its words. That last part matters: a search for "lube oil varnish" should also find passages about "EHC fluid degradation" and "servo-valve sticking", because those are the same problem described differently. Keyword search cannot do that. Vector search can.

Open-WebUI is the chat front end. It is the window I type into: pick a model, ask a question in plain language, get an answer with citations. It also handles the document side — the knowledge base the questions are answered from.

Open-WebUI is the window you talk to; Ollama is the engine behind it. The two jobs stay separate, and both stay on your machine.

Hermes Agent is the agent layer. Where a chat model answers one question at a time, an agent carries out multi-step work: it decides when to use a tool, runs it, looks at the result, and keeps going. In this setup it is the piece that turns a question-and-answer tool into something that can work through a longer task.

The models on the box at the moment are Qwen3-14B as the main answering model, Nous Hermes 2 Mixtral 8x7B for heavier reasoning work, Llama 3.2 in 1B and 3B sizes for light, fast tasks, and nomic-embed-text for embeddings — the model that turns text into the 768-number vectors Qdrant stores. Nothing here is exotic. These are openly available models, and that is the point: the capability does not depend on any vendor's goodwill.

How a question travels through it

The flow has two halves: indexing, which happens once per document, and retrieval, which happens on every question.

Indexing goes like this. A document goes in — a standard, a manual, a determination, a bid package. Its text is extracted and split into passages. Each passage is turned into a vector by the embedding model. The vector goes into Qdrant with its payload: source document, page number, section where known. The original document stays exactly where it was. Qdrant holds the vectors and the references, not a second copy of your library to lose track of.

Retrieval goes like this. I ask a question. The question is embedded the same way, and Qdrant returns the passages nearest in meaning. Those passages — and only those passages — go to the language model with one strict instruction: answer from these passages, and cite them. If the passages do not contain the answer, the honest output is that the archive does not contain it. That discipline is the whole difference between this setup and a public chatbot. A public chatbot, asked about a document it has never seen, will still produce a fluent answer. This setup, asked about a document it has not indexed, has nothing to invent with.

The setup in one picture: index once, then every question is answered strictly from retrieved passages.

What it changed in real work

Three kinds of work changed first, because they are the three kinds that drown in paper.

Plant O&M memory. When a senior engineer leaves a plant, 15 to 20 years of plant history leaves with them. Shift logs, root cause reports, OEM bulletins, and test records hold that history, but only if someone can find the relevant entry. Indexed locally, a younger engineer can query decades of history in seconds and get the log entry or bulletin reference back, not a summary from memory. The plant keeps its memory after the person leaves.

Regulatory work. Cross-checking years of determinations, PPA schedules, and grid code amendments is exactly the kind of task where a buried precedent decides the outcome. A vector search across the full set finds the amendment that a keyword search misses because it uses different wording. The answer still needs a human to read the cited pages and judge them — but the finding part, which used to take days, takes minutes.

Bid evaluations. Auditing multi-bidder submissions against employer specifications and international standards is a comparison task: does this bidder's offered clause match that requirement, and where does it deviate? With the specs, the standards, and all bids indexed, each deviation surfaces with its page reference. Senior engineers stop hunting for paragraphs and spend their time on engineering judgment, which is the part no tool can do.

What it costs, and what it does not

The software costs nothing. Ollama, Qdrant, Open-WebUI, and the models are free and open. The hardware is standard workstation hardware — no enterprise servers, no GPU farm, no monthly per-token bill that grows with every question. That is worth stating plainly, because the usual objection to private AI is cost, and for this scale of setup the objection does not survive contact with the price list.

The real costs are different, and a post like this should name them. Indexing a multi-gigabyte library takes time and disk space. Local models answer more slowly than the biggest cloud models, so a long answer takes longer to arrive. And the maintenance is yours: updates, backups, re-indexing when documents change. There is no vendor to call. Anyone who tells you private AI is free in every sense is selling something.

The honest limits

A local setup has limits, and they should be stated before someone builds one on the strength of this post.

First, local models are not frontier models. On general reasoning and polished writing, the biggest cloud models are ahead. For this job that is the wrong metric. The job is finding the right passage and quoting it accurately, and there retrieval quality and citation discipline matter more than eloquence. But if you need frontier-level reasoning on open-ended questions, a local 14B model will not give it to you.

Second, the setup only knows what you indexed. That is a feature — it cannot invent from outside the archive — but it makes indexing discipline the whole game. A half-indexed library gives half answers with full confidence in appearance. The rule has to be: if it is not indexed, it does not exist for the tool, and the engineer must know that.

Third, scanned documents cap the quality. A badly scanned manual with poor OCR gives poor passages, and no retrieval trick fixes that. Garbage in, garbage out applies here exactly as it does everywhere else.

Fourth, citations still need a human eye before anything contractual goes out. The tool finds. The engineer decides. That line does not move, and any setup that blurs it is dangerous no matter where it runs.

How a local firm can start

The mistake would be trying to index everything on day one. Start with one corpus: one project's bid set, or one plant's manuals, or one regulator's determinations for a single year. Install the four pieces, index that corpus, and test with questions you already know the answers to. If the setup cannot find a clause you know is there, fix the indexing before you expand — the test questions are the quality control.

Then set three rules and keep them. One: nothing confidential leaves the network, ever, for any reason, including "just this once" for a quick check. Two: someone owns the library — who adds documents, who removes superseded versions, when re-indexing happens. Three: no cited answer goes to a client or a regulator without a human opening the cited page and reading it. Those rules are the difference between a tool and a liability.

From there, expansion is mechanical: more corpora, more collections, the same workflow. The setup grows by adding documents, not by redesigning anything.

Why this matters here

Proper AI adoption across Pakistan's technical industry is still way too slow. Globally, consultancies and EPC firms are deploying structured, private AI fast, cutting weeks of technical due diligence down to days. Meanwhile, most local utilities, IPPs, and industrial setups are stuck between manual Ctrl+F searches and pasting sensitive snippets into public chatbots.

The tools are free, the hardware is accessible, and the data never leaves your network. The barrier is not money. It is habit, and know-how, and the belief that AI means either a public chatbot or a multi-million-rupee enterprise platform. There is a third option, and it runs on a workstation.

The boundary that matters: documents, models and answers never leave your own network.

If our local industry wants to keep pace, we need to move past ad-hoc public chatbots and build private AI into engineering workflows — one indexed library at a time.

Download the step-by-step setup guide

I wrote a beginner's guide that walks through this exact setup on a fresh Debian system — basic tools, Docker, Ollama, OpenWebUI, your first model, and your first knowledge base — with copy-paste commands for every step and a graphics-card table with indicative Pakistan prices at the end. Download the guide (PDF)

Further Reading

A Home Network in Three VLANs: Main, IoT, Cameras — An earlier post on this site about building home infrastructure in segments: the same thinking this setup applies to AI, keep things separate and keep them under your own control.

Ollama quickstart — Running models locally, from the official documentation.

Qdrant local quickstart — Running the vector database locally and making a first collection.

Open WebUI documentation — The self-hosted front end, its RAG features, and connecting agents.