Why run AI privately
For many organisations the question is not whether AI is useful but where the data goes. Customer records, contracts, internal reports and source code are exactly the material that makes an assistant valuable, and exactly the material many organisations cannot or will not send to a third-party service. Data residency rules, sector expectations and simple prudence all point the same way.
The good news is that open-weight language models have become capable enough for a wide range of everyday tasks: summarising, drafting, classifying, extracting and answering questions over documents. Run on your own hardware, they keep data inside your boundary by design rather than by contract.
The local stack
A practical private AI stack is smaller than people expect. A runtime such as Ollama serves open-weight models on a single server, with or without a GPU. A retrieval layer, often built on a search engine such as OpenSearch with vector support, lets the model answer from your own documents rather than from memory. An application layer wraps both with authentication, logging and a user interface.
Model choice is a trade-off between quality, speed and hardware. Smaller models run comfortably on a modest CPU server and handle classification and short drafting well. Larger models give better reasoning but need more memory or a GPU. We recommend testing two or three sizes against your real tasks before buying hardware; the right answer is usually smaller than the first instinct.
Plan the hardware honestly. Label any sizing estimate as illustrative until it has been measured on your own workload, because response times depend heavily on document length, concurrency and the model chosen.
Connecting AI to your systems with MCP
An assistant that can only chat is of limited use. The value comes when it can look up a customer, check an invoice or open a ticket. The Model Context Protocol is an open standard for exactly this: it defines how an AI application discovers and calls tools and data sources in a consistent way, so each integration is written once rather than once per model.
Because MCP is open, it fits a no-lock-in approach. The tools you expose today can be used by a different model tomorrow. But it also concentrates power, because a model with tools can act, not merely talk. That makes governance essential rather than optional.
Governance that is actually followed
Good AI governance is mostly ordinary good engineering, written down. The controls worth having from the first day are few and clear:
- The assistant acts with the permissions of the person using it, never with a shared superuser account.
- Tools that change data require confirmation or follow the same approval route as a human change.
- Every prompt, tool call and result is logged and auditable.
- Data sources are listed explicitly; nothing is indexed simply because it was reachable.
- Outputs are labelled as AI-generated where they reach customers or official records.
Start with a narrow, valuable task
The most successful private AI projects begin with one task that is frequent, well defined and easy to check: drafting replies to routine enquiries, pointing out records that need attention, or answering staff questions from an approved handbook. A narrow task makes quality measurable and keeps risk contained. Once the stack, the logging and the governance have proven themselves on one task, extending to the next is straightforward.
Above all, keep a person accountable. AI on your own servers is still AI: it can be confidently wrong. Design the workflow so a human reviews what matters, and treat the model as a capable assistant rather than a decision-maker.