Case study · Reference build

Project S.P.H.E.R.E.: A Sovereign Local AI Reference Build

How a private language model, a vector database, and a retrieval pipeline were assembled so that prompts, documents, and embeddings never touch a public AI API. This is the architecture I adapt for law firms, healthcare organizations, and financial institutions.

By Michael P. Goodman · Published October 18, 2026 · 8 min read

Project type: Internal reference build, not a client engagement
Data boundary: No public LLM API in the data path
Core stack: Ollama, LlamaIndex, Qdrant, FastAPI, React
Platform: Ubuntu GPU nodes

The business problem

Every regulated organization asking about AI arrives at the same wall. The useful tools are public cloud APIs, and the data that would make those tools valuable is the data that cannot leave the building. Privileged client files, protected health records, and nonpublic financial information are exactly what a knowledge assistant needs to read.

The design brief for this build was therefore narrow and strict. Give a team a capable question-answering assistant over its own document collection. Cite the source passage for every answer. Keep every prompt, document, embedding, and log inside infrastructure the operator controls. Make the whole system reproducible, so it can be rebuilt on a client's own hardware.

Three constraints shaped every decision that followed:

Data sovereignty. No third-party model provider may process the data.
Verifiability. Answers must point to the passages they came from, so a human can check them.
Operability. A normal IT team must be able to run, patch, back up, and recover it.

Architecture and stack

The system is a standard private retrieval-augmented generation pipeline. A React interface sends a question to a FastAPI service. LlamaIndex converts the question into a search against a Qdrant vector database, retrieves the most relevant passages, and assembles a prompt. Ollama serves a local model on an Ubuntu GPU node, which writes an answer grounded in those passages. The interface displays the answer next to its sources.

Request flow and data boundary for the Project S.P.H.E.R.E. AI infrastructure.

How a question moves through the system

The user asks a question in the React interface.
FastAPI authenticates the request, applies rate limits, and records an audit entry.
LlamaIndex embeds the question and queries Qdrant for the closest passages, filtered by what that user is allowed to see.
LlamaIndex builds a prompt from the question and the retrieved passages and sends it to Ollama.
Ollama generates the answer on the local GPU using a model whose weights sit on the node.
The interface shows the answer with links to its source passages.

Stack decisions and why they were made

Technology stack for the Project S.P.H.E.R.E. AI infrastructure and the reason for each choice
Layer	Technology	Role in the design
Compute	Ubuntu GPU nodes. Vast.ai rented GPU capacity was used for the build, with the architecture designed to move to client-owned or dedicated hardware.	Rented capacity is a different trust boundary from owned hardware. Regulated deployments run on infrastructure the client controls.
Model serving	Ollama	Runs open-weight language models locally. No request is forwarded to an external model provider.
Orchestration	LlamaIndex	Handles chunking, embedding, retrieval, and prompt assembly so the pipeline stays testable and swappable.
Vector database	Qdrant	Stores embeddings and metadata and supports filtered search, which is how access boundaries are enforced at retrieval time.
API layer	FastAPI	Exposes a typed, documented interface, adds authentication, rate limits, and writes the audit log.
User interface	React	Delivers the chat experience and displays the source passages behind every answer.

Sovereignty and security controls

Sovereignty is a property you design for and then verify. These are the controls I specify for this architecture, and every client deployment is built and tested against the same list:

No external model calls. Egress rules block the inference host from reaching public AI services, so the guarantee is enforced by the network and not by good intentions.
Retrieval-time access control. Passage metadata carries permissions, and Qdrant filters apply them before any text reaches the model.
Audit logging. Each query, the passages retrieved, and the response are logged for later review.
Isolated services. Model serving, the vector store, and the API run as separate services with least-privilege credentials.
Backup and recovery. Index data and configuration are backed up and the rebuild procedure is documented and tested.

A note on trust boundaries. Renting GPU capacity from a marketplace places your workload on hardware someone else operates. That is acceptable for a public research corpus and unacceptable for privileged or regulated records. For regulated clients the same stack is deployed on hardware the client owns or leases exclusively, inside the client's network.

Return on investment

I do not publish invented savings figures for a reference build, and I would not trust a vendor who did. The return from a sovereign AI deployment comes from four levers, and each one can be measured against your own numbers:

Predictable cost. Public APIs charge per token or per seat, so cost rises with usage. A local deployment replaces that metering with hardware you amortize and power you pay for, which makes the bill flat and forecastable.
Avoided exposure. When no third party processes the data, a whole class of vendor risk reviews, data processing addenda, and breach scenarios disappears from your analysis.
Reclaimed professional time. Cited answers over your own documents replace hours of manual searching. The value is the loaded hourly cost of the people who stop doing that searching.
Reusable architecture. A documented, scripted build can be repeated for a second department or office at a fraction of the first deployment's design effort.

The calculation I use in an assessment is simple enough to check on a napkin:

Net annual return =
    (annual API and license spend avoided)
  + (hours reclaimed per year x loaded hourly rate)
  - (hardware amortization + hosting + support)

In an assessment I replace each term with measurements from your organization: your query volume, your document counts, your staff costs, and a hardware quote. You see the model and the assumptions, and you decide whether the project pays back.

What this build proves for regulated organizations

A useful, cited AI assistant can run with no public AI API anywhere in the data path.
The components are mature, open, and replaceable, so you avoid dependence on a single vendor.
The system can be operated by a competent IT team with ordinary infrastructure skills.

The same engineer who designs the AI layer also understands the servers, identity, and networks underneath it. That is the advantage of 28 years in infrastructure, and it is the same discipline behind my IT infrastructure automation work.

Scope and context

Project S.P.H.E.R.E. is also a public, proposal-stage research initiative with its own website at spheredesci.org, and an interactive educational simulator, the SPHERE Virtual Lab. This page covers only the AI infrastructure behind the project's tooling and the reference architecture it demonstrates. It describes a reference build of my own and not a delivered client engagement, and it makes no claim about regulatory certification.