Skip to main content
Michael P. Goodman

Project · Migration pattern Service pattern

AI migration: from public AI APIs to private local infrastructure

Many organizations adopted public AI APIs faster than their data policies could follow. This is the pattern Michael P. Goodman uses to move those workloads onto local models and a private retrieval layer, without a big-bang cutover.

Plan an AI migrationRead the local LLM guide

By Michael P. Goodman · Last reviewed

From
Public AI APIs and SaaS assistants
To
Local LLM and private RAG on owned hardware
Method
Inventory, classify, pilot, run in parallel, cut over
Principle
No client data, prompts or logs touch public AI APIs

Compliance note

Local AI removes third-party cloud API exposure and can simplify a compliance posture. It is not legal advice and not a certification.

AI Migration overview

A migration fails when it starts with the model. It succeeds when it starts with the data. The first job is to find out what is actually going to public AI services today, including the tools nobody formally approved, and what kind of data each one sees.

  1. Inventory. Find every AI service, key and browser tool in use, and the data each touches.
  2. Classify. Sort workloads by sensitivity: what must move, what may stay, what should stop.
  3. Design. Size GPU capacity, choose local models, and define the data boundary and access controls.
  4. Pilot and run in parallel. Serve the same tasks from the local stack, compare quality on your own documents, and keep the old path available.
  5. Cut over and enforce. Move users, then block the public endpoints at the network edge so the boundary is enforced and not just promised.
  6. Harden and transfer. Audit logging, backups, runbooks and training for your team.

Applications that call a hosted model through an OpenAI-style API can often be repointed to a local endpoint with limited code change, which keeps the migration small.

AI Migration screenshots

Screenshots of the comparison and cutover tooling will be added here.

Screenshots are being added

Annotated screenshots of this project will appear here. The architecture diagram below and the linked live or source pages show the current state in the meantime.

AI Migration architecture

The target state replaces external model calls with services inside your boundary, and uses egress rules to prove it.

AI migration target architectureApplications and users call an API gateway inside the organization boundary. The gateway routes to a local model server on an Ubuntu GPU node and to a private retrieval layer with a Qdrant vector database over the organization's documents. Egress rules block public AI endpoints, and audit logs record each request.ORGANIZATION BOUNDARY: egress rules block public AI endpointsApps + usersexisting workflowsAPI gatewayauth + audit logLocal modelsOllama on GPU nodePrivate RAGLlamaIndex + QdrantYour documentscontrolled storage
Target architecture after migration: every model call stays inside the organization boundary.

What gets verified before cutover

  • Quality on your data. Side-by-side results on your own documents and tasks, not public benchmarks.
  • Boundary enforcement. Packet-level confirmation that the inference host cannot reach public AI services.
  • Recovery. A documented, tested path back to the previous setup, and a tested rebuild of the new one.

AI Migration outcomes

What a completed migration should leave you with. Engagement-specific measurements are added as they are confirmed.

A defined data boundary

No prompts, documents or logs processed by a third-party model provider, enforced by the network.

Predictable operating cost

Metered per-token spend is replaced by hardware you amortize and power you pay for.

Operable by your team

Runbooks, audit logging and backups that a competent IT team can run without me.

Technologies used in AI Migration

The components used in the target architecture.

Inference and retrieval

  • Ollama
  • Open-weight local models
  • LlamaIndex
  • Qdrant
  • FastAPI

Platform and controls

  • Ubuntu GPU nodes
  • nftables egress rules
  • Role-based access control
  • Audit logging
  • Backup and recovery runbooks

Related to AI Migration

Secure local LLM deployment

The full hardening guide: GPU sizing, isolation, RBAC and logging.

Private RAG for regulated industries

Retrieval inside the boundary, with source-cited answers.

Case study

The S.P.H.E.R.E. reference build.

AI Migration FAQ

What is an AI migration?

It is the move of AI workloads from public AI APIs and hosted assistants to local models and a private retrieval layer that run on infrastructure you control.

Do I have to switch everything at once?

No. The pattern runs the local stack in parallel with the existing service, compares results on your own data, and then moves workloads in stages.

Does this make us compliant with HIPAA or GLBA?

No single architecture does. Local AI removes third-party cloud API exposure and can simplify your compliance posture, but it is not legal advice and not a certification.

Ready to move AI workloads inside your boundary?

Start with an inventory of what your organization sends to public AI services today.

Plan an AI migrationSee services