Sovereign RAG: hosting AI in-house (self-hosted), when and how

In short

• A "sovereign" RAG keeps your documents within your perimeter: no data is sent to a cloud outside the EU.
• It is technically within reach: open-source embeddings (BGE, E5), a self-hosted vector database (Qdrant, Chroma, pgvector), an open or EU-hosted LLM.
• The honest trade-off: more control, but a bit more operations (GPU, updates, monitoring). You reserve it for the data that justifies it.
• In practice, the best choice is often a mix: the sensitive in-house, the rest on EU managed services.

The problem, concretely

Healthcare, legal, HR, regulated sector, strategic data: for some clients, "the documents go to a US cloud" is a categorical no, sometimes written into a contract or a regulatory obligation. The question then is no longer "is it better?" but "how do we run a serious RAG without anything leaving?".

Why it has become realistic

Two years ago, doing RAG without a US API meant mediocre results. That is no longer true: open-source embedding models (BGE, E5 families) rival the proprietary APIs, and capable open LLMs can run on GPUs rented from a European host. Sovereignty no longer costs you quality.

The sovereign architecture

• Embeddings open-source, run on your servers or an EU host.
• Vector database self-hosted: Qdrant, Chroma, or pgvector inside your PostgreSQL (see choosing the database).
• LLM open and self-hosted, or via a European host with contractual guarantees.
• The whole thing in your VPC or with a sovereign host: no data leaves the perimeter.

The trade-off to own up to (the nuance)

Full self-hosting means more control, but also: GPU to provision, models to update, monitoring to ensure. For many non-sensitive uses, a managed API with EU residency and zero retention remains simpler and just as compliant (see RAG & GDPR). The right reflex is not "everything sovereign" on principle, but "sovereign where the data demands it".

In practice

I design the architecture according to your real level of requirement: often a mix, where the sensitive corpus stays in-house and the rest relies on optimised services. We start by mapping the sensitivity of the data, then choose, without dogma.

Sources

• CNIL · AI & data protection
• Qdrant · open-source vector database (documentation)
• MTEB · embedding models leaderboard (open-source included)