RAG & GDPR: doing AI on your documents without sending your data outside

In short

• A RAG passes your documents through 3 building blocks (embeddings, vector database, LLM): compliance comes down to where each one runs.
• With US APIs by default, your extracts may be retained or transferred outside the EU. For sensitive data, the CNIL points towards on-premise or a European host.
• A compliant RAG is possible without sacrificing quality: open-source embeddings in the EU, a self-hostable database, a European LLM, and above all replication of access rights.
• You calibrate according to real sensitivity, not dogma: not everything needs to be sovereign.

The problem, concretely

"We want a chatbot on our documents, but our data must not go to the United States." That is the first sentence of half my conversations. And it is legitimate: a poorly architected RAG sends, without you realising it, every passage of your contracts or HR files to third-party servers.

Where your data really goes

• The embeddings: to index, each document extract is turned into a vector. With an API (OpenAI, Google), that extract passes through their servers. Depending on the contract, it may be retained, or even transferred outside the EU.
• The vector database: Pinecone or Weaviate Cloud are hosted in the United States by default (EU regions optional); Qdrant and Chroma can run on your infrastructure.
• The LLM: with each question, the relevant passages are sent to the model to write the answer, another potential point of exit.

What the framework says (GDPR, CNIL)

When the database contains sensitive personal data or large volumes, the CNIL recommends favouring on-premise or European hosting: fewer transfers to third parties, less risk of exfiltration. Added to this is the AI Act, which reinforces transparency and data governance obligations.

How to build a compliant RAG (without making it mediocre)

• Open-source embeddings (BGE, E5 families) run on European servers: quality on par with the APIs, without the data leaving.
• A self-hostable vector database (Qdrant, Chroma) or pgvector in your European PostgreSQL.
• The LLM via a European host (or an open model) when sensitivity requires it.
• Replicate access rights: if a document is restricted in your IT system, the assistant must be too. The RAG must never become a way to bypass permissions (see multi-tenant RAG).

The nuance that makes the difference

Not everything needs to be sovereign. For a public product FAQ, a managed API with EU residency and zero retention ("enterprise" offers) is often perfectly acceptable, and simpler. Full sovereignty (see self-hosted RAG) is reserved for truly sensitive data. The mistake would be to over-secure everything and pay for needless complexity, or the reverse.

In practice

I calibrate the confidentiality level during the audit: I map which data touches which building blocks, and I propose the simplest architecture that meets your constraints. Often a mix: sensitive in-house, the rest on EU-optimised services.

Sources

• CNIL · Artificial intelligence & personal data
• DGE · Guide to retrieval-augmented generation (RAG)
• Supabase AI · pgvector & embeddings