Building a RAG Pipeline with Strapi 5 and Vector DBs

To build a Retrieval-Augmented Generation (RAG) pipeline using Strapi 5, you need to extract published content from your headless CMS, split it into semantic chunks, generate vector embeddings, and synchronize those vectors whenever entries change. This setup lets large language models (LLMs) and semantic search engines answer questions grounded in your editorial content without manual retraining.

Running a RAG system over headless content solves hallucination problems, but it introduces synchronization challenges. When an editor updates an article or unpublishes a guide, the vector index must reflect that state immediately. Here is how to architect, build, and maintain an automated RAG sync pipeline between Strapi 5 and a vector database.

The Architecture of a CMS-Driven RAG Pipeline

A RAG architecture powered by Strapi consists of four core stages:

1. Event Detection: Listening to publish, update, and delete actions within the CMS.
2. Transformation and Chunking: Stripping markup, parsing structured components, and dividing documents into meaningful context windows.
3. Embedding Generation: Converting text chunks into high-dimensional vectors using an embedding model.
4. Vector Storage and Metadata Indexing: Inserting or deleting vectors alongside metadata (such as document ID, locale, slug, and category) inside a vector database like pgvector, Pinecone, or Qdrant.

Strapi 5 provides a modern Document Service API that handles drafts, publication states, and internationalization out of the box, making it clean to track content lifecycles.

Choosing Between Webhooks and Lifecycle Subscriptions

There are two primary approaches to triggering vector synchronization from Strapi 5: HTTP webhooks or server-side Document Service lifecycles.

Approach 1: External Webhooks

Strapi includes a built-in webhook engine accessible under Settings → Webhooks. When an author publishes or unpublishes an entry, Strapi dispatches an HTTP POST payload to an external worker (such as an AWS Lambda function, Cloudflare Worker, or Next.js Route Handler). The external worker fetches the full document, chunks it, and updates the vector store.

Webhooks decouple heavy AI embedding calls from the CMS request-response cycle, protecting the Strapi admin panel from slowdowns if the embedding API experiences latency.

Approach 2: Server-Side Document Lifecycles

If you prefer keeping logic within your Strapi codebase or want immediate consistency, you can subscribe to Document Service lifecycles directly in Node.js. In Strapi 5, document lifecycles run during document operations (publish, update, delete, discardDraft).

Here is how to register a subscriber in src/index.js:

module.exports = {
  register() {},
  bootstrap({ strapi }) {
    strapi.documents.use(async (context, next) => {
      // Execute the underlying CMS action
      const result = await next();

      const targetUID = 'api::article.article';
      if (context.uid !== targetUID) {
        return result;
      }

      const { action, params } = context;

      // Handle publication and unpublication events
      if (action === 'publish') {
        await strapi.service('api::article.rag').indexDocument(result);
      } else if (action === 'unpublish' || action === 'delete') {
        await strapi.service('api::article.rag').removeDocument(params.documentId);
      }

      return result;
    });
  },
};

Using a custom service (api::article.rag) lets you manage retry queues or offload indexing jobs to background workers such as BullMQ.

Chunking Strapi Content Components

Vector databases perform best when text segments are compact and focused on a single topic. If you pass an entire 3,000-word article into one vector, specific facts become diluted in the embedding.

In Strapi, rich content often lives inside Dynamic Zones or structured blocks. Splitting content by component boundaries produces more coherent chunks than arbitrary character counts.

Preparing the Text Representation

Before chunking, convert structured blocks or markdown into clean plain text. Preserve hierarchy by prefixing headers to child paragraphs:

function extractArticleText(article) {
  const chunks = [];
  
  // Include title and summary as primary context
  chunks.push(`Title: ${article.title}\nSummary: ${article.description || ''}`);

  // Iterate through Dynamic Zone blocks or rich text fields
  if (Array.isArray(article.content_blocks)) {
    for (const block of article.content_blocks) {
      if (block.__component === 'blocks.text-section') {
        chunks.push(`${block.heading ? block.heading + '\n' : ''}${block.body}`);
      } else if (block.__component === 'blocks.qa-item') {
        chunks.push(`Question: ${block.question}\nAnswer: ${block.answer}`);
      }
    }
  }

  return chunks;
}

Sliding Window and Token Sizing

For lengthy text blocks, apply a sliding window chunker. Aim for 250 to 500 tokens per chunk with a 10% to 15% overlap. Overlap ensures that a sentence split between two chunks does not lose semantic context.

Generating Embeddings and Storing Vectors

Once chunks are prepared, submit them to an embedding model (for example, text-embedding-3-small or self-hosted models like bge-large-en). Store the resulting vectors along with metadata that helps you filter results during retrieval.

Here is a complete Strapi service implementation updating an external vector store:

module.exports = ({ strapi }) => ({
  async indexDocument(document) {
    const chunks = extractArticleText(document);
    
    for (let index = 0; index < chunks.length; index++) {
      const text = chunks[index];
      
      // Generate vector embedding
      const embedding = await generateEmbedding(text);
      
      // Create a deterministic vector ID
      const vectorId = `${document.documentId}_chunk_${index}`;
      
      // Upsert into vector store with metadata
      await vectorClient.upsert({
        id: vectorId,
        values: embedding,
        metadata: {
          documentId: document.documentId,
          chunkIndex: index,
          title: document.title,
          locale: document.locale,
          text: text,
          updatedAt: document.updatedAt,
        },
      });
    }
  },

  async removeDocument(documentId) {
    // Delete all chunk vectors linked to this documentId
    await vectorClient.deleteMany({
      filter: { documentId: { $eq: documentId } }
    });
  },
});

Querying the Vector Database from Next.js

When a user enters a query in your front-end application (such as a search input or chatbot), your Next.js route handler converts that query into an embedding and runs a similarity search against your vector index.

// app/api/search/route.js in Next.js
import { NextResponse } from 'next/server';

export async function POST(request) {
  const { query, locale = 'en' } = await request.json();

  // Convert query string into vector
  const queryVector = await generateEmbedding(query);

  // Search for the top 5 most similar chunks matching the locale
  const matches = await vectorClient.query({
    vector: queryVector,
    topK: 5,
    filter: { locale: { $eq: locale } },
    includeMetadata: true,
  });

  const context = matches.map((match) => match.metadata.text).join('\n\n---\n\n');

  return NextResponse.json({ results: matches, retrievedContext: context });
}

The retrieved context is then passed into your LLM prompt template:

You are a technical assistant. Answer the user's question using only the reference
context below. If the answer cannot be found in the context, state that you do
not know.

Context:
{{retrievedContext}}

Question: {{query}}

Managing Content Revisions and Draft States

One common pitfall in headless RAG pipelines is indexing draft content before it is ready for public consumption. In Strapi 5, entries exist in draft and published states under a single documentId.

Ensure your synchronization logic strictly checks publication status. If your editorial team uses a plugin like Revisionary to keep every version of your Strapi content and compare changes over time, your vector pipeline should only index the version that is actively marked as published, ignoring draft iterations and milestone snapshots.

When an editor unpublishes a document or restores a previous revision, the sync routine must purge existing chunks and re-embed the newly restored state.

Handling Internationalization and Locales

If you run a multilingual Strapi site, each locale represents a distinct document variant sharing the same logical entry or grouped under locale keys. Vector similarity can vary across languages:

1. Multilingual Embedding Models: Use models trained on cross-lingual representations (such as text-embedding-3 or multilingual-e5) if users ask questions in one language to find content in another.
2. Metadata Filtering: If your user interface strictly limits searches to the current language, store locale in your vector metadata and apply an exact filter during query time. This prevents returning mixed-language excerpts in search results.

Best Practices for RAG Maintenance

To keep your retrieval pipeline reliable in production, follow these principles:

Conclusion

Connecting Strapi 5 to a vector database turns your headless CMS into a governed knowledge source for AI-driven search and conversational agents. By utilizing Document Service hooks, chunking structured components cleanly, and maintaining strict metadata filters for publication state and locales, you ensure your LLMs always generate answers from verified, current content.