Automatically flag and block user‑generated content in Strapi 5 with OpenAI Moderation

Answer: In Strapi 5 you can automatically check every piece of user‑generated text (or image) with OpenAI’s Moderation API and reject it in the content workflow by adding a lifecycle hook or by using the AI‑SDK guardrails middleware. Below is a complete, production‑ready guide.

Overview

Strapi 5 stores content as Draft and Published documents. Before a Draft is persisted—or before a Draft is promoted to Published—you can run custom code. The OpenAI Moderation endpoint (POST /v1/moderations) returns a flagged flag and category details for the submitted text. By wiring that call into Strapi’s lifecycle hooks or the Guardrails middleware you get:

• Real‑time rejection of hateful, harassing, or illegal content.
• Detailed reason shown to editors.
• Full audit trail (use the Revisionary plugin to keep a history of moderation decisions).

Prerequisites

• Strapi 5 project running on Node 18‑24 with a supported database.
• OpenAI API key with access to the moderation model (omni-moderation-latest).
• Optional: the paid Revisionary plugin if you want to keep a versioned record of every moderation attempt.

1. Install the OpenAI plugin

Strapi provides an official community plugin that adds an openAI service.

npm install @strapi/plugin-open-ai   # or yarn add @strapi/plugin-open-ai

After installation, restart Strapi so the plugin is loaded.

2. Configure the OpenAI token

Create or edit ./config/plugins.js (or plugins.ts if you use TypeScript) and add the API token:

module.exports = ({ env }) => ({
  // other plugins …
  openAI: {
    config: {
      apiToken: env('OPENAI_API_KEY'), // set in .env
    },
  },
});

Add OPENAI_API_KEY=sk-… to your .env file. The plugin now exposes strapi.plugin('openAI').service('moderation').

3. Choose your integration point

You have two reliable ways to intercept content:

1. Lifecycle hooks – run moderation before the content is saved or published.
2. Guardrails middleware – use the AI‑SDK’s guardrails plugin to block requests to any AI endpoint, which is useful when you expose a custom /api/ai/* route. Both approaches are demonstrated below.

3.1 Using lifecycle hooks

Lifecycle hooks live in ./src/api/<content-type>/content-types/<content-type>/lifecycles.js.

// src/api/article/content-types/article/lifecycles.js
module.exports = {
  async beforeCreate(event) {
    await moderate(event);
  },
  async beforeUpdate(event) {
    await moderate(event);
  },
};

async function moderate(event) {
  const { data } = event.params;
  // Only moderate the fields that contain user text
  const text = `${data.title || ''}\n${data.body || ''}`;
  if (!text.trim()) return; // nothing to check

  const moderation = await strapi
    .plugin('openAI')
    .service('moderation')
    .create({ model: 'omni-moderation-latest', input: text });

  if (moderation.results[0].flagged) {
    const categories = Object.entries(moderation.results[0].categories)
      .filter(([, v]) => v)
      .map(([k]) => k)
      .join(', ');
    throw new Error(
      `Content rejected by OpenAI moderation. Categories: ${categories}`
    );
  }
}

What happens: When an editor creates or updates an article, Strapi calls beforeCreate/beforeUpdate. The hook sends the combined title and body to OpenAI. If the response is flagged, the hook throws an error, which Strapi surfaces in the admin UI as a validation error.

3.2 Using guardrails middleware (AI‑SDK)

First install the AI‑SDK guardrails plugin (part of @strapi/plugin-open-ai or a separate plugin depending on your setup).

npm install @strapi/plugin-ai-sdk   # hypothetical name; adjust to your project

Create a guardrails config in config/plugins.ts:

export default ({ env }) => ({
  guardrails: {
    enabled: true,
    maxInputLength: 10000,
    blockedMessage: 'Your content violates our policy and was blocked.',
    // custom hook defined below
    beforeProcess: async ({ text, route, ctx }) => {
      const response = await fetch('https://api.openai.com/v1/moderations', {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
          Authorization: `Bearer ${env('OPENAI_API_KEY')}`,
        },
        body: JSON.stringify({ model: 'omni-moderation-latest', input: text }),
      });
      const result = await response.json();
      const flagged = result.results[0].flagged;
      if (flagged) {
        const categories = Object.entries(result.results[0].categories)
          .filter(([, v]) => v)
          .map(([k]) => k)
          .join(', ');
        return { blocked: true, reason: `OpenAI moderation flagged: ${categories}` };
      }
      return { blocked: false, sanitized: text };
    },
  },
});

Now any route that uses the AI‑SDK (e.g., /api/ai-sdk/chat) will run the beforeProcess function first. If the content is flagged, the middleware returns a 403 with the blockedMessage.

4. Testing the integration

1. Unit test – mock the OpenAI response using a library like nock and assert that the hook throws on flagged content.
2. Manual test – in the Strapi admin UI, try to create an article containing obvious hate speech (e.g., "I hate all …"). The error should appear instantly.
3. End‑to‑end test – call the public API (POST /api/articles) with a JSON payload that contains flagged text. The HTTP response should be 400 Bad Request with the error message from the hook.

5. Production considerations

• Rate limits – OpenAI’s free tier allows a limited number of moderation calls per minute. Cache recent results if you expect high traffic.
• Image moderation – The moderation endpoint also accepts images (image_url). Extend the moderate function to loop over uploaded image URLs and send them in the same request (max 20 MB per image).
• Audit trail – Use the Revisionary plugin to record each moderation decision. The plugin’s audit log (/docs/reference/audit) will show who attempted to save the content and why it was blocked.
• Error handling – Do not expose raw OpenAI error messages to end users. Map them to a generic “Content not allowed” notice while logging the detailed reason for admins.
• Localization – If your site is multilingual, run moderation on each locale’s text. The Revisionary translation panel (/docs/everyday-use/translations) helps keep locales in sync.

6. Deploying to production

1. Add OPENAI_API_KEY as a secret in your hosting environment (Vercel, Netlify, Railway, etc.).
2. Ensure the plugin files are bundled in your Docker image or build step.
3. Run a smoke test after deployment to verify that moderation still blocks prohibited content.

7. When to choose hooks vs guardrails

Pick the approach that matches your architecture. Many teams start with hooks for editorial content and later add guardrails for public AI endpoints.

Conclusion

By installing the official OpenAI plugin, configuring your API key, and wiring the moderation call into either a lifecycle hook or the AI‑SDK guardrails middleware, you get an automated safety net that blocks toxic user‑generated content before it reaches your database or public API. Combine this with Revisionary’s version history and audit log to keep a tamper‑proof record of every moderation decision.

Further reading

• OpenAI Moderation API reference – https://developers.openai.com/api/reference/resources/moderations/methods/create
• Strapi lifecycle hooks guide – https://docs.strapi.io/cms/migration/v4-to-v5/breaking-changes/lifecycle-hooks-document-service
• Guardrails middleware documentation – https://github.com/PaulBratslavsky/strapi-plugin-ai-sdk/blob/HEAD/docs/guardrails.md
• Strapi plugin development basics – https://docs.strapi.io/cms/plugins-development/plugin-structure