◉RORK LABJP
●MCP — Rork now lets you add and manage your own MCP servers inside Plugins (11/21)●2.5 ROUTE — From 12/1, apps calling gemini-2.5-flash are routed to 3.5 Flash-Lite and billed at its rates (9 days left)●CHANNEL — Users report Android notification channel reads and writes failing in Expo Go●NEW — New: three clues to check before replying to a stuck-on-splash review●EXPORT — Export your projects before you downgrade or cancel a plan●SONNET 4.5 — Apps that call Sonnet 4.5 through the API should switch before it retires on 11/30●MCP — Rork now lets you add and manage your own MCP servers inside Plugins (11/21)●2.5 ROUTE — From 12/1, apps calling gemini-2.5-flash are routed to 3.5 Flash-Lite and billed at its rates (9 days left)●CHANNEL — Users report Android notification channel reads and writes failing in Expo Go●NEW — New: three clues to check before replying to a stuck-on-splash review●EXPORT — Export your projects before you downgrade or cancel a plan●SONNET 4.5 — Apps that call Sonnet 4.5 through the API should switch before it retires on 11/30
Articles/Dev Tools
⬡ Dev Tools/2026-04-30Advanced

Rork × AI Moderation Production Guide: Reporting, Auto-Review, and Tiered Sanctions for UGC Apps

A complete production-grade guide to layering AI moderation, reporting, and tiered sanctions into your Rork UGC app. Includes a working three-layer pipeline using OpenAI Moderation, Perspective API, and Cloudflare Workers Queues.

Rork620UGCContent ModerationOpenAI ModerationPerspective APICloudflare Workers26Trust and Safety

✦ Premium Article

The week after I shipped my first UGC app to the App Store with Rork, I received over thirty abuse reports. It was a small community app I was running solo, but the kinds of violations that showed up were nothing I had anticipated: text containing self-harm signals, photos of children uploaded without consent, impersonation accounts using celebrity names. There were days when I spent six hours just handling reports.

What I learned from that experience is this: if you don't design moderation in three layers from day one, you'll be crushed by manual work the moment you launch. This article walks you through how to build production-grade moderation into a Rork-built UGC app — pre-submission checks, on-submission auto-review, and post-publication report handling — using the actual code I run in production today (Cloudflare Workers + Queue + KV).

We'll cover where OpenAI Moderation API shines and where it falls short, how to choose between it and Perspective API, what Apple and Google explicitly require in their guidelines, and what the EU DSA and regional child-protection laws demand. Everything an indie developer needs at implementation time, in one place.

Why moderation must be designed in three layers

The mistake I made on first launch was trying to consolidate every decision into a single "on-submission auto-review" layer. When you put everything in one place, three problems hit at once: API costs balloon, misjudgments have wide blast radius, and there's no way to handle anything reported after publication.

These days, I design every UGC feature with three distinct layers:

LayerWhen it runsWhat it doesWhat it should not own
Pre-submission (client-side preflight)Before the user taps SubmitFilters obvious noise on the device: empty strings, exceeded length, URL-only posts, emoji spam. A good share of pointless API calls never leave the phone.Any final safety verdict. The client can be tampered with.
On-submission (server-side moderation)Right after the post reaches the APIHands the post to an AI moderation API asynchronously and splits the result into three buckets: publish, hold for review, or reject.Handling anything reported later. Post-publication context is invisible to this layer.
Post-publicationReports and appeals on published contentTakes reports, folds in human-review outcomes, and hosts the appeal channel that many jurisdictions now expect.Pre-publication screening. By this point, someone may already have seen the post.

Deciding what each layer should not own turned out to matter. On my first launch I pushed everything into the on-submission check, and when something went wrong I could not tell where to fix it.

Connect these three layers with Cloudflare Workers Queues and you get a clean balance between cost and response speed. The submission API enqueues and returns immediately, a worker runs the actual judgment in the background and writes the result to KV, and the frontend shows a "Reviewing" state until the verdict is final.

Pre-submission: client-side preflight in Rork

Rork compiles into React Native components, so we can attach a lightweight check at the TextInput and ImagePicker layer. This isn't a security boundary — it's a noise filter that cuts out 99% of obvious garbage before it ever leaves the device.

// app/components/PostComposer.tsx
import { useState } from "react";
import { TextInput, View, Pressable, Text } from "react-native";
 
const NG_PATTERNS = [
  /^https?:\/\/\S+$/,                    // URL-only posts not allowed
  /(.)\1{20,}/,                          // Same character repeated 21+ times
  /[\u{1F300}-\u{1F9FF}]{30,}/u,         // 30+ consecutive emojis
];
 
const MIN_CHARS = 2;
const MAX_CHARS = 1000;
 
type PreflightResult =
  | { ok: true }
  | { ok: false; reason: string };
 
function preflight(text: string): PreflightResult {
  const trimmed = text.trim();
  if (trimmed.length < MIN_CHARS) {
    return { ok: false, reason: "Posts must be at least 2 characters." };
  }
  if (trimmed.length > MAX_CHARS) {
    return { ok: false, reason: `Posts must be under ${MAX_CHARS} characters.` };
  }
  for (const pattern of NG_PATTERNS) {
    if (pattern.test(trimmed)) {
      return { ok: false, reason: "This post format is not allowed." };
    }
  }
  return { ok: true };
}
 
export function PostComposer({ onSubmit }: { onSubmit: (text: string) => void }) {
  const [text, setText] = useState("");
  const [error, setError] = useState<string | null>(null);
 
  const handleSubmit = () => {
    const result = preflight(text);
    if (!result.ok) {
      setError(result.reason);
      return;
    }
    setError(null);
    onSubmit(text.trim());
    setText("");
  };
 
  return (
    <View>
      <TextInput
        value={text}
        onChangeText={setText}
        multiline
        placeholder="What's on your mind?"
        maxLength={MAX_CHARS + 100}
      />
      {error && <Text style={{ color: "#d33" }}>{error}</Text>}
      <Pressable onPress={handleSubmit}>
        <Text>Post</Text>
      </Pressable>
    </View>
  );
}

A critical note: this preflight does not replace server-side validation. The client is tamperable, so the server must run the same checks independently. Preflight exists for UX (instant error feedback, fewer wasteful API calls) and to reduce server load — nothing more.

The reason these three patterns are first in the filter is that they're exactly the patterns I saw concentrated during the early days of my UGC launch. URL-only posts are spammers probing the system, character repetition is script-kiddie smoke testing, and emoji spam is the classic signature of teenage pile-on participation. Early on, a surprisingly large slice of posts matched these patterns, and adding preflight alone visibly lightened the server. That ratio is a gut feeling from the first weeks after launch and will differ a lot by app.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN
✦If your UGC app has been drowning in reports and manual deletions, you'll have a working automated moderation pipeline you can deploy today
✦You'll learn when to use OpenAI Moderation, Perspective API, or a custom model, and how to design a stack that fits your specific product
✦You'll walk away with a three-layer (pre-submission / on-submission / post-publication) moderation architecture, with concrete Rork + Cloudflare Workers code aligned with the EU DSA, child-safety regulations, and Apple/Google guidelines
Secure payment via Stripe · Cancel anytime
✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

or
Unlock all articles with Membership →
Share

Thank You for Reading

Rork Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.

  • ✦Copy-paste ready implementation code
  • ✦New advanced guides published daily
  • ✦$5/mo or $10 for lifetime access
View Membership →

Related Articles

⬡ Dev Tools2026-05-09
Building a Rork A/B Testing Platform with GrowthBook and PostHog from Scratch
Drop Firebase and pair GrowthBook with PostHog to ship an A/B testing stack for your Rork apps: SDK wiring, a KV-cached feature endpoint on Cloudflare Workers, sample-size math, SRM checks, and the operational pitfalls.
⬡ Dev Tools2026-11-19
New Wallpapers Showed Up Everywhere Except the Filters: Ship the Count and the Category Lists Together
I added 67 wallpapers in one batch. They appeared in the full list and in New, but not under a single category filter, because I had raised the ID limit and forgotten to ship the category lists. Here is a pre-bump check, a client-side way to spot silent fallbacks in an Expo app, and a post-release count against the origin.
⬡ Dev Tools2026-11-04
Catching the Few Seconds Before Streaming Audio Starts: A Loading State and a Timeout
You tap play and nothing happens for a moment. Here is how I model that gap with expo-audio: a requested flag, a 400ms delayed spinner, and an 8-second timeout.