Guardrails & Content Moderation

The plain-language idea

Guardrails are the checks and constraints placed around a model to keep its behavior within acceptable bounds. Content moderation is one concrete form of that: filtering for harmful, abusive, or policy-violating content. The part that's easy to under-build is that this needs to run in both directions:

• Input moderation checks what goes into the model — catching abusive, harmful, or clearly policy-violating requests before they ever reach it.
• Output moderation checks what comes back out — catching harmful content the model generates even when the prompt itself looked completely benign.

Input-only filtering misses two real cases. First, a model can produce harmful output from a prompt that wasn't obviously harmful at all — the harm shows up only in the completion, not in anything that was typed in. Second, as this track's lesson on prompt injection covers, content the model reads (a webpage, an uploaded document) can carry injected instructions that never passed through input moderation as "the user's request" at all, but still shape what the model outputs. Output moderation is the layer that catches both of those, which input moderation structurally cannot.

Guardrails aren't limited to a dedicated moderation API, either. A system prompt constraint — telling a model what it's allowed to talk about, or what source of truth to stick to — is itself a lightweight guardrail. It's just enforced by instruction rather than by a separate filtering pass.

See it in a real thing: this app's own anti-hallucination prompts

Praxis's own lib/ai/prompts/tutor.ts and lib/ai/prompts/interview.ts system prompts include a rule like this:

Answer ONLY using the provided lesson context. If the question is outside
it, say you're not certain rather than inventing facts.

That single instruction is a guardrail, doing real work with no dedicated moderation API involved: it constrains the model to a specific, grounded source of truth and explicitly tells it what to do when a question falls outside that boundary, instead of leaving it free to fabricate a plausible-sounding but ungrounded answer. It's lightweight compared to a dedicated content-moderation service, but it's the same underlying idea — a deliberate constraint on model behavior, chosen because the unconstrained default (answer anything, from anywhere) was a real risk (hallucinated tutoring content) worth closing off.

What people usually miss

Watch out

• Output moderation isn't redundant with input moderation. They catch genuinely different failure modes — a benign prompt can still produce harmful output, and injected content the model reads never goes through input moderation as "the request" at all.
• A system prompt instruction is a real guardrail, not a lesser substitute for "real" moderation. Praxis's own grounding rule in tutor.ts/interview.ts is a legitimate, load-bearing example — dismissing prompt-level constraints as not counting misses a whole category of practical, low-overhead risk reduction.
• Guardrails reduce risk; they don't guarantee safety. Both a system prompt constraint and a dedicated moderation classifier can be wrong or bypassed — the same honest framing from the prompt-injection lesson applies here: layered mitigation, not a guarantee.
• Moderation has to run somewhere the user can't skip it. A check that only happens in a client-side UI layer (e.g. a frontend-only content filter) is trivially bypassable by anyone calling the API directly — moderation that matters has to be enforced server-side.

Why this matters in interviews

"Your app only checks user input before sending it to the model — is that enough?" tests whether a candidate can name the specific gap (harmful output from a benign prompt, or injected content bypassing input checks entirely) rather than just agreeing that input filtering sounds sufficient.