The framework
Six layers between every family and Iris.
Each layer is independently sufficient to catch the highest-risk moments. In practice they overlap on purpose — that's what fail-closed means.
- 01
Input sanitisation
Before any user text reaches a model, we strip out obvious prompt-injection patterns ("ignore all previous instructions", role-swap attempts, developer-mode phrases) and cap length. Uploaded documents and calendar titles get the same treatment.
WhyThis isn't the whole defence — the safeguarding layer re-runs on model output too — but it stops the noisiest attacks from ever entering the conversation.
- 02
Deterministic rules
A fast keyword and phrase scan runs on every user turn. It flags suicide risk, self-harm, harm to a child, abuse disclosures, domestic abuse, medical emergencies, medication dosing requests, diagnosis requests, role confusion, prompt injection, and adult topics in Kid Mode.
WhyRules run first because they never fail silently. A false positive costs us a slightly more careful reply and a queue entry — a false negative costs a family. We prefer the first trade every time.
- 03
Classifier pass
For messages that don't trip a hard rule but hint at distress, a lightweight classifier runs to catch the language rules would miss — indirect suicide ideation, quiet requests for medical decisions, safeguarding disclosures wrapped in humour.
WhyThe classifier can escalate a message into the review queue even when the model reply is safe. It never downgrades a rule hit.
- 04
Fail-closed responder
When any blocking category fires, Iris does not hand off to the model. A warm, prewritten response validates the person, names the concern, and signposts UK support — Samaritans 116 123, NSPCC 0808 800 5000, Childline 0800 1111, 999 for immediate danger.
WhyThis means Iris's crisis reply cannot be jailbroken by clever phrasing: the model isn't the one writing it.
- 05
Append-only audit log
Every trigger writes an event to a service-role-only safety_events table with the category, surface (parent chat, parent voice, Kid Mode talk, Kid Mode voice, proposals, trust report), severity, a scrubbed excerpt, and the action taken.
WhyExcerpts have phone numbers, emails, and street addresses redacted before storage, so the audit log is safe to review without exposing families.
- 06
Human review + parent notify
Medium and higher severity events create a ticket in a review queue with an SLA (critical 1 hour, high 4 hours). For any child safety flag in Kid Mode, the parent is notified so a real conversation can happen off the app.
WhyIris is never the only line of defence. A person always sees the hard moments.
Where the framework applies
- Parent chat with Iris
- Parent voice memos (well-being tracker, behaviour logs)
- Kid Mode "talk to Iris" text and voice
- Kid Mode journal voice notes
- Iris proposals sent for parent approval
- Public /trust reports submitted without an account