Most AI chat products follow a simple pattern: send a message to a language model, wait for the response, then show it to the user.
Pip works differently.
A response generated by an AI model is never automatically considered ready for a child. Every chat turn passes through a server-controlled pipeline that applies the family's rules, reviews the child's message, generates an age-appropriate candidate answer, checks that answer for obvious failures, and reviews it again before it can become visible.
If a required safety check cannot complete, the answer is not shown.
We call this failing closed.
This article explains why we built Pip this way, how the different layers work together, and where the limits of the system still are.
Why a system prompt is not enough
The easiest way to create a "kid-friendly AI" is to start with a general-purpose language model and add instructions such as:
You are talking to a child. Be safe, friendly, and age appropriate.
Those instructions are useful. Pip uses detailed instructions too.
But we do not think they are a sufficient safety boundary.
Language models are probabilistic systems. They can misunderstand a request, follow the wrong instruction, produce an answer in the wrong language, repeat themselves, stop halfway through an answer, or generate content that the surrounding application should not accept.
Prompt injection is also a well-known issue in LLM applications. OWASP specifically identifies prompt injection and improper output handling among the major risks developers need to consider when building systems around language models.
For a child-facing product, we wanted the important rules to exist outside the model as well.
That led to a simple architectural principle:
The model can propose an answer. The application decides whether that answer is allowed to reach the child.
That is also why general-purpose products such as ChatGPT belong to a different category than an app built for children from the start. For ChatGPT's own age rules and parental controls, see Is ChatGPT Safe for Kids?.
The Pip chat pipeline
At a high level, one Pip chat turn loads the child's current policy, applies parental and account rules, reviews the new message, generates an age-oriented candidate, checks that candidate, reviews it again, and only then may save and display it.
There are several failure paths along the way. That is intentional.
A failed safety check should not quietly turn into "show the answer anyway."
One chat turn
Every turn, fail closed
Pick a situation and step through the turn. A generated reply is only a candidate until every required stage has succeeded.
Pick a situation
Every required stage succeeds, so the candidate becomes an answer the child can see.
Child sends
Why do leaves change color in autumn?
Not started
App stages are enforced in code. AI stages are a model decision.
1. Parent settings are application rules
Pip lets parents configure things such as:
- Whether chat is currently available
- Quiet hours
- Daily message limits
- Which helpers a child can use
- Whether general chat is available
- The child's preferred communication style
- Additional topics the family wants Pip to avoid
Those settings are evaluated by the application before an answer is generated.
This distinction is important.
If a parent sets quiet hours, Pip does not tell the model, "Please remember that this child should not chat after 9 PM."
The model is not asked at all.
The application checks the current policy and stops the turn.
The same principle applies to daily limits and access to helpers. These are product rules, so they belong in product logic.
2. Some safety boundaries cannot be disabled
Families have different preferences.
One family might be comfortable with age-appropriate discussion of current events. Another may prefer to keep politics and news outside the app.
Pip therefore supports additional parent-controlled topic restrictions.
Some boundaries are different. They are part of Pip's base safety policy and are not optional parental preferences.
These include areas such as:
- Harmful self-harm instructions
- Explicit sexual content and grooming
- Dangerous or illegal instructions
- Dangerous challenges
- Serious privacy and stranger risks
- Scams and phishing
Parent controls can make Pip more restrictive in other areas, but they cannot turn those core protections off.
This creates an effective policy for each child that combines Pip's baseline safeguards with the family's choices.
The first AI decision happens before the answer
If the policy allows the child to chat, the new message goes through an input safety check.
This is a separate task from answering the question.
The evaluator is asked to make a structured decision about what should happen next. Broadly, it can decide that the message should:
- Continue normally
- Be refused
- Receive a supportive response
That third path matters.
Sensitive conversations are not always harmful requests.
A child asking for dangerous instructions should not receive those instructions. A child expressing that they are scared, overwhelmed, or in distress may need a calm response that encourages them to talk to a trusted adult.
Treating every sensitive message as the same kind of violation would produce a worse product.
The input evaluator therefore has a narrow job: decide how the application should handle the request before answer generation begins.
If the message is refused at this stage, the normal answer model is never called.
The answer is generated for that child, not for an abstract user
Once a turn is allowed to continue, Pip builds the context for the answer.
The model receives information that is relevant to how it should respond, including:
- The child's approximate age
- The selected Pip helper
- The preferred tone
- The expected language
- The family's effective topic restrictions
- Whether the situation calls for a supportive response
- A limited amount of relevant conversation history
The result should feel different depending on who is asking.
An explanation for a six-year-old should not sound like an explanation for a sixteen-year-old.
Homework Helper should guide a child through a problem rather than simply doing all of the work. How families can use that helper without replacing the child's own work is covered in AI for Homework: How Kids Can Learn Without Cheating.
Story Time can be imaginative.
Calm Corner should be reassuring without pretending to be a therapist.
These differences are part of the generation instructions, but the generated text is still only a candidate answer.
The child has not received it yet.
Code checks the obvious failures first
Not every bad AI response requires another AI model to identify it.
Before Pip spends another model call reviewing a candidate, the application can catch several obvious problems directly.
For example, Pip can reject candidates that appear to be:
- Empty
- Unusually incomplete
- Repetitive
- Cut off during generation
- Missing a usable child-facing answer
- Stopped abnormally by the model provider
The exact heuristics are implementation details and change as we improve the system.
The architectural idea is more important: deterministic problems should be handled deterministically.
If the application can tell that an answer is broken, there is no reason to ask another AI whether it looks broken.
A failed candidate may be regenerated within a limited retry budget. That retry starts another candidate generation. It does not bypass the remaining checks.
Every candidate gets an output review
A candidate that passes the deterministic checks then goes through another structured review.
This stage looks at both safety and practical answer quality.
Among other things, it can evaluate whether the answer is:
- Safe for the child
- Relevant to the child's question
- Complete enough to be useful
- Written in the expected language
- Readable rather than nonsensical
- Free from problematic repetition
- Compatible with the child's active topic restrictions
Safety and quality are treated differently.
If the reviewer considers the candidate unsafe, that answer is withheld.
Pip does not send the unsafe answer back to the generator with a request to "make this safer" and then assume the rewrite is acceptable. The new candidate would still need to pass the pipeline again.
If the problem is quality rather than safety, such as an incomplete or wrong-language response, Pip can generate another candidate.
This distinction keeps the handling of an unsafe answer stricter than the handling of a merely bad one.
Generation and review are separate jobs
It is tempting to describe this architecture simply as "two-model safety."
That wording is not quite accurate.
The important separation is between generation and review.
The generation stage is optimized for producing a helpful answer for the child.
The review stage is given a different task. It receives the child's request and the candidate answer and must return a structured decision rather than helpful prose.
Those roles can be configured independently and may use different models. They can also use the same underlying model through separate calls.
Using separate calls does not create perfect independence. Models can share blind spots.
What it does create is a second decision boundary between generated text and child-visible text.
That boundary is combined with application-level policy checks and deterministic validation rather than being treated as the entire safety system.
What happens when something goes wrong?
AI systems depend on services that can fail.
A request can time out. A provider can return an error. A structured response can be malformed. A generation can stop unexpectedly.
Those cases need an explicit product behavior.
For Pip, the rule is simple:
If a required review has not succeeded, the candidate answer is not shown.
The child may see a neutral temporary message asking them to try again.
That is intentionally different from a safety refusal.
A service outage does not mean the child's question was unsafe, and the product should not pretend that it was.
At the same time, Pip will not use an unchecked candidate simply because a validator happens to be unavailable.
This is what failing closed means in practice.
Retries cannot skip the pipeline
Retries are surprisingly important in a chat application.
Mobile connections fail. Requests get repeated. A child can tap twice. A server may finish processing a turn even if the app never receives the response.
Pip tracks chat turns in a way that lets repeated requests be handled consistently instead of blindly generating another answer.
More importantly for safety, retry behavior belongs to the pipeline itself.
A retry can produce a new candidate.
It cannot turn a previously rejected candidate into an accepted answer.
It cannot skip the final review.
And if the available retries are exhausted, Pip returns a temporary failure instead of falling back to an answer that did not pass.
Reliability engineering and safety engineering overlap more often than they first appear.
What parents see
Pip separates ordinary product restrictions from moderation events.
If a child cannot chat because it is quiet time or because a daily limit has been reached, that is not treated as a safety incident.
The child receives an appropriate product message and the conversation does not continue.
A moderation denial is different.
When Pip blocks a message or withholds an unsafe generated answer, that event can appear in the parent's Issues view.
The goal is to answer two different questions:
For the child:
"What can I safely see right now?"
For the parent:
"Did Pip encounter something that I may want to know about?"
Content that has been deliberately withheld from the child can remain parent-only.
This is part of Pip's wider design philosophy. Parental supervision should be an explicit feature of the product rather than an invisible promise hidden inside an AI prompt.
Parents who want a practical way to evaluate those properties in any AI app, not only Pip, can use Safe AI for Kids: A Parent's Checklist.
Policy versions make decisions easier to understand
Parental controls change over time.
A parent may enable another helper, change quiet hours, or add a topic restriction next week.
That creates an important question when looking back at a conversation:
Which policy was active when this answer was created?
Pip associates accepted answers with the policy version that governed the turn.
That does not make a response correct or safe by itself. It makes the system easier to reason about and audit.
For us, this kind of detail is important because parental controls should be more than UI state. They should be part of the execution history of the product.
We intentionally do not publish every implementation detail
Transparency does not require publishing a blueprint for probing the system.
In this article we describe the architecture, safety boundaries, failure behavior, and important tradeoffs.
We do not publish the tuning values underneath it: the thresholds, limits, budgets, prompts, routing rules, and infrastructure settings that make the individual checks work.
Those values change as the system evolves. Publishing them would add little for parents while making the system easier to probe.
What we think users should be able to understand is the behavior those mechanisms create.
A parent's quiet-hours rule is enforced before generation.
A generated answer is not immediately trusted.
A required review cannot silently be skipped.
A withheld answer stays withheld from the child.
Those are product properties rather than tuning parameters.
The tradeoffs
Building the pipeline this way is not free.
More checks mean more latency
A typical successful turn involves multiple stages instead of a single request to a language model.
That adds latency.
We use short, specialized validation tasks and deterministic checks where possible to keep that overhead reasonable, but the additional work is real.
For a child-facing product, we think the tradeoff is worthwhile.
More checks mean more cost
The same is true financially.
Reviewing inputs and outputs costs more than showing the first model response.
We designed Pip around the assumption that review is part of serving an answer rather than an optional feature added later.
Parent visibility requires storing conversations
Pip can show parents their children's conversation history because that history is stored by the service.
That creates a privacy tradeoff.
Our current Privacy Policy explains how Pip handles this data. We do not use children's conversations to train AI models, we do not build advertising profiles for children, and Pip is not funded through behavioral advertising.
Parents should still understand that conversation history and moderation features require server-side data storage.
Moderation can still be wrong
No automated safety system is perfect.
Children use slang, misspellings, jokes, short follow-up messages, invented words, and context that can be difficult for a classifier to understand.
A system can block something harmless.
It can also fail to identify something concerning.
Layered checks reduce reliance on one model decision, but they do not eliminate the underlying limitations of generative AI.
That is why we describe Pip as a safer, parent-managed way for children to explore AI rather than as a system that can guarantee perfect safety.
Safety review is not fact-checking
This distinction deserves its own section.
An answer can be safe and still be wrong.
The output review is designed to catch safety and practical quality problems. It is not a universal fact-checking engine.
Pip can still:
- Get a date wrong
- Misunderstand a homework question
- Explain a scientific concept incorrectly
- Confidently state something that is not true
Children should learn that AI answers can require verification.
Parents should also treat medical, mental health, legal, financial, and other high-stakes topics as areas where an AI chat should not replace a qualified person.
Safety and factual accuracy are related problems, but they are not the same problem.
What this architecture does not prove
Publishing an architecture diagram is not the same as proving that a system is perfectly safe.
This article does not claim that:
- Every unsafe request will always be detected
- Every harmless request will always be allowed
- Every accepted answer is factually correct
- Automated review replaces parental supervision
- The Issues view is an emergency monitoring system
- Separate review calls create independent human judgment
- Pip can guarantee risk-free access to generative AI
Frameworks such as NIST's Generative AI Profile treat AI risk management as an ongoing process involving governance, measurement, evaluation, monitoring, and improvement.
We agree with that framing.
The pipeline described here is the architecture we use to reduce risk. Its effectiveness still depends on the models, policies, testing, monitoring, and improvements around it.
Why we are publishing this
Parents are increasingly being asked to make choices between products that all use similar words:
"safe"
"age appropriate"
"family friendly"
"protected"
Those words are difficult to evaluate from the outside.
We would rather explain what happens when a child presses send.
The parental policy is loaded.
The message is reviewed.
The model generates a candidate.
The candidate is checked.
The candidate is reviewed again.
Only then can it become an answer.
If a required part of that process fails, the child does not receive the unchecked candidate.
That is the standard we wanted for Pip.
It is more work than putting a child-friendly prompt in front of a general-purpose chatbot.
We think it should be.
Try Pip
Pip is an AI chat app designed around families rather than individual adult accounts.
Parents set up each child's profile and decide when and how they can use AI. Children get age-oriented chat, Homework Helper, Story Time, Curious Mind, and Calm Corner, while parents keep control over topics, quiet hours, daily limits, conversation history, and safety events.
Frequently asked questions
Does Pip use a separate AI for safety?
Pip separates answer generation from safety evaluation.
The stages can be configured with different models, although they do not have to use different underlying models. What matters architecturally is that generated text does not become child-visible simply because the generation call succeeded.
It still has to pass the surrounding application checks and final review.
Does Pip moderate both the question and the answer?
Yes.
The child's new message is evaluated before normal answer generation.
The generated candidate is then checked again before it can be saved and displayed.
What happens if the safety review fails technically?
The candidate is withheld.
Pip returns a temporary failure rather than interpreting a missing review as approval.
Can parents see content that Pip blocked?
Safety events can be shown to parents through the Issues experience.
Content that is deliberately withheld from a child can be stored with parent-only visibility so the parent can understand what happened without making that content child-visible.
Do quiet hours and daily limits use AI?
No.
Rules such as chat availability, quiet hours, usage limits, and helper access are application-level controls.
They do not depend on the language model deciding whether it wants to follow them.
Does Pip guarantee that every answer is correct?
No.
The pipeline is designed to reduce safety and quality problems. It does not independently verify every factual claim generated by the model.
Children should still be taught to verify important information.
Is Pip completely safe?
No generative AI system can honestly promise that.
Pip combines parent-controlled policies, input review, age-oriented generation, deterministic checks, output review, parent visibility, and fail-closed behavior to reduce risk and keep adults involved.
That is a stronger safety model than simply giving a child access to a general-purpose AI account, but it does not replace supervision or professional help when that is needed.