Most organisations discover they need an AI-reliability policy the way they discover they need a fire extinguisher — after something has already gone wrong. This is the skeleton for building one before that happens: not a legal document, but an operational one, with named owners, a review rhythm and thresholds that turn a vague worry into a repeatable process.
Why an informal approach fails
"Someone will notice if a model says something wrong about us" is not a policy — it's a hope. In practice, no one owns AI-narrative monitoring by default, which means the first person to notice a factual error is usually a customer, a journalist or an investor, not anyone inside the organisation. By the time that happens, the error has had time to compound: repeated across multiple queries, screenshotted, occasionally cited elsewhere as if it were established fact.
A written policy doesn't need to be heavy to close this gap. It needs three things an ad hoc approach never has: a named owner, a fixed cadence, and a clear definition of what counts as urgent versus what can wait for the next review.
There's a second cost beyond the delayed discovery, worth naming explicitly: whoever eventually finds the error — a customer, a journalist, an investor — now also forms an opinion about how the organisation handles this kind of thing. Correcting the false claim itself doesn't undo the impression that no one appeared to be watching.
Define ownership and roles
AI-reliability monitoring tends to fail when it's "everyone's job," because everyone's job is no one's job. A workable structure assigns three distinct roles, which can be one person in a small organisation or a small cross-functional group in a larger one:
- The monitor — runs or reviews the recurring queries across models, logs what changed since the last check, and flags anything that crosses an alert threshold.
- The verifier — checks flagged claims against authoritative sources (registries, filings, official statements) and decides whether something is a genuine error or an acceptable nuance.
- The approver — usually legal, communications or a senior leader, who signs off on any public correction, statement, or escalation to a model provider or counsel.
In a small organisation, one person may hold all three roles. What matters is that the roles are explicit, so nothing depends on someone happening to notice. These roles also map naturally onto functions many organisations already staff for other purposes — the monitor is often whoever already runs press or search monitoring, the verifier often overlaps with legal or investor relations, and the approver is whoever already signs off on public statements. Reusing existing capacity is usually more realistic than building a dedicated team from scratch, at least at first.
Set a review cadence
A cadence turns monitoring from a reactive scramble into a routine. A workable default:
- Monthly — the baseline check: rerun your standard set of queries across your tracked models, log results, compare against the previous cycle.
- Weekly — during sensitive periods: a fundraise, a leadership change, a product launch, active litigation, or any moment when the cost of a stale or wrong AI answer is higher than usual.
- Ad hoc — triggered immediately by an external signal: a journalist's question, a customer flagging something odd, an internal report of a strange AI answer.
The specific frequency matters less than having one written down. A cadence that exists only "whenever someone remembers" isn't a cadence. It's also worth resisting the temptation to over-engineer the cadence before you've run it even once — a simple monthly check, done consistently for a quarter, teaches you more about your actual exposure than an elaborate schedule that collapses the first time someone is on leave.
Don't know what AI says about you today?
A SetRecord audit queries supported AI models about your organisation and measures the gap with verifiable facts — within five business days.
Define alert thresholds
Not every discrepancy needs to trigger a response — treating every minor wording difference as a crisis burns out the people doing the monitoring. A tiered threshold keeps attention on what matters:
- Low — stylistic or tonal variation with no factual error. Log it, no action required.
- Medium — a factual inaccuracy with limited stakes (an outdated title, a minor figure) or appearing in only one model. Queue it for the next scheduled correction cycle.
- High — a factual error with real stakes (financial figures, legal status, safety claims, a named individual) or appearing consistently across multiple models. Trigger the correction workflow immediately, don't wait for the next cycle.
- Critical — a fabricated scandal, legal claim, or safety issue with potential for rapid spread. Escalate to the approver and, where warranted, counsel, the same day it's found.
Writing down one or two real example claims for each tier — drawn from an actual audit, if you've run one — does more to align a team than any abstract description ever will. "Medium" means something different to a lawyer than to a marketer until everyone has agreed on a shared example.
The correction workflow
Once something crosses a threshold, the workflow should be the same every time, so it doesn't have to be reinvented under pressure:
- Document — capture the exact output, prompt, model and date.
- Trace — identify the likely source the model drew on, where feasible.
- Correct — publish or update an authoritative piece of content, or request a direct correction where a channel exists.
- Verify — re-query on the next cycle (and sooner for high/critical items) to confirm the correction actually propagated.
- Close — log the resolution, with dates, so the pattern is visible over time rather than lost in a chat thread.
Assign a rough target timeframe to each step rather than leaving it open-ended — even a loose one, like "document within 24 hours, verify within one review cycle," is enough to stop a flagged item from quietly ageing out of anyone's attention. This workflow should hold regardless of how the item was found, whether by internal monitoring or because someone external mentioned it first — treating an externally-flagged error as a fire drill and an internally-flagged one as routine creates exactly the kind of two-tier response that makes a policy feel arbitrary rather than trustworthy.
A starter policy skeleton you can adapt
Here is a minimal structure to turn into your own one-page policy:
- Scope — which models are tracked, which queries are standard, which topics/individuals get named-entity monitoring.
- Roles — monitor, verifier, approver, named by title, with a backup for each.
- Cadence — monthly baseline, weekly during sensitive periods, ad hoc on external trigger.
- Thresholds — the four tiers above, with example claims for each so the distinction isn't subjective in the moment.
- Workflow — document, trace, correct, verify, close — with an owner and a target timeframe for each step.
- Escalation path — when and how counsel gets involved, and who has authority to approve a public statement.
- Review of the policy itself — revisit the skeleton every six to twelve months; what counted as "critical" a year ago may not match today's exposure.
Resist the urge to make this document comprehensive on day one. A one-page skeleton that's actually followed beats a ten-page policy that gets written once and never opened again — the goal is a living reference, not a compliance artefact.
None of this requires new tooling to start — a shared log and a calendar reminder are enough on day one, and you can layer in dedicated monitoring tooling later as needs grow. What matters is that it's written down, owned, and reviewed on a schedule, the same discipline our own methodology is built on, applied inside your organisation rather than by an outside auditor.