Ask ten organisations to "audit what AI says about us" and you'll get ten different exercises — a few prompts typed into ChatGPT over coffee, a vendor demo, a scramble after one bad answer surfaced in a client meeting. None of it is repeatable, which means none of it can be compared against what you find next quarter. The checklist below is the protocol we run ourselves: twelve checks, in three passes, that turn a vague worry into a defensible baseline.
Before you start
Scope the audit before you run a single query. List every name your organisation answers to — the current legal name, a former name after a rebrand, an acquired subsidiary, a common misspelling — because models will happily confuse you with any of them. Decide which executives, products or certifications are material enough to check on their own.
Then set expectations: this first pass is a baseline, not a verdict. Its purpose is to give you a number and a set of transcripts to compare against next time, not to produce a final judgment on how AI treats your organisation. Save every raw answer as you go — a finding nobody can reproduce is a finding nobody will act on.
Block out more time than you expect. Twelve checks across five models and twenty prompts is roughly a hundred individual answers to read, and reading them carefully — rather than skimming for anything alarming — is what makes the difference between a real baseline and a false sense of comfort.
Phase 1 — coverage (checks 1–4)
1. List your entity's name variants
Write down every legal name, trading name, former name and closely related entity a model might conflate with you. This list becomes the backbone of every query you run afterward.
2. Draft fifteen to twenty-five natural prompts
Go beyond "who is [company]". Include the questions a buyer, journalist or investor would actually type: is X reliable for Y, has X had any issues with Z, how does X compare to its closest competitor.
3. Query at least five distinct assistants
One model tells you about one model's blind spots. Run the identical prompt set against at least five — the major consumer assistants plus any industry-specific tool your buyers are likely to use.
4. Ask indirect questions too
The most damaging errors rarely surface in a direct "who is X" answer. They surface when a model is asked something adjacent — a recommendation, a comparison, a due-diligence question — where it has to infer rather than recite.
Phase 2 — accuracy (checks 5–8)
5. Decompose each answer into atomic claims
Do not grade an answer as a whole. Split it into individual factual assertions — a figure, a date, a title, a location, a partnership — so that one wrong sentence in an otherwise accurate answer doesn't get lost or, worse, sink the whole thing unfairly.
6. Check each claim against a dated primary source
An annual report, an official registry, a dated press release — not a competitor's blog, not a forum thread, not another AI's answer. If you cannot point to the source that would settle a dispute, the claim is not yet checked.
7. Note what's unverifiable, not just what's wrong
A claim with no available source to confirm or deny it is a distinct category from a claim that is flatly contradicted by one. Both matter, but they call for different responses — one needs more public documentation from you, the other needs a correction.
8. Flag entity confusion separately
A model blending you with a namesake, a divested subsidiary or a competitor is not the same failure as inventing a number. Track it separately, because the fix — disambiguating your identity in public content — is different from correcting a fact.
Don't know what AI says about you today?
A SetRecord audit queries supported AI models about your organisation and measures the gap with verifiable facts — within five business days.
Phase 3 — signals and risk (checks 9–12)
9. Record the implicit tone of each answer
Independent of accuracy, note whether the framing leans neutral, favourable or unfavourable. A technically accurate answer can still be built almost entirely from your least flattering coverage.
10. Check which sources the model leans on
When a model cites or appears to draw on sources, note whether they are your own official content or unverified third parties. Heavy reliance on the latter is itself a risk signal, even where no individual claim is wrong yet.
11. Re-run the same prompts to test volatility
Ask the identical question twice, a day or two apart, and compare. Answers that swing significantly with nothing in the world having changed point to an unstable underlying source, which is worth knowing before you build a correction around it.
12. Assess the severity and reach of each error
Not every error deserves the same response. Weigh who is likely to see it, in what context, and what decision it might influence — a wrong founding date is not in the same category as a fabricated legal dispute.
Scoring the results
Turn the twelve checks into a single baseline score rather than a pile of raw notes. A simple approach: score each check 0 to 1, weight checks 5 through 8 and 12 more heavily since they touch accuracy and severity directly, and sum to a baseline out of 100. The absolute number matters less than having one you can compare against three months from now.
Keep the underlying evidence attached to the score. A number with no transcript behind it is not auditable, and an audit that cannot be re-run by someone else is closer to an opinion than a measurement.
Common mistakes to avoid
- Testing only one model. A clean answer from your favourite assistant tells you nothing about the other four your buyers are actually using.
- Sticking to literal "who is X" prompts. The errors that hurt hide in adjacent questions, not the obvious one.
- Treating the audit as a one-off. A baseline with nothing to compare against is a curiosity, not a management tool.
- Not saving raw transcripts. Screenshots and quotes fade from memory; verbatim, timestamped records don't.
- Ignoring severity. A dozen trivial slips and one fabricated controversy should never score the same.
Turning findings into a programme
A checklist run once tells you where you stand today. Run on a cadence — monthly at minimum, weekly through a sensitive period — it tells you whether you are drifting, improving, or holding steady, which is the information that actually justifies a budget line. Automating the repeat runs is what makes that cadence sustainable once the novelty of the first audit wears off.
If the first pass through these twelve checks turns up more than you expected, that is the normal outcome, not a bad one — it is exactly what the checklist is designed to surface early. Have us run it against your organisation and compare notes against your own findings.