A four-step AI audit process for high-stakes outputs

Executive overview

The most dangerous AI output is one that is almost right — clean writing, mostly correct, but with hidden inaccuracies. Standard prompts like "are you sure?" do not catch them.

The fix is a structured four-step audit: finish, split, check, rewrite — each step in a fresh conversation with a new AI instance to prevent bias from carrying over.

Use this only for high-stakes tasks (legal, financial, brand reputation) — 90% of AI use cases don't need it.

When to use the audit process

• Contract review where AI-extracted advice must be verified
• Due diligence on a vendor or investment target
• Vendor proposals where AI-made comparisons carry financial or legal weight
• Any output where errors have significant brand, legal, or financial consequences

Step 1: Finish the output

• Complete the AI-generated artifact (document, spreadsheet, deck) to the point you'd send it
• Only after you're satisfied, ask: is this high stakes?
• If yes, proceed to the audit — not before, not after

Step 2: Split out the claims

• Start a fresh conversation with a high-end model (e.g. Claude Opus, GPT-5)
• Prompt the AI to extract every factual claim as a separate, checkable unit
• Output format: a table with claim number, exact claim, and source it was drawn from
• Do not include tone, wording choices, or advice — only verifiable facts

Step 3: Check claims against the source

• Start another fresh conversation; paste the claims table and the original source material
• The AI labels each claim with one of four categories:
  • Supported — claim matches the source; keep it
  • Conflicts — claim contradicts the source; replace with source information
  • No proof — source doesn't support the claim; likely remove it
  • Needs human judgment — claim involves prediction, market context, or inference the source can't confirm; human decides
• Instruct the AI to use only the provided source — no internet, no model knowledge
• The output is a full audit report with label, source quote, and one-sentence rationale for each claim

Step 4: Rewrite with audit findings

• Start a fourth fresh conversation with the original output and the audit report
• Prompt rules:
  • Keep structure and style of the original
  • Supported claims: leave unchanged
  • Conflicts claims: reword to match the source
  • No proof claims: remove or soften (your choice — adjust the prompt accordingly)
  • Needs human judgment claims: flag for review; AI leaves them in place for you to decide
• Ground the AI explicitly: use only the original write-up and audit, nothing else

Advanced variation: use different models per step

• For the highest-stakes tasks, assign a different AI to each step
• Each model has distinct strengths and biases — rotating them increases the chance of catching errors
• Example: Opus for finish and rewrite, GPT-5 for split, Gemini Pro for check
• Reserve this for the top 1% of tasks where the cost of errors is extreme