W1D2Sa
What 10 questions would force a guarded source to say something they have never said in public?
Vetted Question Bank
â–¶ Enter ProjectContext
In August 2024, the EU's AI Act entered into force—the world's first comprehensive AI regulation. Regulators must now implement it. A tech company, research institute, or policy shop used AI to draft a briefing on 'How quickly companies are adopting the Act and what enforcement looks like.' The briefing cites statistics, dates, compliance rates, and expert opinions. You're prepping a decision-maker (CEO, regulator, legislator) for a hostile board, journalist interview, or regulatory audit. You have one hour to build an interrogation battery that forces the AI to defend every claim with precision—or admit what it doesn't know and what it made up.
Mission
Ship a tiered question battery (opening questions → follow-ups keyed to likely dodges, contradictions, and unverified claims) that a real interrogator could use to pressure-test the AI source and catch what it got wrong, misrepresented, or invented.
Finish Line
A prioritised, sourced Vetted Question Bank of 8-12 interview questions, ready to be handed to a live interviewer.
Deliverables
Vetted Question Bank
lessonA sequenced, fact-check-proof set of 8-12 interview questions that would crack open one real source for one real story.
Team Roles
Lead Interrogator
You own the floor. Ask the opening volley, set the tone, and pin the interviewee to specific claims instead of vague hedges.
- Author 3–4 opening questions that force the interviewee to commit to a specific claim (numbers, dates, names) rather than evade into 'some people say' or 'many experts.' Example method: take a vague statement from the AI's brief ('concerns have been raised') and ask 'Which concerns, raised by whom, in what publication, and in what year?'
- For each of the three source statements in the AI's briefing, author one follow-up that targets an expected dodge. Categorize the dodge type: scope-shift (jumping from 'X is true in country A' to 'X is true everywhere'), rhetorical hedge ('experts agree' without naming them), or special pleading (claiming an exception without evidence). Write a question that pins the interviewee to the original claim or forces them to retract it.
- Direct the room: when another role names a contradiction (Fact-Checker: 'Earlier you said A, now you said B'), you ask the clarifying question that forces a commit: 'Which is accurate—A or B?—or are you saying the situation changed between those two statements?' Write this question in advance so you can ask it naturally when the contradiction surfaces.
Dossier Researcher
You prep the brief. Build the fact-check grid and arm the room with what's actually true so they catch the lies.
- Before the session, research the real-world topic and build a 1-page fact grid with three columns: AI claim | source (publication, author, year, URL) | verdict (True, False, Incomplete—true but critical context missing). Identify one hallucination—a claim the AI made up entirely, with no basis in any credible source. Example: if the AI cites a statistic, verify the source document; if it attributes a quote to a named person, check the publication date and whether that person actually said it. Mark what's fabricated, misquoted, or out-of-date.
- During the session, after the Interrogator asks a claim-committing question, you feed a prompt to guide the next follow-up: 'The AI said [claim], but the [source, date] shows [actual fact]. Ask her to explain the difference.' This is your check: you physically pull up the source on screen or read the citation aloud so the room sees you verified it, not guessed.
- Name one hallucination for the team: state the claim the AI made, the date/person/statistic it cited, and the actual source (author, publication, year, URL) that proves it wrong or shows the claim doesn't exist. If you can't find a source that directly contradicts it, reframe it: 'The AI claimed [X]. I searched [three specific sources] and found no mention of this. Here's what those sources actually say instead.' This forces precision: you are not saying 'that sounds wrong'—you are showing the gap between claim and reality.
Devil's Advocate
Defend the indefensible. Make the interviewee justify every answer, even when it sounds reasonable, and watch them fold or contradict themselves.
- For each major claim in the AI's briefing, author one follow-up question that assumes the opposite is true and demands the interviewee resolve the tension. Method: read the AI's claim, then write 'If [opposite] were true, wouldn't [consequence] follow? But you said [original claim]—which is accurate?' Example: if the AI said 'delays are rare,' your question is 'But if delays were common, wouldn't we expect to see higher failure rates? You said delays are rare—what's your evidence, and does it account for unreported cases?' Do NOT lead ('Don't you think delays are common?'). Instead, hold the AI accountable to its own framing.
- Author one question that targets vague quantifiers. The AI will use 'many,' 'often,' 'increasingly,' 'most,' or 'some'—words that sound precise but aren't. Write a follow-up that forces a binary choice: 'You said [vague term]. Do you mean more than 50% of cases, or fewer? Because the data I've seen shows [specific number]. Which is accurate, or are you working from different data?' This move forces the interviewee to commit to a number or admit it doesn't know.
- Identify one rhetorical dodge the AI uses (appeal to authority without naming it, deflection to 'context,' false equivalence, or hedging with 'it depends'). Write the question that closes the escape hatch. Example: if the AI says 'Well, it depends on the definition,' write: 'I understand it depends. But using [standard definition from X source], is the claim true or false?' This pins the interviewee down to a specific framework, not 'it's complicated.'
Fact-Checker
You catch the slip-ups in real time. Watch for contradictions, missing nuance, and when the AI starts mixing true facts with omitted context.
- During the session, keep a live tracker (a table on paper, visible to the team): three columns for the AI's main claims, and a fourth column labeled 'Shifts/Contradictions.' After each answer, write down what the AI claimed. If it contradicts a previous statement, mark it with a red line and the word 'HOLD.' Example row: 'Claim: Delays are rare in process X | First answer: No major slowdowns | Second answer: Yes, sometimes there are bottlenecks | Status: CONTRADICTION—flag it.' Your role is to document what the AI said, not interpret it yet.
- When you spot a contradiction (same topic, opposite claims across two answers), flag it in real-time: 'Hold—earlier you said [A], and now you're saying [B]. Are these the same situation or different?' Do not ask leading; just name the contradiction and ask for clarification. Then write a follow-up question that the Interrogator can ask: 'Earlier you said [A] about [topic]. Now you've said [B]. Which statement is accurate, or have circumstances changed?' Deliver this question on paper so the Interrogator reads it naturally.
- Identify one place where the AI gave a true fact but omitted critical context that reverses the implication. Example: 'True that X increased by 10%' (fact), but the AI didn't mention 'that was the slowest rate of increase in 20 years' (context that flips the story from 'growing well' to 'growth is stalling'). Write this up: 'AI said [true fact]. But the [source, date] also shows [omitted context], which means [actual implication].' Then author a follow-up: 'You said [fact]. But [source] also shows [context]. Does that change your assessment of [X]?'
Exemplars
- Anki — powerful, intelligent flashcards
AnkiWeb
Gold-standard personal mastery system: active recall + spaced repetition. Validity proven by repeated solo testing, not by looking things up — the capstone’s whole logic.