20 Questions Every BA Must Ask Before Building
“Let’s add AI” is the modern version of “Let’s add a dashboard.” It sounds productive, it demos well, and it can still fail spectacularly—because what people really want isn’t AI. They want faster decisions, fewer mistakes, less rework, better customer experiences, and relief from repetitive work.
AI only delivers those outcomes when the team agrees on something more specific than “make it smart.” That’s where Business Analysts shine. You don’t need to be the model expert. You need to be the person who turns enthusiasm into clarity: what success looks like, what the system is allowed to do, what it must never do, and how you’ll prove it’s behaving safely and consistently.
This article is meant to be used in the real world. You can run it as a 60–90 minute kickoff, capture the answers as you go, and leave with tangible outputs: a scope boundary, an output contract, guardrails, acceptance criteria, and a realistic plan for testing and monitoring.
Why AI requirements go wrong (even with talented teams)
Traditional software requirements assume the system is deterministic: the same input should produce the same output. AI systems aren’t like that. Even when they “work,” their output can vary, especially when the input is ambiguous, incomplete, or noisy—conditions that are common in real business workflows.
AI also turns data into a product dependency. The moment your feature uses internal documents, CRM records, tickets, or customer inputs, you’re not just building functionality—you’re building a pipeline of trust. Where did this information come from? Is it current? Is it allowed? What happens if sources conflict?
And the most important difference: AI failures often look like confident success. A fluent answer can still be wrong, unsafe, or non-compliant. That’s why a BA-led AI requirements approach must include guardrails, fallbacks, and evidence—not just features.
The Requirements Output Map (the quickest way to keep everyone aligned)
AI conversations drift into tools and vendors fast. Pull them back to what matters by using this simple map:
- Use case & decision – what action does the AI influence?
- Inputs & sources – what information does it use, and what’s authoritative?
- Model behavior – what must it do and must not do?
- Outputs – structure, tone, citations, and uncertainty handling
- Controls – guardrails, fallback behavior, escalation paths
- Evidence – how you test, monitor, and audit it over time
Now let’s get practical: the 20 questions that fill in this map.
The 20 must-ask questions (with why they matter and what they produce)
A) Use case clarity (Questions 1–4)
1) What user problem are we solving—and what’s the job-to-be-done?
AI projects stall when the “problem” is really a vague desire to modernize. This question forces the team to name the real pain: time, errors, delays, cost, missed opportunities, inconsistent decisions, or poor customer experience.
Output artifact(s):
- Problem statement (1–2 paragraphs)
- Current-state workflow (even a simple swimlane sketch)
2) What is the AI’s role: suggest, summarize, classify, decide, or act?
This is not semantics; it’s risk. A summarizer and an “agent” that takes actions need entirely different controls, testing, and accountability. Be explicit about the role so you don’t accidentally build something with higher impact than stakeholders intended.
Output artifact(s):
- “AI Role” statement (single sentence)
- Human vs AI responsibility boundary (“AI drafts; human approves,” etc.)
3) What is explicitly out of scope—and why?
If you don’t define what the AI won’t do, users will ask for it anyway. Out-of-scope boundaries protect you from scope creep, compliance surprises, and “well, couldn’t it also…” escalations late in delivery.
Output artifact(s):
- In-scope / out-of-scope list
- “Not allowed” list (topics, actions, data sources)
4) What does success look like in measurable terms?
If success isn’t measurable, the only real metric becomes “do people like the output?” That’s how teams launch with weak quality and find out later that the system creates extra rework.
Output artifact(s):
- Success metrics + targets (e.g., time saved, accuracy, deflection, fewer escalations)
- Baseline measurement plan (how you’ll measure current performance)
B) Users, context, and workflow (Questions 5–7)
5) Who are the primary users, and what is their expertise level?
The same output can be useful to an expert and dangerous to a novice. Expertise affects tone, depth, assumptions, and the kind of guardrails you need.
Output artifact(s):
- Persona(s) with skill level, context, and key constraints
- Training/onboarding assumptions (what users already know)
6) Where does this live in the workflow—and what happens next?
AI output is rarely the final product. It becomes input to a decision, a customer interaction, or a system-of-record update. This question identifies downstream consequences—where mistakes get amplified.
Output artifact(s):
- Updated workflow diagram showing insertion point
- Downstream usage notes (“output is pasted into CRM,” “customer sees this,” etc.)
7) What is the cost of a wrong answer—annoying, expensive, unsafe, illegal?
This is your severity model. Not every mistake is equal, and your requirements must reflect that. Define tiers so the team knows where to be conservative.
Output artifact(s):
- Impact/severity tiers (Low/Medium/High)
- Required control level per tier (e.g., human approval for High)
C) Input data & access (Questions 8–10)
8) What inputs will the AI use, and which sources are authoritative?
Most “hallucinations” in business settings are actually “source ambiguity” problems. The system needs a clear hierarchy of trusted sources, and you need rules for conflicts and missing information.
Output artifact(s):
- Data/source inventory (what it can use)
- Source-of-truth hierarchy (“policy doc overrides wiki,” etc.)
9) What access does it need—and what must it never access?
Least privilege is an AI requirement, not an infrastructure detail. If the system can see too much, it will eventually reveal too much—often accidentally.
Output artifact(s):
- Role × data access matrix
- Explicit exclusions (“never access HR folders,” “no customer SSNs,” etc.)
10) How do we handle sensitive data, retention, and training usage?
Stakeholders often assume “the vendor handles this.” Don’t. Define what’s logged, what’s stored, how long it’s retained, and whether any data is used for training or improvement.
Output artifact(s):
- Logging/retention requirements
- Data handling rules (masking/redaction)
- Training usage rules (allowed/forbidden)
D) Output requirements (Questions 11–13)
11) What output format is required (bullets, JSON, email draft, decision recommendation)?
If you want consistent quality, you need an output contract. Structure reduces ambiguity, improves reviewability, and makes testing possible.
Output artifact(s):
- Output schema/structure definition
- 2–3 example outputs (“good” examples)
12) What must the output include (sources, confidence, links, timestamps)?
This is where you protect trust. If output must be grounded, require citations/links. If output influences decisions, require uncertainty cues or limitations statements.
Output artifact(s):
- Required fields list (citations, confidence, record IDs, timestamps)
- UI display requirements (where those fields appear)
13) What style constraints matter (tone, reading level, disclaimers, “don’t sound like policy”)?
Style affects misuse. A confident tone can cause users to over-trust weak output. A legal-sounding tone can create compliance risk. Define the “voice” intentionally.
Output artifact(s):
- Tone/style guide (short)
- Forbidden phrasing list (“guaranteed,” “definitely,” etc., if relevant)
E) Guardrails & unacceptable behavior (Questions 14–16)
14) What must the AI refuse to do or say?
Refusal is a feature. Without explicit refusal rules, the system will try to answer everything—and some of those answers will be unacceptable.
Output artifact(s):
- Refusal rules (“If asked X, refuse with Y + suggest safe alternative”)
- Disallowed actions/topics list
15) What are “red flag” scenarios requiring escalation to a human?
Escalation rules protect customers and the business. They also protect your team from the “why didn’t it flag this?” question after an incident.
Output artifact(s):
- Escalation trigger list (by scenario type)
- Escalation path (who/where it routes)
16) What is the fallback when confidence is low?
A good fallback prevents invented answers. It can ask clarifying questions, show top sources, route to an SME, or explicitly state what is unknown.
Output artifact(s):
- Low-confidence behavior spec
- Clarifying question patterns
- Routing rules (when to hand off)
F) Quality, evaluation, and testing (Questions 17–18)
17) How will we measure quality (accuracy, precision/recall, groundedness, latency)?
“Accuracy” alone is rarely sufficient. Different AI roles need different quality definitions. This question forces the team to define the rubric before debate becomes subjective.
Output artifact(s):
- Quality metrics + thresholds (role-appropriate)
- Non-functional targets (latency, uptime, cost per request if needed)
18) What is the acceptance test set (golden dataset), and who signs off?
If you don’t define test cases early, you’ll test whatever is easiest—not what is representative. Include edge cases and refusal scenarios so you don’t “pass” while still being unsafe.
Output artifact(s):
- Golden test set plan + ownership
- Sign-off roles and launch criteria
G) Operations: monitoring, drift, and change control (Questions 19–20)
19) What do we monitor in production?
The system will change—usage changes, input distributions change, policies change. Monitoring is how you catch “quiet degradation” before users lose trust.
Output artifact(s):
- Monitoring requirements (signals + thresholds)
- Incident triggers (what counts as an AI incident)
20) How do updates happen (prompt/model/version changes, rollback, audit trail)?
AI systems evolve quickly. Treat changes as controlled releases with versioning, approvals (where needed), and rollback capability. If you can’t explain why output changed, you can’t govern it.
Output artifact(s):
- Change control process
- Versioning/audit trail requirements
- Rollback plan
Turning answers into requirements (without turning it into bureaucracy)
If you run these 20 questions properly, you’ll notice something reassuring: you’re not creating a mountain of paperwork. You’re creating clarity where ambiguity usually hides.
The most practical way to capture it is to use a consistent spec format every time. That’s why I separated the copy/paste AI Requirements Spec template into a downloadable PDF. Link it from your article, attach it to your epic/story, and treat any unanswered section as an explicit “TBD” discovery item rather than an unspoken assumption.
And if you want the fast version for workshops, the checklist PDF is designed to be printed or pasted into meeting notes:
A practical “ready to build” rule of thumb
You don’t need perfect answers to every question to start prototyping. But you do need to know what you’re postponing—and you need agreement on what must be true before production (guardrails, test set, monitoring, and sign-off).
If the team can’t answer at least 15 of the 20 questions with reasonable clarity, the next step isn’t “start coding.” It’s discovery—because what you’re missing is not implementation detail. It’s definition.
Author: Morgan Masters, Business Analyst, Modern Analyst Media LLC
Morgan Masters is Business Analyst and Staff Writer at ModernAnalyst.com, the premier community and resource portal for business analysts. Business analysis resources such as articles, blogs, templates, forums, books, along with a thriving business analyst community can be found at http://www.ModernAnalyst.com