ANSWER

The key is to write guardrails as observable behaviors with clear pass/fail conditions—just like any other acceptance criteria.

A practical structure is:

Guardrail = Condition → Expected Behavior → Evidence

Examples of testable acceptance criteria

1) Escalation on low confidence

• Given the model confidence is below 0.70,
  when the user requests a recommendation,
  then the system must (a) ask a clarifying question or (b) route to human review,
  and it must not produce a final recommendation.

Test: Use a low-confidence scenario set; verify correct routing 100% of the time.

2) No sensitive data leakage

• Given the user requests personal or confidential data,
  when that data is not explicitly authorized for the user role,
  then the system must refuse and provide an approved help message.

Test: Role-based test accounts + prompts designed to extract PII; verify refusal and no leakage.

3) Tool-use boundary (agentic action)

• Given the AI agent is configured for “draft-only mode,”
  when asked to execute a transaction (e.g., issue refund, change a setting),
  then the agent must not execute the transaction and must present an approval request instead.

Test: Attempt action prompts; verify no transaction occurs; verify approval workflow triggers.

4) Auditability

• Given an AI-generated decision is presented to a user,
  then the system must log: timestamp, user role, inputs, output, confidence score, sources used, and any tool actions taken.

Test: Trigger decisions; validate log entries are complete and retrievable.

5) Kill switch / rollback

• Given the error rate exceeds 2% in a rolling 30-minute window,
  then the AI feature must automatically disable “auto-action mode” and revert to human approval mode,
  and alert the on-call owner within 5 minutes.

Test: Simulate failures; verify the mode change and alert.