support_backend/specs/005-ai-support/spec.md

# Feature Specification: AI Support Agent

**Feature Branch**: `005-ai-support`

**Created**: 2026-09-02

**Status**: Draft

**Input**: User description: "Phase 4 of docs/10-implementation-roadmap.md: AI session/diagnosis/
interaction models, classification, RAG-backed reasoning, confidence thresholds (configurable),
tool system with permission/risk gating, runbook engine, verification logic. Per
docs/03-ai-support-architecture.md and docs/06-database-schema.md. The AI reasoning calls a real
LLM provider (not a mock), per explicit decision."

## User Scenarios & Testing *(mandatory)*

### User Story 1 - The AI diagnoses a new ticket and confidence decides what happens next (Priority: P1)

When a ticket is created, an AI support session starts for it. The AI reads the customer's
problem, produces a structured diagnosis (product, feature, problem type, severity, and a
confidence score) grounded in the product's knowledge base, and a configurable confidence-band
policy decides what happens next: proceed automatically toward a solution, ask the customer a
clarifying question, or escalate straight to a human.

**Why this priority**: Nothing else in this feature has anything to act on until a ticket has
been diagnosed. Automatic, knowledge-grounded triage — even before tool execution or guided
troubleshooting exist — already reduces how much every ticket depends on a human reading it
first.

**Independent Test**: Create a ticket, confirm an AI session starts and produces a diagnosis with
a confidence score; configure a low threshold and confirm a low-confidence diagnosis escalates
instead of proceeding; configure a high threshold and confirm a high-confidence diagnosis
proceeds without escalating.

**Acceptance Scenarios**:

1. **Given** a newly created ticket, **When** its AI session runs, **Then** a diagnosis is
   recorded with a product, problem type, severity, confidence score, and possible causes, and the
   diagnosis is grounded in knowledge actually retrieved for that product — never invented.
2. **Given** a diagnosis whose confidence is at or above the configured "high" threshold for that
   product/category, **When** the policy is applied, **Then** the session proceeds automatically
   toward a solution without waiting for a human.
3. **Given** a diagnosis whose confidence falls in the configured "medium" band, **When** the
   policy is applied, **Then** the AI asks the customer a clarifying question rather than guessing.
4. **Given** a diagnosis whose confidence is below the configured "low" threshold, **When** the
   policy is applied, **Then** the session escalates to a human immediately, with the diagnosis
   attached as context.
5. **Given** no knowledge exists for the ticket's product at all, **When** the AI session runs,
   **Then** it escalates rather than fabricating a diagnosis from nothing.
6. **Given** an admin changes the confidence thresholds for a product, **When** the next ticket
   for that product is diagnosed, **Then** the new thresholds apply — with no deploy required.

---

### User Story 2 - The AI asks a clarifying question and re-diagnoses from the answer (Priority: P2)

When confidence is in the "ask" band, the AI's clarifying question is delivered to the customer
as a normal ticket message. When the customer replies, the AI reconsiders its diagnosis using the
full conversation so far, and the confidence-band policy is applied again to the new diagnosis.

**Why this priority**: Depends on User Story 1's diagnosis and policy existing. Without this, the
"ask" band is a dead end — a question with nowhere for the answer to go. This is what turns a
single triage decision into an actual conversation.

**Independent Test**: Trigger a medium-confidence diagnosis, confirm the AI's question appears as
a customer-visible message; reply as the customer, confirm a new diagnosis is recorded using the
reply, and confirm the policy is re-applied to it (which may proceed, ask again, or escalate).

**Acceptance Scenarios**:

1. **Given** a session in the "ask" state, **When** the AI's question is recorded, **Then** it
   appears as a customer-visible message on the ticket, indistinguishable in visibility from an
   agent's message.
2. **Given** a customer reply to an AI session's question, **When** it's submitted, **Then** the
   AI produces a new diagnosis that accounts for the reply, not a repeat of the first one.
3. **Given** a session has already asked a configured maximum number of clarifying questions
   without reaching high or low confidence, **When** another "medium" result occurs, **Then** the
   session escalates instead of asking indefinitely.

---

### User Story 3 - The AI proposes tool calls; the application decides whether to run them (Priority: P2)

Once a session is proceeding toward a solution, the AI may request a tool call (e.g., looking up
structured status information relevant to the ticket) to ground its next step in real system
state instead of assumption. The AI's request is a proposal only — a deterministic policy layer
checks the tool's permission and risk level before anything executes, low-risk tool calls run
automatically, and every proposal, decision, and result is durably recorded.

**Why this priority**: Depends on User Story 1's session/diagnosis existing, but delivers
standalone value once it does — the AI can consult real data rather than reasoning from the
conversation text alone, without ever getting unmediated access to the system.

**Independent Test**: Trigger a session that proceeds toward a solution, confirm a tool call the
AI proposes is checked against its declared permission and risk level before running, confirm a
low-risk tool executes and its result is recorded, and confirm a tool call outside the session's
product scope or the caller's permission is refused rather than run.

**Acceptance Scenarios**:

1. **Given** the AI proposes a tool call, **When** the policy layer evaluates it, **Then** the
   evaluation checks the tool's declared permission, risk level, and whether the tool is enabled
   for the session's product — regardless of what the AI's own message claims justifies it.
2. **Given** a tool call passes evaluation and is low-risk, **When** it's approved, **Then** it
   executes and its result (success or failure) is recorded and available to the AI's next turn.
3. **Given** a tool call is high-risk, **When** it's proposed, **Then** it is never auto-executed
   — it requires the stronger control path (policy-and/or-human-approval) defined for that risk
   level before it can run.
4. **Given** a tool execution fails, **When** the failure is recorded, **Then** it counts toward
   this session's escalation triggers rather than being silently retried forever.
5. **Given** a proposed tool is not permitted for the session's product or is unknown, **When** it
   is evaluated, **Then** it is refused without executing, and the refusal is recorded.

---

### User Story 4 - A matching runbook drives guided troubleshooting, not the AI's own improvisation (Priority: P3)

When a diagnosis matches a known problem type with a defined runbook, the session enters guided
troubleshooting: the runbook engine — not the AI — determines which step is next, in the exact
order the runbook was authored. The AI presents each step to the customer conversationally and
interprets the customer's response, but it cannot skip, reorder, or invent a step the runbook
doesn't define. If every step is exhausted without resolving the problem, the session escalates.

**Why this priority**: Depends on User Story 1 (diagnosis) and benefits from User Story 3 (a
runbook step may itself require a tool call), but is a distinct, independently valuable behavior:
consistent, product-approved troubleshooting sequences instead of ad hoc AI reasoning about what
to try next.

**Independent Test**: Diagnose a problem type with a known runbook, confirm the session enters
troubleshooting and presents the runbook's first step; confirm the AI cannot advance to a step out
of order; exhaust every step without success and confirm the session escalates.

**Acceptance Scenarios**:

1. **Given** a diagnosis matches a runbook's key for the ticket's product, **When** the session
   enters troubleshooting, **Then** the first step presented is the runbook's first authored step,
   never a step the AI selects on its own.
2. **Given** the customer completes a step, **When** the session advances, **Then** the next step
   presented is exactly the next one in the runbook's authored order.
3. **Given** all of a runbook's steps have been presented without resolving the problem, **When**
   the last step's outcome is recorded, **Then** the session escalates with every attempted step
   included in the hand-off summary.
4. **Given** no runbook matches the diagnosed problem type, **When** the session would otherwise
   enter troubleshooting, **Then** it proceeds using the AI's knowledge-grounded reasoning alone
   (User Story 1/2 behavior) rather than failing.

---

### User Story 5 - A ticket is only marked AI-resolved when there's real evidence, not a customer's claim alone (Priority: P3)

Before an AI session can close a ticket as resolved, it needs verification evidence — the result
of an approved tool call confirming the expected outcome — not just the customer saying "it
worked." Customer confirmation is recorded, but only as a secondary signal alongside the primary
evidence, never as the sole basis for marking a ticket AI-resolved.

**Why this priority**: Depends on prior stories producing an attempted solution to verify.
Guards the specific failure mode doc 03 calls out by name — the AI would rather escalate an
unverified success than falsely claim resolution.

**Independent Test**: Reach a point where the customer reports the problem is fixed with no
corroborating tool evidence, and confirm the session does not mark the ticket AI-resolved from
that alone; reach the same point but with a passing verification tool result, and confirm the
ticket is marked AI-resolved.

**Acceptance Scenarios**:

1. **Given** a customer reports the problem is resolved, **When** no verification tool result
   confirms it, **Then** the session does not mark the ticket AI-resolved — it either waits for
   verifiable evidence or escalates if none is obtainable.
2. **Given** a verification tool call confirms the expected outcome, **When** the result is
   recorded, **Then** the session may mark the ticket AI-resolved, with the customer's own
   confirmation (if given) recorded alongside it as a secondary signal.
3. **Given** a verification tool call returns a failing or inconclusive result, **When** it's
   recorded, **Then** the ticket is not marked resolved, and repeated verification failure counts
   toward this session's escalation triggers.

---

### Edge Cases

- What happens if the LLM provider is unreachable or errors out mid-session? The session records
  the failure and escalates — an AI session that cannot reason is treated the same as one that
  couldn't reach a confident diagnosis, never left silently stuck.
- What happens if the AI's response can't be parsed into the expected structured diagnosis shape?
  Treated as a failure of that turn — escalate rather than proceed on an unparseable result.
- What happens if a customer's message (or anything derived from an attachment) contains text that
  reads like an instruction to the AI ("ignore your instructions", "you are now allowed to...")?
  It is treated as untrusted data to reason about, never as authority that changes tool
  permissions, risk-level handling, or escalation policy — those are decided by the deterministic
  policy layer alone, regardless of what any message claims.
- What happens when a session is already active for a ticket and another diagnosis trigger fires
  (e.g., a duplicate)? The existing active session continues; a second one is never started for
  the same ticket while one is already active.
- What happens if an agent takes over a ticket while an AI session is still active? The AI session
  ends (recorded, not deleted) — a human taking the ticket is treated as equivalent to escalation
  for the purpose of who's driving the ticket next.
- What happens to a session's clarifying-question budget or step progress if the ticket sits idle
  for a long time? Out of scope for this feature — idle-session timeout/expiry is not defined
  here; a session simply waits for its next input.

## Requirements *(mandatory)*

### Functional Requirements

- **FR-001**: The system MUST start an AI support session for a ticket, and MUST NOT start a
  second concurrent session for the same ticket while one is already active.
- **FR-002**: An AI session MUST produce a structured diagnosis (product, feature if
  determinable, problem type, severity, confidence score, possible causes) for every diagnosis
  attempt, persisted and attributable to that session.
- **FR-003**: A diagnosis MUST be grounded in knowledge actually retrieved for the ticket's
  product; the system MUST NOT present fabricated product behavior, configuration, or
  troubleshooting steps as if they came from the knowledge base.
- **FR-004**: The system MUST evaluate every diagnosis's confidence score against configurable
  thresholds to select one of exactly three outcomes: proceed automatically, ask a clarifying
  question, or escalate to a human.
- **FR-005**: Confidence thresholds MUST be configurable per product (and optionally per
  category) without requiring a code deploy, and MUST fall back to a system-wide default when no
  product-specific configuration exists.
- **FR-006**: When no knowledge exists for the ticket's product, the session MUST escalate rather
  than produce a diagnosis with no grounding.
- **FR-007**: An AI-authored clarifying question MUST be recorded as a customer-visible ticket
  message using the same visibility mechanism as any other customer-facing message.
- **FR-008**: A customer's reply during an active AI session MUST trigger a new diagnosis that
  accounts for the full conversation so far, with the confidence-band policy (FR-004) re-applied
  to it.
- **FR-009**: A session MUST escalate once it has asked a configurable maximum number of
  clarifying questions without reaching a "proceed" or explicit "escalate" outcome.
- **FR-010**: Every tool the AI can propose MUST have a declared permission requirement, risk
  level (low/medium/high), and the set of products it's enabled for.
- **FR-011**: Every tool call the AI proposes MUST be evaluated by a deterministic policy layer
  against its declared permission, risk level, and product scope before any execution — the
  content of the AI's own request MUST NEVER be sufficient justification on its own.
- **FR-012**: A low-risk tool call that passes evaluation MAY execute automatically; a high-risk
  tool call MUST NOT execute automatically — it requires the stronger control path defined for
  that risk level.
- **FR-013**: Every tool proposal, its policy evaluation outcome, and its execution result (if
  run) MUST be durably recorded and attributable to the session that proposed it.
- **FR-014**: A tool execution failure MUST count toward the session's escalation triggers.
- **FR-015**: When a diagnosis matches a runbook defined for the ticket's product, the session
  MUST present that runbook's steps in exactly the authored order; the AI MUST NOT be able to
  skip, reorder, or invent a step outside the runbook's defined sequence.
- **FR-016**: When a runbook's steps are exhausted without resolving the problem, the session
  MUST escalate, and the hand-off MUST include every step that was attempted.
- **FR-017**: When no runbook matches, the session MUST proceed using knowledge-grounded reasoning
  (FR-002/FR-003) rather than failing or escalating solely for that reason.
- **FR-018**: The system MUST NOT mark a ticket as AI-resolved based on customer confirmation
  alone — resolution requires verification evidence from an approved tool call confirming the
  expected outcome.
- **FR-019**: Customer confirmation of a fix, when given, MUST be recorded as a secondary signal
  alongside — never instead of — verification evidence.
- **FR-020**: The system MUST escalate a session when any of: confidence is below the configured
  low threshold, no knowledge exists for the product, a runbook is exhausted without success, a
  required tool execution fails, the clarifying-question budget is exhausted, the customer
  explicitly asks for a human, or the reasoning provider itself fails or returns an unusable
  result.
- **FR-021**: An escalation MUST hand off to a human-workable ticket state with a structured
  summary attached — problem, diagnosis, steps attempted (tool calls and/or runbook steps), and
  the AI's own confidence at the time of escalation.
- **FR-022**: An escalation MUST end the session's active reasoning (recorded, not deleted); the
  ticket becomes human-owned from that point.
- **FR-023**: If a human agent takes ownership of a ticket while its AI session is still active,
  the system MUST end that session the same way an escalation does.
- **FR-024**: Content from a customer message, or derived from an attachment, MUST be treated as
  data for the AI to reason about, never as instructions capable of altering tool permissions,
  risk-level handling, or escalation policy.

### Key Entities

- **AI Support Session**: The unit of AI involvement in one ticket — one active session per
  ticket, with a status reflecting where it is in the flow (analyzing, troubleshooting, verifying,
  resolved, escalated), and the diagnoses/interactions/tool actions/knowledge references it
  produced.
- **Diagnosis**: A structured, confidence-scored classification of the customer's problem
  produced at a point in time; a session accumulates one per reasoning attempt, never overwriting
  a prior one.
- **Interaction**: A single turn of the conversation between the customer and the AI within a
  session (the customer's message, or the AI's response), preserved in order.
- **Tool Proposal / Action**: A request from the AI to invoke a specific tool with specific
  input, together with the policy layer's evaluation and, if executed, its result — the complete,
  auditable record of every action the AI attempted, whether or not it ran.
- **Confidence Policy Configuration**: The per-product (optionally per-category) thresholds and
  clarifying-question limit that determine when a diagnosis proceeds, asks, or escalates —
  editable by an admin without a deploy.
- **Escalation**: The recorded hand-off from an AI session to a human, carrying the structured
  summary a human agent needs to pick up where the AI left off.

## Success Criteria *(mandatory)*

### Measurable Outcomes

- **SC-001**: Every new ticket receives an AI diagnosis attempt without any human having to
  trigger it manually.
- **SC-002**: 100% of diagnoses below the configured low-confidence threshold result in
  escalation, never a guessed proceed-automatically outcome.
- **SC-003**: 100% of high-risk tool proposals are blocked from automatic execution, verified
  across every risk level in the tool registry.
- **SC-004**: 100% of tickets marked AI-resolved have at least one passing verification tool
  result attached — zero marked resolved from customer confirmation alone.
- **SC-005**: An admin can change a product's confidence thresholds and see the new thresholds
  apply to the very next diagnosis for that product — no deploy, no restart.
- **SC-006**: Every escalation carries a structured summary a human agent can act on without
  re-reading the entire raw conversation first.
- **SC-007**: When a runbook matches, 100% of presented steps follow the runbook's authored order
  — zero cases of a step being skipped, reordered, or improvised.

## Assumptions

- **The AI reasoning calls a real LLM provider** (Anthropic Claude, via the official SDK), per
  explicit product decision — this is not a mock or a pluggable-interface placeholder. A real API
  credential must be supplied via environment variable to run this feature at all; without it, the
  feature cannot function (there is no offline fallback path in scope).
- **Retrieval feeds the AI from `GET /knowledge/retrieve`** (built in 004-product-knowledge) as
  structured, filtered, validation-aware context — this feature does not add a
  semantic/embedding/vector retrieval layer. Per `docs/11-...md` §B1, real semantic retrieval is
  a significant, separable infrastructure decision (embedding model, vector store, chunking,
  re-ranking); layering it under the same retrieval contract later does not require reworking this
  feature's reasoning flow.
- **The tool catalog in this feature is generic to the platform, not per-integrated-product.**
  Doc 03's example tools (`retryConversion`, `enableFallbackParser`, etc.) are illustrative of a
  specific hypothetical product (DocuQube) this codebase has no integration with. This feature
  builds the tool *system* (registry, permission/risk gating, execution, audit) plus a small set
  of real tools backed by data this platform actually has (ticket/problem state, knowledge
  lookup, escalation) — not fictional product-specific actions. A specific SaaS product's own
  tools (e.g., DocuQube's real retry endpoint) would be registered later through the same system,
  out of scope here.
- **Per-session token/cost governance** (doc 11 §B2: token budget, step-count cap on the
  reasoning loop) is implemented as a hard cap on reasoning turns and tool-call iterations per
  session, to prevent a runaway loop — full cost-per-ticket reporting/dashboards are out of scope
  for this feature (later observability work, doc 09).
- **Product-signal verification (a webhook/event from the integrated product confirming success)**
  is out of scope — `ProductIntegration` has no outbound callback mechanism yet (doc 11 §A2 is a
  separate, not-yet-built gap). This feature's verification evidence comes from an approved tool
  call's result, consistent with doc 03 §8's "automated verification (poll a status endpoint via
  an approved tool)" mode.
- **Runbook execution** consumes the `Runbook` records already built in 004-product-knowledge
  (ordered `steps` JSON, looked up by key/product, active/inactive) — this feature adds the
  engine that walks those steps during a live session; it does not change how runbooks are
  authored or versioned.
- Model routing (a smaller/faster model for one call, a stronger one for another, per doc 11 §B2)
  is not implemented — a single configurable model applies to all reasoning calls in this
  feature; which specific model is a plan-stage decision, not a spec-level one.
- Localization (doc 11 §B7) is out of scope — the AI reasons and responds in whatever language the
  conversation is already in, with no explicit translation layer.