support_backend/specs/005-ai-support/spec.md

Feature Specification: AI Support Agent

Feature Branch: 005-ai-support

Created: 2026-09-02

Status: Draft

Input: User description: "Phase 4 of docs/10-implementation-roadmap.md: AI session/diagnosis/ interaction models, classification, RAG-backed reasoning, confidence thresholds (configurable), tool system with permission/risk gating, runbook engine, verification logic. Per docs/03-ai-support-architecture.md and docs/06-database-schema.md. The AI reasoning calls a real LLM provider (not a mock), per explicit decision."

User Scenarios & Testing (mandatory)

User Story 1 - The AI diagnoses a new ticket and confidence decides what happens next (Priority: P1)

When a ticket is created, an AI support session starts for it. The AI reads the customer's problem, produces a structured diagnosis (product, feature, problem type, severity, and a confidence score) grounded in the product's knowledge base, and a configurable confidence-band policy decides what happens next: proceed automatically toward a solution, ask the customer a clarifying question, or escalate straight to a human.

Why this priority: Nothing else in this feature has anything to act on until a ticket has been diagnosed. Automatic, knowledge-grounded triage — even before tool execution or guided troubleshooting exist — already reduces how much every ticket depends on a human reading it first.

Independent Test: Create a ticket, confirm an AI session starts and produces a diagnosis with a confidence score; configure a low threshold and confirm a low-confidence diagnosis escalates instead of proceeding; configure a high threshold and confirm a high-confidence diagnosis proceeds without escalating.

Acceptance Scenarios:

1. Given a newly created ticket, When its AI session runs, Then a diagnosis is recorded with a product, problem type, severity, confidence score, and possible causes, and the diagnosis is grounded in knowledge actually retrieved for that product — never invented.
2. Given a diagnosis whose confidence is at or above the configured "high" threshold for that product/category, When the policy is applied, Then the session proceeds automatically toward a solution without waiting for a human.
3. Given a diagnosis whose confidence falls in the configured "medium" band, When the policy is applied, Then the AI asks the customer a clarifying question rather than guessing.
4. Given a diagnosis whose confidence is below the configured "low" threshold, When the policy is applied, Then the session escalates to a human immediately, with the diagnosis attached as context.
5. Given no knowledge exists for the ticket's product at all, When the AI session runs, Then it escalates rather than fabricating a diagnosis from nothing.
6. Given an admin changes the confidence thresholds for a product, When the next ticket for that product is diagnosed, Then the new thresholds apply — with no deploy required.

---

User Story 2 - The AI asks a clarifying question and re-diagnoses from the answer (Priority: P2)

When confidence is in the "ask" band, the AI's clarifying question is delivered to the customer as a normal ticket message. When the customer replies, the AI reconsiders its diagnosis using the full conversation so far, and the confidence-band policy is applied again to the new diagnosis.

Why this priority: Depends on User Story 1's diagnosis and policy existing. Without this, the "ask" band is a dead end — a question with nowhere for the answer to go. This is what turns a single triage decision into an actual conversation.

Independent Test: Trigger a medium-confidence diagnosis, confirm the AI's question appears as a customer-visible message; reply as the customer, confirm a new diagnosis is recorded using the reply, and confirm the policy is re-applied to it (which may proceed, ask again, or escalate).

Acceptance Scenarios:

1. Given a session in the "ask" state, When the AI's question is recorded, Then it appears as a customer-visible message on the ticket, indistinguishable in visibility from an agent's message.
2. Given a customer reply to an AI session's question, When it's submitted, Then the AI produces a new diagnosis that accounts for the reply, not a repeat of the first one.
3. Given a session has already asked a configured maximum number of clarifying questions without reaching high or low confidence, When another "medium" result occurs, Then the session escalates instead of asking indefinitely.

---

User Story 3 - The AI proposes tool calls; the application decides whether to run them (Priority: P2)

Once a session is proceeding toward a solution, the AI may request a tool call (e.g., looking up structured status information relevant to the ticket) to ground its next step in real system state instead of assumption. The AI's request is a proposal only — a deterministic policy layer checks the tool's permission and risk level before anything executes, low-risk tool calls run automatically, and every proposal, decision, and result is durably recorded.

Why this priority: Depends on User Story 1's session/diagnosis existing, but delivers standalone value once it does — the AI can consult real data rather than reasoning from the conversation text alone, without ever getting unmediated access to the system.

Independent Test: Trigger a session that proceeds toward a solution, confirm a tool call the AI proposes is checked against its declared permission and risk level before running, confirm a low-risk tool executes and its result is recorded, and confirm a tool call outside the session's product scope or the caller's permission is refused rather than run.

Acceptance Scenarios:

1. Given the AI proposes a tool call, When the policy layer evaluates it, Then the evaluation checks the tool's declared permission, risk level, and whether the tool is enabled for the session's product — regardless of what the AI's own message claims justifies it.
2. Given a tool call passes evaluation and is low-risk, When it's approved, Then it executes and its result (success or failure) is recorded and available to the AI's next turn.
3. Given a tool call is high-risk, When it's proposed, Then it is never auto-executed — it requires the stronger control path (policy-and/or-human-approval) defined for that risk level before it can run.
4. Given a tool execution fails, When the failure is recorded, Then it counts toward this session's escalation triggers rather than being silently retried forever.
5. Given a proposed tool is not permitted for the session's product or is unknown, When it is evaluated, Then it is refused without executing, and the refusal is recorded.

---

User Story 4 - A matching runbook drives guided troubleshooting, not the AI's own improvisation (Priority: P3)

When a diagnosis matches a known problem type with a defined runbook, the session enters guided troubleshooting: the runbook engine — not the AI — determines which step is next, in the exact order the runbook was authored. The AI presents each step to the customer conversationally and interprets the customer's response, but it cannot skip, reorder, or invent a step the runbook doesn't define. If every step is exhausted without resolving the problem, the session escalates.

Why this priority: Depends on User Story 1 (diagnosis) and benefits from User Story 3 (a runbook step may itself require a tool call), but is a distinct, independently valuable behavior: consistent, product-approved troubleshooting sequences instead of ad hoc AI reasoning about what to try next.

Independent Test: Diagnose a problem type with a known runbook, confirm the session enters troubleshooting and presents the runbook's first step; confirm the AI cannot advance to a step out of order; exhaust every step without success and confirm the session escalates.

Acceptance Scenarios:

1. Given a diagnosis matches a runbook's key for the ticket's product, When the session enters troubleshooting, Then the first step presented is the runbook's first authored step, never a step the AI selects on its own.
2. Given the customer completes a step, When the session advances, Then the next step presented is exactly the next one in the runbook's authored order.
3. Given all of a runbook's steps have been presented without resolving the problem, When the last step's outcome is recorded, Then the session escalates with every attempted step included in the hand-off summary.
4. Given no runbook matches the diagnosed problem type, When the session would otherwise enter troubleshooting, Then it proceeds using the AI's knowledge-grounded reasoning alone (User Story 1/2 behavior) rather than failing.

---

User Story 5 - A ticket is only marked AI-resolved when there's real evidence, not a customer's claim alone (Priority: P3)

Before an AI session can close a ticket as resolved, it needs verification evidence — the result of an approved tool call confirming the expected outcome — not just the customer saying "it worked." Customer confirmation is recorded, but only as a secondary signal alongside the primary evidence, never as the sole basis for marking a ticket AI-resolved.

Why this priority: Depends on prior stories producing an attempted solution to verify. Guards the specific failure mode doc 03 calls out by name — the AI would rather escalate an unverified success than falsely claim resolution.

Independent Test: Reach a point where the customer reports the problem is fixed with no corroborating tool evidence, and confirm the session does not mark the ticket AI-resolved from that alone; reach the same point but with a passing verification tool result, and confirm the ticket is marked AI-resolved.

Acceptance Scenarios:

1. Given a customer reports the problem is resolved, When no verification tool result confirms it, Then the session does not mark the ticket AI-resolved — it either waits for verifiable evidence or escalates if none is obtainable.
2. Given a verification tool call confirms the expected outcome, When the result is recorded, Then the session may mark the ticket AI-resolved, with the customer's own confirmation (if given) recorded alongside it as a secondary signal.
3. Given a verification tool call returns a failing or inconclusive result, When it's recorded, Then the ticket is not marked resolved, and repeated verification failure counts toward this session's escalation triggers.

---

Edge Cases

• What happens if the LLM provider is unreachable or errors out mid-session? The session records the failure and escalates — an AI session that cannot reason is treated the same as one that couldn't reach a confident diagnosis, never left silently stuck.
• What happens if the AI's response can't be parsed into the expected structured diagnosis shape? Treated as a failure of that turn — escalate rather than proceed on an unparseable result.
• What happens if a customer's message (or anything derived from an attachment) contains text that reads like an instruction to the AI ("ignore your instructions", "you are now allowed to...")? It is treated as untrusted data to reason about, never as authority that changes tool permissions, risk-level handling, or escalation policy — those are decided by the deterministic policy layer alone, regardless of what any message claims.
• What happens when a session is already active for a ticket and another diagnosis trigger fires (e.g., a duplicate)? The existing active session continues; a second one is never started for the same ticket while one is already active.
• What happens if an agent takes over a ticket while an AI session is still active? The AI session ends (recorded, not deleted) — a human taking the ticket is treated as equivalent to escalation for the purpose of who's driving the ticket next.
• What happens to a session's clarifying-question budget or step progress if the ticket sits idle for a long time? Out of scope for this feature — idle-session timeout/expiry is not defined here; a session simply waits for its next input.

Requirements (mandatory)

Functional Requirements

• FR-001: The system MUST start an AI support session for a ticket, and MUST NOT start a second concurrent session for the same ticket while one is already active.
• FR-002: An AI session MUST produce a structured diagnosis (product, feature if determinable, problem type, severity, confidence score, possible causes) for every diagnosis attempt, persisted and attributable to that session.
• FR-003: A diagnosis MUST be grounded in knowledge actually retrieved for the ticket's product; the system MUST NOT present fabricated product behavior, configuration, or troubleshooting steps as if they came from the knowledge base.
• FR-004: The system MUST evaluate every diagnosis's confidence score against configurable thresholds to select one of exactly three outcomes: proceed automatically, ask a clarifying question, or escalate to a human.
• FR-005: Confidence thresholds MUST be configurable per product (and optionally per category) without requiring a code deploy, and MUST fall back to a system-wide default when no product-specific configuration exists.
• FR-006: When no knowledge exists for the ticket's product, the session MUST escalate rather than produce a diagnosis with no grounding.
• FR-007: An AI-authored clarifying question MUST be recorded as a customer-visible ticket message using the same visibility mechanism as any other customer-facing message.
• FR-008: A customer's reply during an active AI session MUST trigger a new diagnosis that accounts for the full conversation so far, with the confidence-band policy (FR-004) re-applied to it.
• FR-009: A session MUST escalate once it has asked a configurable maximum number of clarifying questions without reaching a "proceed" or explicit "escalate" outcome.
• FR-010: Every tool the AI can propose MUST have a declared permission requirement, risk level (low/medium/high), and the set of products it's enabled for.
• FR-011: Every tool call the AI proposes MUST be evaluated by a deterministic policy layer against its declared permission, risk level, and product scope before any execution — the content of the AI's own request MUST NEVER be sufficient justification on its own.
• FR-012: A low-risk tool call that passes evaluation MAY execute automatically; a high-risk tool call MUST NOT execute automatically — it requires the stronger control path defined for that risk level.
• FR-013: Every tool proposal, its policy evaluation outcome, and its execution result (if run) MUST be durably recorded and attributable to the session that proposed it.
• FR-014: A tool execution failure MUST count toward the session's escalation triggers.
• FR-015: When a diagnosis matches a runbook defined for the ticket's product, the session MUST present that runbook's steps in exactly the authored order; the AI MUST NOT be able to skip, reorder, or invent a step outside the runbook's defined sequence.
• FR-016: When a runbook's steps are exhausted without resolving the problem, the session MUST escalate, and the hand-off MUST include every step that was attempted.
• FR-017: When no runbook matches, the session MUST proceed using knowledge-grounded reasoning (FR-002/FR-003) rather than failing or escalating solely for that reason.
• FR-018: The system MUST NOT mark a ticket as AI-resolved based on customer confirmation alone — resolution requires verification evidence from an approved tool call confirming the expected outcome.
• FR-019: Customer confirmation of a fix, when given, MUST be recorded as a secondary signal alongside — never instead of — verification evidence.
• FR-020: The system MUST escalate a session when any of: confidence is below the configured low threshold, no knowledge exists for the product, a runbook is exhausted without success, a required tool execution fails, the clarifying-question budget is exhausted, the customer explicitly asks for a human, or the reasoning provider itself fails or returns an unusable result.
• FR-021: An escalation MUST hand off to a human-workable ticket state with a structured summary attached — problem, diagnosis, steps attempted (tool calls and/or runbook steps), and the AI's own confidence at the time of escalation.
• FR-022: An escalation MUST end the session's active reasoning (recorded, not deleted); the ticket becomes human-owned from that point.
• FR-023: If a human agent takes ownership of a ticket while its AI session is still active, the system MUST end that session the same way an escalation does.
• FR-024: Content from a customer message, or derived from an attachment, MUST be treated as data for the AI to reason about, never as instructions capable of altering tool permissions, risk-level handling, or escalation policy.

Key Entities

• AI Support Session: The unit of AI involvement in one ticket — one active session per ticket, with a status reflecting where it is in the flow (analyzing, troubleshooting, verifying, resolved, escalated), and the diagnoses/interactions/tool actions/knowledge references it produced.
• Diagnosis: A structured, confidence-scored classification of the customer's problem produced at a point in time; a session accumulates one per reasoning attempt, never overwriting a prior one.
• Interaction: A single turn of the conversation between the customer and the AI within a session (the customer's message, or the AI's response), preserved in order.
• Tool Proposal / Action: A request from the AI to invoke a specific tool with specific input, together with the policy layer's evaluation and, if executed, its result — the complete, auditable record of every action the AI attempted, whether or not it ran.
• Confidence Policy Configuration: The per-product (optionally per-category) thresholds and clarifying-question limit that determine when a diagnosis proceeds, asks, or escalates — editable by an admin without a deploy.
• Escalation: The recorded hand-off from an AI session to a human, carrying the structured summary a human agent needs to pick up where the AI left off.

Success Criteria (mandatory)

Measurable Outcomes

• SC-001: Every new ticket receives an AI diagnosis attempt without any human having to trigger it manually.
• SC-002: 100% of diagnoses below the configured low-confidence threshold result in escalation, never a guessed proceed-automatically outcome.
• SC-003: 100% of high-risk tool proposals are blocked from automatic execution, verified across every risk level in the tool registry.
• SC-004: 100% of tickets marked AI-resolved have at least one passing verification tool result attached — zero marked resolved from customer confirmation alone.
• SC-005: An admin can change a product's confidence thresholds and see the new thresholds apply to the very next diagnosis for that product — no deploy, no restart.
• SC-006: Every escalation carries a structured summary a human agent can act on without re-reading the entire raw conversation first.
• SC-007: When a runbook matches, 100% of presented steps follow the runbook's authored order — zero cases of a step being skipped, reordered, or improvised.

Assumptions

• The AI reasoning calls a real LLM provider (Anthropic Claude, via the official SDK), per explicit product decision — this is not a mock or a pluggable-interface placeholder. A real API credential must be supplied via environment variable to run this feature at all; without it, the feature cannot function (there is no offline fallback path in scope).
• Retrieval feeds the AI from GET /knowledge/retrieve (built in 004-product-knowledge) as structured, filtered, validation-aware context — this feature does not add a semantic/embedding/vector retrieval layer. Per docs/11-...md §B1, real semantic retrieval is a significant, separable infrastructure decision (embedding model, vector store, chunking, re-ranking); layering it under the same retrieval contract later does not require reworking this feature's reasoning flow.
• The tool catalog in this feature is generic to the platform, not per-integrated-product. Doc 03's example tools (retryConversion, enableFallbackParser, etc.) are illustrative of a specific hypothetical product (DocuQube) this codebase has no integration with. This feature builds the tool system (registry, permission/risk gating, execution, audit) plus a small set of real tools backed by data this platform actually has (ticket/problem state, knowledge lookup, escalation) — not fictional product-specific actions. A specific SaaS product's own tools (e.g., DocuQube's real retry endpoint) would be registered later through the same system, out of scope here.
• Per-session token/cost governance (doc 11 §B2: token budget, step-count cap on the reasoning loop) is implemented as a hard cap on reasoning turns and tool-call iterations per session, to prevent a runaway loop — full cost-per-ticket reporting/dashboards are out of scope for this feature (later observability work, doc 09).
• Product-signal verification (a webhook/event from the integrated product confirming success) is out of scope — ProductIntegration has no outbound callback mechanism yet (doc 11 §A2 is a separate, not-yet-built gap). This feature's verification evidence comes from an approved tool call's result, consistent with doc 03 §8's "automated verification (poll a status endpoint via an approved tool)" mode.
• Runbook execution consumes the Runbook records already built in 004-product-knowledge (ordered steps JSON, looked up by key/product, active/inactive) — this feature adds the engine that walks those steps during a live session; it does not change how runbooks are authored or versioned.
• Model routing (a smaller/faster model for one call, a stronger one for another, per doc 11 §B2) is not implemented — a single configurable model applies to all reasoning calls in this feature; which specific model is a plan-stage decision, not a spec-level one.
• Localization (doc 11 §B7) is out of scope — the AI reasons and responds in whatever language the conversation is already in, with no explicit translation layer.