From 9586b872b7d5f03273a506eee8fd725d60b3da00 Mon Sep 17 00:00:00 2001 From: saqib mir Date: Wed, 2 Sep 2026 16:43:51 +0530 Subject: [PATCH] docs: spec for AI support agent feature (005) Phase 4 of the roadmap: AI session/diagnosis, confidence-band policy, permission/risk-gated tool system, runbook execution, and evidence-based verification. Per explicit decision, reasoning integrates a real LLM provider (Anthropic Claude) rather than a mock. Co-Authored-By: Claude Sonnet 5 --- .../005-ai-support/checklists/requirements.md | 49 +++ specs/005-ai-support/spec.md | 343 ++++++++++++++++++ 2 files changed, 392 insertions(+) create mode 100644 specs/005-ai-support/checklists/requirements.md create mode 100644 specs/005-ai-support/spec.md diff --git a/specs/005-ai-support/checklists/requirements.md b/specs/005-ai-support/checklists/requirements.md new file mode 100644 index 0000000..27dd441 --- /dev/null +++ b/specs/005-ai-support/checklists/requirements.md @@ -0,0 +1,49 @@ +# Specification Quality Checklist: AI Support Agent + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-09-02 +**Feature**: [spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- Scope is Phase 4 per `docs/10-implementation-roadmap.md`: AI session/diagnosis/interaction + models, classification, RAG-backed reasoning (consuming 004's filtered retrieval, not adding a + new semantic layer — see Assumptions), configurable confidence thresholds, a permission/risk- + gated tool system, the runbook execution engine, and evidence-based verification. +- The "Assumptions" section makes explicit which doc 03 concepts are illustrative-only for this + codebase (the DocuQube-specific example tools) versus which are actually built (the tool + system itself, with a small set of real, platform-native tools). +- Per explicit product decision, the AI reasoning integration calls a real LLM provider + (Anthropic Claude) rather than a mock or a pluggable-first interface — this is a spec-level + assumption, not deferred to plan, because it changes what "done" and "independently testable" + mean for every user story here (a real credential is required to verify any of them). +- Out of scope, explicitly: semantic/vector retrieval (doc 11 §B1, deferred again — the same + deferral 004 made, now to a still-later phase), product-signal webhook verification (doc 11 + §A2, not yet built anywhere in this codebase), model routing/fallback (doc 11 §B2), full cost + dashboards (doc 09), localization (doc 11 §B7), and idle-session timeout/expiry. +- All items pass; no revision iterations were needed. diff --git a/specs/005-ai-support/spec.md b/specs/005-ai-support/spec.md new file mode 100644 index 0000000..29c54a6 --- /dev/null +++ b/specs/005-ai-support/spec.md @@ -0,0 +1,343 @@ +# Feature Specification: AI Support Agent + +**Feature Branch**: `005-ai-support` + +**Created**: 2026-09-02 + +**Status**: Draft + +**Input**: User description: "Phase 4 of docs/10-implementation-roadmap.md: AI session/diagnosis/ +interaction models, classification, RAG-backed reasoning, confidence thresholds (configurable), +tool system with permission/risk gating, runbook engine, verification logic. Per +docs/03-ai-support-architecture.md and docs/06-database-schema.md. The AI reasoning calls a real +LLM provider (not a mock), per explicit decision." + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - The AI diagnoses a new ticket and confidence decides what happens next (Priority: P1) + +When a ticket is created, an AI support session starts for it. The AI reads the customer's +problem, produces a structured diagnosis (product, feature, problem type, severity, and a +confidence score) grounded in the product's knowledge base, and a configurable confidence-band +policy decides what happens next: proceed automatically toward a solution, ask the customer a +clarifying question, or escalate straight to a human. + +**Why this priority**: Nothing else in this feature has anything to act on until a ticket has +been diagnosed. Automatic, knowledge-grounded triage — even before tool execution or guided +troubleshooting exist — already reduces how much every ticket depends on a human reading it +first. + +**Independent Test**: Create a ticket, confirm an AI session starts and produces a diagnosis with +a confidence score; configure a low threshold and confirm a low-confidence diagnosis escalates +instead of proceeding; configure a high threshold and confirm a high-confidence diagnosis +proceeds without escalating. + +**Acceptance Scenarios**: + +1. **Given** a newly created ticket, **When** its AI session runs, **Then** a diagnosis is + recorded with a product, problem type, severity, confidence score, and possible causes, and the + diagnosis is grounded in knowledge actually retrieved for that product — never invented. +2. **Given** a diagnosis whose confidence is at or above the configured "high" threshold for that + product/category, **When** the policy is applied, **Then** the session proceeds automatically + toward a solution without waiting for a human. +3. **Given** a diagnosis whose confidence falls in the configured "medium" band, **When** the + policy is applied, **Then** the AI asks the customer a clarifying question rather than guessing. +4. **Given** a diagnosis whose confidence is below the configured "low" threshold, **When** the + policy is applied, **Then** the session escalates to a human immediately, with the diagnosis + attached as context. +5. **Given** no knowledge exists for the ticket's product at all, **When** the AI session runs, + **Then** it escalates rather than fabricating a diagnosis from nothing. +6. **Given** an admin changes the confidence thresholds for a product, **When** the next ticket + for that product is diagnosed, **Then** the new thresholds apply — with no deploy required. + +--- + +### User Story 2 - The AI asks a clarifying question and re-diagnoses from the answer (Priority: P2) + +When confidence is in the "ask" band, the AI's clarifying question is delivered to the customer +as a normal ticket message. When the customer replies, the AI reconsiders its diagnosis using the +full conversation so far, and the confidence-band policy is applied again to the new diagnosis. + +**Why this priority**: Depends on User Story 1's diagnosis and policy existing. Without this, the +"ask" band is a dead end — a question with nowhere for the answer to go. This is what turns a +single triage decision into an actual conversation. + +**Independent Test**: Trigger a medium-confidence diagnosis, confirm the AI's question appears as +a customer-visible message; reply as the customer, confirm a new diagnosis is recorded using the +reply, and confirm the policy is re-applied to it (which may proceed, ask again, or escalate). + +**Acceptance Scenarios**: + +1. **Given** a session in the "ask" state, **When** the AI's question is recorded, **Then** it + appears as a customer-visible message on the ticket, indistinguishable in visibility from an + agent's message. +2. **Given** a customer reply to an AI session's question, **When** it's submitted, **Then** the + AI produces a new diagnosis that accounts for the reply, not a repeat of the first one. +3. **Given** a session has already asked a configured maximum number of clarifying questions + without reaching high or low confidence, **When** another "medium" result occurs, **Then** the + session escalates instead of asking indefinitely. + +--- + +### User Story 3 - The AI proposes tool calls; the application decides whether to run them (Priority: P2) + +Once a session is proceeding toward a solution, the AI may request a tool call (e.g., looking up +structured status information relevant to the ticket) to ground its next step in real system +state instead of assumption. The AI's request is a proposal only — a deterministic policy layer +checks the tool's permission and risk level before anything executes, low-risk tool calls run +automatically, and every proposal, decision, and result is durably recorded. + +**Why this priority**: Depends on User Story 1's session/diagnosis existing, but delivers +standalone value once it does — the AI can consult real data rather than reasoning from the +conversation text alone, without ever getting unmediated access to the system. + +**Independent Test**: Trigger a session that proceeds toward a solution, confirm a tool call the +AI proposes is checked against its declared permission and risk level before running, confirm a +low-risk tool executes and its result is recorded, and confirm a tool call outside the session's +product scope or the caller's permission is refused rather than run. + +**Acceptance Scenarios**: + +1. **Given** the AI proposes a tool call, **When** the policy layer evaluates it, **Then** the + evaluation checks the tool's declared permission, risk level, and whether the tool is enabled + for the session's product — regardless of what the AI's own message claims justifies it. +2. **Given** a tool call passes evaluation and is low-risk, **When** it's approved, **Then** it + executes and its result (success or failure) is recorded and available to the AI's next turn. +3. **Given** a tool call is high-risk, **When** it's proposed, **Then** it is never auto-executed + — it requires the stronger control path (policy-and/or-human-approval) defined for that risk + level before it can run. +4. **Given** a tool execution fails, **When** the failure is recorded, **Then** it counts toward + this session's escalation triggers rather than being silently retried forever. +5. **Given** a proposed tool is not permitted for the session's product or is unknown, **When** it + is evaluated, **Then** it is refused without executing, and the refusal is recorded. + +--- + +### User Story 4 - A matching runbook drives guided troubleshooting, not the AI's own improvisation (Priority: P3) + +When a diagnosis matches a known problem type with a defined runbook, the session enters guided +troubleshooting: the runbook engine — not the AI — determines which step is next, in the exact +order the runbook was authored. The AI presents each step to the customer conversationally and +interprets the customer's response, but it cannot skip, reorder, or invent a step the runbook +doesn't define. If every step is exhausted without resolving the problem, the session escalates. + +**Why this priority**: Depends on User Story 1 (diagnosis) and benefits from User Story 3 (a +runbook step may itself require a tool call), but is a distinct, independently valuable behavior: +consistent, product-approved troubleshooting sequences instead of ad hoc AI reasoning about what +to try next. + +**Independent Test**: Diagnose a problem type with a known runbook, confirm the session enters +troubleshooting and presents the runbook's first step; confirm the AI cannot advance to a step out +of order; exhaust every step without success and confirm the session escalates. + +**Acceptance Scenarios**: + +1. **Given** a diagnosis matches a runbook's key for the ticket's product, **When** the session + enters troubleshooting, **Then** the first step presented is the runbook's first authored step, + never a step the AI selects on its own. +2. **Given** the customer completes a step, **When** the session advances, **Then** the next step + presented is exactly the next one in the runbook's authored order. +3. **Given** all of a runbook's steps have been presented without resolving the problem, **When** + the last step's outcome is recorded, **Then** the session escalates with every attempted step + included in the hand-off summary. +4. **Given** no runbook matches the diagnosed problem type, **When** the session would otherwise + enter troubleshooting, **Then** it proceeds using the AI's knowledge-grounded reasoning alone + (User Story 1/2 behavior) rather than failing. + +--- + +### User Story 5 - A ticket is only marked AI-resolved when there's real evidence, not a customer's claim alone (Priority: P3) + +Before an AI session can close a ticket as resolved, it needs verification evidence — the result +of an approved tool call confirming the expected outcome — not just the customer saying "it +worked." Customer confirmation is recorded, but only as a secondary signal alongside the primary +evidence, never as the sole basis for marking a ticket AI-resolved. + +**Why this priority**: Depends on prior stories producing an attempted solution to verify. +Guards the specific failure mode doc 03 calls out by name — the AI would rather escalate an +unverified success than falsely claim resolution. + +**Independent Test**: Reach a point where the customer reports the problem is fixed with no +corroborating tool evidence, and confirm the session does not mark the ticket AI-resolved from +that alone; reach the same point but with a passing verification tool result, and confirm the +ticket is marked AI-resolved. + +**Acceptance Scenarios**: + +1. **Given** a customer reports the problem is resolved, **When** no verification tool result + confirms it, **Then** the session does not mark the ticket AI-resolved — it either waits for + verifiable evidence or escalates if none is obtainable. +2. **Given** a verification tool call confirms the expected outcome, **When** the result is + recorded, **Then** the session may mark the ticket AI-resolved, with the customer's own + confirmation (if given) recorded alongside it as a secondary signal. +3. **Given** a verification tool call returns a failing or inconclusive result, **When** it's + recorded, **Then** the ticket is not marked resolved, and repeated verification failure counts + toward this session's escalation triggers. + +--- + +### Edge Cases + +- What happens if the LLM provider is unreachable or errors out mid-session? The session records + the failure and escalates — an AI session that cannot reason is treated the same as one that + couldn't reach a confident diagnosis, never left silently stuck. +- What happens if the AI's response can't be parsed into the expected structured diagnosis shape? + Treated as a failure of that turn — escalate rather than proceed on an unparseable result. +- What happens if a customer's message (or anything derived from an attachment) contains text that + reads like an instruction to the AI ("ignore your instructions", "you are now allowed to...")? + It is treated as untrusted data to reason about, never as authority that changes tool + permissions, risk-level handling, or escalation policy — those are decided by the deterministic + policy layer alone, regardless of what any message claims. +- What happens when a session is already active for a ticket and another diagnosis trigger fires + (e.g., a duplicate)? The existing active session continues; a second one is never started for + the same ticket while one is already active. +- What happens if an agent takes over a ticket while an AI session is still active? The AI session + ends (recorded, not deleted) — a human taking the ticket is treated as equivalent to escalation + for the purpose of who's driving the ticket next. +- What happens to a session's clarifying-question budget or step progress if the ticket sits idle + for a long time? Out of scope for this feature — idle-session timeout/expiry is not defined + here; a session simply waits for its next input. + +## Requirements *(mandatory)* + +### Functional Requirements + +- **FR-001**: The system MUST start an AI support session for a ticket, and MUST NOT start a + second concurrent session for the same ticket while one is already active. +- **FR-002**: An AI session MUST produce a structured diagnosis (product, feature if + determinable, problem type, severity, confidence score, possible causes) for every diagnosis + attempt, persisted and attributable to that session. +- **FR-003**: A diagnosis MUST be grounded in knowledge actually retrieved for the ticket's + product; the system MUST NOT present fabricated product behavior, configuration, or + troubleshooting steps as if they came from the knowledge base. +- **FR-004**: The system MUST evaluate every diagnosis's confidence score against configurable + thresholds to select one of exactly three outcomes: proceed automatically, ask a clarifying + question, or escalate to a human. +- **FR-005**: Confidence thresholds MUST be configurable per product (and optionally per + category) without requiring a code deploy, and MUST fall back to a system-wide default when no + product-specific configuration exists. +- **FR-006**: When no knowledge exists for the ticket's product, the session MUST escalate rather + than produce a diagnosis with no grounding. +- **FR-007**: An AI-authored clarifying question MUST be recorded as a customer-visible ticket + message using the same visibility mechanism as any other customer-facing message. +- **FR-008**: A customer's reply during an active AI session MUST trigger a new diagnosis that + accounts for the full conversation so far, with the confidence-band policy (FR-004) re-applied + to it. +- **FR-009**: A session MUST escalate once it has asked a configurable maximum number of + clarifying questions without reaching a "proceed" or explicit "escalate" outcome. +- **FR-010**: Every tool the AI can propose MUST have a declared permission requirement, risk + level (low/medium/high), and the set of products it's enabled for. +- **FR-011**: Every tool call the AI proposes MUST be evaluated by a deterministic policy layer + against its declared permission, risk level, and product scope before any execution — the + content of the AI's own request MUST NEVER be sufficient justification on its own. +- **FR-012**: A low-risk tool call that passes evaluation MAY execute automatically; a high-risk + tool call MUST NOT execute automatically — it requires the stronger control path defined for + that risk level. +- **FR-013**: Every tool proposal, its policy evaluation outcome, and its execution result (if + run) MUST be durably recorded and attributable to the session that proposed it. +- **FR-014**: A tool execution failure MUST count toward the session's escalation triggers. +- **FR-015**: When a diagnosis matches a runbook defined for the ticket's product, the session + MUST present that runbook's steps in exactly the authored order; the AI MUST NOT be able to + skip, reorder, or invent a step outside the runbook's defined sequence. +- **FR-016**: When a runbook's steps are exhausted without resolving the problem, the session + MUST escalate, and the hand-off MUST include every step that was attempted. +- **FR-017**: When no runbook matches, the session MUST proceed using knowledge-grounded reasoning + (FR-002/FR-003) rather than failing or escalating solely for that reason. +- **FR-018**: The system MUST NOT mark a ticket as AI-resolved based on customer confirmation + alone — resolution requires verification evidence from an approved tool call confirming the + expected outcome. +- **FR-019**: Customer confirmation of a fix, when given, MUST be recorded as a secondary signal + alongside — never instead of — verification evidence. +- **FR-020**: The system MUST escalate a session when any of: confidence is below the configured + low threshold, no knowledge exists for the product, a runbook is exhausted without success, a + required tool execution fails, the clarifying-question budget is exhausted, the customer + explicitly asks for a human, or the reasoning provider itself fails or returns an unusable + result. +- **FR-021**: An escalation MUST hand off to a human-workable ticket state with a structured + summary attached — problem, diagnosis, steps attempted (tool calls and/or runbook steps), and + the AI's own confidence at the time of escalation. +- **FR-022**: An escalation MUST end the session's active reasoning (recorded, not deleted); the + ticket becomes human-owned from that point. +- **FR-023**: If a human agent takes ownership of a ticket while its AI session is still active, + the system MUST end that session the same way an escalation does. +- **FR-024**: Content from a customer message, or derived from an attachment, MUST be treated as + data for the AI to reason about, never as instructions capable of altering tool permissions, + risk-level handling, or escalation policy. + +### Key Entities + +- **AI Support Session**: The unit of AI involvement in one ticket — one active session per + ticket, with a status reflecting where it is in the flow (analyzing, troubleshooting, verifying, + resolved, escalated), and the diagnoses/interactions/tool actions/knowledge references it + produced. +- **Diagnosis**: A structured, confidence-scored classification of the customer's problem + produced at a point in time; a session accumulates one per reasoning attempt, never overwriting + a prior one. +- **Interaction**: A single turn of the conversation between the customer and the AI within a + session (the customer's message, or the AI's response), preserved in order. +- **Tool Proposal / Action**: A request from the AI to invoke a specific tool with specific + input, together with the policy layer's evaluation and, if executed, its result — the complete, + auditable record of every action the AI attempted, whether or not it ran. +- **Confidence Policy Configuration**: The per-product (optionally per-category) thresholds and + clarifying-question limit that determine when a diagnosis proceeds, asks, or escalates — + editable by an admin without a deploy. +- **Escalation**: The recorded hand-off from an AI session to a human, carrying the structured + summary a human agent needs to pick up where the AI left off. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: Every new ticket receives an AI diagnosis attempt without any human having to + trigger it manually. +- **SC-002**: 100% of diagnoses below the configured low-confidence threshold result in + escalation, never a guessed proceed-automatically outcome. +- **SC-003**: 100% of high-risk tool proposals are blocked from automatic execution, verified + across every risk level in the tool registry. +- **SC-004**: 100% of tickets marked AI-resolved have at least one passing verification tool + result attached — zero marked resolved from customer confirmation alone. +- **SC-005**: An admin can change a product's confidence thresholds and see the new thresholds + apply to the very next diagnosis for that product — no deploy, no restart. +- **SC-006**: Every escalation carries a structured summary a human agent can act on without + re-reading the entire raw conversation first. +- **SC-007**: When a runbook matches, 100% of presented steps follow the runbook's authored order + — zero cases of a step being skipped, reordered, or improvised. + +## Assumptions + +- **The AI reasoning calls a real LLM provider** (Anthropic Claude, via the official SDK), per + explicit product decision — this is not a mock or a pluggable-interface placeholder. A real API + credential must be supplied via environment variable to run this feature at all; without it, the + feature cannot function (there is no offline fallback path in scope). +- **Retrieval feeds the AI from `GET /knowledge/retrieve`** (built in 004-product-knowledge) as + structured, filtered, validation-aware context — this feature does not add a + semantic/embedding/vector retrieval layer. Per `docs/11-...md` §B1, real semantic retrieval is + a significant, separable infrastructure decision (embedding model, vector store, chunking, + re-ranking); layering it under the same retrieval contract later does not require reworking this + feature's reasoning flow. +- **The tool catalog in this feature is generic to the platform, not per-integrated-product.** + Doc 03's example tools (`retryConversion`, `enableFallbackParser`, etc.) are illustrative of a + specific hypothetical product (DocuQube) this codebase has no integration with. This feature + builds the tool *system* (registry, permission/risk gating, execution, audit) plus a small set + of real tools backed by data this platform actually has (ticket/problem state, knowledge + lookup, escalation) — not fictional product-specific actions. A specific SaaS product's own + tools (e.g., DocuQube's real retry endpoint) would be registered later through the same system, + out of scope here. +- **Per-session token/cost governance** (doc 11 §B2: token budget, step-count cap on the + reasoning loop) is implemented as a hard cap on reasoning turns and tool-call iterations per + session, to prevent a runaway loop — full cost-per-ticket reporting/dashboards are out of scope + for this feature (later observability work, doc 09). +- **Product-signal verification (a webhook/event from the integrated product confirming success)** + is out of scope — `ProductIntegration` has no outbound callback mechanism yet (doc 11 §A2 is a + separate, not-yet-built gap). This feature's verification evidence comes from an approved tool + call's result, consistent with doc 03 §8's "automated verification (poll a status endpoint via + an approved tool)" mode. +- **Runbook execution** consumes the `Runbook` records already built in 004-product-knowledge + (ordered `steps` JSON, looked up by key/product, active/inactive) — this feature adds the + engine that walks those steps during a live session; it does not change how runbooks are + authored or versioned. +- Model routing (a smaller/faster model for one call, a stronger one for another, per doc 11 §B2) + is not implemented — a single configurable model applies to all reasoning calls in this + feature; which specific model is a plan-stage decision, not a spec-level one. +- Localization (doc 11 §B7) is out of scope — the AI reasons and responds in whatever language the + conversation is already in, with no explicit translation layer.