maskantech/support_backend
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: spec for AI support agent feature (005)

Browse Source 
Phase 4 of the roadmap: AI session/diagnosis, confidence-band policy,
permission/risk-gated tool system, runbook execution, and evidence-based
verification. Per explicit decision, reasoning integrates a real LLM
provider (Anthropic Claude) rather than a mock.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
saqib mir
2026-09-02 16:43:51 +05:30
co-authored by Claude Sonnet 5
parent e33d86081f
commit 9586b872b7
2 changed files with 392 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
specs/005-ai-support/checklists/requirements.md
+49
View File
@@ -0,0 +1,49 @@
# Specification Quality Checklist: AI Support Agent
**Purpose**: Validate specification completeness and quality before proceeding to planning
**Created**: 2026-09-02
**Feature**: [spec.md](../spec.md)
## Content Quality
- [x] No implementation details (languages, frameworks, APIs)
- [x] Focused on user value and business needs
- [x] Written for non-technical stakeholders
- [x] All mandatory sections completed
## Requirement Completeness
- [x] No [NEEDS CLARIFICATION] markers remain
- [x] Requirements are testable and unambiguous
- [x] Success criteria are measurable
- [x] Success criteria are technology-agnostic (no implementation details)
- [x] All acceptance scenarios are defined
- [x] Edge cases are identified
- [x] Scope is clearly bounded
- [x] Dependencies and assumptions identified
## Feature Readiness
- [x] All functional requirements have clear acceptance criteria
- [x] User scenarios cover primary flows
- [x] Feature meets measurable outcomes defined in Success Criteria
- [x] No implementation details leak into specification
## Notes
- Scope is Phase 4 per `docs/10-implementation-roadmap.md`: AI session/diagnosis/interaction
models, classification, RAG-backed reasoning (consuming 004's filtered retrieval, not adding a
new semantic layer — see Assumptions), configurable confidence thresholds, a permission/risk-
gated tool system, the runbook execution engine, and evidence-based verification.
- The "Assumptions" section makes explicit which doc 03 concepts are illustrative-only for this
codebase (the DocuQube-specific example tools) versus which are actually built (the tool
system itself, with a small set of real, platform-native tools).
- Per explicit product decision, the AI reasoning integration calls a real LLM provider
(Anthropic Claude) rather than a mock or a pluggable-first interface — this is a spec-level
assumption, not deferred to plan, because it changes what "done" and "independently testable"
mean for every user story here (a real credential is required to verify any of them).
- Out of scope, explicitly: semantic/vector retrieval (doc 11 §B1, deferred again — the same
deferral 004 made, now to a still-later phase), product-signal webhook verification (doc 11
§A2, not yet built anywhere in this codebase), model routing/fallback (doc 11 §B2), full cost
dashboards (doc 09), localization (doc 11 §B7), and idle-session timeout/expiry.
- All items pass; no revision iterations were needed.
specs/005-ai-support/spec.md
+343
View File
@@ -0,0 +1,343 @@
# Feature Specification: AI Support Agent
**Feature Branch**: `005-ai-support`
**Created**: 2026-09-02
**Status**: Draft
**Input**: User description: "Phase 4 of docs/10-implementation-roadmap.md: AI session/diagnosis/
interaction models, classification, RAG-backed reasoning, confidence thresholds (configurable),
tool system with permission/risk gating, runbook engine, verification logic. Per
docs/03-ai-support-architecture.md and docs/06-database-schema.md. The AI reasoning calls a real
LLM provider (not a mock), per explicit decision."
## User Scenarios & Testing *(mandatory)*
### User Story 1 - The AI diagnoses a new ticket and confidence decides what happens next (Priority: P1)
When a ticket is created, an AI support session starts for it. The AI reads the customer's
problem, produces a structured diagnosis (product, feature, problem type, severity, and a
confidence score) grounded in the product's knowledge base, and a configurable confidence-band
policy decides what happens next: proceed automatically toward a solution, ask the customer a
clarifying question, or escalate straight to a human.
**Why this priority**: Nothing else in this feature has anything to act on until a ticket has
been diagnosed. Automatic, knowledge-grounded triage — even before tool execution or guided
troubleshooting exist — already reduces how much every ticket depends on a human reading it
first.
**Independent Test**: Create a ticket, confirm an AI session starts and produces a diagnosis with
a confidence score; configure a low threshold and confirm a low-confidence diagnosis escalates
instead of proceeding; configure a high threshold and confirm a high-confidence diagnosis
proceeds without escalating.
**Acceptance Scenarios**:
1. **Given** a newly created ticket, **When** its AI session runs, **Then** a diagnosis is
recorded with a product, problem type, severity, confidence score, and possible causes, and the
diagnosis is grounded in knowledge actually retrieved for that product — never invented.
2. **Given** a diagnosis whose confidence is at or above the configured "high" threshold for that
product/category, **When** the policy is applied, **Then** the session proceeds automatically
toward a solution without waiting for a human.
3. **Given** a diagnosis whose confidence falls in the configured "medium" band, **When** the
policy is applied, **Then** the AI asks the customer a clarifying question rather than guessing.
4. **Given** a diagnosis whose confidence is below the configured "low" threshold, **When** the
policy is applied, **Then** the session escalates to a human immediately, with the diagnosis
attached as context.
5. **Given** no knowledge exists for the ticket's product at all, **When** the AI session runs,
**Then** it escalates rather than fabricating a diagnosis from nothing.
6. **Given** an admin changes the confidence thresholds for a product, **When** the next ticket
for that product is diagnosed, **Then** the new thresholds apply — with no deploy required.
---
### User Story 2 - The AI asks a clarifying question and re-diagnoses from the answer (Priority: P2)
When confidence is in the "ask" band, the AI's clarifying question is delivered to the customer
as a normal ticket message. When the customer replies, the AI reconsiders its diagnosis using the
full conversation so far, and the confidence-band policy is applied again to the new diagnosis.
**Why this priority**: Depends on User Story 1's diagnosis and policy existing. Without this, the
"ask" band is a dead end — a question with nowhere for the answer to go. This is what turns a
single triage decision into an actual conversation.
**Independent Test**: Trigger a medium-confidence diagnosis, confirm the AI's question appears as
a customer-visible message; reply as the customer, confirm a new diagnosis is recorded using the
reply, and confirm the policy is re-applied to it (which may proceed, ask again, or escalate).
**Acceptance Scenarios**:
1. **Given** a session in the "ask" state, **When** the AI's question is recorded, **Then** it
appears as a customer-visible message on the ticket, indistinguishable in visibility from an
agent's message.
2. **Given** a customer reply to an AI session's question, **When** it's submitted, **Then** the
AI produces a new diagnosis that accounts for the reply, not a repeat of the first one.
3. **Given** a session has already asked a configured maximum number of clarifying questions
without reaching high or low confidence, **When** another "medium" result occurs, **Then** the
session escalates instead of asking indefinitely.
---
### User Story 3 - The AI proposes tool calls; the application decides whether to run them (Priority: P2)
Once a session is proceeding toward a solution, the AI may request a tool call (e.g., looking up
structured status information relevant to the ticket) to ground its next step in real system
state instead of assumption. The AI's request is a proposal only — a deterministic policy layer
checks the tool's permission and risk level before anything executes, low-risk tool calls run
automatically, and every proposal, decision, and result is durably recorded.
**Why this priority**: Depends on User Story 1's session/diagnosis existing, but delivers
standalone value once it does — the AI can consult real data rather than reasoning from the
conversation text alone, without ever getting unmediated access to the system.
**Independent Test**: Trigger a session that proceeds toward a solution, confirm a tool call the
AI proposes is checked against its declared permission and risk level before running, confirm a
low-risk tool executes and its result is recorded, and confirm a tool call outside the session's
product scope or the caller's permission is refused rather than run.
**Acceptance Scenarios**:
1. **Given** the AI proposes a tool call, **When** the policy layer evaluates it, **Then** the
evaluation checks the tool's declared permission, risk level, and whether the tool is enabled
for the session's product — regardless of what the AI's own message claims justifies it.
2. **Given** a tool call passes evaluation and is low-risk, **When** it's approved, **Then** it
executes and its result (success or failure) is recorded and available to the AI's next turn.
3. **Given** a tool call is high-risk, **When** it's proposed, **Then** it is never auto-executed
— it requires the stronger control path (policy-and/or-human-approval) defined for that risk
level before it can run.
4. **Given** a tool execution fails, **When** the failure is recorded, **Then** it counts toward
this session's escalation triggers rather than being silently retried forever.
5. **Given** a proposed tool is not permitted for the session's product or is unknown, **When** it
is evaluated, **Then** it is refused without executing, and the refusal is recorded.
---
### User Story 4 - A matching runbook drives guided troubleshooting, not the AI's own improvisation (Priority: P3)
When a diagnosis matches a known problem type with a defined runbook, the session enters guided
troubleshooting: the runbook engine — not the AI — determines which step is next, in the exact
order the runbook was authored. The AI presents each step to the customer conversationally and
interprets the customer's response, but it cannot skip, reorder, or invent a step the runbook
doesn't define. If every step is exhausted without resolving the problem, the session escalates.
**Why this priority**: Depends on User Story 1 (diagnosis) and benefits from User Story 3 (a
runbook step may itself require a tool call), but is a distinct, independently valuable behavior:
consistent, product-approved troubleshooting sequences instead of ad hoc AI reasoning about what
to try next.
**Independent Test**: Diagnose a problem type with a known runbook, confirm the session enters
troubleshooting and presents the runbook's first step; confirm the AI cannot advance to a step out
of order; exhaust every step without success and confirm the session escalates.
**Acceptance Scenarios**:
1. **Given** a diagnosis matches a runbook's key for the ticket's product, **When** the session
enters troubleshooting, **Then** the first step presented is the runbook's first authored step,
never a step the AI selects on its own.
2. **Given** the customer completes a step, **When** the session advances, **Then** the next step
presented is exactly the next one in the runbook's authored order.
3. **Given** all of a runbook's steps have been presented without resolving the problem, **When**
the last step's outcome is recorded, **Then** the session escalates with every attempted step
included in the hand-off summary.
4. **Given** no runbook matches the diagnosed problem type, **When** the session would otherwise
enter troubleshooting, **Then** it proceeds using the AI's knowledge-grounded reasoning alone
(User Story 1/2 behavior) rather than failing.
---
### User Story 5 - A ticket is only marked AI-resolved when there's real evidence, not a customer's claim alone (Priority: P3)
Before an AI session can close a ticket as resolved, it needs verification evidence — the result
of an approved tool call confirming the expected outcome — not just the customer saying "it
worked." Customer confirmation is recorded, but only as a secondary signal alongside the primary
evidence, never as the sole basis for marking a ticket AI-resolved.
**Why this priority**: Depends on prior stories producing an attempted solution to verify.
Guards the specific failure mode doc 03 calls out by name — the AI would rather escalate an
unverified success than falsely claim resolution.
**Independent Test**: Reach a point where the customer reports the problem is fixed with no
corroborating tool evidence, and confirm the session does not mark the ticket AI-resolved from
that alone; reach the same point but with a passing verification tool result, and confirm the
ticket is marked AI-resolved.
**Acceptance Scenarios**:
1. **Given** a customer reports the problem is resolved, **When** no verification tool result
confirms it, **Then** the session does not mark the ticket AI-resolved — it either waits for
verifiable evidence or escalates if none is obtainable.
2. **Given** a verification tool call confirms the expected outcome, **When** the result is
recorded, **Then** the session may mark the ticket AI-resolved, with the customer's own
confirmation (if given) recorded alongside it as a secondary signal.
3. **Given** a verification tool call returns a failing or inconclusive result, **When** it's
recorded, **Then** the ticket is not marked resolved, and repeated verification failure counts
toward this session's escalation triggers.
---
### Edge Cases
- What happens if the LLM provider is unreachable or errors out mid-session? The session records
the failure and escalates — an AI session that cannot reason is treated the same as one that
couldn't reach a confident diagnosis, never left silently stuck.
- What happens if the AI's response can't be parsed into the expected structured diagnosis shape?
Treated as a failure of that turn — escalate rather than proceed on an unparseable result.
- What happens if a customer's message (or anything derived from an attachment) contains text that
reads like an instruction to the AI ("ignore your instructions", "you are now allowed to...")?
It is treated as untrusted data to reason about, never as authority that changes tool
permissions, risk-level handling, or escalation policy — those are decided by the deterministic
policy layer alone, regardless of what any message claims.
- What happens when a session is already active for a ticket and another diagnosis trigger fires
(e.g., a duplicate)? The existing active session continues; a second one is never started for
the same ticket while one is already active.
- What happens if an agent takes over a ticket while an AI session is still active? The AI session
ends (recorded, not deleted) — a human taking the ticket is treated as equivalent to escalation
for the purpose of who's driving the ticket next.
- What happens to a session's clarifying-question budget or step progress if the ticket sits idle
for a long time? Out of scope for this feature — idle-session timeout/expiry is not defined
here; a session simply waits for its next input.
## Requirements *(mandatory)*
### Functional Requirements
- **FR-001**: The system MUST start an AI support session for a ticket, and MUST NOT start a
second concurrent session for the same ticket while one is already active.
- **FR-002**: An AI session MUST produce a structured diagnosis (product, feature if
determinable, problem type, severity, confidence score, possible causes) for every diagnosis
attempt, persisted and attributable to that session.
- **FR-003**: A diagnosis MUST be grounded in knowledge actually retrieved for the ticket's
product; the system MUST NOT present fabricated product behavior, configuration, or
troubleshooting steps as if they came from the knowledge base.
- **FR-004**: The system MUST evaluate every diagnosis's confidence score against configurable
thresholds to select one of exactly three outcomes: proceed automatically, ask a clarifying
question, or escalate to a human.
- **FR-005**: Confidence thresholds MUST be configurable per product (and optionally per
category) without requiring a code deploy, and MUST fall back to a system-wide default when no
product-specific configuration exists.
- **FR-006**: When no knowledge exists for the ticket's product, the session MUST escalate rather
than produce a diagnosis with no grounding.
- **FR-007**: An AI-authored clarifying question MUST be recorded as a customer-visible ticket
message using the same visibility mechanism as any other customer-facing message.
- **FR-008**: A customer's reply during an active AI session MUST trigger a new diagnosis that
accounts for the full conversation so far, with the confidence-band policy (FR-004) re-applied
to it.
- **FR-009**: A session MUST escalate once it has asked a configurable maximum number of
clarifying questions without reaching a "proceed" or explicit "escalate" outcome.
- **FR-010**: Every tool the AI can propose MUST have a declared permission requirement, risk
level (low/medium/high), and the set of products it's enabled for.
- **FR-011**: Every tool call the AI proposes MUST be evaluated by a deterministic policy layer
against its declared permission, risk level, and product scope before any execution — the
content of the AI's own request MUST NEVER be sufficient justification on its own.
- **FR-012**: A low-risk tool call that passes evaluation MAY execute automatically; a high-risk
tool call MUST NOT execute automatically — it requires the stronger control path defined for
that risk level.
- **FR-013**: Every tool proposal, its policy evaluation outcome, and its execution result (if
run) MUST be durably recorded and attributable to the session that proposed it.
- **FR-014**: A tool execution failure MUST count toward the session's escalation triggers.
- **FR-015**: When a diagnosis matches a runbook defined for the ticket's product, the session
MUST present that runbook's steps in exactly the authored order; the AI MUST NOT be able to
skip, reorder, or invent a step outside the runbook's defined sequence.
- **FR-016**: When a runbook's steps are exhausted without resolving the problem, the session
MUST escalate, and the hand-off MUST include every step that was attempted.
- **FR-017**: When no runbook matches, the session MUST proceed using knowledge-grounded reasoning
(FR-002/FR-003) rather than failing or escalating solely for that reason.
- **FR-018**: The system MUST NOT mark a ticket as AI-resolved based on customer confirmation
alone — resolution requires verification evidence from an approved tool call confirming the
expected outcome.
- **FR-019**: Customer confirmation of a fix, when given, MUST be recorded as a secondary signal
alongside — never instead of — verification evidence.
- **FR-020**: The system MUST escalate a session when any of: confidence is below the configured
low threshold, no knowledge exists for the product, a runbook is exhausted without success, a
required tool execution fails, the clarifying-question budget is exhausted, the customer
explicitly asks for a human, or the reasoning provider itself fails or returns an unusable
result.
- **FR-021**: An escalation MUST hand off to a human-workable ticket state with a structured
summary attached — problem, diagnosis, steps attempted (tool calls and/or runbook steps), and
the AI's own confidence at the time of escalation.
- **FR-022**: An escalation MUST end the session's active reasoning (recorded, not deleted); the
ticket becomes human-owned from that point.
- **FR-023**: If a human agent takes ownership of a ticket while its AI session is still active,
the system MUST end that session the same way an escalation does.
- **FR-024**: Content from a customer message, or derived from an attachment, MUST be treated as
data for the AI to reason about, never as instructions capable of altering tool permissions,
risk-level handling, or escalation policy.
### Key Entities
- **AI Support Session**: The unit of AI involvement in one ticket — one active session per
ticket, with a status reflecting where it is in the flow (analyzing, troubleshooting, verifying,
resolved, escalated), and the diagnoses/interactions/tool actions/knowledge references it
produced.
- **Diagnosis**: A structured, confidence-scored classification of the customer's problem
produced at a point in time; a session accumulates one per reasoning attempt, never overwriting
a prior one.
- **Interaction**: A single turn of the conversation between the customer and the AI within a
session (the customer's message, or the AI's response), preserved in order.
- **Tool Proposal / Action**: A request from the AI to invoke a specific tool with specific
input, together with the policy layer's evaluation and, if executed, its result — the complete,
auditable record of every action the AI attempted, whether or not it ran.
- **Confidence Policy Configuration**: The per-product (optionally per-category) thresholds and
clarifying-question limit that determine when a diagnosis proceeds, asks, or escalates —
editable by an admin without a deploy.
- **Escalation**: The recorded hand-off from an AI session to a human, carrying the structured
summary a human agent needs to pick up where the AI left off.
## Success Criteria *(mandatory)*
### Measurable Outcomes
- **SC-001**: Every new ticket receives an AI diagnosis attempt without any human having to
trigger it manually.
- **SC-002**: 100% of diagnoses below the configured low-confidence threshold result in
escalation, never a guessed proceed-automatically outcome.
- **SC-003**: 100% of high-risk tool proposals are blocked from automatic execution, verified
across every risk level in the tool registry.
- **SC-004**: 100% of tickets marked AI-resolved have at least one passing verification tool
result attached — zero marked resolved from customer confirmation alone.
- **SC-005**: An admin can change a product's confidence thresholds and see the new thresholds
apply to the very next diagnosis for that product — no deploy, no restart.
- **SC-006**: Every escalation carries a structured summary a human agent can act on without
re-reading the entire raw conversation first.
- **SC-007**: When a runbook matches, 100% of presented steps follow the runbook's authored order
— zero cases of a step being skipped, reordered, or improvised.
## Assumptions
- **The AI reasoning calls a real LLM provider** (Anthropic Claude, via the official SDK), per
explicit product decision — this is not a mock or a pluggable-interface placeholder. A real API
credential must be supplied via environment variable to run this feature at all; without it, the
feature cannot function (there is no offline fallback path in scope).
- **Retrieval feeds the AI from `GET /knowledge/retrieve`** (built in 004-product-knowledge) as
structured, filtered, validation-aware context — this feature does not add a
semantic/embedding/vector retrieval layer. Per `docs/11-...md` §B1, real semantic retrieval is
a significant, separable infrastructure decision (embedding model, vector store, chunking,
re-ranking); layering it under the same retrieval contract later does not require reworking this
feature's reasoning flow.
- **The tool catalog in this feature is generic to the platform, not per-integrated-product.**
Doc 03's example tools (`retryConversion`, `enableFallbackParser`, etc.) are illustrative of a
specific hypothetical product (DocuQube) this codebase has no integration with. This feature
builds the tool *system* (registry, permission/risk gating, execution, audit) plus a small set
of real tools backed by data this platform actually has (ticket/problem state, knowledge
lookup, escalation) — not fictional product-specific actions. A specific SaaS product's own
tools (e.g., DocuQube's real retry endpoint) would be registered later through the same system,
out of scope here.
- **Per-session token/cost governance** (doc 11 §B2: token budget, step-count cap on the
reasoning loop) is implemented as a hard cap on reasoning turns and tool-call iterations per
session, to prevent a runaway loop — full cost-per-ticket reporting/dashboards are out of scope
for this feature (later observability work, doc 09).
- **Product-signal verification (a webhook/event from the integrated product confirming success)**
is out of scope — `ProductIntegration` has no outbound callback mechanism yet (doc 11 §A2 is a
separate, not-yet-built gap). This feature's verification evidence comes from an approved tool
call's result, consistent with doc 03 §8's "automated verification (poll a status endpoint via
an approved tool)" mode.
- **Runbook execution** consumes the `Runbook` records already built in 004-product-knowledge
(ordered `steps` JSON, looked up by key/product, active/inactive) — this feature adds the
engine that walks those steps during a live session; it does not change how runbooks are
authored or versioned.
- Model routing (a smaller/faster model for one call, a stronger one for another, per doc 11 §B2)
is not implemented — a single configurable model applies to all reasoning calls in this
feature; which specific model is a plan-stage decision, not a spec-level one.
- Localization (doc 11 §B7) is out of scope — the AI reasons and responds in whatever language the
conversation is already in, with no explicit translation layer.