support_backend/specs/005-ai-support/research.md

# Phase 0 Research: AI Support Agent

## Decision: LLM provider integration — Anthropic SDK, manual loop, two calls per reasoning turn

- **Decision**: Use `@anthropic-ai/sdk` directly (`new Anthropic()`, credential from
  `ANTHROPIC_API_KEY`). Each session turn is **two** model calls, not one, plus a deterministic
  step between them:
  1. **Classify+diagnose** — `client.messages.parse()` with `output_config.format` (Zod schema
     via `zodOutputFormat`) against the conversation so far. No tools. Output: the structured
     `AIDiagnosis` shape (product/feature/problemType/severity/confidence/possibleCauses).
     Structured output and tool use are not combined in the same call — keeping diagnosis as a
     pure structured-output call means it can never emit a stray tool proposal, and keeps the
     confidence score honest (it's the model's stated belief about the classification, not
     entangled with whatever it also did with tools that turn).
  2. Deterministically (no model call): call `knowledgeService.retrieve(...)` (004, imported
     through `ai-support/knowledge`'s public `index.ts` — Principle III; an in-process call, not
     an HTTP loopback to this same service's own `GET /knowledge/retrieve` route) scoped to the
     diagnosis's product/feature and the ticket's category, and apply the confidence-band policy
     (FR-004) to the diagnosis. This decides the branch: ask / proceed / escalate — this decision
     is application code, never delegated to the model.
  3. **Reasoning/response** — only on the "proceed" branch (or to word a clarifying question on
     the "ask" branch): a regular `client.messages.create()` call, given the diagnosis, the
     retrieved knowledge (as context, explicitly framed as data), the conversation so far, and —
     only on "proceed" — the tool registry's currently-enabled tools for this product. The model
     may return `tool_use` blocks here; a manual loop (not the SDK's beta tool runner) executes
     them, because policy evaluation has to happen **before** execution and has to be a first-
     class, auditable step of its own — see the tool-gating decision below — which the tool
     runner's `run()`-function-level gating pattern would bury inside each tool rather than
     express as a shared, visible gate.
- **Rationale**: Directly implements doc 03 §1's four-step flow (Classification → Knowledge
  Retrieval → Reasoning → Next Action) as written, rather than collapsing it into one prompt that
  both classifies and acts — which would make confidence-band policy (a MUST per FR-004) something
  the model influences by how it phrases one big answer, instead of something applied
  deterministically to a discrete classification output.
- **Alternatives considered**: One combined call producing diagnosis + response + tool calls
  together — rejected; makes it impossible to apply the confidence-band policy *before* the model
  has already committed to a response/tool calls, which is backwards from FR-004's "confidence
  decides what happens next." The SDK's beta tool runner — rejected for the reasoning call
  specifically (not for tool definition, which still uses the same `Anthropic.Tool` shape); the
  runner's per-tool `run()` gating pattern would scatter the policy check across each tool
  function instead of keeping it as one shared, auditable evaluation step ahead of any execution,
  which is what Constitution Principle IV ("AI recommends, policy decides") and FR-011 actually
  require — a visible decision point, not a convention every tool has to individually remember.

## Decision: Model, thinking, and effort — configurable, not hardcoded

- **Decision**: `AI_SUPPORT_MODEL` env var (default `claude-opus-5`), `AI_SUPPORT_EFFORT` env var
  (default `medium`, one of `low`/`medium`/`high`/`xhigh`/`max`). Every call uses
  `thinking: { type: "adaptive" }` (Claude Opus 5 runs adaptive thinking by default; this makes
  it explicit and keeps the code correct if the configured model changes) and
  `output_config: { effort: AI_SUPPORT_EFFORT }`.
- **Rationale**: Doc 11 §B2 explicitly asks for model choice to be "a configurable policy, not a
  hardcoded model name" — this satisfies that with the simplest mechanism that fits this feature's
  scope (one configured model for all calls; see Assumptions in spec.md for why per-call model
  routing is out of scope). Defaulting to `claude-opus-5` follows this codebase's standing default
  for new Claude integrations; the env var lets it be changed without a code change if the
  operator wants a different cost/quality tradeoff.
- **Alternatives considered**: Hardcoding the model string inline at each call site — rejected,
  directly contradicts doc 11 §B2 and Constitution Principle II (config over hardcoding).

## Decision: Confidence-band policy — new `AIConfidencePolicy` config table, most-specific-match fallback

- **Decision**: A new model, `AIConfidencePolicy` (`productId String?`, `categoryId String?`,
  `highThreshold Float`, `lowThreshold Float`, `maxClarifyingQuestions Int`), unique on
  `(productId, categoryId)`. Lookup order: exact `(productId, categoryId)` match → `(productId,
  null)` match → hardcoded system defaults from env
  (`AI_SUPPORT_DEFAULT_HIGH_CONFIDENCE`/`AI_SUPPORT_DEFAULT_LOW_CONFIDENCE`/
  `AI_SUPPORT_DEFAULT_MAX_CLARIFYING_QUESTIONS`). A diagnosis with `confidence >= highThreshold` →
  proceed; `confidence < lowThreshold` → escalate; otherwise → ask.
- **Rationale**: FR-005 requires per-product (optionally per-category) configuration "without
  requiring a code deploy" — a DB-backed config row an admin can write via an endpoint satisfies
  that directly, and the most-specific-match-with-fallback pattern means an operator never has to
  pre-seed every product before the feature works; the env-var defaults exist for exactly the
  "no configuration exists yet" case (FR-005's explicit fallback requirement).
- **Alternatives considered**: A single global config row with no per-product override — rejected,
  doesn't satisfy FR-005's "per product (and optionally per category)" requirement, and doc 03 §4
  explicitly frames thresholds as "tunable per product/category." Storing thresholds as JSON on
  `Product` itself — rejected; a dedicated table keeps the same
  admin-CRUD-with-versioning-free-writes shape as every other config surface in this codebase and
  supports the category-scoped override doc 03 asks for without denormalizing `Product`.

## Decision: Tool system — code-defined registry, DB-backed per-product enablement not needed for this phase's tool set

- **Decision**: Tools are defined in code (name, description, Zod input schema, permission
  string, risk level, `auditRequired`) as a small, fixed registry —
  `src/modules/ai-support/tools/constants/tool-registry.ts`. Four tools ship in this feature:
  - `getTicketSnapshot` (low risk) — reads the ticket, its problem, and recent messages; real,
    read-only, always available.
  - `searchProductKnowledge` (low risk) — re-runs `knowledgeService.retrieve(...)` (same
    in-process call as above) with a model-supplied feature/category refinement mid-conversation;
    real, read-only.
  - `verifyProductResolution` (low risk) — **a documented, fail-closed placeholder**, exactly the
    same pattern already established by `ticketing/attachments`'s
    `UnimplementedPlaceholderScanner`: it always returns `{ confirmed: false, status: "unknown"
    }`, never a fabricated success, because there is no real product-side signal to check yet
    (doc 11 §A2 — outbound webhooks from an integrated product don't exist in this codebase).
    This is what makes FR-018 ("never mark AI-resolved from customer confirmation alone") hold
    mechanically today: until a real per-product status check replaces this placeholder,
    verification can never auto-pass, so genuinely automatic AI-resolution won't happen in
    practice — which is the correct, honest behavior for a system with no real evidence source
    yet, not a bug to work around.
  - `escalateToHuman` (low risk, always policy-approved) — a real, functional action: ends the
    session the same way FR-020/FR-022 describe, lets the AI act on an explicit customer request
    for a human as a first-class, audited proposal rather than free-text the app has to sniff for.
  A fifth, **high-risk** tool, `overrideTicketPriority` (mutates `Ticket.priority`/`severity`
  based on the AI's assessment), exists in the registry specifically so SC-003 ("100% of
  high-risk proposals blocked from automatic execution... across every risk level") is actually
  exercised by a real tool, not vacuously true. Per FR-012, its risk tier requires the "stronger
  control path" — this feature implements that path as: the proposal and policy evaluation are
  recorded as `pending_approval` and the action never executes automatically. A human-approval UI
  is Phase 10 (Agent/Admin UI) work, out of scope here — this is the same class of explicitly
  documented known limitation as `fastify.authenticate`'s auth stub, not a silent gap.
  Per-product enablement uses each tool's static `supportedProducts` field (`"*"` for
  all-products tools, or an explicit external-product-id allowlist) — no separate DB table for
  tool enablement in this phase, since FR-010/FR-011 only require the scope to be *declared* and
  *checked*, not admin-editable without a deploy (unlike confidence thresholds, which FR-005
  explicitly requires to be).
- **Rationale**: Matches FR-010 (permission/risk/product-scope declared per tool) and FR-013
  (every proposal + evaluation + result durably recorded) exactly, while keeping the registry a
  plain, typed, code-reviewed artifact — appropriate for a small, fixed tool set, consistent with
  how Zod schemas generally aren't meant to be data-driven in this codebase. Reusing the
  fail-closed-placeholder pattern for `verifyProductResolution` is a direct, deliberate echo of
  an already-accepted precedent in this same codebase, not a new pattern being introduced.
- **Alternatives considered**: A DB-backed dynamic tool registry (schemas as JSON, admin-editable)
  — rejected as significant unrequired complexity; nothing in spec.md's FRs asks for tools
  themselves to be admin-configurable, only for thresholds (FR-005) to be. Inventing a working
  fake product API to make `verifyProductResolution` "really" verify something — rejected; would
  misrepresent evidence this system doesn't actually have, which is precisely what FR-018 exists
  to prevent.

## Decision: Deterministic policy gate — one shared evaluation function, ahead of any execution

- **Decision**: `evaluateToolProposal(tool, sessionContext)` is a single function every proposed
  tool call passes through before anything executes: checks (a) the tool exists in the registry,
  (b) `supportedProducts` includes the session's product, (c) risk level — `low` → approved
  automatically; `medium`/`high` → recorded `pending_approval`, not executed. The AI's own message
  text is never consulted by this function — only the tool name and the session's actual product/
  permission context (FR-011, FR-024).
- **Rationale**: This is Constitution Principle IV made concrete for this feature, and directly
  answers doc 11 §A4's prompt-injection concern: since the gate never reads free-text content,
  content injected into a customer message or attachment has no path to influence what executes,
  no matter how it's phrased.
- **Alternatives considered**: Per-tool ad hoc checks inside each tool's handler — rejected, same
  reasoning as the tool-runner rejection above: a shared gate is auditable and impossible to
  accidentally skip for a new tool; scattered checks are not.

## Decision: Session triggering — enqueued on ticket creation, advanced over HTTP per customer turn

- **Decision**: `TicketsService.createFromInboundRequest` (003-ticketing,
  `src/modules/ticketing/tickets/service/tickets.service.ts`) enqueues a
  `QueueName.AI_SESSION` job (`{ ticketId }`) right after a **new** ticket is created (not on an
  idempotent replay). A new worker (`src/jobs/ai-session/index.ts`, registered in
  `src/bootstrap/queue.bootstrap.ts` alongside the existing attachment worker) picks it up and
  runs the first diagnosis turn asynchronously. Every subsequent turn (a customer's reply, a
  runbook step's outcome) is driven by an explicit HTTP call —
  `POST /tickets/:ticketId/ai-session/messages` — which runs synchronously and returns the AI's
  next turn in the same response.
- **Rationale**: The first turn happens off the hot path of `POST /v1/support/requests` (002's
  inbound integration boundary) — that caller shouldn't wait on an LLM round-trip just to get a
  ticket-created acknowledgment, and this codebase already has a real, working queue+worker
  pattern for exactly this shape of "do this after the request returns" work (the attachment
  malware-scan worker). Subsequent turns are naturally request/response — a customer reply is
  already an HTTP call into this system (matching 003's existing `POST .../messages` shape), and
  there's no reason to make the AI's response to it async when the caller is already waiting for
  an HTTP response.
- **Alternatives considered**: Running the first diagnosis synchronously inside
  `createFromInboundRequest` — rejected; would add LLM latency (and a new failure mode: an LLM
  timeout) to every inbound SaaS integration request, which is the trust-boundary endpoint 002
  already established as latency-sensitive. Polling instead of a queued worker — rejected, this
  codebase already has BullMQ wired up for exactly this "background work after a DB write"
  purpose.

## Decision: `AISupportSession.status` drives `Ticket.status` through the existing state machine

- **Decision**: `src/modules/ticketing/tickets/mapper/ticket-state-machine.ts` (built in
  003-ticketing, before this feature existed) already defines `NEW → AI_ANALYZING →
  AI_TROUBLESHOOTING → AI_VERIFYING → AI_RESOLVED`, with `HUMAN_ESCALATION` reachable from every
  AI_* state and `HUMAN_ESCALATION → IN_PROGRESS` as the human hand-off edge — a near-exact match
  for doc 06's `AISupportSession.status` enum (`analyzing | troubleshooting | verifying |
  resolved | escalated`). This feature does not invent a parallel status concept: every time a
  session's own status changes, it calls the **existing**
  `ticketsService.updateStatus(ticketId, newTicketStatus, expectedVersion, 'ai')` (003) to drive
  the ticket through the matching status (`analyzing→AI_ANALYZING`,
  `troubleshooting→AI_TROUBLESHOOTING`, `verifying→AI_VERIFYING`, `resolved→AI_RESOLVED`,
  `escalated→HUMAN_ESCALATION`), reusing 003's own optimistic-concurrency handling
  (`expectedVersion`/`409`) rather than adding a second one. The session's own `status` field
  still exists separately (data-model.md) because it carries session-scoped values the ticket
  state machine doesn't need to know about (`ended_by_agent` — see FR-023 below — never appears
  on `Ticket`), but for every value the two share, the ticket is the caller-visible source of
  truth and the session record is the AI-internal detail behind it.
- **Rationale**: `Ticket.status` is what every other part of this codebase (agents, SLA,
  orchestration once built, the ticket-status contract in 003) already reads to know where a
  ticket stands — a session-only status field that never touched `Ticket.status` would make the
  ticket lie about its own state while an AI session was quietly doing something else internally.
  Reusing 003's state machine and its `updateStatus` method also means this feature inherits
  003's already-tested transition validation and concurrency guarantee for free, rather than
  re-deriving both.
- **Alternatives considered**: A session-only status with no `Ticket.status` linkage — rejected
  per the rationale above. Building a second, AI-specific transition table — rejected; 003's
  table already defines exactly these states and edges, and doc 06's `AISupportSession.status`
  values were clearly authored to match it in the first place.
- **FR-023 implementation note**: `ticketsService.updateStatus` gains one additional check — when
  it's called with an actor other than `'ai'` (a human agent action) while an `AISupportSession`
  for that ticket is still active, the session is ended (`status: ended_by_agent`) as part of the
  same call, before the ticket's own status update commits. This is the concrete mechanism behind
  "a human agent takes ownership ends the AI session the same way an escalation does" (FR-023) —
  there's no separate "agent claims ticket" endpoint yet (orchestration/assignment is a later
  phase), so any human-actor status transition is the signal this feature has available today.

## Decision: Runbook engine — the app selects the step, the model only phrases and interprets it

- **Decision**: `AISupportSession` gains `activeRunbookKey String?` and `currentStepIndex Int?`
  (refining doc 06's conceptual `AISupportSession`, same "refine during Phase 1 modeling"
  convention 004 already established for `KnowledgeEntry`/`Runbook`). When a diagnosis's
  `problemType` matches an active `Runbook.key` for the ticket's product (004's
  `RunbooksRepository.findCurrentByKey`), the session sets `activeRunbookKey` and
  `currentStepIndex = 0`. The **application** reads `steps[currentStepIndex]` from the runbook's
  JSON and passes only that one step's content into the reasoning call's prompt as an instruction
  ("present this step, then interpret the customer's response against it") — the model is never
  given the full step list or asked to choose a step. On an outcome that means "try the next
  step," the app increments `currentStepIndex` deterministically; the model cannot set or
  advance the index itself (it has no tool for that).
- **Rationale**: This is the literal requirement in FR-015/doc 03 §6 ("the AI cannot skip,
  reorder, or invent a step the runbook doesn't define") — the only way to guarantee that
  mechanically is for the index to be state the application owns and advances, with the model
  never seeing (and therefore never able to act on) any step but the current one.
- **Alternatives considered**: Giving the model the full runbook and trusting a system-prompt
  instruction ("only present steps in order") — rejected; doc 03 §6 explicitly says not to trust
  the LLM to "improvise" here, and a prompt instruction is not a guarantee, it's a request the
  model could depart from under distribution shift or adversarial input (doc 11 §A4).

## Decision: Verification and resolution — a session can only close on tool evidence

- **Decision**: `AISupportSession.status` (and, via the decision above, `Ticket.status`)
  transitions `verifying → resolved` (`AI_VERIFYING → AI_RESOLVED`) only when a
  `verifyProductResolution` tool result exists on the session with `confirmed: true` — which,
  given that tool's current fail-closed placeholder implementation (above), means resolution
  through this exact tool never actually auto-fires yet in this deployment. Customer confirmation
  (a message recorded during the session) is stored and surfaced in the escalation/resolution
  summary, but the status transition's guard checks tool evidence only, never message content.
  Once `Ticket.status` reaches `AI_RESOLVED`, this feature's own responsibility ends — whether the
  ticket then moves to `RESOLUTION_PENDING_CUSTOMER` or straight to `RESOLVED` is 003-ticketing's
  existing generic status-update surface, not something this feature further automates.
- **Rationale**: FR-018/FR-019 verbatim — resolution requires verification evidence, customer
  confirmation is secondary-only. Guarding the state transition on a structured tool-result field
  (not on parsing what the AI "said" about the outcome) keeps this enforceable in code, not just
  in prompt instructions.
- **Alternatives considered**: Letting the reasoning call's own structured output include a
  `resolved: boolean` field the app trusts — rejected; this delegates a MUST-level policy decision
  (FR-018) to the model's own judgment, exactly the inversion Constitution Principle IV forbids.

## Decision: Module placement — `sessions`, `tools`, `troubleshooting`, `escalation` under `ai-support`

- **Decision**: Four new submodules under the existing `src/modules/ai-support/` group (which
  004 created with only `knowledge` populated): `sessions/` (`AISupportSession`, `AIDiagnosis`,
  `AIInteraction`, `AIConfidencePolicy`, the two-call reasoning orchestration), `tools/` (the
  registry, the policy gate, `AIAction`/`AIActionResult`), `troubleshooting/` (the runbook-step
  engine, consuming 004's `RunbooksRepository` through `knowledge`'s public `index.ts`),
  `escalation/` (hand-off summary construction, ending a session, ticket-ownership hand-off).
  Doc 07's fuller list (`agents/diagnosis/tool-execution/verification`) is intentionally *not*
  built as separate submodules — `diagnosis` and `verification` are concerns inside `sessions`
  and `tools` respectively (an `AIDiagnosis` is produced *by* a session turn, not by an
  independent subsystem; verification is one tool's evaluated result, not a separate engine), and
  `tool-execution` is the same concern as `tools` split for no reason this feature's requirements
  give. `agents` (a registry of distinct AI "personas"/agent configs) has no requirement in
  spec.md at all — this feature has exactly one reasoning agent, not a multi-agent registry.
- **Rationale**: Same reasoning 004 already used for not pre-building every doc 07 submodule
  speculatively — build what the current feature's FRs actually require, not the full documented
  taxonomy ahead of need.
- **Alternatives considered**: One flat `ai-support/agent/` module holding everything — rejected;
  four genuinely distinct responsibilities (session/diagnosis orchestration, tool policy/
  execution, runbook stepping, escalation hand-off) benefit from the same
  controller/service/repository separation every other module in this codebase already uses, and
  cross-module imports must go through `index.ts` either way (Principle III) — collapsing them
  into one module wouldn't reduce real coupling, just hide the boundaries.

## Decision: Admin endpoint authentication and message visibility — reuse existing conventions

- **Decision**: The confidence-policy admin CRUD endpoint (`PUT
  /admin/products/:externalProductId/ai-policy`, optionally `.../categories/:categoryId/ai-policy`)
  is gated by `fastify.authenticate`, same known-limitation stub as every prior admin surface.
  AI-authored messages are written through 003-ticketing's existing `messagesService.post(...)`
  with `type: 'AI_MESSAGE'` — that type and its customer-visible mapping already exist in the
  message-type→visibility map (`specs/003-ticketing/research.md`); this feature adds no new
  message-visibility rule.
- **Rationale**: Consistency with established precedent; introducing a different auth mechanism
  or a parallel message-writing path for this feature alone would be unjustified inconsistency.
- **Alternatives considered**: None — direct reuse of existing, already-accepted conventions.