docs: task breakdown for AI support agent feature
53 tasks across setup, foundational schema/env/LLM-client work, and five user stories (diagnosis+confidence policy, clarification loop, gated tool system, runbook engine, evidence-based verification), plus the two constitution-required standing E2E scenarios in Polish. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
saqib mir
co-authored by
Claude Sonnet 5
Claude Sonnet 5
parent
eceb00632d
commit
49eaa4bc58
@@ -0,0 +1,419 @@
|
||||
---
|
||||
description: "Task list for 005-ai-support"
|
||||
---
|
||||
|
||||
# Tasks: AI Support Agent
|
||||
|
||||
**Input**: Design documents from `specs/005-ai-support/`
|
||||
|
||||
**Prerequisites**: [plan.md](./plan.md), [spec.md](./spec.md), [research.md](./research.md),
|
||||
[data-model.md](./data-model.md), [contracts/ai-support-contract.md](./contracts/ai-support-contract.md),
|
||||
[quickstart.md](./quickstart.md)
|
||||
|
||||
**Tests**: Included as first-class tasks. Unlike 004 (thin Prisma queries with no pure-logic
|
||||
surface), this feature has several genuinely extractable pure functions (confidence-band
|
||||
decision, tool policy gate, runbook step-advancement) that get real unit tests, plus integration
|
||||
tests that — for the first time in this codebase — depend on a real external network call (a real
|
||||
`ANTHROPIC_API_KEY`) and incur real per-run API cost, not just Docker-local infra. The
|
||||
constitution's Testing gate also requires this feature to add: AI tool-permission tests, and the
|
||||
two standing E2E scenarios (AI resolves directly / AI escalates to human) that were impossible
|
||||
before this feature existed.
|
||||
|
||||
**Organization**: Tasks are grouped by user story (US1 = P1 diagnosis/confidence-policy, US2 = P2
|
||||
clarification loop, US3 = P2 tool system, US4 = P3 runbook engine, US5 = P3 verification/
|
||||
resolution).
|
||||
|
||||
## Format: `[ID] [P?] [Story] Description`
|
||||
|
||||
All file paths are relative to `supporthub-api/` (repo root).
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Setup
|
||||
|
||||
- [ ] T001 [P] Scaffold `src/modules/ai-support/sessions/` and `src/modules/ai-support/tools/`
|
||||
with the standard module shape (`controller/`, `routes/`, `schema/`, `repository/`,
|
||||
`service/`, `types/`, `mapper/`, `constants/`, `index.ts`)
|
||||
- [ ] T002 [P] Scaffold `src/modules/ai-support/troubleshooting/` and
|
||||
`src/modules/ai-support/escalation/` with the reduced shape plan.md specifies for
|
||||
internal-only submodules (`service/`, `types/`, `index.ts` — no `routes/controller/schema`,
|
||||
since neither has its own HTTP surface)
|
||||
- [ ] T003 [P] Scaffold `src/infrastructure/ai/` (Anthropic client singleton) and
|
||||
`src/jobs/ai-session/` (empty worker module, populated in Phase 3)
|
||||
- [ ] T004 Add `@anthropic-ai/sdk` as a runtime dependency (`npm install @anthropic-ai/sdk`)
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Foundational (Blocking Prerequisites)
|
||||
|
||||
**Purpose**: Schema, env config, and the LLM client every user story depends on.
|
||||
|
||||
**⚠️ CRITICAL**: No user-story stage work can begin until this phase is complete.
|
||||
|
||||
- [ ] T005 Add `AISupportSession`, `AIDiagnosis`, `AIInteraction`, `AIAction`, `AIActionResult`,
|
||||
`AIKnowledgeReference`, `AIConfidencePolicy` models to `prisma/schema.prisma` per
|
||||
data-model.md, plus the `Ticket.aiSessions` back-relation (depends on T001-T003)
|
||||
- [ ] T006 Run `npm run prisma:generate` and create the migration (`npm run prisma:migrate`) for
|
||||
T005 (depends on T005)
|
||||
- [ ] T007 Add env vars to `src/config/env.ts`: `ANTHROPIC_API_KEY` (`z.string().optional()` —
|
||||
the app must still boot and every non-AI test must still pass without it; the Anthropic
|
||||
client wrapper (T008) is what throws a clear, explicit error if a reasoning call is
|
||||
attempted with it unset — spec.md Assumptions' "no offline fallback path" is enforced at
|
||||
the point of use, not by making every test fixture supply a fake credential),
|
||||
`AI_SUPPORT_MODEL` (default `claude-opus-5`), `AI_SUPPORT_EFFORT` (default `medium`),
|
||||
`AI_SUPPORT_DEFAULT_HIGH_CONFIDENCE` (default `0.75`),
|
||||
`AI_SUPPORT_DEFAULT_LOW_CONFIDENCE` (default `0.4`),
|
||||
`AI_SUPPORT_DEFAULT_MAX_CLARIFYING_QUESTIONS` (default `2`),
|
||||
`AI_SUPPORT_MAX_REASONING_ITERATIONS_PER_TURN` (default `4` — doc 11 §B2's runaway-loop
|
||||
cap on tool-call iterations within one reasoning turn). Mirror the new vars into
|
||||
`.env.example` (with `ANTHROPIC_API_KEY=CHANGE_ME` and a comment that a real key is
|
||||
required for this feature to function) — `.env.development`/`.env.test` are gitignored, so
|
||||
note in the task (not the repo) that the user must add a real key there themselves
|
||||
(depends on T004)
|
||||
- [ ] T008 Add the Anthropic client singleton in
|
||||
`src/infrastructure/ai/anthropic.client.ts` — constructs `new Anthropic()` (credential
|
||||
resolved from `ANTHROPIC_API_KEY` per the SDK's own env resolution), exports the
|
||||
configured `model`/`effort` from env, and a guard that throws a clear `AppError` if a
|
||||
reasoning call is attempted with no key configured (depends on T007)
|
||||
|
||||
**Checkpoint**: Schema migrated, env validated, LLM client ready. User stories can now be built.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: User Story 1 - The AI diagnoses a new ticket and confidence decides what happens next (Priority: P1) 🎯 MVP
|
||||
|
||||
**Goal**: A ticket's first AI turn — diagnose (real, structured, knowledge-grounded LLM call),
|
||||
apply the confidence-band policy, and reach one of proceed/ask/escalate, with `Ticket.status`
|
||||
correctly reflecting the outcome through the existing state machine.
|
||||
|
||||
**Independent Test**: Quickstart Scenario 1.
|
||||
|
||||
### Tests for User Story 1
|
||||
|
||||
- [ ] T009 [P] [US1] Unit tests for the confidence-band decision function — proceed/ask/escalate
|
||||
boundary values, most-specific-`(productId, categoryId)`-match-with-fallback resolution — in
|
||||
`tests/unit/ai-support/confidence-band.test.ts`
|
||||
- [ ] T010 [US1] Integration test covering Quickstart Scenario 1 (diagnosis recorded with
|
||||
confidence; low `highThreshold` still proceeds; high `lowThreshold` escalates; no-knowledge
|
||||
product escalates) against a real Postgres **and a real Anthropic API call** in
|
||||
`tests/integration/ai-diagnosis.test.ts` (depends on T006, T008; requires a real
|
||||
`ANTHROPIC_API_KEY` in the test environment to run)
|
||||
|
||||
### Implementation for User Story 1
|
||||
|
||||
- [ ] T011 [US1] Add `AIConfidencePolicyRepository`/`AIConfidencePolicyService`
|
||||
(most-specific-match lookup: `(productId, categoryId)` → `(productId, null)` → env
|
||||
defaults; upsert) in `src/modules/ai-support/sessions/repository/confidence-policy.repository.ts`
|
||||
+ `service/confidence-policy.service.ts` (depends on T006)
|
||||
- [ ] T012 [P] [US1] Add the pure confidence-band decision function
|
||||
`decideConfidenceBand(confidence, policy) => 'proceed' | 'ask' | 'escalate'` in
|
||||
`src/modules/ai-support/sessions/service/confidence-band.ts` (no dependencies — pure
|
||||
function, can be written and unit-tested in parallel with T011)
|
||||
- [ ] T013 [US1] Add `AISupportSessionRepository` (create; findActiveByTicketId — enforces
|
||||
FR-001's one-active-session rule; update status/runbook fields/counters) in
|
||||
`src/modules/ai-support/sessions/repository/session.repository.ts` (depends on T006)
|
||||
- [ ] T014 [US1] Add `AIDiagnosisRepository` (create; findLatestBySession) in
|
||||
`src/modules/ai-support/sessions/repository/diagnosis.repository.ts` (depends on T006)
|
||||
- [ ] T015 [US1] Add the diagnosis LLM call — `zodOutputFormat` schema matching data-model.md's
|
||||
`AIDiagnosis` shape, `client.messages.parse()` against the ticket's problem statement +
|
||||
conversation-so-far, no tools — in `src/modules/ai-support/sessions/service/diagnose.ts`
|
||||
(depends on T008)
|
||||
- [ ] T016 [US1] Add `EscalationService.escalate(sessionId, reason)` — builds the structured
|
||||
summary (problem, diagnosis, steps attempted so far, confidence — FR-021), calls
|
||||
`ticketsService.updateStatus(ticketId, 'HUMAN_ESCALATION', expectedVersion, 'ai')` (003,
|
||||
reused per research.md), ends the session (`status: escalated`) — in
|
||||
`src/modules/ai-support/escalation/service/escalation.service.ts` (depends on T013; this is
|
||||
needed by US1 itself, since "escalate" is one of US1's three outcomes — not deferred to a
|
||||
later story)
|
||||
- [ ] T017 [US1] Add `SessionsService.runFirstTurn(ticketId)`: create the session
|
||||
(`status: analyzing`, mirrored onto `Ticket.status: AI_ANALYZING` via
|
||||
`ticketsService.updateStatus(..., 'ai')`), run T015's diagnosis call, call
|
||||
`knowledgeService.retrieve(...)` (004, through `ai-support/knowledge`'s `index.ts` —
|
||||
research.md) and record `AIKnowledgeReference` rows for what was actually retrieved, escalate
|
||||
via T016 if no knowledge exists (FR-006) or apply T011/T012's confidence-band decision
|
||||
otherwise: `escalate` → T016; `ask` → produce a clarifying question (plain
|
||||
`client.messages.create()` call framing the diagnosis + retrieved knowledge, no tools yet)
|
||||
and post it via `messagesService.post(ticketId, 'ai', 'AI_MESSAGE', ...)` (003, reused);
|
||||
`proceed` → transition to `status: troubleshooting`
|
||||
(`Ticket.status: AI_TROUBLESHOOTING`) — full tool/runbook wiring for the proceed branch
|
||||
lands in US3/US4, so for this story `proceed` only needs to reach the correct status, not
|
||||
yet call any tool — in `src/modules/ai-support/sessions/service/session.service.ts`
|
||||
(depends on T011, T012, T013, T014, T015, T016)
|
||||
- [ ] T018 [US1] Add Zod schema + `PUT`/`GET /admin/products/:externalProductId/ai-policy` routes
|
||||
(gated by `fastify.authenticate`, per contracts/ai-support-contract.md) in
|
||||
`src/modules/ai-support/sessions/schema/` + `routes/`, registered from `src/api/routes.ts`
|
||||
(depends on T011)
|
||||
- [ ] T019 [US1] Add the `AI_SESSION` worker — `registerAiSessionWorker()` in
|
||||
`src/jobs/ai-session/index.ts`, calling `sessionsService.runFirstTurn(ticketId)` — and
|
||||
register it in `src/bootstrap/queue.bootstrap.ts` alongside the existing attachment worker
|
||||
(depends on T017)
|
||||
- [ ] T020 [US1] Modify `TicketsService.createFromInboundRequest`
|
||||
(`src/modules/ticketing/tickets/service/tickets.service.ts`) to enqueue a
|
||||
`QueueName.AI_SESSION` job (`{ ticketId: ticket.id }`) when `wasExisting` is `false`
|
||||
(research.md "Session triggering" — never on an idempotent replay) (depends on T019)
|
||||
- [ ] T021 [US1] Modify `TicketsService.updateStatus` to end any active `AISupportSession` for the
|
||||
ticket (`status: ended_by_agent`) when called with an actor other than `'ai'` — FR-023's
|
||||
concrete mechanism (research.md) — before the status update itself commits (depends on
|
||||
T013)
|
||||
- [ ] T022 [US1] Run Quickstart Scenario 1 locally (with a real `ANTHROPIC_API_KEY`) and confirm
|
||||
all 5 steps pass
|
||||
|
||||
**Checkpoint**: Every new ticket gets a real, knowledge-grounded diagnosis, and confidence
|
||||
correctly decides proceed/ask/escalate, with `Ticket.status` reflecting it. This alone is a
|
||||
usable automatic-triage surface even before clarification, tools, runbooks, or verification exist.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: User Story 2 - The AI asks a clarifying question and re-diagnoses from the answer (Priority: P2)
|
||||
|
||||
**Goal**: The "ask" branch becomes a real back-and-forth instead of a dead end — a customer reply
|
||||
triggers a new diagnosis, with the confidence-band policy re-applied and the question-count cap
|
||||
enforced.
|
||||
|
||||
**Independent Test**: Quickstart Scenario 2.
|
||||
|
||||
### Tests for User Story 2
|
||||
|
||||
- [ ] T023 [US2] Integration test covering Quickstart Scenario 2 (question posted as
|
||||
customer-visible `AI_MESSAGE`; reply triggers a second `AIDiagnosis`; policy re-applied;
|
||||
question cap reached → escalate) against a real Postgres and a real Anthropic API call in
|
||||
`tests/integration/ai-clarification.test.ts` (depends on T022)
|
||||
|
||||
### Implementation for User Story 2
|
||||
|
||||
- [ ] T024 [US2] Add Zod schema + `POST /tickets/:ticketId/ai-session/messages` route (`404` if
|
||||
no active session — contracts/ai-support-contract.md guarantee 1) in
|
||||
`src/modules/ai-support/sessions/schema/` + `routes/` + `controller/` (depends on T017)
|
||||
- [ ] T025 [US2] Add `SessionsService.handleCustomerReply(ticketId, message)`: record the reply
|
||||
as an `AIInteraction` (`role: customer`) and a `TicketMessage` (`type: CUSTOMER_MESSAGE`,
|
||||
via the existing `messagesService`), re-run T015's diagnosis call with the full
|
||||
conversation, increment `clarifyingQuestionsAsked` when continuing from an `ask` outcome,
|
||||
escalate via T016 once `clarifyingQuestionsAsked >= policy.maxClarifyingQuestions` (FR-009)
|
||||
regardless of the new diagnosis's own confidence, otherwise re-apply the confidence-band
|
||||
decision as in T017 — in `src/modules/ai-support/sessions/service/session.service.ts`
|
||||
(depends on T024)
|
||||
- [ ] T026 [US2] Add Zod schema + `GET /tickets/:ticketId/ai-session` read route (status, latest
|
||||
diagnosis, interaction history) in `sessions/schema/` + `routes/` + `controller/` (depends
|
||||
on T013, T014)
|
||||
- [ ] T027 [US2] Run Quickstart Scenario 2 locally and confirm it passes
|
||||
|
||||
**Checkpoint**: US1 and US2 together deliver a full diagnose → clarify → re-diagnose loop with a
|
||||
correctly enforced question budget.
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: User Story 3 - The AI proposes tool calls; the application decides whether to run them (Priority: P2)
|
||||
|
||||
**Goal**: The `proceed` branch actually does something — the AI can request real, permission/
|
||||
risk-gated tool calls, evaluated by one shared deterministic gate the AI's own text can never
|
||||
influence.
|
||||
|
||||
**Independent Test**: Quickstart Scenario 3. This story also satisfies the constitution's
|
||||
required "AI tool-permission tests" category (Testing, Observability & CI/CD Gates).
|
||||
|
||||
### Tests for User Story 3
|
||||
|
||||
- [ ] T028 [P] [US3] Unit tests for `evaluateToolProposal` — unknown tool refused, product not in
|
||||
`supportedProducts` refused, low-risk auto-approved, medium/high-risk `pending_approval`,
|
||||
and an explicit case asserting the AI's own proposal/justification text is never read by
|
||||
the gate (prompt-injection resistance, FR-024) — in
|
||||
`tests/unit/ai-support/tool-policy-gate.test.ts`
|
||||
- [ ] T029 [US3] Integration test covering Quickstart Scenario 3 (low-risk tool executes and is
|
||||
recorded; `overrideTicketPriority` proposal is `pending_approval` with no result; an
|
||||
out-of-scope tool proposal is `refused`) against a real Postgres and a real Anthropic API
|
||||
call in `tests/integration/ai-tool-actions.test.ts` (depends on T022)
|
||||
|
||||
### Implementation for User Story 3
|
||||
|
||||
- [ ] T030 [P] [US3] Add the tool registry — `getTicketSnapshot` (low), `searchProductKnowledge`
|
||||
(low), `verifyProductResolution` (low, fail-closed placeholder — research.md, mirrors
|
||||
`UnimplementedPlaceholderScanner`), `escalateToHuman` (low, always policy-approved),
|
||||
`overrideTicketPriority` (high) — with Zod input schemas, `permission`, `riskLevel`,
|
||||
`supportedProducts`, `auditRequired` per tool — in
|
||||
`src/modules/ai-support/tools/constants/tool-registry.ts` (depends on T003)
|
||||
- [ ] T031 [US3] Add `evaluateToolProposal(toolName, sessionContext)` — the shared deterministic
|
||||
gate (research.md): unknown-tool / out-of-scope → `refused`; `low` → `approved`;
|
||||
`medium`/`high` → `pending_approval`. Reads only the tool name and session's product/
|
||||
permission context, never the AI's proposal text — in
|
||||
`src/modules/ai-support/tools/service/policy-gate.ts` (depends on T030)
|
||||
- [ ] T032 [US3] Add `AIActionRepository`/`AIActionResultRepository` (create action + evaluation
|
||||
outcome; create result when executed; list by session, newest first) in
|
||||
`src/modules/ai-support/tools/repository/` (depends on T006)
|
||||
- [ ] T033 [US3] Add real tool execution handlers — `getTicketSnapshot` (reads
|
||||
`Ticket`+`Problem`+recent `TicketMessage`s via existing repositories), `searchProductKnowledge`
|
||||
(calls `knowledgeService.retrieve(...)`), `verifyProductResolution` (always returns
|
||||
`{ confirmed: false, status: 'unknown' }` — documented placeholder), `escalateToHuman`
|
||||
(calls T016's `EscalationService.escalate`) — in
|
||||
`src/modules/ai-support/tools/service/tool-executor.ts` (depends on T031; `overrideTicketPriority`
|
||||
has no execution handler yet — it can never reach `approved`, so it's never called)
|
||||
- [ ] T034 [US3] Add `ToolsService.proposeAndEvaluate(sessionId, toolUseBlocks)`: for each
|
||||
proposed `tool_use` block, run T031's gate, persist the `AIAction` (T032), execute + persist
|
||||
an `AIActionResult` (T032) only when `approved`, and increment the failure count toward
|
||||
escalation triggers on a failed result (FR-014) — in
|
||||
`src/modules/ai-support/tools/service/tools.service.ts` (depends on T032, T033)
|
||||
- [ ] T035 [US3] Wire the reasoning/response call into `SessionsService`'s `proceed` branch
|
||||
(research.md "two calls per reasoning turn," step 3): a `client.messages.create()` call
|
||||
with the tool registry's `Anthropic.Tool[]` definitions, the diagnosis + retrieved
|
||||
knowledge + conversation as context; route any `tool_use` blocks through T034, feed
|
||||
`tool_result` blocks back for a follow-up call, capped at
|
||||
`AI_SUPPORT_MAX_REASONING_ITERATIONS_PER_TURN` iterations (doc 11 §B2) — in
|
||||
`src/modules/ai-support/sessions/service/session.service.ts` (depends on T034)
|
||||
- [ ] T036 [US3] Add Zod schema + `GET /tickets/:ticketId/ai-session/actions` route in
|
||||
`sessions/schema/` + `routes/` + `controller/` (or `tools/` — whichever module owns the
|
||||
route registers it; the data comes from T032's repository either way) (depends on T032)
|
||||
- [ ] T037 [US3] Run Quickstart Scenario 3 locally and confirm it passes
|
||||
|
||||
**Checkpoint**: US1-US3 together deliver diagnose → clarify → act-through-gated-tools, with every
|
||||
proposal, decision, and result durably recorded and auditable.
|
||||
|
||||
---
|
||||
|
||||
## Phase 6: User Story 4 - A matching runbook drives guided troubleshooting, not the AI's own improvisation (Priority: P3)
|
||||
|
||||
**Goal**: When a diagnosis matches a runbook, the application — not the model — controls which
|
||||
step is next.
|
||||
|
||||
**Independent Test**: Quickstart Scenario 4.
|
||||
|
||||
### Tests for User Story 4
|
||||
|
||||
- [ ] T038 [P] [US4] Unit tests for the pure runbook step-advancement function — advances by
|
||||
exactly one on a "try next step" outcome, reports exhaustion after the last step, never
|
||||
skips or resets — in `tests/unit/ai-support/runbook-step-advance.test.ts`
|
||||
- [ ] T039 [US4] Integration test covering Quickstart Scenario 4 (matching runbook sets
|
||||
`activeRunbookKey`/`currentStepIndex: 0`; advances exactly one step per turn; exhaustion
|
||||
escalates with every attempted step listed) against a real Postgres and a real Anthropic
|
||||
API call in `tests/integration/ai-runbook-troubleshooting.test.ts` (depends on T022)
|
||||
|
||||
### Implementation for User Story 4
|
||||
|
||||
- [ ] T040 [P] [US4] Add the pure step-advancement function
|
||||
`advanceRunbookStep(steps, currentStepIndex, outcome) => { nextIndex } | { exhausted: true }`
|
||||
in `src/modules/ai-support/troubleshooting/service/step-advance.ts` (no dependencies)
|
||||
- [ ] T041 [US4] Add `RunbookEngineService.matchRunbook(problemType, productId)` — calls
|
||||
`runbooksService.findCurrentByKey(...)` (004, through `ai-support/knowledge`'s `index.ts`)
|
||||
— in `src/modules/ai-support/troubleshooting/service/runbook-engine.service.ts` (depends on
|
||||
T003)
|
||||
- [ ] T042 [US4] Wire T041/T040 into `SessionsService`: after a `proceed` diagnosis, attempt
|
||||
T041's match; if found, set `activeRunbookKey`/`currentStepIndex: 0` on the session; the
|
||||
reasoning call (T035) receives **only** `steps[currentStepIndex]` in its prompt context,
|
||||
never the full list; a customer's step-outcome reply advances via T040, and exhaustion
|
||||
(`{ exhausted: true }`) escalates via T016 with every attempted step in the summary
|
||||
(FR-016) — in `src/modules/ai-support/sessions/service/session.service.ts` (depends on
|
||||
T041)
|
||||
- [ ] T043 [US4] Run Quickstart Scenario 4 locally and confirm it passes
|
||||
|
||||
**Checkpoint**: When a runbook matches, troubleshooting follows its authored order exactly — the
|
||||
model presents and interprets, the application sequences.
|
||||
|
||||
---
|
||||
|
||||
## Phase 7: User Story 5 - A ticket is only marked AI-resolved when there's real evidence, not a customer's claim alone (Priority: P3)
|
||||
|
||||
**Goal**: The `resolved`/`AI_RESOLVED` transition is guarded on real tool evidence, never on
|
||||
message content.
|
||||
|
||||
**Independent Test**: Quickstart Scenario 5. This story is also required to complete the
|
||||
constitution's standing "(A) AI resolves directly" E2E scenario (Phase 8 wires the test itself,
|
||||
since it depends on every prior story existing).
|
||||
|
||||
### Tests for User Story 5
|
||||
|
||||
- [ ] T044 [US5] Integration test covering Quickstart Scenario 5 (customer claims fixed with no
|
||||
tool evidence → not resolved; `verifyProductResolution`'s placeholder never confirms →
|
||||
session doesn't auto-resolve) against a real Postgres and a real Anthropic API call in
|
||||
`tests/integration/ai-verification.test.ts` (depends on T022)
|
||||
|
||||
### Implementation for User Story 5
|
||||
|
||||
- [ ] T045 [US5] Add the resolution guard in `SessionsService`: a session only transitions
|
||||
`troubleshooting/verifying → resolved` (mirrored to `Ticket.status: AI_VERIFYING →
|
||||
AI_RESOLVED`) when an `AIActionResult` from `verifyProductResolution` with
|
||||
`confirmed: true` exists for the session (T032's repository) — a customer's "it's fixed"
|
||||
reply is recorded as an `AIInteraction` only and never inspected by this guard (FR-018/
|
||||
FR-019) — in `src/modules/ai-support/sessions/service/session.service.ts` (depends on T033)
|
||||
- [ ] T046 [US5] Run Quickstart Scenario 5 locally and confirm it passes
|
||||
|
||||
**Checkpoint**: All five user stories work independently and together — the full diagnose →
|
||||
clarify → act → troubleshoot → verify flow, with policy (never the model) deciding every
|
||||
MUST-level outcome.
|
||||
|
||||
---
|
||||
|
||||
## Phase 8: Polish & Cross-Cutting Concerns
|
||||
|
||||
- [ ] T047 [P] Integration test for the prompt-injection edge case (quickstart.md — a customer
|
||||
reply containing an instruction-like string never changes `evaluationOutcome` for a
|
||||
subsequent high-risk proposal) in `tests/integration/ai-tool-actions.test.ts` (extends
|
||||
T029's file) or a new `tests/integration/ai-prompt-injection.test.ts` (depends on T037)
|
||||
- [ ] T048 The constitution's standing E2E scenario (A) — "AI resolves directly: problem →
|
||||
knowledge → guided troubleshooting → verification → AI-resolved" — as a single, real,
|
||||
end-to-end integration test spanning US1/US3/US4/US5 in one session in
|
||||
`tests/integration/e2e-ai-resolves.test.ts` (depends on T022, T037, T043, T046)
|
||||
- [ ] T049 The constitution's standing E2E scenario (B) — "AI escalates to human: problem →
|
||||
failed AI troubleshooting → escalation → [ticket reaches HUMAN_ESCALATION, ready for
|
||||
orchestration/assignment when that phase exists]" — in
|
||||
`tests/integration/e2e-ai-escalates.test.ts` (depends on T022, T037, T043)
|
||||
- [ ] T050 [P] Add an "AI Support" section to `README.md` describing the session lifecycle, the
|
||||
confidence-policy config surface, the tool registry (including the two documented known
|
||||
limitations: `verifyProductResolution`'s fail-closed placeholder and
|
||||
`overrideTicketPriority`'s permanently-`pending_approval` state pending a future approval
|
||||
UI), and the runbook engine
|
||||
- [ ] T051 [P] Update `specs/005-ai-support/checklists/requirements.md` Notes with any
|
||||
implementation-time findings
|
||||
- [ ] T052 Run `npx tsx scripts/check-architecture.ts` and `npm run lint`/`npm run typecheck`
|
||||
- [ ] T053 Full regression: `npm run test:unit` (scoped to `tests/unit`) to confirm nothing broke
|
||||
elsewhere
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Execution Order
|
||||
|
||||
### Phase Dependencies
|
||||
|
||||
- **Setup (Phase 1)**: No dependencies
|
||||
- **Foundational (Phase 2)**: Depends on Setup — BLOCKS all user stories
|
||||
- **User Story 1 (Phase 3)**: Depends on Foundational — no dependency on US2-US5
|
||||
- **User Story 2 (Phase 4)**: Depends on US1 (extends `SessionsService`, reuses T015-T017) —
|
||||
genuinely not independent of US1's diagnosis/session machinery, unlike 004's stories
|
||||
- **User Story 3 (Phase 5)**: Depends on US1 (the `proceed` branch it fills in) — independent of
|
||||
US2's clarification loop
|
||||
- **User Story 4 (Phase 6)**: Depends on US1 (the `proceed` branch) and benefits from, but doesn't
|
||||
strictly require, US3 (a runbook step could in principle need a tool call — not required by any
|
||||
FR here, so US4 doesn't block on US3 completing)
|
||||
- **User Story 5 (Phase 7)**: Depends on US3 (needs `verifyProductResolution`'s execution handler,
|
||||
T033) and US1's session status machinery
|
||||
- **Polish (Phase 8)**: Depends on all five user stories
|
||||
|
||||
### Parallel Opportunities
|
||||
|
||||
- T001/T002/T003 (independent scaffolding)
|
||||
- T009 alongside T011-T016 once T006 exists (unit test doesn't need the real implementation)
|
||||
- T012 (pure function) can be written independently of T011
|
||||
- T028 alongside T030-T034
|
||||
- T030 (registry) alongside T031's early drafting, though T031 needs T030's exports to compile
|
||||
- T038 (pure function) independent of T041
|
||||
- T047/T050/T051 in Polish
|
||||
|
||||
---
|
||||
|
||||
## Implementation Strategy
|
||||
|
||||
### MVP First (User Story 1 Only)
|
||||
|
||||
1. Setup + Foundational (T001-T008)
|
||||
2. User Story 1 (T009-T022)
|
||||
3. **STOP and VALIDATE**: Quickstart Scenario 1 passes — every new ticket gets a real,
|
||||
knowledge-grounded diagnosis, and confidence correctly decides proceed/ask/escalate, with
|
||||
`Ticket.status` reflecting it end to end. Usable automatic-triage value even before
|
||||
clarification, tools, runbooks, or verification exist.
|
||||
|
||||
### Incremental Delivery
|
||||
|
||||
1. Setup + Foundational → schema migrated, LLM client ready
|
||||
2. Add User Story 1 → every ticket gets a real AI diagnosis (MVP)
|
||||
3. Add User Story 2 → the "ask" branch becomes a real conversation
|
||||
4. Add User Story 3 → the "proceed" branch can act, through a gate the AI can't talk its way past
|
||||
5. Add User Story 4 → matched problems get consistent, product-approved troubleshooting sequences
|
||||
6. Add User Story 5 → resolution requires real evidence, closing the loop safely
|
||||
7. Polish → the two constitution-required standing E2E scenarios, docs, full regression
|
||||
Reference in New Issue
Block a user