maskantech/support_backend
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
49eaa4bc583b0611874c926d214b9b0868ac0f38
support_backend/specs/005-ai-support/tasks.md
T
saqib mirandClaude Sonnet 5 49eaa4bc58 docs: task breakdown for AI support agent feature 
53 tasks across setup, foundational schema/env/LLM-client work, and five
user stories (diagnosis+confidence policy, clarification loop, gated tool
system, runbook engine, evidence-based verification), plus the two
constitution-required standing E2E scenarios in Polish.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-02 16:56:42 +05:30

25 KiB
Raw Blame History

description
description
Task list for 005-ai-support

Tasks: AI Support Agent

Input: Design documents from specs/005-ai-support/

Prerequisites: plan.md, spec.md, research.md, data-model.md, contracts/ai-support-contract.md, quickstart.md

Tests: Included as first-class tasks. Unlike 004 (thin Prisma queries with no pure-logic surface), this feature has several genuinely extractable pure functions (confidence-band decision, tool policy gate, runbook step-advancement) that get real unit tests, plus integration tests that — for the first time in this codebase — depend on a real external network call (a real ANTHROPIC_API_KEY) and incur real per-run API cost, not just Docker-local infra. The constitution's Testing gate also requires this feature to add: AI tool-permission tests, and the two standing E2E scenarios (AI resolves directly / AI escalates to human) that were impossible before this feature existed.

Organization: Tasks are grouped by user story (US1 = P1 diagnosis/confidence-policy, US2 = P2 clarification loop, US3 = P2 tool system, US4 = P3 runbook engine, US5 = P3 verification/ resolution).

Format: [ID] [P?] [Story] Description

All file paths are relative to supporthub-api/ (repo root).

Phase 1: Setup

  • T001 [P] Scaffold src/modules/ai-support/sessions/ and src/modules/ai-support/tools/ with the standard module shape (controller/, routes/, schema/, repository/, service/, types/, mapper/, constants/, index.ts)
  • T002 [P] Scaffold src/modules/ai-support/troubleshooting/ and src/modules/ai-support/escalation/ with the reduced shape plan.md specifies for internal-only submodules (service/, types/, index.ts — no routes/controller/schema, since neither has its own HTTP surface)
  • T003 [P] Scaffold src/infrastructure/ai/ (Anthropic client singleton) and src/jobs/ai-session/ (empty worker module, populated in Phase 3)
  • T004 Add @anthropic-ai/sdk as a runtime dependency (npm install @anthropic-ai/sdk)

Phase 2: Foundational (Blocking Prerequisites)

Purpose: Schema, env config, and the LLM client every user story depends on.

⚠️ CRITICAL: No user-story stage work can begin until this phase is complete.

  • T005 Add AISupportSession, AIDiagnosis, AIInteraction, AIAction, AIActionResult, AIKnowledgeReference, AIConfidencePolicy models to prisma/schema.prisma per data-model.md, plus the Ticket.aiSessions back-relation (depends on T001-T003)
  • T006 Run npm run prisma:generate and create the migration (npm run prisma:migrate) for T005 (depends on T005)
  • T007 Add env vars to src/config/env.ts: ANTHROPIC_API_KEY (z.string().optional() — the app must still boot and every non-AI test must still pass without it; the Anthropic client wrapper (T008) is what throws a clear, explicit error if a reasoning call is attempted with it unset — spec.md Assumptions' "no offline fallback path" is enforced at the point of use, not by making every test fixture supply a fake credential), AI_SUPPORT_MODEL (default claude-opus-5), AI_SUPPORT_EFFORT (default medium), AI_SUPPORT_DEFAULT_HIGH_CONFIDENCE (default 0.75), AI_SUPPORT_DEFAULT_LOW_CONFIDENCE (default 0.4), AI_SUPPORT_DEFAULT_MAX_CLARIFYING_QUESTIONS (default 2), AI_SUPPORT_MAX_REASONING_ITERATIONS_PER_TURN (default 4 — doc 11 §B2's runaway-loop cap on tool-call iterations within one reasoning turn). Mirror the new vars into .env.example (with ANTHROPIC_API_KEY=CHANGE_ME and a comment that a real key is required for this feature to function) — .env.development/.env.test are gitignored, so note in the task (not the repo) that the user must add a real key there themselves (depends on T004)
  • T008 Add the Anthropic client singleton in src/infrastructure/ai/anthropic.client.ts — constructs new Anthropic() (credential resolved from ANTHROPIC_API_KEY per the SDK's own env resolution), exports the configured model/effort from env, and a guard that throws a clear AppError if a reasoning call is attempted with no key configured (depends on T007)

Checkpoint: Schema migrated, env validated, LLM client ready. User stories can now be built.

Phase 3: User Story 1 - The AI diagnoses a new ticket and confidence decides what happens next (Priority: P1) 🎯 MVP

Goal: A ticket's first AI turn — diagnose (real, structured, knowledge-grounded LLM call), apply the confidence-band policy, and reach one of proceed/ask/escalate, with Ticket.status correctly reflecting the outcome through the existing state machine.

Independent Test: Quickstart Scenario 1.

Tests for User Story 1

  • T009 [P] [US1] Unit tests for the confidence-band decision function — proceed/ask/escalate boundary values, most-specific-(productId, categoryId)-match-with-fallback resolution — in tests/unit/ai-support/confidence-band.test.ts
  • T010 [US1] Integration test covering Quickstart Scenario 1 (diagnosis recorded with confidence; low highThreshold still proceeds; high lowThreshold escalates; no-knowledge product escalates) against a real Postgres and a real Anthropic API call in tests/integration/ai-diagnosis.test.ts (depends on T006, T008; requires a real ANTHROPIC_API_KEY in the test environment to run)

Implementation for User Story 1

  • T011 [US1] Add AIConfidencePolicyRepository/AIConfidencePolicyService (most-specific-match lookup: (productId, categoryId) → (productId, null) → env defaults; upsert) in src/modules/ai-support/sessions/repository/confidence-policy.repository.ts + service/confidence-policy.service.ts (depends on T006)
  • T012 [P] [US1] Add the pure confidence-band decision function decideConfidenceBand(confidence, policy) => 'proceed' | 'ask' | 'escalate' in src/modules/ai-support/sessions/service/confidence-band.ts (no dependencies — pure function, can be written and unit-tested in parallel with T011)
  • T013 [US1] Add AISupportSessionRepository (create; findActiveByTicketId — enforces FR-001's one-active-session rule; update status/runbook fields/counters) in src/modules/ai-support/sessions/repository/session.repository.ts (depends on T006)
  • T014 [US1] Add AIDiagnosisRepository (create; findLatestBySession) in src/modules/ai-support/sessions/repository/diagnosis.repository.ts (depends on T006)
  • T015 [US1] Add the diagnosis LLM call — zodOutputFormat schema matching data-model.md's AIDiagnosis shape, client.messages.parse() against the ticket's problem statement + conversation-so-far, no tools — in src/modules/ai-support/sessions/service/diagnose.ts (depends on T008)
  • T016 [US1] Add EscalationService.escalate(sessionId, reason) — builds the structured summary (problem, diagnosis, steps attempted so far, confidence — FR-021), calls ticketsService.updateStatus(ticketId, 'HUMAN_ESCALATION', expectedVersion, 'ai') (003, reused per research.md), ends the session (status: escalated) — in src/modules/ai-support/escalation/service/escalation.service.ts (depends on T013; this is needed by US1 itself, since "escalate" is one of US1's three outcomes — not deferred to a later story)
  • T017 [US1] Add SessionsService.runFirstTurn(ticketId): create the session (status: analyzing, mirrored onto Ticket.status: AI_ANALYZING via ticketsService.updateStatus(..., 'ai')), run T015's diagnosis call, call knowledgeService.retrieve(...) (004, through ai-support/knowledge's index.ts — research.md) and record AIKnowledgeReference rows for what was actually retrieved, escalate via T016 if no knowledge exists (FR-006) or apply T011/T012's confidence-band decision otherwise: escalate → T016; ask → produce a clarifying question (plain client.messages.create() call framing the diagnosis + retrieved knowledge, no tools yet) and post it via messagesService.post(ticketId, 'ai', 'AI_MESSAGE', ...) (003, reused); proceed → transition to status: troubleshooting (Ticket.status: AI_TROUBLESHOOTING) — full tool/runbook wiring for the proceed branch lands in US3/US4, so for this story proceed only needs to reach the correct status, not yet call any tool — in src/modules/ai-support/sessions/service/session.service.ts (depends on T011, T012, T013, T014, T015, T016)
  • T018 [US1] Add Zod schema + PUT/GET /admin/products/:externalProductId/ai-policy routes (gated by fastify.authenticate, per contracts/ai-support-contract.md) in src/modules/ai-support/sessions/schema/ + routes/, registered from src/api/routes.ts (depends on T011)
  • T019 [US1] Add the AI_SESSION worker — registerAiSessionWorker() in src/jobs/ai-session/index.ts, calling sessionsService.runFirstTurn(ticketId) — and register it in src/bootstrap/queue.bootstrap.ts alongside the existing attachment worker (depends on T017)
  • T020 [US1] Modify TicketsService.createFromInboundRequest (src/modules/ticketing/tickets/service/tickets.service.ts) to enqueue a QueueName.AI_SESSION job ({ ticketId: ticket.id }) when wasExisting is false (research.md "Session triggering" — never on an idempotent replay) (depends on T019)
  • T021 [US1] Modify TicketsService.updateStatus to end any active AISupportSession for the ticket (status: ended_by_agent) when called with an actor other than 'ai' — FR-023's concrete mechanism (research.md) — before the status update itself commits (depends on T013)
  • T022 [US1] Run Quickstart Scenario 1 locally (with a real ANTHROPIC_API_KEY) and confirm all 5 steps pass

Checkpoint: Every new ticket gets a real, knowledge-grounded diagnosis, and confidence correctly decides proceed/ask/escalate, with Ticket.status reflecting it. This alone is a usable automatic-triage surface even before clarification, tools, runbooks, or verification exist.

Phase 4: User Story 2 - The AI asks a clarifying question and re-diagnoses from the answer (Priority: P2)

Goal: The "ask" branch becomes a real back-and-forth instead of a dead end — a customer reply triggers a new diagnosis, with the confidence-band policy re-applied and the question-count cap enforced.

Independent Test: Quickstart Scenario 2.

Tests for User Story 2

  • T023 [US2] Integration test covering Quickstart Scenario 2 (question posted as customer-visible AI_MESSAGE; reply triggers a second AIDiagnosis; policy re-applied; question cap reached → escalate) against a real Postgres and a real Anthropic API call in tests/integration/ai-clarification.test.ts (depends on T022)

Implementation for User Story 2

  • T024 [US2] Add Zod schema + POST /tickets/:ticketId/ai-session/messages route (404 if no active session — contracts/ai-support-contract.md guarantee 1) in src/modules/ai-support/sessions/schema/ + routes/ + controller/ (depends on T017)
  • T025 [US2] Add SessionsService.handleCustomerReply(ticketId, message): record the reply as an AIInteraction (role: customer) and a TicketMessage (type: CUSTOMER_MESSAGE, via the existing messagesService), re-run T015's diagnosis call with the full conversation, increment clarifyingQuestionsAsked when continuing from an ask outcome, escalate via T016 once clarifyingQuestionsAsked >= policy.maxClarifyingQuestions (FR-009) regardless of the new diagnosis's own confidence, otherwise re-apply the confidence-band decision as in T017 — in src/modules/ai-support/sessions/service/session.service.ts (depends on T024)
  • T026 [US2] Add Zod schema + GET /tickets/:ticketId/ai-session read route (status, latest diagnosis, interaction history) in sessions/schema/ + routes/ + controller/ (depends on T013, T014)
  • T027 [US2] Run Quickstart Scenario 2 locally and confirm it passes

Checkpoint: US1 and US2 together deliver a full diagnose → clarify → re-diagnose loop with a correctly enforced question budget.

Phase 5: User Story 3 - The AI proposes tool calls; the application decides whether to run them (Priority: P2)

Goal: The proceed branch actually does something — the AI can request real, permission/ risk-gated tool calls, evaluated by one shared deterministic gate the AI's own text can never influence.

Independent Test: Quickstart Scenario 3. This story also satisfies the constitution's required "AI tool-permission tests" category (Testing, Observability & CI/CD Gates).

Tests for User Story 3

  • T028 [P] [US3] Unit tests for evaluateToolProposal — unknown tool refused, product not in supportedProducts refused, low-risk auto-approved, medium/high-risk pending_approval, and an explicit case asserting the AI's own proposal/justification text is never read by the gate (prompt-injection resistance, FR-024) — in tests/unit/ai-support/tool-policy-gate.test.ts
  • T029 [US3] Integration test covering Quickstart Scenario 3 (low-risk tool executes and is recorded; overrideTicketPriority proposal is pending_approval with no result; an out-of-scope tool proposal is refused) against a real Postgres and a real Anthropic API call in tests/integration/ai-tool-actions.test.ts (depends on T022)

Implementation for User Story 3

  • T030 [P] [US3] Add the tool registry — getTicketSnapshot (low), searchProductKnowledge (low), verifyProductResolution (low, fail-closed placeholder — research.md, mirrors UnimplementedPlaceholderScanner), escalateToHuman (low, always policy-approved), overrideTicketPriority (high) — with Zod input schemas, permission, riskLevel, supportedProducts, auditRequired per tool — in src/modules/ai-support/tools/constants/tool-registry.ts (depends on T003)
  • T031 [US3] Add evaluateToolProposal(toolName, sessionContext) — the shared deterministic gate (research.md): unknown-tool / out-of-scope → refused; low → approved; medium/high → pending_approval. Reads only the tool name and session's product/ permission context, never the AI's proposal text — in src/modules/ai-support/tools/service/policy-gate.ts (depends on T030)
  • T032 [US3] Add AIActionRepository/AIActionResultRepository (create action + evaluation outcome; create result when executed; list by session, newest first) in src/modules/ai-support/tools/repository/ (depends on T006)
  • T033 [US3] Add real tool execution handlers — getTicketSnapshot (reads Ticket+Problem+recent TicketMessages via existing repositories), searchProductKnowledge (calls knowledgeService.retrieve(...)), verifyProductResolution (always returns { confirmed: false, status: 'unknown' } — documented placeholder), escalateToHuman (calls T016's EscalationService.escalate) — in src/modules/ai-support/tools/service/tool-executor.ts (depends on T031; overrideTicketPriority has no execution handler yet — it can never reach approved, so it's never called)
  • T034 [US3] Add ToolsService.proposeAndEvaluate(sessionId, toolUseBlocks): for each proposed tool_use block, run T031's gate, persist the AIAction (T032), execute + persist an AIActionResult (T032) only when approved, and increment the failure count toward escalation triggers on a failed result (FR-014) — in src/modules/ai-support/tools/service/tools.service.ts (depends on T032, T033)
  • T035 [US3] Wire the reasoning/response call into SessionsService's proceed branch (research.md "two calls per reasoning turn," step 3): a client.messages.create() call with the tool registry's Anthropic.Tool[] definitions, the diagnosis + retrieved knowledge + conversation as context; route any tool_use blocks through T034, feed tool_result blocks back for a follow-up call, capped at AI_SUPPORT_MAX_REASONING_ITERATIONS_PER_TURN iterations (doc 11 §B2) — in src/modules/ai-support/sessions/service/session.service.ts (depends on T034)
  • T036 [US3] Add Zod schema + GET /tickets/:ticketId/ai-session/actions route in sessions/schema/ + routes/ + controller/ (or tools/ — whichever module owns the route registers it; the data comes from T032's repository either way) (depends on T032)
  • T037 [US3] Run Quickstart Scenario 3 locally and confirm it passes

Checkpoint: US1-US3 together deliver diagnose → clarify → act-through-gated-tools, with every proposal, decision, and result durably recorded and auditable.

Phase 6: User Story 4 - A matching runbook drives guided troubleshooting, not the AI's own improvisation (Priority: P3)

Goal: When a diagnosis matches a runbook, the application — not the model — controls which step is next.

Independent Test: Quickstart Scenario 4.

Tests for User Story 4

  • T038 [P] [US4] Unit tests for the pure runbook step-advancement function — advances by exactly one on a "try next step" outcome, reports exhaustion after the last step, never skips or resets — in tests/unit/ai-support/runbook-step-advance.test.ts
  • T039 [US4] Integration test covering Quickstart Scenario 4 (matching runbook sets activeRunbookKey/currentStepIndex: 0; advances exactly one step per turn; exhaustion escalates with every attempted step listed) against a real Postgres and a real Anthropic API call in tests/integration/ai-runbook-troubleshooting.test.ts (depends on T022)

Implementation for User Story 4

  • T040 [P] [US4] Add the pure step-advancement function advanceRunbookStep(steps, currentStepIndex, outcome) => { nextIndex } | { exhausted: true } in src/modules/ai-support/troubleshooting/service/step-advance.ts (no dependencies)
  • T041 [US4] Add RunbookEngineService.matchRunbook(problemType, productId) — calls runbooksService.findCurrentByKey(...) (004, through ai-support/knowledge's index.ts) — in src/modules/ai-support/troubleshooting/service/runbook-engine.service.ts (depends on T003)
  • T042 [US4] Wire T041/T040 into SessionsService: after a proceed diagnosis, attempt T041's match; if found, set activeRunbookKey/currentStepIndex: 0 on the session; the reasoning call (T035) receives only steps[currentStepIndex] in its prompt context, never the full list; a customer's step-outcome reply advances via T040, and exhaustion ({ exhausted: true }) escalates via T016 with every attempted step in the summary (FR-016) — in src/modules/ai-support/sessions/service/session.service.ts (depends on T041)
  • T043 [US4] Run Quickstart Scenario 4 locally and confirm it passes

Checkpoint: When a runbook matches, troubleshooting follows its authored order exactly — the model presents and interprets, the application sequences.

Phase 7: User Story 5 - A ticket is only marked AI-resolved when there's real evidence, not a customer's claim alone (Priority: P3)

Goal: The resolved/AI_RESOLVED transition is guarded on real tool evidence, never on message content.

Independent Test: Quickstart Scenario 5. This story is also required to complete the constitution's standing "(A) AI resolves directly" E2E scenario (Phase 8 wires the test itself, since it depends on every prior story existing).

Tests for User Story 5

  • T044 [US5] Integration test covering Quickstart Scenario 5 (customer claims fixed with no tool evidence → not resolved; verifyProductResolution's placeholder never confirms → session doesn't auto-resolve) against a real Postgres and a real Anthropic API call in tests/integration/ai-verification.test.ts (depends on T022)

Implementation for User Story 5

  • T045 [US5] Add the resolution guard in SessionsService: a session only transitions troubleshooting/verifying → resolved (mirrored to Ticket.status: AI_VERIFYING → AI_RESOLVED) when an AIActionResult from verifyProductResolution with confirmed: true exists for the session (T032's repository) — a customer's "it's fixed" reply is recorded as an AIInteraction only and never inspected by this guard (FR-018/ FR-019) — in src/modules/ai-support/sessions/service/session.service.ts (depends on T033)
  • T046 [US5] Run Quickstart Scenario 5 locally and confirm it passes

Checkpoint: All five user stories work independently and together — the full diagnose → clarify → act → troubleshoot → verify flow, with policy (never the model) deciding every MUST-level outcome.

Phase 8: Polish & Cross-Cutting Concerns

  • T047 [P] Integration test for the prompt-injection edge case (quickstart.md — a customer reply containing an instruction-like string never changes evaluationOutcome for a subsequent high-risk proposal) in tests/integration/ai-tool-actions.test.ts (extends T029's file) or a new tests/integration/ai-prompt-injection.test.ts (depends on T037)
  • T048 The constitution's standing E2E scenario (A) — "AI resolves directly: problem → knowledge → guided troubleshooting → verification → AI-resolved" — as a single, real, end-to-end integration test spanning US1/US3/US4/US5 in one session in tests/integration/e2e-ai-resolves.test.ts (depends on T022, T037, T043, T046)
  • T049 The constitution's standing E2E scenario (B) — "AI escalates to human: problem → failed AI troubleshooting → escalation → [ticket reaches HUMAN_ESCALATION, ready for orchestration/assignment when that phase exists]" — in tests/integration/e2e-ai-escalates.test.ts (depends on T022, T037, T043)
  • T050 [P] Add an "AI Support" section to README.md describing the session lifecycle, the confidence-policy config surface, the tool registry (including the two documented known limitations: verifyProductResolution's fail-closed placeholder and overrideTicketPriority's permanently-pending_approval state pending a future approval UI), and the runbook engine
  • T051 [P] Update specs/005-ai-support/checklists/requirements.md Notes with any implementation-time findings
  • T052 Run npx tsx scripts/check-architecture.ts and npm run lint/npm run typecheck
  • T053 Full regression: npm run test:unit (scoped to tests/unit) to confirm nothing broke elsewhere

Dependencies & Execution Order

Phase Dependencies

  • Setup (Phase 1): No dependencies
  • Foundational (Phase 2): Depends on Setup — BLOCKS all user stories
  • User Story 1 (Phase 3): Depends on Foundational — no dependency on US2-US5
  • User Story 2 (Phase 4): Depends on US1 (extends SessionsService, reuses T015-T017) — genuinely not independent of US1's diagnosis/session machinery, unlike 004's stories
  • User Story 3 (Phase 5): Depends on US1 (the proceed branch it fills in) — independent of US2's clarification loop
  • User Story 4 (Phase 6): Depends on US1 (the proceed branch) and benefits from, but doesn't strictly require, US3 (a runbook step could in principle need a tool call — not required by any FR here, so US4 doesn't block on US3 completing)
  • User Story 5 (Phase 7): Depends on US3 (needs verifyProductResolution's execution handler, T033) and US1's session status machinery
  • Polish (Phase 8): Depends on all five user stories

Parallel Opportunities

  • T001/T002/T003 (independent scaffolding)
  • T009 alongside T011-T016 once T006 exists (unit test doesn't need the real implementation)
  • T012 (pure function) can be written independently of T011
  • T028 alongside T030-T034
  • T030 (registry) alongside T031's early drafting, though T031 needs T030's exports to compile
  • T038 (pure function) independent of T041
  • T047/T050/T051 in Polish

Implementation Strategy

MVP First (User Story 1 Only)

  1. Setup + Foundational (T001-T008)
  2. User Story 1 (T009-T022)
  3. STOP and VALIDATE: Quickstart Scenario 1 passes — every new ticket gets a real, knowledge-grounded diagnosis, and confidence correctly decides proceed/ask/escalate, with Ticket.status reflecting it end to end. Usable automatic-triage value even before clarification, tools, runbooks, or verification exist.

Incremental Delivery

  1. Setup + Foundational → schema migrated, LLM client ready
  2. Add User Story 1 → every ticket gets a real AI diagnosis (MVP)
  3. Add User Story 2 → the "ask" branch becomes a real conversation
  4. Add User Story 3 → the "proceed" branch can act, through a gate the AI can't talk its way past
  5. Add User Story 4 → matched problems get consistent, product-approved troubleshooting sequences
  6. Add User Story 5 → resolution requires real evidence, closing the loop safely
  7. Polish → the two constitution-required standing E2E scenarios, docs, full regression
Reference in New Issue View Git Blame Copy Permalink