feat: implement AI support agent (005) — diagnosis, tools, runbooks, verification
Real Anthropic Claude integration per explicit product decision: a ticket's AI session diagnoses the problem via a structured-output call, applies a DB-configurable confidence-band policy (FR-005), and on "proceed" reasons and acts through a small permission/risk-gated tool system (FR-011/FR-012), optionally walking a matching runbook step by step with the application — never the model — owning the step index (FR-015/FR-016). Resolution requires real tool evidence, never customer claims alone (FR-018) — verifyProductResolution is a documented fail-closed placeholder mirroring the existing malware-scanner precedent, since no real per-product operational signal exists yet. AISupportSession.status mirrors onto Ticket.status through 003-ticketing's existing AI_ANALYZING/AI_TROUBLESHOOTING/AI_VERIFYING/AI_RESOLVED/ HUMAN_ESCALATION state machine, discovered during planning to have been built anticipating this exact feature. Two circular module dependencies (escalation<->sessions, tools<->sessions) were designed around rather than found as bugs: escalation is a pure summary formatter with no state dependencies of its own, and tools stays a clean leaf module with zero dependency on ai-support/sessions. Ticket creation enqueues the first diagnosis turn via the existing queue infrastructure (off the hot path of the inbound SaaS integration endpoint); a human actor changing ticket status ends the AI session via the event-bus scaffold that existed in this codebase but had never been wired to anything. A real Prisma limitation was found and fixed before it reached tests: compound-unique upsert rejects null for a nullable key column, so AIConfidencePolicy uses find-then-update/create instead, same fix class 004 already used for the same underlying limitation. Adds 9 unit tests (confidence-band, tool-policy-gate, runbook-step- advance) and 6 integration test files, including the two constitution- required standing E2E scenarios. AI-independent tests were run against real Postgres/Redis/MinIO (88 passed, 0 failed across the full suite, including every pre-existing 002/003/004 test). The AI-dependent tests compile and skip cleanly via describe.skipIf but were not run against a live model — no ANTHROPIC_API_KEY was available in this session; a real key must be supplied before this feature can actually run. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
saqib mir
co-authored by
Claude Sonnet 5
Claude Sonnet 5
parent
49eaa4bc58
commit
82d02bcdcd
@@ -59,3 +59,56 @@
|
||||
than leaving `AISupportSession.status` as an isolated field the rest of the system can't see —
|
||||
see research.md "AISupportSession.status drives Ticket.status through the existing state
|
||||
machine".
|
||||
|
||||
## Implementation notes (added during /speckit-implement)
|
||||
|
||||
- **Two circular module dependencies were designed around during implementation, not discovered
|
||||
as bugs after the fact**: (1) `escalation` initially needed `sessions`' repositories to end a
|
||||
session and sync ticket status, while `sessions` needed `escalation` to build the hand-off
|
||||
summary — resolved by making `EscalationService.buildSummary` a pure formatter with no
|
||||
repository/service dependencies of its own; `sessions` now owns ending its own session state
|
||||
and the ticket-status sync directly. (2) The `GET .../ai-session/actions` route initially lived
|
||||
in `tools` and imported `sessions` to resolve ticketId → session, which would have collided
|
||||
with `sessions`' own dependency on `tools` (for `proposeAndEvaluate`) — moved the route into
|
||||
`sessions` instead, which already owns that resolution; `tools` stays a clean leaf module with
|
||||
no dependency on `ai-support/sessions` at all.
|
||||
- **Found and fixed a real Prisma bug before it reached tests**: `AIConfidencePolicy.upsert`
|
||||
initially used Prisma's generated `productId_categoryId` compound-unique `where` shape, which
|
||||
rejects `null` for the (nullable) `categoryId` column at the client-API level ("Argument
|
||||
categoryId must not be null") even though the DB-level unique index itself permits it. Fixed by
|
||||
switching to `findFirst` + `update`/`create` instead of `upsert` — the same class of fix
|
||||
`KnowledgeRepository.updateCurrent` (004) already used for the same underlying Prisma
|
||||
limitation, discovered independently here.
|
||||
- **`ticket-state-machine.ts`'s AI_* statuses required two hooks into 003-ticketing's
|
||||
`tickets.service.ts`** to actually be driven correctly: (1) `createFromInboundRequest` enqueues
|
||||
the `AI_SESSION` job directly via `queueManager` (no import of `ai-support/sessions` — the
|
||||
worker, not the enqueue call, is what depends on it), and (2) `updateStatus` now publishes a
|
||||
`DomainEventName.TICKET_UPDATED` domain event unconditionally after every status change, using
|
||||
the event-bus scaffold (`src/events/`) that existed in this codebase from the original
|
||||
scaffold but had never been wired to anything — `ai-support/sessions` subscribes to it
|
||||
(registered in `src/events/handlers/index.ts`) to implement FR-023 (a human actor ends the AI
|
||||
session) without `tickets` ever needing to know `ai-support/sessions` exists.
|
||||
- **The runbook-matching convention is a real, disclosed scope decision, not an oversight**: a
|
||||
runbook's `key` is matched directly against the diagnosis's `problemType` string (no fuzzy
|
||||
matching, no separate mapping table) — admins author runbook keys to match the exact
|
||||
`problemType` vocabulary the AI's diagnosis call produces. This is simple and works, but is
|
||||
inherently a naming-convention contract between the diagnosis system prompt and runbook
|
||||
authoring, not a robust semantic match — documented in `session.service.ts`'s
|
||||
`enterTroubleshooting` and in research.md.
|
||||
- 9 unit tests (confidence-band, tool-policy-gate, runbook-step-advance) and 9 integration test
|
||||
files were added. The AI-independent ones (`ai-confidence-policy.test.ts`, the deterministic
|
||||
tool-policy-gate re-check in `ai-tools-and-runbook.test.ts`, and the message-routing guard in
|
||||
`ai-clarification.test.ts`) run unconditionally and were verified passing against a real
|
||||
Postgres/Redis/MinIO. The remaining integration tests and the two constitution-required
|
||||
standing E2E scenarios (`e2e-ai-flows.test.ts`) require a real `ANTHROPIC_API_KEY` and are
|
||||
gated with `describe.skipIf` so the suite skips them cleanly (not a failure) rather than
|
||||
requiring every contributor to hold a live credential just to run the test suite — they were
|
||||
written and confirmed to compile and skip correctly, but not yet run against a live model in
|
||||
this environment (no key was available this session). The "AI resolves directly" E2E test
|
||||
additionally exercises the resolution-guard transition deterministically (via
|
||||
`SessionsService.recheckVerification`, a new seam also intended for a future real
|
||||
product-signal webhook) rather than relying solely on live-model non-determinism to reach that
|
||||
state.
|
||||
- Full regression (all 17 pre-existing integration test files plus every new one) was run
|
||||
together against real Docker-provisioned Postgres/Redis/MinIO: 88 passed, 9 skipped (the
|
||||
AI-key-gated ones), 0 failed.
|
||||
|
||||
@@ -31,16 +31,16 @@ All file paths are relative to `supporthub-api/` (repo root).
|
||||
|
||||
## Phase 1: Setup
|
||||
|
||||
- [ ] T001 [P] Scaffold `src/modules/ai-support/sessions/` and `src/modules/ai-support/tools/`
|
||||
- [x] T001 [P] Scaffold `src/modules/ai-support/sessions/` and `src/modules/ai-support/tools/`
|
||||
with the standard module shape (`controller/`, `routes/`, `schema/`, `repository/`,
|
||||
`service/`, `types/`, `mapper/`, `constants/`, `index.ts`)
|
||||
- [ ] T002 [P] Scaffold `src/modules/ai-support/troubleshooting/` and
|
||||
- [x] T002 [P] Scaffold `src/modules/ai-support/troubleshooting/` and
|
||||
`src/modules/ai-support/escalation/` with the reduced shape plan.md specifies for
|
||||
internal-only submodules (`service/`, `types/`, `index.ts` — no `routes/controller/schema`,
|
||||
since neither has its own HTTP surface)
|
||||
- [ ] T003 [P] Scaffold `src/infrastructure/ai/` (Anthropic client singleton) and
|
||||
- [x] T003 [P] Scaffold `src/infrastructure/ai/` (Anthropic client singleton) and
|
||||
`src/jobs/ai-session/` (empty worker module, populated in Phase 3)
|
||||
- [ ] T004 Add `@anthropic-ai/sdk` as a runtime dependency (`npm install @anthropic-ai/sdk`)
|
||||
- [x] T004 Add `@anthropic-ai/sdk` as a runtime dependency (`npm install @anthropic-ai/sdk`)
|
||||
|
||||
---
|
||||
|
||||
@@ -50,12 +50,12 @@ All file paths are relative to `supporthub-api/` (repo root).
|
||||
|
||||
**⚠️ CRITICAL**: No user-story stage work can begin until this phase is complete.
|
||||
|
||||
- [ ] T005 Add `AISupportSession`, `AIDiagnosis`, `AIInteraction`, `AIAction`, `AIActionResult`,
|
||||
- [x] T005 Add `AISupportSession`, `AIDiagnosis`, `AIInteraction`, `AIAction`, `AIActionResult`,
|
||||
`AIKnowledgeReference`, `AIConfidencePolicy` models to `prisma/schema.prisma` per
|
||||
data-model.md, plus the `Ticket.aiSessions` back-relation (depends on T001-T003)
|
||||
- [ ] T006 Run `npm run prisma:generate` and create the migration (`npm run prisma:migrate`) for
|
||||
- [x] T006 Run `npm run prisma:generate` and create the migration (`npm run prisma:migrate`) for
|
||||
T005 (depends on T005)
|
||||
- [ ] T007 Add env vars to `src/config/env.ts`: `ANTHROPIC_API_KEY` (`z.string().optional()` —
|
||||
- [x] T007 Add env vars to `src/config/env.ts`: `ANTHROPIC_API_KEY` (`z.string().optional()` —
|
||||
the app must still boot and every non-AI test must still pass without it; the Anthropic
|
||||
client wrapper (T008) is what throws a clear, explicit error if a reasoning call is
|
||||
attempted with it unset — spec.md Assumptions' "no offline fallback path" is enforced at
|
||||
@@ -70,7 +70,7 @@ All file paths are relative to `supporthub-api/` (repo root).
|
||||
required for this feature to function) — `.env.development`/`.env.test` are gitignored, so
|
||||
note in the task (not the repo) that the user must add a real key there themselves
|
||||
(depends on T004)
|
||||
- [ ] T008 Add the Anthropic client singleton in
|
||||
- [x] T008 Add the Anthropic client singleton in
|
||||
`src/infrastructure/ai/anthropic.client.ts` — constructs `new Anthropic()` (credential
|
||||
resolved from `ANTHROPIC_API_KEY` per the SDK's own env resolution), exports the
|
||||
configured `model`/`effort` from env, and a guard that throws a clear `AppError` if a
|
||||
@@ -90,10 +90,10 @@ correctly reflecting the outcome through the existing state machine.
|
||||
|
||||
### Tests for User Story 1
|
||||
|
||||
- [ ] T009 [P] [US1] Unit tests for the confidence-band decision function — proceed/ask/escalate
|
||||
- [x] T009 [P] [US1] Unit tests for the confidence-band decision function — proceed/ask/escalate
|
||||
boundary values, most-specific-`(productId, categoryId)`-match-with-fallback resolution — in
|
||||
`tests/unit/ai-support/confidence-band.test.ts`
|
||||
- [ ] T010 [US1] Integration test covering Quickstart Scenario 1 (diagnosis recorded with
|
||||
- [x] T010 [US1] Integration test covering Quickstart Scenario 1 (diagnosis recorded with
|
||||
confidence; low `highThreshold` still proceeds; high `lowThreshold` escalates; no-knowledge
|
||||
product escalates) against a real Postgres **and a real Anthropic API call** in
|
||||
`tests/integration/ai-diagnosis.test.ts` (depends on T006, T008; requires a real
|
||||
@@ -101,31 +101,31 @@ correctly reflecting the outcome through the existing state machine.
|
||||
|
||||
### Implementation for User Story 1
|
||||
|
||||
- [ ] T011 [US1] Add `AIConfidencePolicyRepository`/`AIConfidencePolicyService`
|
||||
- [x] T011 [US1] Add `AIConfidencePolicyRepository`/`AIConfidencePolicyService`
|
||||
(most-specific-match lookup: `(productId, categoryId)` → `(productId, null)` → env
|
||||
defaults; upsert) in `src/modules/ai-support/sessions/repository/confidence-policy.repository.ts`
|
||||
+ `service/confidence-policy.service.ts` (depends on T006)
|
||||
- [ ] T012 [P] [US1] Add the pure confidence-band decision function
|
||||
- [x] T012 [P] [US1] Add the pure confidence-band decision function
|
||||
`decideConfidenceBand(confidence, policy) => 'proceed' | 'ask' | 'escalate'` in
|
||||
`src/modules/ai-support/sessions/service/confidence-band.ts` (no dependencies — pure
|
||||
function, can be written and unit-tested in parallel with T011)
|
||||
- [ ] T013 [US1] Add `AISupportSessionRepository` (create; findActiveByTicketId — enforces
|
||||
- [x] T013 [US1] Add `AISupportSessionRepository` (create; findActiveByTicketId — enforces
|
||||
FR-001's one-active-session rule; update status/runbook fields/counters) in
|
||||
`src/modules/ai-support/sessions/repository/session.repository.ts` (depends on T006)
|
||||
- [ ] T014 [US1] Add `AIDiagnosisRepository` (create; findLatestBySession) in
|
||||
- [x] T014 [US1] Add `AIDiagnosisRepository` (create; findLatestBySession) in
|
||||
`src/modules/ai-support/sessions/repository/diagnosis.repository.ts` (depends on T006)
|
||||
- [ ] T015 [US1] Add the diagnosis LLM call — `zodOutputFormat` schema matching data-model.md's
|
||||
- [x] T015 [US1] Add the diagnosis LLM call — `zodOutputFormat` schema matching data-model.md's
|
||||
`AIDiagnosis` shape, `client.messages.parse()` against the ticket's problem statement +
|
||||
conversation-so-far, no tools — in `src/modules/ai-support/sessions/service/diagnose.ts`
|
||||
(depends on T008)
|
||||
- [ ] T016 [US1] Add `EscalationService.escalate(sessionId, reason)` — builds the structured
|
||||
- [x] T016 [US1] Add `EscalationService.escalate(sessionId, reason)` — builds the structured
|
||||
summary (problem, diagnosis, steps attempted so far, confidence — FR-021), calls
|
||||
`ticketsService.updateStatus(ticketId, 'HUMAN_ESCALATION', expectedVersion, 'ai')` (003,
|
||||
reused per research.md), ends the session (`status: escalated`) — in
|
||||
`src/modules/ai-support/escalation/service/escalation.service.ts` (depends on T013; this is
|
||||
needed by US1 itself, since "escalate" is one of US1's three outcomes — not deferred to a
|
||||
later story)
|
||||
- [ ] T017 [US1] Add `SessionsService.runFirstTurn(ticketId)`: create the session
|
||||
- [x] T017 [US1] Add `SessionsService.runFirstTurn(ticketId)`: create the session
|
||||
(`status: analyzing`, mirrored onto `Ticket.status: AI_ANALYZING` via
|
||||
`ticketsService.updateStatus(..., 'ai')`), run T015's diagnosis call, call
|
||||
`knowledgeService.retrieve(...)` (004, through `ai-support/knowledge`'s `index.ts` —
|
||||
@@ -139,24 +139,29 @@ correctly reflecting the outcome through the existing state machine.
|
||||
lands in US3/US4, so for this story `proceed` only needs to reach the correct status, not
|
||||
yet call any tool — in `src/modules/ai-support/sessions/service/session.service.ts`
|
||||
(depends on T011, T012, T013, T014, T015, T016)
|
||||
- [ ] T018 [US1] Add Zod schema + `PUT`/`GET /admin/products/:externalProductId/ai-policy` routes
|
||||
- [x] T018 [US1] Add Zod schema + `PUT`/`GET /admin/products/:externalProductId/ai-policy` routes
|
||||
(gated by `fastify.authenticate`, per contracts/ai-support-contract.md) in
|
||||
`src/modules/ai-support/sessions/schema/` + `routes/`, registered from `src/api/routes.ts`
|
||||
(depends on T011)
|
||||
- [ ] T019 [US1] Add the `AI_SESSION` worker — `registerAiSessionWorker()` in
|
||||
- [x] T019 [US1] Add the `AI_SESSION` worker — `registerAiSessionWorker()` in
|
||||
`src/jobs/ai-session/index.ts`, calling `sessionsService.runFirstTurn(ticketId)` — and
|
||||
register it in `src/bootstrap/queue.bootstrap.ts` alongside the existing attachment worker
|
||||
(depends on T017)
|
||||
- [ ] T020 [US1] Modify `TicketsService.createFromInboundRequest`
|
||||
- [x] T020 [US1] Modify `TicketsService.createFromInboundRequest`
|
||||
(`src/modules/ticketing/tickets/service/tickets.service.ts`) to enqueue a
|
||||
`QueueName.AI_SESSION` job (`{ ticketId: ticket.id }`) when `wasExisting` is `false`
|
||||
(research.md "Session triggering" — never on an idempotent replay) (depends on T019)
|
||||
- [ ] T021 [US1] Modify `TicketsService.updateStatus` to end any active `AISupportSession` for the
|
||||
- [x] T021 [US1] Modify `TicketsService.updateStatus` to end any active `AISupportSession` for the
|
||||
ticket (`status: ended_by_agent`) when called with an actor other than `'ai'` — FR-023's
|
||||
concrete mechanism (research.md) — before the status update itself commits (depends on
|
||||
T013)
|
||||
- [ ] T022 [US1] Run Quickstart Scenario 1 locally (with a real `ANTHROPIC_API_KEY`) and confirm
|
||||
all 5 steps pass
|
||||
- [~] T022 [US1] ~~Run Quickstart Scenario 1 locally~~ — blocked: no real `ANTHROPIC_API_KEY` was
|
||||
available in this environment/session. `tests/integration/ai-diagnosis.test.ts` implements
|
||||
this exact scenario and is verified to compile and skip cleanly
|
||||
(`describe.skipIf(!hasRealApiKey)`); it has not yet been run against a live model. Every
|
||||
AI-independent path (schema, routing, the confidence-policy admin surface, the
|
||||
deterministic tool gate) was verified against real Postgres/Redis/MinIO — see checklists/
|
||||
requirements.md "Implementation notes."
|
||||
|
||||
**Checkpoint**: Every new ticket gets a real, knowledge-grounded diagnosis, and confidence
|
||||
correctly decides proceed/ask/escalate, with `Ticket.status` reflecting it. This alone is a
|
||||
@@ -174,17 +179,17 @@ enforced.
|
||||
|
||||
### Tests for User Story 2
|
||||
|
||||
- [ ] T023 [US2] Integration test covering Quickstart Scenario 2 (question posted as
|
||||
- [x] T023 [US2] Integration test covering Quickstart Scenario 2 (question posted as
|
||||
customer-visible `AI_MESSAGE`; reply triggers a second `AIDiagnosis`; policy re-applied;
|
||||
question cap reached → escalate) against a real Postgres and a real Anthropic API call in
|
||||
`tests/integration/ai-clarification.test.ts` (depends on T022)
|
||||
|
||||
### Implementation for User Story 2
|
||||
|
||||
- [ ] T024 [US2] Add Zod schema + `POST /tickets/:ticketId/ai-session/messages` route (`404` if
|
||||
- [x] T024 [US2] Add Zod schema + `POST /tickets/:ticketId/ai-session/messages` route (`404` if
|
||||
no active session — contracts/ai-support-contract.md guarantee 1) in
|
||||
`src/modules/ai-support/sessions/schema/` + `routes/` + `controller/` (depends on T017)
|
||||
- [ ] T025 [US2] Add `SessionsService.handleCustomerReply(ticketId, message)`: record the reply
|
||||
- [x] T025 [US2] Add `SessionsService.handleCustomerReply(ticketId, message)`: record the reply
|
||||
as an `AIInteraction` (`role: customer`) and a `TicketMessage` (`type: CUSTOMER_MESSAGE`,
|
||||
via the existing `messagesService`), re-run T015's diagnosis call with the full
|
||||
conversation, increment `clarifyingQuestionsAsked` when continuing from an `ask` outcome,
|
||||
@@ -192,10 +197,12 @@ enforced.
|
||||
regardless of the new diagnosis's own confidence, otherwise re-apply the confidence-band
|
||||
decision as in T017 — in `src/modules/ai-support/sessions/service/session.service.ts`
|
||||
(depends on T024)
|
||||
- [ ] T026 [US2] Add Zod schema + `GET /tickets/:ticketId/ai-session` read route (status, latest
|
||||
- [x] T026 [US2] Add Zod schema + `GET /tickets/:ticketId/ai-session` read route (status, latest
|
||||
diagnosis, interaction history) in `sessions/schema/` + `routes/` + `controller/` (depends
|
||||
on T013, T014)
|
||||
- [ ] T027 [US2] Run Quickstart Scenario 2 locally and confirm it passes
|
||||
- [~] T027 [US2] ~~Run Quickstart Scenario 2 locally~~ — same blocker as T022. Implemented in
|
||||
`tests/integration/ai-clarification.test.ts`; its AI-independent routing guard (guarantee
|
||||
1: `404` on a reply with no active session) was run and passes.
|
||||
|
||||
**Checkpoint**: US1 and US2 together deliver a full diagnose → clarify → re-diagnose loop with a
|
||||
correctly enforced question budget.
|
||||
@@ -213,55 +220,60 @@ required "AI tool-permission tests" category (Testing, Observability & CI/CD Gat
|
||||
|
||||
### Tests for User Story 3
|
||||
|
||||
- [ ] T028 [P] [US3] Unit tests for `evaluateToolProposal` — unknown tool refused, product not in
|
||||
- [x] T028 [P] [US3] Unit tests for `evaluateToolProposal` — unknown tool refused, product not in
|
||||
`supportedProducts` refused, low-risk auto-approved, medium/high-risk `pending_approval`,
|
||||
and an explicit case asserting the AI's own proposal/justification text is never read by
|
||||
the gate (prompt-injection resistance, FR-024) — in
|
||||
`tests/unit/ai-support/tool-policy-gate.test.ts`
|
||||
- [ ] T029 [US3] Integration test covering Quickstart Scenario 3 (low-risk tool executes and is
|
||||
- [x] T029 [US3] Integration test covering Quickstart Scenario 3 (low-risk tool executes and is
|
||||
recorded; `overrideTicketPriority` proposal is `pending_approval` with no result; an
|
||||
out-of-scope tool proposal is `refused`) against a real Postgres and a real Anthropic API
|
||||
call in `tests/integration/ai-tool-actions.test.ts` (depends on T022)
|
||||
call — implemented in `tests/integration/ai-tools-and-runbook.test.ts` (combined with
|
||||
T039/US4, since both stories' scenarios share the same session-reaches-troubleshooting
|
||||
setup) rather than the originally-planned `ai-tool-actions.test.ts` (depends on T022)
|
||||
|
||||
### Implementation for User Story 3
|
||||
|
||||
- [ ] T030 [P] [US3] Add the tool registry — `getTicketSnapshot` (low), `searchProductKnowledge`
|
||||
- [x] T030 [P] [US3] Add the tool registry — `getTicketSnapshot` (low), `searchProductKnowledge`
|
||||
(low), `verifyProductResolution` (low, fail-closed placeholder — research.md, mirrors
|
||||
`UnimplementedPlaceholderScanner`), `escalateToHuman` (low, always policy-approved),
|
||||
`overrideTicketPriority` (high) — with Zod input schemas, `permission`, `riskLevel`,
|
||||
`supportedProducts`, `auditRequired` per tool — in
|
||||
`src/modules/ai-support/tools/constants/tool-registry.ts` (depends on T003)
|
||||
- [ ] T031 [US3] Add `evaluateToolProposal(toolName, sessionContext)` — the shared deterministic
|
||||
- [x] T031 [US3] Add `evaluateToolProposal(toolName, sessionContext)` — the shared deterministic
|
||||
gate (research.md): unknown-tool / out-of-scope → `refused`; `low` → `approved`;
|
||||
`medium`/`high` → `pending_approval`. Reads only the tool name and session's product/
|
||||
permission context, never the AI's proposal text — in
|
||||
`src/modules/ai-support/tools/service/policy-gate.ts` (depends on T030)
|
||||
- [ ] T032 [US3] Add `AIActionRepository`/`AIActionResultRepository` (create action + evaluation
|
||||
- [x] T032 [US3] Add `AIActionRepository`/`AIActionResultRepository` (create action + evaluation
|
||||
outcome; create result when executed; list by session, newest first) in
|
||||
`src/modules/ai-support/tools/repository/` (depends on T006)
|
||||
- [ ] T033 [US3] Add real tool execution handlers — `getTicketSnapshot` (reads
|
||||
- [x] T033 [US3] Add real tool execution handlers — `getTicketSnapshot` (reads
|
||||
`Ticket`+`Problem`+recent `TicketMessage`s via existing repositories), `searchProductKnowledge`
|
||||
(calls `knowledgeService.retrieve(...)`), `verifyProductResolution` (always returns
|
||||
`{ confirmed: false, status: 'unknown' }` — documented placeholder), `escalateToHuman`
|
||||
(calls T016's `EscalationService.escalate`) — in
|
||||
`src/modules/ai-support/tools/service/tool-executor.ts` (depends on T031; `overrideTicketPriority`
|
||||
has no execution handler yet — it can never reach `approved`, so it's never called)
|
||||
- [ ] T034 [US3] Add `ToolsService.proposeAndEvaluate(sessionId, toolUseBlocks)`: for each
|
||||
- [x] T034 [US3] Add `ToolsService.proposeAndEvaluate(sessionId, toolUseBlocks)`: for each
|
||||
proposed `tool_use` block, run T031's gate, persist the `AIAction` (T032), execute + persist
|
||||
an `AIActionResult` (T032) only when `approved`, and increment the failure count toward
|
||||
escalation triggers on a failed result (FR-014) — in
|
||||
`src/modules/ai-support/tools/service/tools.service.ts` (depends on T032, T033)
|
||||
- [ ] T035 [US3] Wire the reasoning/response call into `SessionsService`'s `proceed` branch
|
||||
- [x] T035 [US3] Wire the reasoning/response call into `SessionsService`'s `proceed` branch
|
||||
(research.md "two calls per reasoning turn," step 3): a `client.messages.create()` call
|
||||
with the tool registry's `Anthropic.Tool[]` definitions, the diagnosis + retrieved
|
||||
knowledge + conversation as context; route any `tool_use` blocks through T034, feed
|
||||
`tool_result` blocks back for a follow-up call, capped at
|
||||
`AI_SUPPORT_MAX_REASONING_ITERATIONS_PER_TURN` iterations (doc 11 §B2) — in
|
||||
`src/modules/ai-support/sessions/service/session.service.ts` (depends on T034)
|
||||
- [ ] T036 [US3] Add Zod schema + `GET /tickets/:ticketId/ai-session/actions` route in
|
||||
- [x] T036 [US3] Add Zod schema + `GET /tickets/:ticketId/ai-session/actions` route in
|
||||
`sessions/schema/` + `routes/` + `controller/` (or `tools/` — whichever module owns the
|
||||
route registers it; the data comes from T032's repository either way) (depends on T032)
|
||||
- [ ] T037 [US3] Run Quickstart Scenario 3 locally and confirm it passes
|
||||
- [~] T037 [US3] ~~Run Quickstart Scenario 3 locally~~ — same blocker as T022. Implemented in
|
||||
`tests/integration/ai-tools-and-runbook.test.ts`; its AI-independent half (SC-003 and the
|
||||
full low-risk-tool-list re-check against the real registry, no LLM call) was run and
|
||||
passes.
|
||||
|
||||
**Checkpoint**: US1-US3 together deliver diagnose → clarify → act-through-gated-tools, with every
|
||||
proposal, decision, and result durably recorded and auditable.
|
||||
@@ -277,31 +289,38 @@ step is next.
|
||||
|
||||
### Tests for User Story 4
|
||||
|
||||
- [ ] T038 [P] [US4] Unit tests for the pure runbook step-advancement function — advances by
|
||||
exactly one on a "try next step" outcome, reports exhaustion after the last step, never
|
||||
skips or resets — in `tests/unit/ai-support/runbook-step-advance.test.ts`
|
||||
- [ ] T039 [US4] Integration test covering Quickstart Scenario 4 (matching runbook sets
|
||||
- [x] T038 [P] [US4] Unit tests for the pure runbook step-advancement function — implemented with
|
||||
a boolean `resolved` signal rather than the originally-sketched 3-way outcome enum (the
|
||||
classifier that produces the signal — classify-step-outcome.ts, US1's diagnose.ts sibling —
|
||||
only ever needs "did this step resolve it or not"; a 3-way enum added no behavior a 2-way
|
||||
one didn't already cover). Advances by exactly one when not resolved and steps remain,
|
||||
reports exhaustion after the last step, never skips or resets — in
|
||||
`tests/unit/ai-support/runbook-step-advance.test.ts`
|
||||
- [x] T039 [US4] Integration test covering Quickstart Scenario 4 (matching runbook sets
|
||||
`activeRunbookKey`/`currentStepIndex: 0`; advances exactly one step per turn; exhaustion
|
||||
escalates with every attempted step listed) against a real Postgres and a real Anthropic
|
||||
API call in `tests/integration/ai-runbook-troubleshooting.test.ts` (depends on T022)
|
||||
API call — implemented in `tests/integration/ai-tools-and-runbook.test.ts` (combined with
|
||||
T029/US3) rather than the originally-planned `ai-runbook-troubleshooting.test.ts` (depends
|
||||
on T022)
|
||||
|
||||
### Implementation for User Story 4
|
||||
|
||||
- [ ] T040 [P] [US4] Add the pure step-advancement function
|
||||
- [x] T040 [P] [US4] Add the pure step-advancement function
|
||||
`advanceRunbookStep(steps, currentStepIndex, outcome) => { nextIndex } | { exhausted: true }`
|
||||
in `src/modules/ai-support/troubleshooting/service/step-advance.ts` (no dependencies)
|
||||
- [ ] T041 [US4] Add `RunbookEngineService.matchRunbook(problemType, productId)` — calls
|
||||
- [x] T041 [US4] Add `RunbookEngineService.matchRunbook(problemType, productId)` — calls
|
||||
`runbooksService.findCurrentByKey(...)` (004, through `ai-support/knowledge`'s `index.ts`)
|
||||
— in `src/modules/ai-support/troubleshooting/service/runbook-engine.service.ts` (depends on
|
||||
T003)
|
||||
- [ ] T042 [US4] Wire T041/T040 into `SessionsService`: after a `proceed` diagnosis, attempt
|
||||
- [x] T042 [US4] Wire T041/T040 into `SessionsService`: after a `proceed` diagnosis, attempt
|
||||
T041's match; if found, set `activeRunbookKey`/`currentStepIndex: 0` on the session; the
|
||||
reasoning call (T035) receives **only** `steps[currentStepIndex]` in its prompt context,
|
||||
never the full list; a customer's step-outcome reply advances via T040, and exhaustion
|
||||
(`{ exhausted: true }`) escalates via T016 with every attempted step in the summary
|
||||
(FR-016) — in `src/modules/ai-support/sessions/service/session.service.ts` (depends on
|
||||
T041)
|
||||
- [ ] T043 [US4] Run Quickstart Scenario 4 locally and confirm it passes
|
||||
- [~] T043 [US4] ~~Run Quickstart Scenario 4 locally~~ — same blocker as T022. Implemented as part
|
||||
of `tests/integration/ai-tools-and-runbook.test.ts`.
|
||||
|
||||
**Checkpoint**: When a runbook matches, troubleshooting follows its authored order exactly — the
|
||||
model presents and interprets, the application sequences.
|
||||
@@ -319,20 +338,23 @@ since it depends on every prior story existing).
|
||||
|
||||
### Tests for User Story 5
|
||||
|
||||
- [ ] T044 [US5] Integration test covering Quickstart Scenario 5 (customer claims fixed with no
|
||||
- [x] T044 [US5] Integration test covering Quickstart Scenario 5 (customer claims fixed with no
|
||||
tool evidence → not resolved; `verifyProductResolution`'s placeholder never confirms →
|
||||
session doesn't auto-resolve) against a real Postgres and a real Anthropic API call in
|
||||
`tests/integration/ai-verification.test.ts` (depends on T022)
|
||||
session doesn't auto-resolve) against a real Postgres and a real Anthropic API call —
|
||||
implemented in `tests/integration/ai-verification-and-escalation.test.ts` (combined with
|
||||
T047's prompt-injection case) rather than the originally-planned `ai-verification.test.ts`
|
||||
(depends on T022)
|
||||
|
||||
### Implementation for User Story 5
|
||||
|
||||
- [ ] T045 [US5] Add the resolution guard in `SessionsService`: a session only transitions
|
||||
- [x] T045 [US5] Add the resolution guard in `SessionsService`: a session only transitions
|
||||
`troubleshooting/verifying → resolved` (mirrored to `Ticket.status: AI_VERIFYING →
|
||||
AI_RESOLVED`) when an `AIActionResult` from `verifyProductResolution` with
|
||||
`confirmed: true` exists for the session (T032's repository) — a customer's "it's fixed"
|
||||
reply is recorded as an `AIInteraction` only and never inspected by this guard (FR-018/
|
||||
FR-019) — in `src/modules/ai-support/sessions/service/session.service.ts` (depends on T033)
|
||||
- [ ] T046 [US5] Run Quickstart Scenario 5 locally and confirm it passes
|
||||
- [~] T046 [US5] ~~Run Quickstart Scenario 5 locally~~ — same blocker as T022. Implemented in
|
||||
`tests/integration/ai-verification-and-escalation.test.ts`.
|
||||
|
||||
**Checkpoint**: All five user stories work independently and together — the full diagnose →
|
||||
clarify → act → troubleshoot → verify flow, with policy (never the model) deciding every
|
||||
@@ -342,27 +364,29 @@ MUST-level outcome.
|
||||
|
||||
## Phase 8: Polish & Cross-Cutting Concerns
|
||||
|
||||
- [ ] T047 [P] Integration test for the prompt-injection edge case (quickstart.md — a customer
|
||||
- [x] T047 [P] Integration test for the prompt-injection edge case (quickstart.md — a customer
|
||||
reply containing an instruction-like string never changes `evaluationOutcome` for a
|
||||
subsequent high-risk proposal) in `tests/integration/ai-tool-actions.test.ts` (extends
|
||||
T029's file) or a new `tests/integration/ai-prompt-injection.test.ts` (depends on T037)
|
||||
- [ ] T048 The constitution's standing E2E scenario (A) — "AI resolves directly: problem →
|
||||
knowledge → guided troubleshooting → verification → AI-resolved" — as a single, real,
|
||||
end-to-end integration test spanning US1/US3/US4/US5 in one session in
|
||||
`tests/integration/e2e-ai-resolves.test.ts` (depends on T022, T037, T043, T046)
|
||||
- [ ] T049 The constitution's standing E2E scenario (B) — "AI escalates to human: problem →
|
||||
failed AI troubleshooting → escalation → [ticket reaches HUMAN_ESCALATION, ready for
|
||||
orchestration/assignment when that phase exists]" — in
|
||||
`tests/integration/e2e-ai-escalates.test.ts` (depends on T022, T037, T043)
|
||||
- [ ] T050 [P] Add an "AI Support" section to `README.md` describing the session lifecycle, the
|
||||
subsequent high-risk proposal) — implemented as its own `it(...)` in
|
||||
`tests/integration/ai-verification-and-escalation.test.ts` rather than a separate file
|
||||
(depends on T037). Not yet run against a live model — same blocker as T022.
|
||||
- [x] T048/T049 Both constitution-required standing E2E scenarios — (A) "AI resolves directly" and
|
||||
(B) "AI escalates to human" — implemented together in a single file,
|
||||
`tests/integration/e2e-ai-flows.test.ts` (one `describe` block, one shared product/knowledge
|
||||
fixture, two `it`s), rather than two separate files as originally planned; the two scenarios
|
||||
share enough setup that splitting them added file overhead without adding coverage. (A)
|
||||
also exercises the resolution-guard transition deterministically via the new
|
||||
`SessionsService.recheckVerification` seam, rather than relying solely on live-model
|
||||
non-determinism to reach the "verifying" state naturally. Not yet run against a live model —
|
||||
same blocker as T022 (depends on T022, T037, T043, T046).
|
||||
- [x] T050 [P] Add an "AI Support" section to `README.md` describing the session lifecycle, the
|
||||
confidence-policy config surface, the tool registry (including the two documented known
|
||||
limitations: `verifyProductResolution`'s fail-closed placeholder and
|
||||
`overrideTicketPriority`'s permanently-`pending_approval` state pending a future approval
|
||||
UI), and the runbook engine
|
||||
- [ ] T051 [P] Update `specs/005-ai-support/checklists/requirements.md` Notes with any
|
||||
- [x] T051 [P] Update `specs/005-ai-support/checklists/requirements.md` Notes with any
|
||||
implementation-time findings
|
||||
- [ ] T052 Run `npx tsx scripts/check-architecture.ts` and `npm run lint`/`npm run typecheck`
|
||||
- [ ] T053 Full regression: `npm run test:unit` (scoped to `tests/unit`) to confirm nothing broke
|
||||
- [x] T052 Run `npx tsx scripts/check-architecture.ts` and `npm run lint`/`npm run typecheck`
|
||||
- [x] T053 Full regression: `npm run test:unit` (scoped to `tests/unit`) to confirm nothing broke
|
||||
elsewhere
|
||||
|
||||
---
|
||||
|
||||
Reference in New Issue
Block a user