maskantech/support_backend
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
82d02bcdcd23b7406b231877f54df4fe69004b54
support_backend/specs/005-ai-support/checklists/requirements.md
T
saqib mirandClaude Sonnet 5 82d02bcdcd feat: implement AI support agent (005) — diagnosis, tools, runbooks, verification 
Real Anthropic Claude integration per explicit product decision: a
ticket's AI session diagnoses the problem via a structured-output call,
applies a DB-configurable confidence-band policy (FR-005), and on
"proceed" reasons and acts through a small permission/risk-gated tool
system (FR-011/FR-012), optionally walking a matching runbook step by
step with the application — never the model — owning the step index
(FR-015/FR-016). Resolution requires real tool evidence, never customer
claims alone (FR-018) — verifyProductResolution is a documented
fail-closed placeholder mirroring the existing malware-scanner precedent,
since no real per-product operational signal exists yet.

AISupportSession.status mirrors onto Ticket.status through 003-ticketing's
existing AI_ANALYZING/AI_TROUBLESHOOTING/AI_VERIFYING/AI_RESOLVED/
HUMAN_ESCALATION state machine, discovered during planning to have been
built anticipating this exact feature. Two circular module dependencies
(escalation<->sessions, tools<->sessions) were designed around rather than
found as bugs: escalation is a pure summary formatter with no state
dependencies of its own, and tools stays a clean leaf module with zero
dependency on ai-support/sessions. Ticket creation enqueues the first
diagnosis turn via the existing queue infrastructure (off the hot path of
the inbound SaaS integration endpoint); a human actor changing ticket
status ends the AI session via the event-bus scaffold that existed in
this codebase but had never been wired to anything.

A real Prisma limitation was found and fixed before it reached tests:
compound-unique upsert rejects null for a nullable key column, so
AIConfidencePolicy uses find-then-update/create instead, same fix class
004 already used for the same underlying limitation.

Adds 9 unit tests (confidence-band, tool-policy-gate, runbook-step-
advance) and 6 integration test files, including the two constitution-
required standing E2E scenarios. AI-independent tests were run against
real Postgres/Redis/MinIO (88 passed, 0 failed across the full suite,
including every pre-existing 002/003/004 test). The AI-dependent tests
compile and skip cleanly via describe.skipIf but were not run against a
live model — no ANTHROPIC_API_KEY was available in this session; a real
key must be supplied before this feature can actually run.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-02 17:44:52 +05:30

7.5 KiB
Raw Blame History

Specification Quality Checklist: AI Support Agent

Purpose: Validate specification completeness and quality before proceeding to planning Created: 2026-09-02 Feature: spec.md

Content Quality

  • No implementation details (languages, frameworks, APIs)
  • Focused on user value and business needs
  • Written for non-technical stakeholders
  • All mandatory sections completed

Requirement Completeness

  • No [NEEDS CLARIFICATION] markers remain
  • Requirements are testable and unambiguous
  • Success criteria are measurable
  • Success criteria are technology-agnostic (no implementation details)
  • All acceptance scenarios are defined
  • Edge cases are identified
  • Scope is clearly bounded
  • Dependencies and assumptions identified

Feature Readiness

  • All functional requirements have clear acceptance criteria
  • User scenarios cover primary flows
  • Feature meets measurable outcomes defined in Success Criteria
  • No implementation details leak into specification

Notes

  • Scope is Phase 4 per docs/10-implementation-roadmap.md: AI session/diagnosis/interaction models, classification, RAG-backed reasoning (consuming 004's filtered retrieval, not adding a new semantic layer — see Assumptions), configurable confidence thresholds, a permission/risk- gated tool system, the runbook execution engine, and evidence-based verification.
  • The "Assumptions" section makes explicit which doc 03 concepts are illustrative-only for this codebase (the DocuQube-specific example tools) versus which are actually built (the tool system itself, with a small set of real, platform-native tools).
  • Per explicit product decision, the AI reasoning integration calls a real LLM provider (Anthropic Claude) rather than a mock or a pluggable-first interface — this is a spec-level assumption, not deferred to plan, because it changes what "done" and "independently testable" mean for every user story here (a real credential is required to verify any of them).
  • Out of scope, explicitly: semantic/vector retrieval (doc 11 §B1, deferred again — the same deferral 004 made, now to a still-later phase), product-signal webhook verification (doc 11 §A2, not yet built anywhere in this codebase), model routing/fallback (doc 11 §B2), full cost dashboards (doc 09), localization (doc 11 §B7), and idle-session timeout/expiry.
  • All items pass; no revision iterations were needed.

Planning notes (added during /speckit-plan research)

  • Found a real integration point, not a gap: 003-ticketing's ticket-state-machine.ts already defines AI_ANALYZING/AI_TROUBLESHOOTING/AI_VERIFYING/AI_RESOLVED/ HUMAN_ESCALATION ticket statuses — a near-exact match for doc 06's AISupportSession.status enum, clearly authored anticipating this feature. research.md's design was corrected during planning (before any code was written) to drive Ticket.status through this existing state machine via the existing ticketsService.updateStatus(...), rather than leaving AISupportSession.status as an isolated field the rest of the system can't see — see research.md "AISupportSession.status drives Ticket.status through the existing state machine".

Implementation notes (added during /speckit-implement)

  • Two circular module dependencies were designed around during implementation, not discovered as bugs after the fact: (1) escalation initially needed sessions' repositories to end a session and sync ticket status, while sessions needed escalation to build the hand-off summary — resolved by making EscalationService.buildSummary a pure formatter with no repository/service dependencies of its own; sessions now owns ending its own session state and the ticket-status sync directly. (2) The GET .../ai-session/actions route initially lived in tools and imported sessions to resolve ticketId → session, which would have collided with sessions' own dependency on tools (for proposeAndEvaluate) — moved the route into sessions instead, which already owns that resolution; tools stays a clean leaf module with no dependency on ai-support/sessions at all.
  • Found and fixed a real Prisma bug before it reached tests: AIConfidencePolicy.upsert initially used Prisma's generated productId_categoryId compound-unique where shape, which rejects null for the (nullable) categoryId column at the client-API level ("Argument categoryId must not be null") even though the DB-level unique index itself permits it. Fixed by switching to findFirst + update/create instead of upsert — the same class of fix KnowledgeRepository.updateCurrent (004) already used for the same underlying Prisma limitation, discovered independently here.
  • ticket-state-machine.ts's AI_ statuses required two hooks into 003-ticketing's tickets.service.ts* to actually be driven correctly: (1) createFromInboundRequest enqueues the AI_SESSION job directly via queueManager (no import of ai-support/sessions — the worker, not the enqueue call, is what depends on it), and (2) updateStatus now publishes a DomainEventName.TICKET_UPDATED domain event unconditionally after every status change, using the event-bus scaffold (src/events/) that existed in this codebase from the original scaffold but had never been wired to anything — ai-support/sessions subscribes to it (registered in src/events/handlers/index.ts) to implement FR-023 (a human actor ends the AI session) without tickets ever needing to know ai-support/sessions exists.
  • The runbook-matching convention is a real, disclosed scope decision, not an oversight: a runbook's key is matched directly against the diagnosis's problemType string (no fuzzy matching, no separate mapping table) — admins author runbook keys to match the exact problemType vocabulary the AI's diagnosis call produces. This is simple and works, but is inherently a naming-convention contract between the diagnosis system prompt and runbook authoring, not a robust semantic match — documented in session.service.ts's enterTroubleshooting and in research.md.
  • 9 unit tests (confidence-band, tool-policy-gate, runbook-step-advance) and 9 integration test files were added. The AI-independent ones (ai-confidence-policy.test.ts, the deterministic tool-policy-gate re-check in ai-tools-and-runbook.test.ts, and the message-routing guard in ai-clarification.test.ts) run unconditionally and were verified passing against a real Postgres/Redis/MinIO. The remaining integration tests and the two constitution-required standing E2E scenarios (e2e-ai-flows.test.ts) require a real ANTHROPIC_API_KEY and are gated with describe.skipIf so the suite skips them cleanly (not a failure) rather than requiring every contributor to hold a live credential just to run the test suite — they were written and confirmed to compile and skip correctly, but not yet run against a live model in this environment (no key was available this session). The "AI resolves directly" E2E test additionally exercises the resolution-guard transition deterministically (via SessionsService.recheckVerification, a new seam also intended for a future real product-signal webhook) rather than relying solely on live-model non-determinism to reach that state.
  • Full regression (all 17 pre-existing integration test files plus every new one) was run together against real Docker-provisioned Postgres/Redis/MinIO: 88 passed, 9 skipped (the AI-key-gated ones), 0 failed.
Reference in New Issue View Git Blame Copy Permalink