diff --git a/specs/015-reporting-dashboards/contracts/reports-api-contract.md b/specs/015-reporting-dashboards/contracts/reports-api-contract.md new file mode 100644 index 0000000..10982d3 --- /dev/null +++ b/specs/015-reporting-dashboards/contracts/reports-api-contract.md @@ -0,0 +1,59 @@ +# Contract: Reporting API + +All four routes require a valid staff session with role `ADMIN` (`requireRole('ADMIN')`), the +same gate every admin-only surface uses since 010-identity-auth. All return the standard +envelope: `{ success: true, data: , meta: null }` on success, `{ success: false, error: +{code, message, details} }` on failure — no change to this codebase's existing response +convention. + +## `GET /admin/reports/management` + +**Query**: `from?`, `to?` (ISO dates). + +**200**: `ManagementDashboard` (data-model.md). + +**400** `VALIDATION_ERROR`: `from` is after `to`. + +**401/403**: missing/invalid session, or a non-`ADMIN` role. + +## `GET /admin/reports/product/:externalProductId` + +**Path**: `externalProductId` — the SaaS-facing product identifier (same convention every other +admin product-scoped route already uses, e.g. `GET /admin/products/:externalProductId/knowledge` +from 004-product-knowledge). + +**Query**: `from?`, `to?`. + +**200**: `ProductDashboard`. + +**404** `NOT_FOUND`: no product with that `externalProductId` (FR-006 — never an empty-but-200 +response for an unknown product). + +**400** `VALIDATION_ERROR`: `from` is after `to`. + +## `GET /admin/reports/support` + +**Query**: `from?`, `to?` (applies only to the performance figures — workload/SLA-risk/breached +are always current, per data-model.md's `SupportDashboard.generatedAt`). + +**200**: `SupportDashboard`. + +## `GET /admin/reports/ai` + +**Query**: `from?`, `to?`. + +**200**: `AiDashboard`. + +## Guarantees + +1. Every rate/average field is `number | null` — `null` means no qualifying data existed in the + requested range (FR-007). A consumer must never see `NaN` or a silently-substituted `0` for + "no data." +2. Every count field is a plain `number`, always present, `0` is a legitimate, meaningful value + for a count (distinct from the `null`-for-no-data rule above, which applies only to + rates/averages). +3. `from`/`to` in every response echo the *resolved* range actually used (including the default, + when omitted) — a caller never has to separately know what "the default" was. +4. No route in this contract mutates any data — a repeated identical request returns the same + shape (though not necessarily identical figures, since the underlying data can change between + requests) with no side effect. diff --git a/specs/015-reporting-dashboards/data-model.md b/specs/015-reporting-dashboards/data-model.md new file mode 100644 index 0000000..0596553 --- /dev/null +++ b/specs/015-reporting-dashboards/data-model.md @@ -0,0 +1,93 @@ +# Data Model: Reporting and Analytics Dashboards + +## New Prisma Model + +### `ErrorCodeLookup` + +Append-only audit record — see research.md §6 for why this is the one new table this feature +needs. + +| Field | Type | Notes | +|---|---|---| +| `id` | `String @id @default(cuid())` | | +| `errorCodeId` | `String` | FK → `ErrorCode.id` | +| `productId` | `String` | FK → `Product.id` — denormalized from `errorCode.productId` so the Product dashboard's range query never needs to join back through `ErrorCode` just to filter by product | +| `createdAt` | `DateTime @default(now())` | | + +Indexes: `@@index([productId, createdAt])` (the Product dashboard's own access pattern). + +No `updatedAt`, no soft-delete, no unique constraint — every lookup is its own row, duplicates +across time are the entire point (frequency is what "top errors" measures). + +## Response Shapes (not persisted — computed per request) + +### Management dashboard — `GET /admin/reports/management` + +```ts +interface ManagementDashboard { + range: { from: string; to: string }; // ISO 8601, echoes the resolved (possibly defaulted) range + totalCases: number; + aiResolved: number; + humanEscalated: number; + resolved: number; + open: number; + slaCompliance: { met: number; breached: number; rate: number | null }; // rate = met / (met + breached) + escalationCount: number; + averageResponseSeconds: number | null; + averageResolutionSeconds: number | null; +} +``` + +### Product dashboard — `GET /admin/reports/product/:externalProductId` + +```ts +interface ProductDashboard { + productId: string; // externalProductId, echoed back + range: { from: string; to: string }; + supportVolume: number; + problemsByCategory: Array<{ categoryId: string | null; count: number }>; + recurringProblems: Array<{ categoryId: string | null; count: number }>; // same data, top N, descending + aiResolutionRate: number | null; + humanEscalationRate: number | null; + topErrors: Array<{ code: string; count: number }>; // top N, descending +} +``` + +### Support dashboard — `GET /admin/reports/support` + +```ts +interface SupportDashboard { + generatedAt: string; // workload/risk are point-in-time, not range-scoped (research.md §2) + range: { from: string; to: string }; // still applies to the performance figures below + workloadByAgent: Array<{ agentId: string; openAssignments: number }>; + slaAtRisk: number; + slaBreached: number; + escalationCount: number; + averageResponseSeconds: number | null; + averageResolutionSeconds: number | null; +} +``` + +### AI dashboard — `GET /admin/reports/ai` + +```ts +interface AiDashboard { + range: { from: string; to: string }; + totalSessions: number; + aiResolutionRate: number | null; + humanHandoffRate: number | null; + failedTroubleshootingEscalationRate: number | null; + knowledgeMatchRate: number | null; + confidenceDistribution: { proceed: number; ask: number; escalate: number }; + toolInvocations: { success: number; failed: number }; +} +``` + +## Query Parameters (all four routes) + +| Param | Type | Notes | +|---|---|---| +| `from` | ISO date, optional | Defaults to `to - REPORTING_DEFAULT_WINDOW_DAYS` | +| `to` | ISO date, optional | Defaults to now | + +`from > to` is a 400 `VALIDATION_ERROR` (spec.md Edge Cases), not silently swapped. diff --git a/specs/015-reporting-dashboards/plan.md b/specs/015-reporting-dashboards/plan.md new file mode 100644 index 0000000..ade5492 --- /dev/null +++ b/specs/015-reporting-dashboards/plan.md @@ -0,0 +1,137 @@ +# Implementation Plan: Reporting and Analytics Dashboards + +**Branch**: `015-reporting-dashboards` | **Date**: 2026-09-09 | **Spec**: [spec.md](./spec.md) + +**Input**: Feature specification from `specs/015-reporting-dashboards/spec.md` + +## Summary + +Wires the pre-scaffolded, unused `platform/reports` module into four real, admin-gated, +read-only aggregation endpoints (Management, Product, Support, AI) matching +`docs/09-testing-observability-cicd.md`'s own dashboard table — each computed synchronously, +on request, directly from existing durable tables (Ticket, Problem, SLARun, EscalationEvent, +AISupportSession, AIDiagnosis, AIAction, Resolution, Assignment). The one new piece of state is +a small durable `ErrorCodeLookup` audit table, needed only because no existing record lets "top +errors" be computed historically (014-full-observability's own equivalent is a process-lifetime +Prometheus counter, unusable for a dated report). No presentation layer — see spec.md's +Assumptions for why `supporthub-web` work is a separate follow-on. + +## Technical Context + +**Language/Version**: TypeScript 5.4 / Node.js 20+ (unchanged). + +**Primary Dependencies**: None new — Prisma's own `groupBy`/`count`/`aggregate`/`findMany`, no +raw SQL (research.md §5), reusing `decideConfidenceBand` (005-ai-support) and the +`Resolution.resolvedBy` convention (014-full-observability) rather than reimplementing either. + +**Storage**: One new table, `ErrorCodeLookup` (`id`, `errorCodeId` FK, `productId` FK, +`createdAt`) — append-only, no update/delete path, indexed `(productId, createdAt)` for the +Product dashboard's range-scoped ranking query. No change to any existing table. + +**Testing**: Vitest — unit tests for the "no data → `null`, never `NaN`" averaging helper and the +confidence-bucketing reuse; integration tests against real Postgres/Redis driving each +dashboard's real underlying data (tickets in various terminal states, SLA runs met/breached, +escalation events, AI sessions/diagnoses/actions, error-code lookups) and asserting every +returned figure against hand-computed expected values — the same rigor and mixed +HTTP-driven/direct-repository setup style as 014's `business-metrics.test.ts`. + +**Target Platform**: Same Fastify modular monolith. Rewrites `platform/reports` (service, +new controller, new routes, new schema for the date-range/product-id query params) from its +current one-stub-method state into the real module. Adds one line to +`ai-support/knowledge/service/error-codes.service.ts`'s existing `findKnownIssuesByErrorCode` +(the same call site 014 already instrumented) to also write the new durable audit row. + +**Project Type**: Backend service — single project. + +**Performance Goals**: Every dashboard query is bounded by the requested date range (default 30 +days, config) and, where a full-row fetch is needed for in-application averaging (research.md +§5), only the two timestamp columns needed for that specific average — never a full-table scan +with no range filter. Acceptable at current data volumes per spec.md's own Assumptions; +pre-aggregation is explicitly deferred to if/when load testing (a separate, not-yet-started +Phase 11 sub-area) shows it's actually needed. + +**Constraints**: FR-006 — an unknown `productId` on the Product dashboard is a 404, never an +empty-but-200 response. FR-007 — every rate/average is `number | null`, `null` meaning "no +qualifying data," computed by checking the qualifying count before ever dividing. FR-008 — every +route requires `requireRole('ADMIN')`, the same gate every admin surface uses since +010-identity-auth. + +**Scale/Scope**: Four new `GET` routes, one new Prisma model + migration, four new service +methods (one per dashboard) replacing the single stub method, one new schema file for query-param +validation, three new env-configured values (Constitution Principle II). No new module — this +extends `platform/reports`, already the correct architectural home. + +## Constitution Check + +*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.* + +| Principle / Section | Check | Result | +|---|---|---| +| I. SaaS Is the Sole Identity & Access Authority | Not applicable — no identity/access surface touched; every figure is derived from SupportHub's own domain data (tickets, problems, SLA, escalation, AI sessions), squarely inside SupportHub's own sole-authority domain per this principle's own second sentence. | PASS | +| II. Configuration Over Hardcoding | The default reporting window, the SLA-risk threshold, and the top-N ranking limit are all new env-configured values (`REPORTING_DEFAULT_WINDOW_DAYS`, `REPORTING_SLA_RISK_THRESHOLD_MINUTES`, `REPORTING_TOP_N_LIMIT`), never hardcoded — matches spec.md's own Assumptions and the roadmap's "never hardcode a placeholder value and ship it as final." | PASS | +| III. Layered Architecture With Enforced Module Boundaries | All new code lives inside `platform/reports` (already its correct home) following Route → Schema → Controller → Service → Repository → Prisma; cross-module reads (tickets, AI support, orchestration, SLA/escalation, problem resolution) go through each owning module's own public `index.ts`, the same precedent every prior feature this session established (e.g. `tool-executor.ts` reading `ticketsService` from `@/modules/ticketing/tickets`). | PASS | +| IV. AI Recommends, Deterministic Policy Decides | Not applicable — no AI tool-execution or decision logic changed; the AI dashboard only reports on outcomes the existing, already-deterministic confidence-band/tool-policy code already produced. | PASS — N/A | +| V. Evidence-Based Verification | Not applicable — no resolution-recording logic changed. | PASS — N/A | +| VI. Durable Audit & History | The one new table (`ErrorCodeLookup`) is itself an append-only audit record, directly in this principle's spirit — "which error codes came up, when" becomes durably answerable for the first time. | PASS | +| VII. Concurrency-Safe, Durable Job Handling | Not applicable — read-only aggregation queries, no job handlers, no assignment/SLA state mutated. | PASS — N/A | +| VIII. Problem and Ticket Are Separate, Related Entities | Respected — the Product dashboard's problem-type breakdown queries `Problem` directly, never conflating it with `Ticket`. | PASS | +| Technology & Platform Constraints | No new dependencies; one new Prisma model via the established non-interactive migration workflow (`prisma migrate diff` → hand-written `migration.sql` → `prisma migrate deploy`) this session has used for every prior schema change. | PASS | + +No violations requiring Complexity Tracking justification. + +## Project Structure + +### Documentation (this feature) + +```text +specs/015-reporting-dashboards/ +├── plan.md +├── research.md +├── data-model.md +├── quickstart.md +├── contracts/ +│ └── reports-api-contract.md +└── tasks.md +``` + +### Source Code (repository root) + +```text +supporthub-api/ +├── prisma/ +│ ├── schema.prisma # MODIFIED — new ErrorCodeLookup model +│ └── migrations/ +│ └── _add_error_code_lookup/migration.sql # NEW +├── src/ +│ ├── config/ +│ │ └── env.ts / reporting.ts (or similar) # MODIFIED — 3 new env-configured values +│ └── modules/ +│ ├── platform/ +│ │ └── reports/ # REWRITTEN (was a 1-method stub) +│ │ ├── controller/ +│ │ ├── mapper/ # date-range parsing/defaulting, averaging helper +│ │ ├── repository/ # the 4 dashboards' Prisma queries +│ │ ├── routes/ +│ │ ├── schema/ # query-param validation +│ │ ├── service/ +│ │ └── index.ts +│ └── ai-support/ +│ └── knowledge/ +│ ├── repository/ # MODIFIED — errorCodeLookupRepository +│ └── service/ +│ └── error-codes.service.ts # MODIFIED — one new line at the existing +│ lookup call site +└── tests/ + ├── unit/platform/reports/ # averaging/no-data-null helper, confidence + │ bucketing reuse + └── integration/platform-reports/ # all four dashboards against real data +``` + +**Structure Decision**: Single project, no new module — `platform/reports` already exists as the +correct architectural home and simply needs its real implementation built out, following the +same Route → Schema → Controller → Service → Repository → Prisma layering every other module +already uses. + +## Complexity Tracking + +*No constitution violations — table intentionally omitted.* diff --git a/specs/015-reporting-dashboards/quickstart.md b/specs/015-reporting-dashboards/quickstart.md new file mode 100644 index 0000000..69f692f --- /dev/null +++ b/specs/015-reporting-dashboards/quickstart.md @@ -0,0 +1,52 @@ +# Quickstart: Reporting and Analytics Dashboards + +Manual verification steps for each user story, against a running instance backed by real +Postgres/Redis, logged in as an ADMIN. + +## Scenario 1 — Management dashboard (User Story 1) + +1. Create several tickets within a known date range: some reaching `AI_RESOLVED`/`RESOLVED` via + an AI session, some escalated to a human and resolved via `resolutionsService.record`, some + left open. +2. Let one ticket's SLA run complete on time and another breach (via the existing breach sweep). +3. `GET /admin/reports/management?from=&to=`. +4. **Expected**: `totalCases`, `aiResolved`, `humanEscalated`, `resolved`, `open` all match what + was actually created; `slaCompliance.met`/`.breached` match the two SLA outcomes; + `averageResponseSeconds`/`averageResolutionSeconds` are non-null and plausible. +5. Request the same endpoint for a date range with no activity at all. +6. **Expected**: every count is `0`, every rate/average is `null`, not an error. + +## Scenario 2 — Product dashboard (User Story 2) + +1. Create tickets for two distinct products in the same range, one with a categorized problem. +2. Look up a known error code for one product several times, a different code once. +3. `GET /admin/reports/product/:externalProductId` for each product. +4. **Expected**: each product's `supportVolume`/`problemsByCategory`/`aiResolutionRate` reflect + only its own tickets; `topErrors` ranks the more-frequently-looked-up code first. +5. Request the endpoint for a nonexistent `externalProductId`. +6. **Expected**: `404 NOT_FOUND`, not an empty `200`. + +## Scenario 3 — Support dashboard (User Story 3) + +1. Assign several tickets across two agents (some via the real orchestration flow). +2. Let one ticket's SLA run sit within `REPORTING_SLA_RISK_THRESHOLD_MINUTES` of its resolution + due date without breaching. +3. `GET /admin/reports/support`. +4. **Expected**: `workloadByAgent` matches each agent's real current open-assignment count; + `slaAtRisk` counts exactly the near-due run, distinct from `slaBreached`. + +## Scenario 4 — AI dashboard (User Story 4) + +1. Run AI sessions to a mix of terminal outcomes (`resolved`, `escalated`), with some tool + invocations succeeding and others failing, and diagnoses spanning a range of confidence + values. +2. `GET /admin/reports/ai`. +3. **Expected**: `aiResolutionRate`/`humanHandoffRate` reflect the real outcome mix; + `confidenceDistribution` buckets match `decideConfidenceBand`'s own classification of each + diagnosis's stored confidence against the system-default thresholds; `toolInvocations` + reflects the real success/failure counts. + +## What "done" looks like + +All four scenarios pass against a real Postgres/Redis, every figure independently verified +against hand-computed expected values, and no route is reachable by a non-admin session. diff --git a/specs/015-reporting-dashboards/research.md b/specs/015-reporting-dashboards/research.md new file mode 100644 index 0000000..a87ee31 --- /dev/null +++ b/specs/015-reporting-dashboards/research.md @@ -0,0 +1,143 @@ +# Research: Reporting and Analytics Dashboards + +## 1. Where this lives + +**Decision**: Wire up the existing, pre-scaffolded `src/modules/platform/reports` module (today +just `ReportsService.generateSummaryReport()` returning `{}`, confirmed unused anywhere) rather +than creating a new module. Its four real methods (`getManagementDashboard`, +`getProductDashboard`, `getSupportDashboard`, `getAiDashboard`) replace the one stub method. +Routes live at `GET /admin/reports/management`, `GET /admin/reports/product/:externalProductId`, +`GET /admin/reports/support`, `GET /admin/reports/ai`, admin-gated the same way every other +admin-only endpoint since 010-identity-auth already is (`requireRole('ADMIN')`). + +**Why not the `ANALYTICS` queue** (`src/jobs/analytics`, also pre-scaffolded, also inert): a +queued background job fits pre-computing a report nobody is currently waiting on; a dashboard +request is someone waiting right now for an answer. Per spec.md's Assumptions, this first cut is +synchronous, direct-query aggregation — the queue stub stays exactly as inert as it already was, +untouched by this feature. + +## 2. Per-dashboard queries + +All four use Prisma's `groupBy`/`count`/`aggregate`, scoped by `createdAt` (or the +milestone-specific timestamp named below) within `[from, to]`, computed directly against the +tables that already own each fact — no new table, no denormalized rollup. + +### Management (FR-001) + +| Figure | Source | +|---|---| +| Total cases | `Ticket.count({ createdAt in range })` | +| AI resolved | `Ticket.count({ createdAt in range, status: 'AI_RESOLVED' })` — a ticket that reached `AI_RESOLVED` and stayed there or moved straight to `RESOLVED` without a `Resolution.resolvedBy` other than `'ai'`; see §4 below for the exact "who resolved it" rule shared with the Product dashboard | +| Human escalated | `Ticket.count({ createdAt in range, status in [HUMAN_ESCALATION, IN_PROGRESS, WAITING_FOR_CUSTOMER, RESOLUTION_PENDING_CUSTOMER, RESOLVED, CLOSED, REOPENED] })` minus AI-only-path tickets — i.e. any ticket that ever reached `HUMAN_ESCALATION`; the state machine research.md already establishes this as one-way (003-ticketing) | +| Resolved (either path) | `Ticket.count({ createdAt in range, status in [RESOLVED, CLOSED] })` | +| Open | `Ticket.count({ createdAt in range, status not in [RESOLVED, CLOSED] })` | +| SLA compliance / breach count | `SLARun.groupBy(['status'], { ticket: { createdAt in range } })`, `status: 'completed'` = met, `'breached'` = breached (mirrors 014's own metric semantics — see 014 research.md §5's "read status before the overwrite" caveat, which applies equally here: a `'breached'`-then-`'completed'` run is still counted breached, by reading the `breachedAt`/`firstResponseBreachedAt` timestamps rather than only the current `status` string) | +| Escalation count | `EscalationEvent.count({ createdAt in range })` | +| Average response time | `avg(firstAgentMessage.createdAt - ticket.createdAt)` over tickets with at least one `AGENT_MESSAGE` in range — computed in application code over a bounded query result (see §5, no raw SQL) | +| Average resolution time | `avg(resolution.createdAt - ticket.createdAt)` over tickets with a `Resolution` row in range | + +### Product (FR-002) + +Same shape as Management, `WHERE Ticket.productId = :productId` (resolved from the given +`externalProductId`, 404 if not found — FR-006), plus: + +| Figure | Source | +|---|---| +| Problem-type breakdown | `Problem.groupBy(['categoryId'], { productId, createdAt in range })` | +| Recurring problems | Same grouped result, sorted descending, top N (config, default 10) | +| Top errors | `reuses 014's own instrumentation point conceptually but queries fresh` — no, see §6: there is no persisted "error code lookup" table, only 014's in-memory Prometheus counter, which is NOT queryable historically. Resolved by adding a durable audit read instead: see §6. | + +### Support (FR-003) + +| Figure | Source | +|---|---| +| Per-agent workload | `Assignment.groupBy(['agentId'], { isCurrent: true })` — a snapshot of *right now*, not date-ranged (workload is inherently current, not historical — spec.md's own framing: "how much work is currently assigned") | +| SLA risk / breached | `SLARun.findMany({ status: 'running', resolutionDueAt: {gte: now} })` filtered in application code by "due within `SLA_RISK_THRESHOLD_MINUTES` of now" for risk, vs. `status: 'breached'` for already-breached | +| Escalation count | Same as Management, unfiltered by product | +| Response/resolution performance | Same computation as Management's averages | + +### AI (FR-004) + +| Figure | Source | +|---|---| +| AI resolution rate / human-handoff rate | `AISupportSession.groupBy(['status'], { startedAt in range })` — `resolved` vs. `escalated`/`ended_by_agent` as a share of total terminal sessions | +| Failed-troubleshooting-then-escalated rate | Sessions with `status: 'escalated'` that have at least one `AIInteraction`/`AIAction` recording a failed troubleshooting attempt — see 005-ai-support's own runbook-step-outcome classification (`classifyStepOutcome`), reused rather than reinvented | +| Knowledge-match rate | `AIKnowledgeReference` presence per session (`recordMany` is only ever called with actual retrieval results — 005-ai-support's own `diagnose.ts`) vs. sessions with zero references recorded | +| Confidence distribution | `AIDiagnosis.findMany({ createdAt in range })`, bucketed in application code against `aiConfig.defaultHighConfidence`/`defaultLowConfidence` (see §7 — NOT a per-diagnosis resolved policy) | +| Tool success/failure | `AIAction` joined to `AIActionResult`, grouped by `result.status` | + +## 3. "No data" convention (FR-007) + +**Decision**: every rate/average field is `number | null` — `null` means "no qualifying records +in range," distinguished in the response shape from a genuine `0` (e.g., a real 0% AI resolution +rate because everything escalated is a valid, meaningful `0`; "nobody's data exists yet" is +`null`). Application code computes every average by fetching the qualifying count first and +returning `null` before ever dividing, never relying on `0/0` producing `NaN` and hoping a caller +notices. + +## 4. "Who resolved it" — reused from 014, not reinvented + +014-full-observability's own event subscriber already established the authoritative rule: a +ticket's `Resolution.resolvedBy` field (`"ai"` | an `agentId`) is the single source of truth for +whether a resolution was AI- or human-driven (014 research.md §5). This feature's Management/ +Product dashboards reuse the exact same join (`Resolution.findMany` scoped to the range, +`resolvedBy === 'ai'` vs. not) rather than re-deriving it from `Ticket.status` transitions a +second, potentially-inconsistent way. + +## 5. No raw SQL + +**Decision**: every duration average (response time, resolution time) is computed by fetching +the bounded set of qualifying rows (ticket `createdAt` + the milestone timestamp) via Prisma and +averaging in application code, not a raw `$queryRaw` computing `AVG(EXTRACT(EPOCH FROM ...))` in +SQL. At the data volumes spec.md's Assumptions accept for this first cut (no pre-aggregation, +synchronous queries), a bounded per-range fetch is simple, type-safe, and testable without +hand-writing SQL — consistent with this codebase's near-total avoidance of `$queryRaw` elsewhere +(confirmed by grep: no existing module uses it for reporting-shaped queries). + +## 6. Top errors needs a durable, queryable record — a real gap 014 left open + +014-full-observability's `supporthub_known_error_lookups_total` Prometheus counter is +process-lifetime, in-memory, and reset on every restart — useless for "top errors in the last 30 +days." Since no durable "error code lookup" record exists anywhere in this codebase today (the +existing `error-codes.service.ts` just reads `KnownIssue`/`ErrorCode` rows, never records that a +lookup happened), this feature adds one small, focused piece of new state: a durable +`ErrorCodeLookup` audit row (`errorCodeId`, `productId`, `createdAt`), written by +`error-codes.service.ts`'s already-existing `findKnownIssuesByErrorCode` (the same call site +014 instrumented for its own live counter — this feature adds one more line there, a durable +write alongside the existing live-metric increment, not a replacement for it). This is the one +schema change this feature needs; every other dashboard figure is computed from tables that +already exist. + +## 7. Confidence distribution uses the system default threshold, not a per-diagnosis policy + +**Decision**: bucket every `AIDiagnosis.confidence` value in range against the env-configured +system-wide defaults (`aiConfig.defaultHighConfidence`/`defaultLowConfidence`), the same +`decideConfidenceBand` pure function 005-ai-support already exports — reused directly, not +reimplemented. + +**Why not resolve each diagnosis's actual applicable per-product/category policy** (what the +live reasoning path itself does): `AIDiagnosis.product`/`feature` are the AI's own free-text +classification output, not foreign keys to `Product`/`Category` — there is no reliable, existing +join from a diagnosis row back to which `ConfidencePolicy` row actually applied to it at the time +without speculatively string-matching free text against product names, which this codebase does +nowhere else and which research.md declines to invent here. A dashboard-level aggregate +distribution using the system-wide default is an honest, documented simplification (spec.md +Assumptions) — precise enough to show a meaningful shape without fabricating a false precision +the data doesn't actually support. + +## 8. New configuration (Constitution Principle II — nothing hardcoded) + +| Env var | Default | Used by | +|---|---|---| +| `REPORTING_DEFAULT_WINDOW_DAYS` | `30` | Every dashboard's `from`/`to` default when omitted (FR-005) | +| `REPORTING_SLA_RISK_THRESHOLD_MINUTES` | `60` | Support dashboard's "at risk" classification (FR-003) | +| `REPORTING_TOP_N_LIMIT` | `10` | Product dashboard's recurring-problems/top-errors ranking length | + +## 9. Test strategy + +Integration tests create real tickets/problems/SLA runs/escalation events/AI sessions/diagnoses/ +actions/error-code lookups directly against real Postgres (mixing real HTTP-driven setup where a +realistic flow matters and direct repository/Prisma writes where only the aggregation math is +under test — the same mix 014's own `business-metrics.test.ts` used), then request each +dashboard endpoint and assert every figure against hand-computed expected values. Unit tests +cover the "no data → null, never NaN" guard and the confidence-bucketing pure-function reuse. diff --git a/specs/015-reporting-dashboards/spec.md b/specs/015-reporting-dashboards/spec.md index cf22b91..c1b3225 100644 --- a/specs/015-reporting-dashboards/spec.md +++ b/specs/015-reporting-dashboards/spec.md @@ -130,8 +130,9 @@ confirming every figure matches the real session data. dashboard is requested, **Then** tool success/failure figures reflect the real invocation outcomes. 3. **Given** diagnoses were recorded with a range of confidence values, **When** the dashboard is - requested, **Then** the confidence distribution groups them into the same high/medium/low - bands the AI support module itself already uses (005-ai-support), not a newly-invented scheme. + requested, **Then** the confidence distribution groups them into the same proceed/ask/escalate + bands the AI support module's own confidence-policy service already classifies each diagnosis + into (005-ai-support), not a newly-invented scheme. 4. **Given** some AI sessions' knowledge-retrieval step found matching entries and others found none, **When** the dashboard is requested, **Then** knowledge-match rate reflects the real match/no-match mix. @@ -175,7 +176,7 @@ confirming every figure matches the real session data. - **FR-004**: System MUST provide an AI dashboard summarizing, for a given date range: AI resolution rate, rate of sessions that escalated after at least one failed troubleshooting attempt, knowledge-match rate, a distribution of diagnosis confidence across the existing - high/medium/low bands, tool invocation success/failure counts, and human-handoff rate. + proceed/ask/escalate bands, tool invocation success/failure counts, and human-handoff rate. - **FR-005**: Every dashboard endpoint MUST accept an optional `from`/`to` date range; when omitted, it MUST default to a documented trailing window rather than scanning unbounded history. @@ -186,10 +187,17 @@ confirming every figure matches the real session data. be reported as an explicit "no data" value, never a computed `0`, `null`, or `NaN`. - **FR-008**: All four dashboard endpoints MUST be admin-gated, consistent with every other admin-only reporting/configuration surface in this codebase. -- **FR-009**: This feature MUST NOT alter what any existing endpoint, event, or table stores — - every figure is derived read-only from data already durably recorded by the modules that own - it (003 ticketing, 005 AI support, 007 orchestration, 008 SLA/escalation, 009 problem - resolution). +- **FR-009**: This feature MUST NOT alter the meaning or shape of any existing endpoint, event, or + table — nearly every figure is derived read-only from data already durably recorded by the + modules that own it (003 ticketing, 005 AI support, 007 orchestration, 008 SLA/escalation, 009 + problem resolution). The one exception is FR-011: a small new durable record needed only + because no existing table can answer "which error codes are looked up most" historically. +- **FR-011**: System MUST durably record each known-error-code lookup (product, error code, + timestamp) at the point it already happens (the existing error-code lookup call site) so the + Product dashboard's "top errors" ranking (FR-002) can be computed historically — the + equivalent live, in-process counter this project already exposes on `/metrics` (014-full- + observability) is process-lifetime and reset on every restart, unusable for a historical + dashboard. - **FR-010**: This feature is backend-only; presenting these figures in a UI is a separate, explicitly out-of-scope follow-on (see Assumptions). @@ -200,8 +208,11 @@ confirming every figure matches the real session data. recomputed fresh on every request from existing durable records. - **Date range**: An inclusive `from`/`to` pair (calendar dates or timestamps) scoping every aggregation query; not a stored entity, a request parameter. -- **Confidence band**: The existing high/medium/low classification 005-ai-support already applies - to a diagnosis's confidence score — reused here for the AI dashboard's distribution, not +- **Error code lookup record** (new, FR-011): a durable, append-only audit row — which product, + which error code, when — written at the existing lookup call site; exists solely so "top + errors" can be computed over a historical range, never read or written anywhere else. +- **Confidence band**: The existing proceed/ask/escalate classification 005-ai-support already + applies to a diagnosis's confidence score — reused here for the AI dashboard's distribution, not redefined. ## Success Criteria *(mandatory)*