diff --git a/specs/015-reporting-dashboards/checklists/requirements.md b/specs/015-reporting-dashboards/checklists/requirements.md new file mode 100644 index 0000000..f411e68 --- /dev/null +++ b/specs/015-reporting-dashboards/checklists/requirements.md @@ -0,0 +1,49 @@ +# Specification Quality Checklist: Reporting and Analytics Dashboards + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-09-09 +**Feature**: [spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- This is `docs/10-implementation-roadmap.md`'s own Phase 11, third sub-area, per explicit user + direction (013 was the security pass, 014 was full observability). Backend-first scope + (Assumptions) follows the same pattern already established three times this session + (010-identity-auth, 011-agent-ticket-queue, and 014-full-observability's own frontend-free + scope) — a `supporthub-web` dashboard UI is a natural, separate follow-on, not re-litigated + here via a fresh question. +- The pre-scaffolded-but-inert `platform/reports` module (`ReportsService.generateSummaryReport` + currently returns `{}`) and the `ANALYTICS` queue stub (`src/jobs/analytics`, logs only) were + both confirmed via direct code inspection before writing this spec — the same + "provisioned before this session's rebuild but never wired up" pattern found repeatedly this + session. This feature wires up the former; the Assumptions section explicitly keeps the latter + out of scope (synchronous queries, no pre-aggregation job, for this first cut). +- All items pass; no revision iterations were needed. No [NEEDS CLARIFICATION] markers were + required — every open question (default date window, SLA-risk threshold, top-N limit) had a + reasonable, documented, CONFIGURABLE default (see Assumptions), matching the roadmap's own + "never hardcode a placeholder value and ship it as final" instruction. diff --git a/specs/015-reporting-dashboards/spec.md b/specs/015-reporting-dashboards/spec.md new file mode 100644 index 0000000..cf22b91 --- /dev/null +++ b/specs/015-reporting-dashboards/spec.md @@ -0,0 +1,246 @@ +# Feature Specification: Reporting and Analytics Dashboards + +**Feature Branch**: `015-reporting-dashboards` + +**Created**: 2026-09-09 + +**Status**: Draft + +**Input**: User description: "Reporting and analytics dashboards: real, read-only aggregation endpoints backing the four dashboards named in docs/09-testing-observability-cicd.md (Management, Product, Support, AI) — wiring up the pre-scaffolded but never-implemented platform/reports module into actual database-backed aggregation queries, admin-gated, with a date-range filter." + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - Management sees organization-wide support health (Priority: P1) + +An admin or team lead opens a single view showing how support is doing overall for a chosen +period: how many cases came in, how many were resolved (by AI vs. by a human), how many are +still open, whether SLA commitments are being met, and how escalation is trending. + +**Why this priority**: This is the one dashboard covering the whole roadmap's own top-level +success criteria (`docs/10-implementation-roadmap.md`'s checklist) in one place — the first +thing anyone asks about a support operation is "how are we doing," and today there is no way to +answer that except querying the database by hand. + +**Independent Test**: Can be fully tested by creating a known set of tickets in various terminal +states (AI-resolved, human-resolved, still open) plus a mix of met/breached SLA runs and +escalations within a chosen date range, then requesting the Management dashboard for that range +and confirming every figure matches what was actually created. + +**Acceptance Scenarios**: + +1. **Given** a mix of tickets created within a chosen date range — some AI-resolved, some + human-resolved, some still open — **When** the Management dashboard is requested for that + range, **Then** total cases, AI-resolved count, human-escalated count, resolved count, and + open count all match the actual data exactly. +2. **Given** SLA runs that completed on time and others that breached within the range, + **When** the dashboard is requested, **Then** SLA compliance (a rate) and SLA breach count + both reflect the real outcomes. +3. **Given** some tickets have a recorded first agent response and a resolution timestamp, + **When** the dashboard is requested, **Then** average response time and average resolution + time are computed only from tickets that actually reached those milestones within the range + (a still-open ticket contributes to "open count" but never a fabricated resolution time). +4. **Given** a date range with zero activity, **When** the dashboard is requested, **Then** every + count is zero and every average is reported as "no data" rather than a computed zero or a + division-by-zero error. + +--- + +### User Story 2 - See support broken down by product (Priority: P1) + +An admin viewing support data for a specific product (or comparing products) sees volume, +problem-type breakdown, which problems recur most, how well AI is resolving that product's +issues versus escalating them, and which error codes come up most often. + +**Why this priority**: SupportHub serves multiple SaaS products (Constitution Principle I); a +number that isn't broken out by product hides which integration actually needs attention — this +is as fundamental as the Management view, just sliced differently. + +**Independent Test**: Can be fully tested by creating tickets/problems/error-code lookups across +two distinct products within a date range, requesting the Product dashboard for each product, +and confirming each one's figures include only its own product's data. + +**Acceptance Scenarios**: + +1. **Given** tickets exist for two different products in the same date range, **When** the + Product dashboard is requested scoped to one product, **Then** support volume and every other + figure reflect only that product's tickets, never the other product's. +2. **Given** problems in several categories for one product, **When** the dashboard is + requested, **Then** the problem-type breakdown and "recurring problems" ranking both reflect + the real category distribution, most-frequent first. +3. **Given** a mix of AI-resolved and human-escalated tickets for one product, **When** the + dashboard is requested, **Then** AI resolution rate and human escalation rate are both + computed as a percentage of that product's own total, not the platform-wide total. +4. **Given** several known-error-code lookups for one product, some codes looked up more than + others, **When** the dashboard is requested, **Then** "top errors" lists those codes ranked by + lookup frequency. + +--- + +### User Story 3 - Support sees team workload and performance (Priority: P2) + +An admin or team lead sees how much work is currently assigned across agents, which tickets are +at SLA risk, how much escalation is happening, and how quickly the team is responding to and +resolving tickets. + +**Why this priority**: This view is about ongoing operational load, not historical trend — useful +for day-to-day team management, but the organization can already see whether it's healthy +overall from User Story 1 without this one; P2 reflects that it adds an operational lens rather +than a new class of information. + +**Independent Test**: Can be fully tested by assigning several tickets to known agents (some +close to SLA breach, some not), then requesting the Support dashboard and confirming workload +per agent and the SLA-risk count both match reality. + +**Acceptance Scenarios**: + +1. **Given** several tickets are currently assigned across two agents, **When** the Support + dashboard is requested, **Then** each agent's current open-assignment count matches what was + actually assigned to them (not a stale count from a previous, now-unassigned period). +2. **Given** a ticket's SLA run is running and past a configurable risk threshold of its + resolution due date (but not yet breached), **When** the dashboard is requested, **Then** it + is counted as "at risk," distinct from both "on track" and "breached." +3. **Given** response and resolution durations for several resolved tickets in the period, + **When** the dashboard is requested, **Then** response-performance and resolution-performance + figures are computed only from tickets that actually reached those milestones. + +--- + +### User Story 4 - See how well the AI is performing (Priority: P2) + +An admin sees, for a chosen period, how often the AI resolves issues on its own versus escalating +them, how often its attempted troubleshooting fails outright, how often it finds relevant +knowledge, how confident its diagnoses tend to be, how reliably its tools succeed, and how often +it ultimately hands off to a human. + +**Why this priority**: This is the dashboard that validates the AI-first design's core premise +(Constitution Principle IV) is actually working in practice — valuable, but a narrower audience +than the org-wide and per-product views above, hence P2. + +**Independent Test**: Can be fully tested by running several AI sessions to different terminal +outcomes (resolved, escalated, escalated-after-failed-troubleshooting) with a mix of tool +successes/failures and confidence levels recorded, then requesting the AI dashboard and +confirming every figure matches the real session data. + +**Acceptance Scenarios**: + +1. **Given** a mix of AI sessions ending resolved vs. escalated in the period, **When** the AI + dashboard is requested, **Then** AI resolution rate and human-handoff rate both reflect the + real outcome mix as percentages of total sessions. +2. **Given** some AI tool invocations succeeded and others failed in the period, **When** the + dashboard is requested, **Then** tool success/failure figures reflect the real invocation + outcomes. +3. **Given** diagnoses were recorded with a range of confidence values, **When** the dashboard is + requested, **Then** the confidence distribution groups them into the same high/medium/low + bands the AI support module itself already uses (005-ai-support), not a newly-invented scheme. +4. **Given** some AI sessions' knowledge-retrieval step found matching entries and others found + none, **When** the dashboard is requested, **Then** knowledge-match rate reflects the real + match/no-match mix. + +--- + +### Edge Cases + +- What happens when no `from`/`to` date range is given? Defaults to a reasonable trailing window + (see Assumptions) rather than scanning the entire history unbounded on every request. +- What happens when `from` is after `to`? Rejected as a validation error, not silently swapped or + silently returning empty data. +- What happens when a requested `productId` (Product dashboard) doesn't exist? Rejected with a + clear not-found error, not an empty-but-200 response that looks like "this product has zero + activity." +- What happens when an average would divide by zero (no tickets reached that milestone in the + range)? Reported as an explicit "no data" value, never `NaN`, `null` silently coerced to `0`, + or a thrown error. +- What happens when a ticket's SLA run was paused for part of the period? SLA-risk/compliance + figures use the run's own already-durable due dates (008-sla-escalation's pause/resume + already accounts for paused time) rather than this feature re-deriving elapsed time itself. +- Who can see these dashboards? Same admin-only gate as every other admin configuration/reporting + surface introduced since 010-identity-auth — no new role is introduced. + +## Requirements *(mandatory)* + +### Functional Requirements + +- **FR-001**: System MUST provide a Management dashboard summarizing, for a given date range: + total cases created, cases resolved by AI, cases escalated to a human, total resolved + (either path), total still open, SLA compliance rate, SLA breach count, escalation count, + average first-response time, and average resolution time. +- **FR-002**: System MUST provide a Product dashboard summarizing, for a given date range and a + specific product: support volume, a breakdown by problem category, a ranked list of the most + recurring problem categories, AI resolution rate, human escalation rate, and a ranked list of + the most frequently looked-up error codes. +- **FR-003**: System MUST provide a Support dashboard summarizing, for a given date range: + current per-agent open-assignment workload, count of tickets at SLA risk (past a configurable + risk threshold of their resolution due date but not yet breached), count of tickets already + breached, escalation count, average response performance, and average resolution performance. +- **FR-004**: System MUST provide an AI dashboard summarizing, for a given date range: AI + resolution rate, rate of sessions that escalated after at least one failed troubleshooting + attempt, knowledge-match rate, a distribution of diagnosis confidence across the existing + high/medium/low bands, tool invocation success/failure counts, and human-handoff rate. +- **FR-005**: Every dashboard endpoint MUST accept an optional `from`/`to` date range; when + omitted, it MUST default to a documented trailing window rather than scanning unbounded + history. +- **FR-006**: The Product dashboard MUST require a valid `productId` and MUST reject an unknown + one with a clear not-found error rather than returning an empty-but-successful response. +- **FR-007**: Every rate/average figure MUST be computed only from tickets/sessions/runs that + actually reached the relevant milestone within the range; a metric with no qualifying data MUST + be reported as an explicit "no data" value, never a computed `0`, `null`, or `NaN`. +- **FR-008**: All four dashboard endpoints MUST be admin-gated, consistent with every other + admin-only reporting/configuration surface in this codebase. +- **FR-009**: This feature MUST NOT alter what any existing endpoint, event, or table stores — + every figure is derived read-only from data already durably recorded by the modules that own + it (003 ticketing, 005 AI support, 007 orchestration, 008 SLA/escalation, 009 problem + resolution). +- **FR-010**: This feature is backend-only; presenting these figures in a UI is a separate, + explicitly out-of-scope follow-on (see Assumptions). + +### Key Entities + +- **Dashboard response**: A read-only, computed JSON summary for one of the four dashboards over + a requested date range (and, for the Product dashboard, one product) — never itself persisted; + recomputed fresh on every request from existing durable records. +- **Date range**: An inclusive `from`/`to` pair (calendar dates or timestamps) scoping every + aggregation query; not a stored entity, a request parameter. +- **Confidence band**: The existing high/medium/low classification 005-ai-support already applies + to a diagnosis's confidence score — reused here for the AI dashboard's distribution, not + redefined. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: For any chosen date range, every figure on all four dashboards can be independently + verified against the underlying ticket/session/SLA-run/escalation-event records and matches + exactly — no discrepancy between what a dashboard reports and what actually happened. +- **SC-002**: An admin can answer "how is support doing right now" (Management), "how is this + specific product doing" (Product), "who's overloaded and what's at risk" (Support), and "is the + AI actually helping" (AI) each from a single request, with no manual database query needed. +- **SC-003**: A dashboard request for a period with no matching activity returns clean, explicit + "no data" results in well under a second — never an error, a stall, or a misleading zero. + +## Assumptions + +- **Presentation is out of scope for this feature.** The user's own explicit direction was to + build the backend aggregation capability first (the established pattern this project has + followed for every prior feature that touched both repos — identity/auth, the agent ticket + queue, and full observability were each built backend-first). A `supporthub-web` dashboard UI + consuming these endpoints is a natural, separate follow-on, not bundled into this spec. +- The default trailing window when no date range is given is the last 30 days, matching common + reporting-dashboard convention; CONFIGURABLE via the same admin-config env-driven pattern this + project already uses for every other business-policy value (Constitution Principle II), not + hardcoded as a magic number in application logic. +- "SLA risk" needs a threshold (how close to the due date counts as "at risk") that the business + has not specified — CONFIGURABLE, not invented as a hardcoded percentage, consistent with + `docs/10-implementation-roadmap.md`'s own "never hardcode a placeholder value and ship it as + final" instruction. +- These endpoints compute their figures synchronously, on request, directly from the existing + tables — no new pre-aggregation table, no scheduled batch job, and no use of the pre-scaffolded + `ANALYTICS` queue (`src/jobs/analytics`), which remains an inert stub outside this feature's + scope. Live query performance at current data volumes is assumed adequate; a future feature can + introduce pre-aggregation if and when it's actually needed (load/concurrency testing, a + separate not-yet-started Phase 11 sub-area, is where that question would be validated). +- "Top errors"/"recurring problems" rankings return a bounded top-N list (CONFIGURABLE limit, + defaulting to 10) rather than the full distribution, matching how a dashboard is actually + consumed. +- Dashboard responses are computed fresh per request (no caching layer) — acceptable given the + assumed data volumes and consistent with not prematurely optimizing ahead of the load-testing + phase.