docs(015-reporting-dashboards): plan, research, data model, contract, quickstart
Documents the exact Prisma query per dashboard figure, the one new durable table this feature needs (ErrorCodeLookup — 014's own equivalent metric is process-lifetime, unusable for a historical report), the "no data -> null, never NaN" convention, and why the AI dashboard's confidence distribution deliberately uses the system-default threshold rather than resolving a per-diagnosis policy (AIDiagnosis has no reliable FK back to which policy applied). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
saqib mir
co-authored by
Claude Sonnet 5
Claude Sonnet 5
parent
c4a2faa6e3
commit
4a159725c2
@@ -0,0 +1,59 @@
|
|||||||
|
# Contract: Reporting API
|
||||||
|
|
||||||
|
All four routes require a valid staff session with role `ADMIN` (`requireRole('ADMIN')`), the
|
||||||
|
same gate every admin-only surface uses since 010-identity-auth. All return the standard
|
||||||
|
envelope: `{ success: true, data: <shape>, meta: null }` on success, `{ success: false, error:
|
||||||
|
{code, message, details} }` on failure — no change to this codebase's existing response
|
||||||
|
convention.
|
||||||
|
|
||||||
|
## `GET /admin/reports/management`
|
||||||
|
|
||||||
|
**Query**: `from?`, `to?` (ISO dates).
|
||||||
|
|
||||||
|
**200**: `ManagementDashboard` (data-model.md).
|
||||||
|
|
||||||
|
**400** `VALIDATION_ERROR`: `from` is after `to`.
|
||||||
|
|
||||||
|
**401/403**: missing/invalid session, or a non-`ADMIN` role.
|
||||||
|
|
||||||
|
## `GET /admin/reports/product/:externalProductId`
|
||||||
|
|
||||||
|
**Path**: `externalProductId` — the SaaS-facing product identifier (same convention every other
|
||||||
|
admin product-scoped route already uses, e.g. `GET /admin/products/:externalProductId/knowledge`
|
||||||
|
from 004-product-knowledge).
|
||||||
|
|
||||||
|
**Query**: `from?`, `to?`.
|
||||||
|
|
||||||
|
**200**: `ProductDashboard`.
|
||||||
|
|
||||||
|
**404** `NOT_FOUND`: no product with that `externalProductId` (FR-006 — never an empty-but-200
|
||||||
|
response for an unknown product).
|
||||||
|
|
||||||
|
**400** `VALIDATION_ERROR`: `from` is after `to`.
|
||||||
|
|
||||||
|
## `GET /admin/reports/support`
|
||||||
|
|
||||||
|
**Query**: `from?`, `to?` (applies only to the performance figures — workload/SLA-risk/breached
|
||||||
|
are always current, per data-model.md's `SupportDashboard.generatedAt`).
|
||||||
|
|
||||||
|
**200**: `SupportDashboard`.
|
||||||
|
|
||||||
|
## `GET /admin/reports/ai`
|
||||||
|
|
||||||
|
**Query**: `from?`, `to?`.
|
||||||
|
|
||||||
|
**200**: `AiDashboard`.
|
||||||
|
|
||||||
|
## Guarantees
|
||||||
|
|
||||||
|
1. Every rate/average field is `number | null` — `null` means no qualifying data existed in the
|
||||||
|
requested range (FR-007). A consumer must never see `NaN` or a silently-substituted `0` for
|
||||||
|
"no data."
|
||||||
|
2. Every count field is a plain `number`, always present, `0` is a legitimate, meaningful value
|
||||||
|
for a count (distinct from the `null`-for-no-data rule above, which applies only to
|
||||||
|
rates/averages).
|
||||||
|
3. `from`/`to` in every response echo the *resolved* range actually used (including the default,
|
||||||
|
when omitted) — a caller never has to separately know what "the default" was.
|
||||||
|
4. No route in this contract mutates any data — a repeated identical request returns the same
|
||||||
|
shape (though not necessarily identical figures, since the underlying data can change between
|
||||||
|
requests) with no side effect.
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
# Data Model: Reporting and Analytics Dashboards
|
||||||
|
|
||||||
|
## New Prisma Model
|
||||||
|
|
||||||
|
### `ErrorCodeLookup`
|
||||||
|
|
||||||
|
Append-only audit record — see research.md §6 for why this is the one new table this feature
|
||||||
|
needs.
|
||||||
|
|
||||||
|
| Field | Type | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| `id` | `String @id @default(cuid())` | |
|
||||||
|
| `errorCodeId` | `String` | FK → `ErrorCode.id` |
|
||||||
|
| `productId` | `String` | FK → `Product.id` — denormalized from `errorCode.productId` so the Product dashboard's range query never needs to join back through `ErrorCode` just to filter by product |
|
||||||
|
| `createdAt` | `DateTime @default(now())` | |
|
||||||
|
|
||||||
|
Indexes: `@@index([productId, createdAt])` (the Product dashboard's own access pattern).
|
||||||
|
|
||||||
|
No `updatedAt`, no soft-delete, no unique constraint — every lookup is its own row, duplicates
|
||||||
|
across time are the entire point (frequency is what "top errors" measures).
|
||||||
|
|
||||||
|
## Response Shapes (not persisted — computed per request)
|
||||||
|
|
||||||
|
### Management dashboard — `GET /admin/reports/management`
|
||||||
|
|
||||||
|
```ts
|
||||||
|
interface ManagementDashboard {
|
||||||
|
range: { from: string; to: string }; // ISO 8601, echoes the resolved (possibly defaulted) range
|
||||||
|
totalCases: number;
|
||||||
|
aiResolved: number;
|
||||||
|
humanEscalated: number;
|
||||||
|
resolved: number;
|
||||||
|
open: number;
|
||||||
|
slaCompliance: { met: number; breached: number; rate: number | null }; // rate = met / (met + breached)
|
||||||
|
escalationCount: number;
|
||||||
|
averageResponseSeconds: number | null;
|
||||||
|
averageResolutionSeconds: number | null;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Product dashboard — `GET /admin/reports/product/:externalProductId`
|
||||||
|
|
||||||
|
```ts
|
||||||
|
interface ProductDashboard {
|
||||||
|
productId: string; // externalProductId, echoed back
|
||||||
|
range: { from: string; to: string };
|
||||||
|
supportVolume: number;
|
||||||
|
problemsByCategory: Array<{ categoryId: string | null; count: number }>;
|
||||||
|
recurringProblems: Array<{ categoryId: string | null; count: number }>; // same data, top N, descending
|
||||||
|
aiResolutionRate: number | null;
|
||||||
|
humanEscalationRate: number | null;
|
||||||
|
topErrors: Array<{ code: string; count: number }>; // top N, descending
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Support dashboard — `GET /admin/reports/support`
|
||||||
|
|
||||||
|
```ts
|
||||||
|
interface SupportDashboard {
|
||||||
|
generatedAt: string; // workload/risk are point-in-time, not range-scoped (research.md §2)
|
||||||
|
range: { from: string; to: string }; // still applies to the performance figures below
|
||||||
|
workloadByAgent: Array<{ agentId: string; openAssignments: number }>;
|
||||||
|
slaAtRisk: number;
|
||||||
|
slaBreached: number;
|
||||||
|
escalationCount: number;
|
||||||
|
averageResponseSeconds: number | null;
|
||||||
|
averageResolutionSeconds: number | null;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### AI dashboard — `GET /admin/reports/ai`
|
||||||
|
|
||||||
|
```ts
|
||||||
|
interface AiDashboard {
|
||||||
|
range: { from: string; to: string };
|
||||||
|
totalSessions: number;
|
||||||
|
aiResolutionRate: number | null;
|
||||||
|
humanHandoffRate: number | null;
|
||||||
|
failedTroubleshootingEscalationRate: number | null;
|
||||||
|
knowledgeMatchRate: number | null;
|
||||||
|
confidenceDistribution: { proceed: number; ask: number; escalate: number };
|
||||||
|
toolInvocations: { success: number; failed: number };
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Query Parameters (all four routes)
|
||||||
|
|
||||||
|
| Param | Type | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| `from` | ISO date, optional | Defaults to `to - REPORTING_DEFAULT_WINDOW_DAYS` |
|
||||||
|
| `to` | ISO date, optional | Defaults to now |
|
||||||
|
|
||||||
|
`from > to` is a 400 `VALIDATION_ERROR` (spec.md Edge Cases), not silently swapped.
|
||||||
@@ -0,0 +1,137 @@
|
|||||||
|
# Implementation Plan: Reporting and Analytics Dashboards
|
||||||
|
|
||||||
|
**Branch**: `015-reporting-dashboards` | **Date**: 2026-09-09 | **Spec**: [spec.md](./spec.md)
|
||||||
|
|
||||||
|
**Input**: Feature specification from `specs/015-reporting-dashboards/spec.md`
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Wires the pre-scaffolded, unused `platform/reports` module into four real, admin-gated,
|
||||||
|
read-only aggregation endpoints (Management, Product, Support, AI) matching
|
||||||
|
`docs/09-testing-observability-cicd.md`'s own dashboard table — each computed synchronously,
|
||||||
|
on request, directly from existing durable tables (Ticket, Problem, SLARun, EscalationEvent,
|
||||||
|
AISupportSession, AIDiagnosis, AIAction, Resolution, Assignment). The one new piece of state is
|
||||||
|
a small durable `ErrorCodeLookup` audit table, needed only because no existing record lets "top
|
||||||
|
errors" be computed historically (014-full-observability's own equivalent is a process-lifetime
|
||||||
|
Prometheus counter, unusable for a dated report). No presentation layer — see spec.md's
|
||||||
|
Assumptions for why `supporthub-web` work is a separate follow-on.
|
||||||
|
|
||||||
|
## Technical Context
|
||||||
|
|
||||||
|
**Language/Version**: TypeScript 5.4 / Node.js 20+ (unchanged).
|
||||||
|
|
||||||
|
**Primary Dependencies**: None new — Prisma's own `groupBy`/`count`/`aggregate`/`findMany`, no
|
||||||
|
raw SQL (research.md §5), reusing `decideConfidenceBand` (005-ai-support) and the
|
||||||
|
`Resolution.resolvedBy` convention (014-full-observability) rather than reimplementing either.
|
||||||
|
|
||||||
|
**Storage**: One new table, `ErrorCodeLookup` (`id`, `errorCodeId` FK, `productId` FK,
|
||||||
|
`createdAt`) — append-only, no update/delete path, indexed `(productId, createdAt)` for the
|
||||||
|
Product dashboard's range-scoped ranking query. No change to any existing table.
|
||||||
|
|
||||||
|
**Testing**: Vitest — unit tests for the "no data → `null`, never `NaN`" averaging helper and the
|
||||||
|
confidence-bucketing reuse; integration tests against real Postgres/Redis driving each
|
||||||
|
dashboard's real underlying data (tickets in various terminal states, SLA runs met/breached,
|
||||||
|
escalation events, AI sessions/diagnoses/actions, error-code lookups) and asserting every
|
||||||
|
returned figure against hand-computed expected values — the same rigor and mixed
|
||||||
|
HTTP-driven/direct-repository setup style as 014's `business-metrics.test.ts`.
|
||||||
|
|
||||||
|
**Target Platform**: Same Fastify modular monolith. Rewrites `platform/reports` (service,
|
||||||
|
new controller, new routes, new schema for the date-range/product-id query params) from its
|
||||||
|
current one-stub-method state into the real module. Adds one line to
|
||||||
|
`ai-support/knowledge/service/error-codes.service.ts`'s existing `findKnownIssuesByErrorCode`
|
||||||
|
(the same call site 014 already instrumented) to also write the new durable audit row.
|
||||||
|
|
||||||
|
**Project Type**: Backend service — single project.
|
||||||
|
|
||||||
|
**Performance Goals**: Every dashboard query is bounded by the requested date range (default 30
|
||||||
|
days, config) and, where a full-row fetch is needed for in-application averaging (research.md
|
||||||
|
§5), only the two timestamp columns needed for that specific average — never a full-table scan
|
||||||
|
with no range filter. Acceptable at current data volumes per spec.md's own Assumptions;
|
||||||
|
pre-aggregation is explicitly deferred to if/when load testing (a separate, not-yet-started
|
||||||
|
Phase 11 sub-area) shows it's actually needed.
|
||||||
|
|
||||||
|
**Constraints**: FR-006 — an unknown `productId` on the Product dashboard is a 404, never an
|
||||||
|
empty-but-200 response. FR-007 — every rate/average is `number | null`, `null` meaning "no
|
||||||
|
qualifying data," computed by checking the qualifying count before ever dividing. FR-008 — every
|
||||||
|
route requires `requireRole('ADMIN')`, the same gate every admin surface uses since
|
||||||
|
010-identity-auth.
|
||||||
|
|
||||||
|
**Scale/Scope**: Four new `GET` routes, one new Prisma model + migration, four new service
|
||||||
|
methods (one per dashboard) replacing the single stub method, one new schema file for query-param
|
||||||
|
validation, three new env-configured values (Constitution Principle II). No new module — this
|
||||||
|
extends `platform/reports`, already the correct architectural home.
|
||||||
|
|
||||||
|
## Constitution Check
|
||||||
|
|
||||||
|
*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
|
||||||
|
|
||||||
|
| Principle / Section | Check | Result |
|
||||||
|
|---|---|---|
|
||||||
|
| I. SaaS Is the Sole Identity & Access Authority | Not applicable — no identity/access surface touched; every figure is derived from SupportHub's own domain data (tickets, problems, SLA, escalation, AI sessions), squarely inside SupportHub's own sole-authority domain per this principle's own second sentence. | PASS |
|
||||||
|
| II. Configuration Over Hardcoding | The default reporting window, the SLA-risk threshold, and the top-N ranking limit are all new env-configured values (`REPORTING_DEFAULT_WINDOW_DAYS`, `REPORTING_SLA_RISK_THRESHOLD_MINUTES`, `REPORTING_TOP_N_LIMIT`), never hardcoded — matches spec.md's own Assumptions and the roadmap's "never hardcode a placeholder value and ship it as final." | PASS |
|
||||||
|
| III. Layered Architecture With Enforced Module Boundaries | All new code lives inside `platform/reports` (already its correct home) following Route → Schema → Controller → Service → Repository → Prisma; cross-module reads (tickets, AI support, orchestration, SLA/escalation, problem resolution) go through each owning module's own public `index.ts`, the same precedent every prior feature this session established (e.g. `tool-executor.ts` reading `ticketsService` from `@/modules/ticketing/tickets`). | PASS |
|
||||||
|
| IV. AI Recommends, Deterministic Policy Decides | Not applicable — no AI tool-execution or decision logic changed; the AI dashboard only reports on outcomes the existing, already-deterministic confidence-band/tool-policy code already produced. | PASS — N/A |
|
||||||
|
| V. Evidence-Based Verification | Not applicable — no resolution-recording logic changed. | PASS — N/A |
|
||||||
|
| VI. Durable Audit & History | The one new table (`ErrorCodeLookup`) is itself an append-only audit record, directly in this principle's spirit — "which error codes came up, when" becomes durably answerable for the first time. | PASS |
|
||||||
|
| VII. Concurrency-Safe, Durable Job Handling | Not applicable — read-only aggregation queries, no job handlers, no assignment/SLA state mutated. | PASS — N/A |
|
||||||
|
| VIII. Problem and Ticket Are Separate, Related Entities | Respected — the Product dashboard's problem-type breakdown queries `Problem` directly, never conflating it with `Ticket`. | PASS |
|
||||||
|
| Technology & Platform Constraints | No new dependencies; one new Prisma model via the established non-interactive migration workflow (`prisma migrate diff` → hand-written `migration.sql` → `prisma migrate deploy`) this session has used for every prior schema change. | PASS |
|
||||||
|
|
||||||
|
No violations requiring Complexity Tracking justification.
|
||||||
|
|
||||||
|
## Project Structure
|
||||||
|
|
||||||
|
### Documentation (this feature)
|
||||||
|
|
||||||
|
```text
|
||||||
|
specs/015-reporting-dashboards/
|
||||||
|
├── plan.md
|
||||||
|
├── research.md
|
||||||
|
├── data-model.md
|
||||||
|
├── quickstart.md
|
||||||
|
├── contracts/
|
||||||
|
│ └── reports-api-contract.md
|
||||||
|
└── tasks.md
|
||||||
|
```
|
||||||
|
|
||||||
|
### Source Code (repository root)
|
||||||
|
|
||||||
|
```text
|
||||||
|
supporthub-api/
|
||||||
|
├── prisma/
|
||||||
|
│ ├── schema.prisma # MODIFIED — new ErrorCodeLookup model
|
||||||
|
│ └── migrations/
|
||||||
|
│ └── <timestamp>_add_error_code_lookup/migration.sql # NEW
|
||||||
|
├── src/
|
||||||
|
│ ├── config/
|
||||||
|
│ │ └── env.ts / reporting.ts (or similar) # MODIFIED — 3 new env-configured values
|
||||||
|
│ └── modules/
|
||||||
|
│ ├── platform/
|
||||||
|
│ │ └── reports/ # REWRITTEN (was a 1-method stub)
|
||||||
|
│ │ ├── controller/
|
||||||
|
│ │ ├── mapper/ # date-range parsing/defaulting, averaging helper
|
||||||
|
│ │ ├── repository/ # the 4 dashboards' Prisma queries
|
||||||
|
│ │ ├── routes/
|
||||||
|
│ │ ├── schema/ # query-param validation
|
||||||
|
│ │ ├── service/
|
||||||
|
│ │ └── index.ts
|
||||||
|
│ └── ai-support/
|
||||||
|
│ └── knowledge/
|
||||||
|
│ ├── repository/ # MODIFIED — errorCodeLookupRepository
|
||||||
|
│ └── service/
|
||||||
|
│ └── error-codes.service.ts # MODIFIED — one new line at the existing
|
||||||
|
│ lookup call site
|
||||||
|
└── tests/
|
||||||
|
├── unit/platform/reports/ # averaging/no-data-null helper, confidence
|
||||||
|
│ bucketing reuse
|
||||||
|
└── integration/platform-reports/ # all four dashboards against real data
|
||||||
|
```
|
||||||
|
|
||||||
|
**Structure Decision**: Single project, no new module — `platform/reports` already exists as the
|
||||||
|
correct architectural home and simply needs its real implementation built out, following the
|
||||||
|
same Route → Schema → Controller → Service → Repository → Prisma layering every other module
|
||||||
|
already uses.
|
||||||
|
|
||||||
|
## Complexity Tracking
|
||||||
|
|
||||||
|
*No constitution violations — table intentionally omitted.*
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
# Quickstart: Reporting and Analytics Dashboards
|
||||||
|
|
||||||
|
Manual verification steps for each user story, against a running instance backed by real
|
||||||
|
Postgres/Redis, logged in as an ADMIN.
|
||||||
|
|
||||||
|
## Scenario 1 — Management dashboard (User Story 1)
|
||||||
|
|
||||||
|
1. Create several tickets within a known date range: some reaching `AI_RESOLVED`/`RESOLVED` via
|
||||||
|
an AI session, some escalated to a human and resolved via `resolutionsService.record`, some
|
||||||
|
left open.
|
||||||
|
2. Let one ticket's SLA run complete on time and another breach (via the existing breach sweep).
|
||||||
|
3. `GET /admin/reports/management?from=<range start>&to=<range end>`.
|
||||||
|
4. **Expected**: `totalCases`, `aiResolved`, `humanEscalated`, `resolved`, `open` all match what
|
||||||
|
was actually created; `slaCompliance.met`/`.breached` match the two SLA outcomes;
|
||||||
|
`averageResponseSeconds`/`averageResolutionSeconds` are non-null and plausible.
|
||||||
|
5. Request the same endpoint for a date range with no activity at all.
|
||||||
|
6. **Expected**: every count is `0`, every rate/average is `null`, not an error.
|
||||||
|
|
||||||
|
## Scenario 2 — Product dashboard (User Story 2)
|
||||||
|
|
||||||
|
1. Create tickets for two distinct products in the same range, one with a categorized problem.
|
||||||
|
2. Look up a known error code for one product several times, a different code once.
|
||||||
|
3. `GET /admin/reports/product/:externalProductId` for each product.
|
||||||
|
4. **Expected**: each product's `supportVolume`/`problemsByCategory`/`aiResolutionRate` reflect
|
||||||
|
only its own tickets; `topErrors` ranks the more-frequently-looked-up code first.
|
||||||
|
5. Request the endpoint for a nonexistent `externalProductId`.
|
||||||
|
6. **Expected**: `404 NOT_FOUND`, not an empty `200`.
|
||||||
|
|
||||||
|
## Scenario 3 — Support dashboard (User Story 3)
|
||||||
|
|
||||||
|
1. Assign several tickets across two agents (some via the real orchestration flow).
|
||||||
|
2. Let one ticket's SLA run sit within `REPORTING_SLA_RISK_THRESHOLD_MINUTES` of its resolution
|
||||||
|
due date without breaching.
|
||||||
|
3. `GET /admin/reports/support`.
|
||||||
|
4. **Expected**: `workloadByAgent` matches each agent's real current open-assignment count;
|
||||||
|
`slaAtRisk` counts exactly the near-due run, distinct from `slaBreached`.
|
||||||
|
|
||||||
|
## Scenario 4 — AI dashboard (User Story 4)
|
||||||
|
|
||||||
|
1. Run AI sessions to a mix of terminal outcomes (`resolved`, `escalated`), with some tool
|
||||||
|
invocations succeeding and others failing, and diagnoses spanning a range of confidence
|
||||||
|
values.
|
||||||
|
2. `GET /admin/reports/ai`.
|
||||||
|
3. **Expected**: `aiResolutionRate`/`humanHandoffRate` reflect the real outcome mix;
|
||||||
|
`confidenceDistribution` buckets match `decideConfidenceBand`'s own classification of each
|
||||||
|
diagnosis's stored confidence against the system-default thresholds; `toolInvocations`
|
||||||
|
reflects the real success/failure counts.
|
||||||
|
|
||||||
|
## What "done" looks like
|
||||||
|
|
||||||
|
All four scenarios pass against a real Postgres/Redis, every figure independently verified
|
||||||
|
against hand-computed expected values, and no route is reachable by a non-admin session.
|
||||||
@@ -0,0 +1,143 @@
|
|||||||
|
# Research: Reporting and Analytics Dashboards
|
||||||
|
|
||||||
|
## 1. Where this lives
|
||||||
|
|
||||||
|
**Decision**: Wire up the existing, pre-scaffolded `src/modules/platform/reports` module (today
|
||||||
|
just `ReportsService.generateSummaryReport()` returning `{}`, confirmed unused anywhere) rather
|
||||||
|
than creating a new module. Its four real methods (`getManagementDashboard`,
|
||||||
|
`getProductDashboard`, `getSupportDashboard`, `getAiDashboard`) replace the one stub method.
|
||||||
|
Routes live at `GET /admin/reports/management`, `GET /admin/reports/product/:externalProductId`,
|
||||||
|
`GET /admin/reports/support`, `GET /admin/reports/ai`, admin-gated the same way every other
|
||||||
|
admin-only endpoint since 010-identity-auth already is (`requireRole('ADMIN')`).
|
||||||
|
|
||||||
|
**Why not the `ANALYTICS` queue** (`src/jobs/analytics`, also pre-scaffolded, also inert): a
|
||||||
|
queued background job fits pre-computing a report nobody is currently waiting on; a dashboard
|
||||||
|
request is someone waiting right now for an answer. Per spec.md's Assumptions, this first cut is
|
||||||
|
synchronous, direct-query aggregation — the queue stub stays exactly as inert as it already was,
|
||||||
|
untouched by this feature.
|
||||||
|
|
||||||
|
## 2. Per-dashboard queries
|
||||||
|
|
||||||
|
All four use Prisma's `groupBy`/`count`/`aggregate`, scoped by `createdAt` (or the
|
||||||
|
milestone-specific timestamp named below) within `[from, to]`, computed directly against the
|
||||||
|
tables that already own each fact — no new table, no denormalized rollup.
|
||||||
|
|
||||||
|
### Management (FR-001)
|
||||||
|
|
||||||
|
| Figure | Source |
|
||||||
|
|---|---|
|
||||||
|
| Total cases | `Ticket.count({ createdAt in range })` |
|
||||||
|
| AI resolved | `Ticket.count({ createdAt in range, status: 'AI_RESOLVED' })` — a ticket that reached `AI_RESOLVED` and stayed there or moved straight to `RESOLVED` without a `Resolution.resolvedBy` other than `'ai'`; see §4 below for the exact "who resolved it" rule shared with the Product dashboard |
|
||||||
|
| Human escalated | `Ticket.count({ createdAt in range, status in [HUMAN_ESCALATION, IN_PROGRESS, WAITING_FOR_CUSTOMER, RESOLUTION_PENDING_CUSTOMER, RESOLVED, CLOSED, REOPENED] })` minus AI-only-path tickets — i.e. any ticket that ever reached `HUMAN_ESCALATION`; the state machine research.md already establishes this as one-way (003-ticketing) |
|
||||||
|
| Resolved (either path) | `Ticket.count({ createdAt in range, status in [RESOLVED, CLOSED] })` |
|
||||||
|
| Open | `Ticket.count({ createdAt in range, status not in [RESOLVED, CLOSED] })` |
|
||||||
|
| SLA compliance / breach count | `SLARun.groupBy(['status'], { ticket: { createdAt in range } })`, `status: 'completed'` = met, `'breached'` = breached (mirrors 014's own metric semantics — see 014 research.md §5's "read status before the overwrite" caveat, which applies equally here: a `'breached'`-then-`'completed'` run is still counted breached, by reading the `breachedAt`/`firstResponseBreachedAt` timestamps rather than only the current `status` string) |
|
||||||
|
| Escalation count | `EscalationEvent.count({ createdAt in range })` |
|
||||||
|
| Average response time | `avg(firstAgentMessage.createdAt - ticket.createdAt)` over tickets with at least one `AGENT_MESSAGE` in range — computed in application code over a bounded query result (see §5, no raw SQL) |
|
||||||
|
| Average resolution time | `avg(resolution.createdAt - ticket.createdAt)` over tickets with a `Resolution` row in range |
|
||||||
|
|
||||||
|
### Product (FR-002)
|
||||||
|
|
||||||
|
Same shape as Management, `WHERE Ticket.productId = :productId` (resolved from the given
|
||||||
|
`externalProductId`, 404 if not found — FR-006), plus:
|
||||||
|
|
||||||
|
| Figure | Source |
|
||||||
|
|---|---|
|
||||||
|
| Problem-type breakdown | `Problem.groupBy(['categoryId'], { productId, createdAt in range })` |
|
||||||
|
| Recurring problems | Same grouped result, sorted descending, top N (config, default 10) |
|
||||||
|
| Top errors | `reuses 014's own instrumentation point conceptually but queries fresh` — no, see §6: there is no persisted "error code lookup" table, only 014's in-memory Prometheus counter, which is NOT queryable historically. Resolved by adding a durable audit read instead: see §6. |
|
||||||
|
|
||||||
|
### Support (FR-003)
|
||||||
|
|
||||||
|
| Figure | Source |
|
||||||
|
|---|---|
|
||||||
|
| Per-agent workload | `Assignment.groupBy(['agentId'], { isCurrent: true })` — a snapshot of *right now*, not date-ranged (workload is inherently current, not historical — spec.md's own framing: "how much work is currently assigned") |
|
||||||
|
| SLA risk / breached | `SLARun.findMany({ status: 'running', resolutionDueAt: {gte: now} })` filtered in application code by "due within `SLA_RISK_THRESHOLD_MINUTES` of now" for risk, vs. `status: 'breached'` for already-breached |
|
||||||
|
| Escalation count | Same as Management, unfiltered by product |
|
||||||
|
| Response/resolution performance | Same computation as Management's averages |
|
||||||
|
|
||||||
|
### AI (FR-004)
|
||||||
|
|
||||||
|
| Figure | Source |
|
||||||
|
|---|---|
|
||||||
|
| AI resolution rate / human-handoff rate | `AISupportSession.groupBy(['status'], { startedAt in range })` — `resolved` vs. `escalated`/`ended_by_agent` as a share of total terminal sessions |
|
||||||
|
| Failed-troubleshooting-then-escalated rate | Sessions with `status: 'escalated'` that have at least one `AIInteraction`/`AIAction` recording a failed troubleshooting attempt — see 005-ai-support's own runbook-step-outcome classification (`classifyStepOutcome`), reused rather than reinvented |
|
||||||
|
| Knowledge-match rate | `AIKnowledgeReference` presence per session (`recordMany` is only ever called with actual retrieval results — 005-ai-support's own `diagnose.ts`) vs. sessions with zero references recorded |
|
||||||
|
| Confidence distribution | `AIDiagnosis.findMany({ createdAt in range })`, bucketed in application code against `aiConfig.defaultHighConfidence`/`defaultLowConfidence` (see §7 — NOT a per-diagnosis resolved policy) |
|
||||||
|
| Tool success/failure | `AIAction` joined to `AIActionResult`, grouped by `result.status` |
|
||||||
|
|
||||||
|
## 3. "No data" convention (FR-007)
|
||||||
|
|
||||||
|
**Decision**: every rate/average field is `number | null` — `null` means "no qualifying records
|
||||||
|
in range," distinguished in the response shape from a genuine `0` (e.g., a real 0% AI resolution
|
||||||
|
rate because everything escalated is a valid, meaningful `0`; "nobody's data exists yet" is
|
||||||
|
`null`). Application code computes every average by fetching the qualifying count first and
|
||||||
|
returning `null` before ever dividing, never relying on `0/0` producing `NaN` and hoping a caller
|
||||||
|
notices.
|
||||||
|
|
||||||
|
## 4. "Who resolved it" — reused from 014, not reinvented
|
||||||
|
|
||||||
|
014-full-observability's own event subscriber already established the authoritative rule: a
|
||||||
|
ticket's `Resolution.resolvedBy` field (`"ai"` | an `agentId`) is the single source of truth for
|
||||||
|
whether a resolution was AI- or human-driven (014 research.md §5). This feature's Management/
|
||||||
|
Product dashboards reuse the exact same join (`Resolution.findMany` scoped to the range,
|
||||||
|
`resolvedBy === 'ai'` vs. not) rather than re-deriving it from `Ticket.status` transitions a
|
||||||
|
second, potentially-inconsistent way.
|
||||||
|
|
||||||
|
## 5. No raw SQL
|
||||||
|
|
||||||
|
**Decision**: every duration average (response time, resolution time) is computed by fetching
|
||||||
|
the bounded set of qualifying rows (ticket `createdAt` + the milestone timestamp) via Prisma and
|
||||||
|
averaging in application code, not a raw `$queryRaw` computing `AVG(EXTRACT(EPOCH FROM ...))` in
|
||||||
|
SQL. At the data volumes spec.md's Assumptions accept for this first cut (no pre-aggregation,
|
||||||
|
synchronous queries), a bounded per-range fetch is simple, type-safe, and testable without
|
||||||
|
hand-writing SQL — consistent with this codebase's near-total avoidance of `$queryRaw` elsewhere
|
||||||
|
(confirmed by grep: no existing module uses it for reporting-shaped queries).
|
||||||
|
|
||||||
|
## 6. Top errors needs a durable, queryable record — a real gap 014 left open
|
||||||
|
|
||||||
|
014-full-observability's `supporthub_known_error_lookups_total` Prometheus counter is
|
||||||
|
process-lifetime, in-memory, and reset on every restart — useless for "top errors in the last 30
|
||||||
|
days." Since no durable "error code lookup" record exists anywhere in this codebase today (the
|
||||||
|
existing `error-codes.service.ts` just reads `KnownIssue`/`ErrorCode` rows, never records that a
|
||||||
|
lookup happened), this feature adds one small, focused piece of new state: a durable
|
||||||
|
`ErrorCodeLookup` audit row (`errorCodeId`, `productId`, `createdAt`), written by
|
||||||
|
`error-codes.service.ts`'s already-existing `findKnownIssuesByErrorCode` (the same call site
|
||||||
|
014 instrumented for its own live counter — this feature adds one more line there, a durable
|
||||||
|
write alongside the existing live-metric increment, not a replacement for it). This is the one
|
||||||
|
schema change this feature needs; every other dashboard figure is computed from tables that
|
||||||
|
already exist.
|
||||||
|
|
||||||
|
## 7. Confidence distribution uses the system default threshold, not a per-diagnosis policy
|
||||||
|
|
||||||
|
**Decision**: bucket every `AIDiagnosis.confidence` value in range against the env-configured
|
||||||
|
system-wide defaults (`aiConfig.defaultHighConfidence`/`defaultLowConfidence`), the same
|
||||||
|
`decideConfidenceBand` pure function 005-ai-support already exports — reused directly, not
|
||||||
|
reimplemented.
|
||||||
|
|
||||||
|
**Why not resolve each diagnosis's actual applicable per-product/category policy** (what the
|
||||||
|
live reasoning path itself does): `AIDiagnosis.product`/`feature` are the AI's own free-text
|
||||||
|
classification output, not foreign keys to `Product`/`Category` — there is no reliable, existing
|
||||||
|
join from a diagnosis row back to which `ConfidencePolicy` row actually applied to it at the time
|
||||||
|
without speculatively string-matching free text against product names, which this codebase does
|
||||||
|
nowhere else and which research.md declines to invent here. A dashboard-level aggregate
|
||||||
|
distribution using the system-wide default is an honest, documented simplification (spec.md
|
||||||
|
Assumptions) — precise enough to show a meaningful shape without fabricating a false precision
|
||||||
|
the data doesn't actually support.
|
||||||
|
|
||||||
|
## 8. New configuration (Constitution Principle II — nothing hardcoded)
|
||||||
|
|
||||||
|
| Env var | Default | Used by |
|
||||||
|
|---|---|---|
|
||||||
|
| `REPORTING_DEFAULT_WINDOW_DAYS` | `30` | Every dashboard's `from`/`to` default when omitted (FR-005) |
|
||||||
|
| `REPORTING_SLA_RISK_THRESHOLD_MINUTES` | `60` | Support dashboard's "at risk" classification (FR-003) |
|
||||||
|
| `REPORTING_TOP_N_LIMIT` | `10` | Product dashboard's recurring-problems/top-errors ranking length |
|
||||||
|
|
||||||
|
## 9. Test strategy
|
||||||
|
|
||||||
|
Integration tests create real tickets/problems/SLA runs/escalation events/AI sessions/diagnoses/
|
||||||
|
actions/error-code lookups directly against real Postgres (mixing real HTTP-driven setup where a
|
||||||
|
realistic flow matters and direct repository/Prisma writes where only the aggregation math is
|
||||||
|
under test — the same mix 014's own `business-metrics.test.ts` used), then request each
|
||||||
|
dashboard endpoint and assert every figure against hand-computed expected values. Unit tests
|
||||||
|
cover the "no data → null, never NaN" guard and the confidence-bucketing pure-function reuse.
|
||||||
@@ -130,8 +130,9 @@ confirming every figure matches the real session data.
|
|||||||
dashboard is requested, **Then** tool success/failure figures reflect the real invocation
|
dashboard is requested, **Then** tool success/failure figures reflect the real invocation
|
||||||
outcomes.
|
outcomes.
|
||||||
3. **Given** diagnoses were recorded with a range of confidence values, **When** the dashboard is
|
3. **Given** diagnoses were recorded with a range of confidence values, **When** the dashboard is
|
||||||
requested, **Then** the confidence distribution groups them into the same high/medium/low
|
requested, **Then** the confidence distribution groups them into the same proceed/ask/escalate
|
||||||
bands the AI support module itself already uses (005-ai-support), not a newly-invented scheme.
|
bands the AI support module's own confidence-policy service already classifies each diagnosis
|
||||||
|
into (005-ai-support), not a newly-invented scheme.
|
||||||
4. **Given** some AI sessions' knowledge-retrieval step found matching entries and others found
|
4. **Given** some AI sessions' knowledge-retrieval step found matching entries and others found
|
||||||
none, **When** the dashboard is requested, **Then** knowledge-match rate reflects the real
|
none, **When** the dashboard is requested, **Then** knowledge-match rate reflects the real
|
||||||
match/no-match mix.
|
match/no-match mix.
|
||||||
@@ -175,7 +176,7 @@ confirming every figure matches the real session data.
|
|||||||
- **FR-004**: System MUST provide an AI dashboard summarizing, for a given date range: AI
|
- **FR-004**: System MUST provide an AI dashboard summarizing, for a given date range: AI
|
||||||
resolution rate, rate of sessions that escalated after at least one failed troubleshooting
|
resolution rate, rate of sessions that escalated after at least one failed troubleshooting
|
||||||
attempt, knowledge-match rate, a distribution of diagnosis confidence across the existing
|
attempt, knowledge-match rate, a distribution of diagnosis confidence across the existing
|
||||||
high/medium/low bands, tool invocation success/failure counts, and human-handoff rate.
|
proceed/ask/escalate bands, tool invocation success/failure counts, and human-handoff rate.
|
||||||
- **FR-005**: Every dashboard endpoint MUST accept an optional `from`/`to` date range; when
|
- **FR-005**: Every dashboard endpoint MUST accept an optional `from`/`to` date range; when
|
||||||
omitted, it MUST default to a documented trailing window rather than scanning unbounded
|
omitted, it MUST default to a documented trailing window rather than scanning unbounded
|
||||||
history.
|
history.
|
||||||
@@ -186,10 +187,17 @@ confirming every figure matches the real session data.
|
|||||||
be reported as an explicit "no data" value, never a computed `0`, `null`, or `NaN`.
|
be reported as an explicit "no data" value, never a computed `0`, `null`, or `NaN`.
|
||||||
- **FR-008**: All four dashboard endpoints MUST be admin-gated, consistent with every other
|
- **FR-008**: All four dashboard endpoints MUST be admin-gated, consistent with every other
|
||||||
admin-only reporting/configuration surface in this codebase.
|
admin-only reporting/configuration surface in this codebase.
|
||||||
- **FR-009**: This feature MUST NOT alter what any existing endpoint, event, or table stores —
|
- **FR-009**: This feature MUST NOT alter the meaning or shape of any existing endpoint, event, or
|
||||||
every figure is derived read-only from data already durably recorded by the modules that own
|
table — nearly every figure is derived read-only from data already durably recorded by the
|
||||||
it (003 ticketing, 005 AI support, 007 orchestration, 008 SLA/escalation, 009 problem
|
modules that own it (003 ticketing, 005 AI support, 007 orchestration, 008 SLA/escalation, 009
|
||||||
resolution).
|
problem resolution). The one exception is FR-011: a small new durable record needed only
|
||||||
|
because no existing table can answer "which error codes are looked up most" historically.
|
||||||
|
- **FR-011**: System MUST durably record each known-error-code lookup (product, error code,
|
||||||
|
timestamp) at the point it already happens (the existing error-code lookup call site) so the
|
||||||
|
Product dashboard's "top errors" ranking (FR-002) can be computed historically — the
|
||||||
|
equivalent live, in-process counter this project already exposes on `/metrics` (014-full-
|
||||||
|
observability) is process-lifetime and reset on every restart, unusable for a historical
|
||||||
|
dashboard.
|
||||||
- **FR-010**: This feature is backend-only; presenting these figures in a UI is a separate,
|
- **FR-010**: This feature is backend-only; presenting these figures in a UI is a separate,
|
||||||
explicitly out-of-scope follow-on (see Assumptions).
|
explicitly out-of-scope follow-on (see Assumptions).
|
||||||
|
|
||||||
@@ -200,8 +208,11 @@ confirming every figure matches the real session data.
|
|||||||
recomputed fresh on every request from existing durable records.
|
recomputed fresh on every request from existing durable records.
|
||||||
- **Date range**: An inclusive `from`/`to` pair (calendar dates or timestamps) scoping every
|
- **Date range**: An inclusive `from`/`to` pair (calendar dates or timestamps) scoping every
|
||||||
aggregation query; not a stored entity, a request parameter.
|
aggregation query; not a stored entity, a request parameter.
|
||||||
- **Confidence band**: The existing high/medium/low classification 005-ai-support already applies
|
- **Error code lookup record** (new, FR-011): a durable, append-only audit row — which product,
|
||||||
to a diagnosis's confidence score — reused here for the AI dashboard's distribution, not
|
which error code, when — written at the existing lookup call site; exists solely so "top
|
||||||
|
errors" can be computed over a historical range, never read or written anywhere else.
|
||||||
|
- **Confidence band**: The existing proceed/ask/escalate classification 005-ai-support already
|
||||||
|
applies to a diagnosis's confidence score — reused here for the AI dashboard's distribution, not
|
||||||
redefined.
|
redefined.
|
||||||
|
|
||||||
## Success Criteria *(mandatory)*
|
## Success Criteria *(mandatory)*
|
||||||
|
|||||||
Reference in New Issue
Block a user