53 lines
2.8 KiB
Markdown
53 lines
2.8 KiB
Markdown
|
# Quickstart: Reporting and Analytics Dashboards
|
||
|
|
|
||
|
|
Manual verification steps for each user story, against a running instance backed by real
|
||
|
|
Postgres/Redis, logged in as an ADMIN.
|
||
|
|
|
||
|
|
## Scenario 1 — Management dashboard (User Story 1)
|
||
|
|
|
||
|
|
1. Create several tickets within a known date range: some reaching `AI_RESOLVED`/`RESOLVED` via
|
||
|
|
an AI session, some escalated to a human and resolved via `resolutionsService.record`, some
|
||
|
|
left open.
|
||
|
|
2. Let one ticket's SLA run complete on time and another breach (via the existing breach sweep).
|
||
|
|
3. `GET /admin/reports/management?from=<range start>&to=<range end>`.
|
||
|
|
4. **Expected**: `totalCases`, `aiResolved`, `humanEscalated`, `resolved`, `open` all match what
|
||
|
|
was actually created; `slaCompliance.met`/`.breached` match the two SLA outcomes;
|
||
|
|
`averageResponseSeconds`/`averageResolutionSeconds` are non-null and plausible.
|
||
|
|
5. Request the same endpoint for a date range with no activity at all.
|
||
|
|
6. **Expected**: every count is `0`, every rate/average is `null`, not an error.
|
||
|
|
|
||
|
|
## Scenario 2 — Product dashboard (User Story 2)
|
||
|
|
|
||
|
|
1. Create tickets for two distinct products in the same range, one with a categorized problem.
|
||
|
|
2. Look up a known error code for one product several times, a different code once.
|
||
|
|
3. `GET /admin/reports/product/:externalProductId` for each product.
|
||
|
|
4. **Expected**: each product's `supportVolume`/`problemsByCategory`/`aiResolutionRate` reflect
|
||
|
|
only its own tickets; `topErrors` ranks the more-frequently-looked-up code first.
|
||
|
|
5. Request the endpoint for a nonexistent `externalProductId`.
|
||
|
|
6. **Expected**: `404 NOT_FOUND`, not an empty `200`.
|
||
|
|
|
||
|
|
## Scenario 3 — Support dashboard (User Story 3)
|
||
|
|
|
||
|
|
1. Assign several tickets across two agents (some via the real orchestration flow).
|
||
|
|
2. Let one ticket's SLA run sit within `REPORTING_SLA_RISK_THRESHOLD_MINUTES` of its resolution
|
||
|
|
due date without breaching.
|
||
|
|
3. `GET /admin/reports/support`.
|
||
|
|
4. **Expected**: `workloadByAgent` matches each agent's real current open-assignment count;
|
||
|
|
`slaAtRisk` counts exactly the near-due run, distinct from `slaBreached`.
|
||
|
|
|
||
|
|
## Scenario 4 — AI dashboard (User Story 4)
|
||
|
|
|
||
|
|
1. Run AI sessions to a mix of terminal outcomes (`resolved`, `escalated`), with some tool
|
||
|
|
invocations succeeding and others failing, and diagnoses spanning a range of confidence
|
||
|
|
values.
|
||
|
|
2. `GET /admin/reports/ai`.
|
||
|
|
3. **Expected**: `aiResolutionRate`/`humanHandoffRate` reflect the real outcome mix;
|
||
|
|
`confidenceDistribution` buckets match `decideConfidenceBand`'s own classification of each
|
||
|
|
diagnosis's stored confidence against the system-default thresholds; `toolInvocations`
|
||
|
|
reflects the real success/failure counts.
|
||
|
|
|
||
|
|
## What "done" looks like
|
||
|
|
|
||
|
|
All four scenarios pass against a real Postgres/Redis, every figure independently verified
|
||
|
|
against hand-computed expected values, and no route is reachable by a non-admin session.
|