Documents implementation-time findings in the requirements checklist: all
three suspected races were confirmed real then fixed, the ticket-status
mechanism needed no fix, a real pre-existing test-infrastructure issue
(throwaway DB ticket-code collisions at high accumulated volume) was found
and resolved by resetting the throwaway database and replaying its full
migration history, two full-suite-only integration failures were confirmed
as pre-existing cross-file contamination (not a regression), and the
load-test tooling surfaced a real Anthropic API cost consideration for
ticket creation itself. All 23 tasks marked complete.
Full quality gate green: typecheck, lint, architecture check, full unit
suite (119/119), full integration suite against a freshly reset throwaway
database (122/124 — the 2 failures are the project's own already-accepted
MinIO baseline), and all 6 concurrency test files (11/11).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
tests/load/autocannon.config.ts is a thin shared wrapper over autocannon's
programmatic API producing this feature's own report shape (throughput,
latency p50/p90/p99, non-2xx, rate-limited count) — printed and written to
tests/load/reports/ (gitignored) for every run. No pass/fail threshold is
applied (FR-009): throughput/latency targets are an OPEN BUSINESS DECISION
per the roadmap's own convention, never invented.
Three scripts cover the named critical endpoint groups: ticket-creation
(pure DB path), admin-reporting (015-reporting-dashboards, pure DB path),
and ai-support-flow (005-ai-support's real Anthropic API calls — clearly
flagged as real, billed cost, run only at a small bounded amount rather than
an open-ended duration). All three were run once at a small scale against the
real dev server to confirm the tooling works end-to-end and cleans up fully
after itself (verified via direct DB checks, not just script exit codes).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
tests/concurrency/ticket-status-race.test.ts fires 20 genuinely concurrent
TicketsRepository.updateStatus calls from the same starting version against
real Postgres. Passes on the first run, confirming (rather than assuming)
003-ticketing's existing atomic version-checked updateMany already holds
under real concurrency — no implementation change needed (research.md §4).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
tests/concurrency/escalation-idempotency.test.ts fires the same escalation
trigger (escalationService.handleBreach) concurrently more than once for the
same ticket against real Postgres, asserting exactly one EscalationEvent and
one current Assignment result every time — verified across 10 repeated runs
(SC-003). The database-level unique-violation is visibly caught and absorbed
in the logs, confirming the fix (previous commit) actually engages under a
genuine race rather than being untested code.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Completes the escalation-event repository/service changes from the prior
commit: normalizes the exactOptionalPropertyTypes mismatch in the
findFirst fallback lookup, and updates every existing unit test
(sla-pause-resume, sla-breach-detection, sla-compliance-metric,
escalation-rule-match) to the new SlaRunRepository.updateWithVersion and
EscalationEventRepository.create({event, wasNewlyCreated}) signatures.
Full typecheck/lint/architecture-check clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
SlaRunRepository.updateWithVersion replaces the old unguarded update(),
mirroring TicketsRepository.updateStatus's exact atomic-updateMany pattern.
pause/resume/complete now retry (bounded) against fresh state on a version
conflict; the breach sweep skips a run that lost the race to a concurrent
pause/resume/complete rather than clobbering it, deferring to the next
scheduled pass. Verified against real Postgres: concurrent pause/resume/sweep
activity against the same run now always leaves it in one
internally-consistent state, across repeated runs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
AssignmentRepository.createAssignment now catches the
assignments_one_current_per_ticket unique-violation and retries the whole
supersede-then-create transaction (bounded, with jitter) instead of
propagating a raw P2002 to the caller. Verified against real Postgres: before
this fix, 20 genuinely concurrent assignment attempts on the same ticket
reliably threw an unhandled unique-constraint error; after it, exactly one
current assignment results every time across 10 repeated runs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds autocannon as a devDependency for the load-test tooling (T001), and the
shared schema migration T002 blocks: SLARun.version for optimistic
concurrency, plus two Postgres partial unique indexes
(assignments_one_current_per_ticket, escalation_events_ticket_rule_unique)
guarding against the assignment and escalation races research.md documents.
Applied directly to both the real dev DB and the throwaway test DB.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Foundational phase (T002) covers the one shared schema migration US1/US2/US3
depend on; US4 (proof-only, no schema change) and US5 (load-test tooling)
have no dependency on it and can proceed independently.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
research.md nails down the exact mechanism for each real race the audit
found: a partial unique index for assignment double-assignment, a
Ticket-style version counter for SLA pause/resume/sweep, and a partial
unique index for escalation-rule idempotency — each traced to the specific
repository/service code that has the gap today. data-model.md and plan.md
carry the resulting schema and repository-contract changes; quickstart.md
defines the real-infra verification steps for each user story.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Five user stories: assignment double-assignment race, SLA pause/resume race,
escalation idempotency, ticket optimistic-concurrency proof, and HTTP
load/throughput testing tooling. Scoped from a targeted audit of existing
concurrency guarantees rather than guesswork — see spec.md's Assumptions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ManagementRepository/ProductReportRepository.countEverEscalatedToHuman checked
a list of terminal statuses that ticket-state-machine.ts's own transition
table shows are reachable from BOTH the AI-resolved path and the
human-escalation path once they converge (RESOLUTION_PENDING_CUSTOMER,
RESOLVED, CLOSED, REOPENED). Every AI-resolved ticket was being double-counted
as human-escalated too — confirmed live against real seeded dev data
(humanEscalated: 34 out of totalCases: 34, an impossible 100%).
Fixed by keying off assignments: { some: {} } instead, since
orchestrationService.handleHumanEscalation is the only code path that ever
creates an Assignment row. Updated management-dashboard.test.ts's own
human-resolved fixture to create a real Assignment row, since it previously
relied on the now-fixed buggy status-based signal without one.
Found via manual verification against a real running dev server while
building supporthub-web's 002-reporting-dashboards-ui, not by any existing
automated test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Wires the pre-scaffolded, unused platform/reports module (ReportsService
.generateSummaryReport previously returned {}) into four real, admin-
gated dashboards matching docs/09-testing-observability-cicd.md's own
table:
- GET /admin/reports/management: total cases, AI-resolved, human-
escalated, resolved/open, SLA compliance/breaches, escalation count,
average response/resolution time.
- GET /admin/reports/product/:externalProductId: support volume,
problem-category breakdown, recurring problems, AI-resolution/human-
escalation rate, top error codes.
- GET /admin/reports/support: current per-agent workload, SLA at-risk/
breached counts, escalation count, response/resolution performance.
- GET /admin/reports/ai: AI resolution/human-handoff rate, failed-
troubleshooting-then-escalated rate, knowledge-match rate, confidence
distribution (reusing 005-ai-support's own decideConfidenceBand),
tool invocation success/failure.
Every rate/average is number|null -- null means no qualifying data in
range, never a computed NaN or a misleading 0. Adds one new durable
table, ErrorCodeLookup, since 014-full-observability's own equivalent
metric is a process-lifetime Prometheus counter unusable for a
historical "top errors" report.
Verified end-to-end against real Postgres/Redis: every figure checked
against hand-computed expected values, including a no-activity range
(all-zero counts, all-null rates) and cross-product isolation.
Also fixes a real regression the new ErrorCodeLookup FK caused in the
pre-existing known-issues.test.ts (its afterAll deleted ErrorCode rows
before the now-referencing lookup rows).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
28 tasks across a shared Foundational phase (schema, config, shared
rate/date-range helpers, module scaffolding) and 4 independently-testable
dashboard user stories.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Documents the exact Prisma query per dashboard figure, the one new
durable table this feature needs (ErrorCodeLookup — 014's own equivalent
metric is process-lifetime, unusable for a historical report), the
"no data -> null, never NaN" convention, and why the AI dashboard's
confidence distribution deliberately uses the system-default threshold
rather than resolving a per-diagnosis policy (AIDiagnosis has no
reliable FK back to which policy applied).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Phase 11's third sub-area (reporting/analytics dashboards), per explicit
user direction. Backend-first scope (four read-only aggregation
endpoints wiring up the pre-scaffolded platform/reports module),
following the same backend-before-frontend pattern already established
for 010/011/014 this session — a supporthub-web dashboard UI is a
separate, not-yet-started follow-on.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
business-metrics.test.ts's "human resolution" case drove a ticket
through a real HUMAN_ESCALATION transition via ticketsService.updateStatus,
which triggers the real orchestration subscriber's default ROUND_ROBIN
auto-assignment against every agent in the shared throwaway database —
reproduced deterministically landing on agent-ticket-queue.test.ts's own
dedicated agent. Fixed by driving the intermediate transitions directly
through ticketsRepository.updateStatus (no domain-event publish),
reserving the real, event-publishing call for only the final RESOLVED
transition the metric subscriber needs to observe.
Also documents (checklist Notes), without fixing, a separate pre-existing
issue confirmed unrelated to this feature via git checkout to the clean
013-auth-hardening tip: nearly every integration test file's product ID
collapses to the same 4-letter ticket-code prefix ("TEST"), so enough
concurrent TEST_*-prefixed files can exceed the fixed retry ceiling on
ticket-code generation and surface as a real 500 — a 003-ticketing
concern, out of scope here.
Marks all 30 tasks.md items complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Completes User Story 4's eleven named metrics: first-response-time
(messages.service.ts's post(), guarded against double-counting a
ticket's second agent message), SLA compliance (sla.service.ts's
complete()/runBreachDetectionSweep(), with a guard so a run already
breached by the sweep is never also counted "met" when it later
resolves), most-common-errors (error-codes.service.ts, counted only
once a code is confirmed real), and tool-failure-rate/knowledge-
effectiveness (tools.service.ts's single executeTool call site).
Verified end-to-end against real Postgres/Redis by scraping the real
/metrics endpoint before and after driving each metric's actual
underlying event through the real service layer — including a genuine
tool-execution failure (a nonexistent ticket ID) rather than a
simulated one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
User Story 3: initializes a real OpenTelemetry TracerProvider (previously
inert — getTracer() returned a no-op tracer with nothing ever exported).
Adds ticket.create, ai.escalation, and orchestration.assignment spans
covering both FR-006 cross-module paths, verified via a real, in-memory
test exporter that confirms actual trace/parent-span nesting, not mocked.
Also registers an AsyncLocalStorageContextManager
(@opentelemetry/context-async-hooks) — without one, OTel's context API is
a no-op that doesn't propagate across the await boundaries this feature's
own event-bus subscribers rely on for span nesting; caught by the first
version of the tracing integration test actually failing on real
parent/child assertions, not assumed.
Graceful degradation (FR-007) verified against a real, deliberately
unreachable OTLP endpoint: the SDK's own background export path (what
production actually exercises) never produces an unhandled rejection.
Starts on the 11 named business-health metrics: AI session
resolved/escalated outcomes (session.repository.ts, the single choke
point every branch in session.service.ts funnels through), human-vs-AI
resolution + resolution-time (a new TICKET_UPDATED/RESOLVED subscriber),
escalation rate (a new subscriber on ESCALATION_TRIGGERED, published
unconditionally since 008 but never previously consumed), and recurring
problems (tickets.service.ts's existing problem-creation call site).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
User Stories 1-2: every completed request (including 404s and early
replies from other hooks) now emits exactly one structured access-log
line, and every log line produced during that request's handling shares
its requestId/correlationId via a new AsyncLocalStorage-backed Pino mixin
— with zero changes to any existing log call site. The previously-dead
supporthub_http_request_duration_seconds histogram now actually receives
observations, so error rate and latency per route are computable from
/metrics alone.
Also bumps @opentelemetry/sdk-trace-base 1.x -> 2.x to align with the two
new tracing dependencies added in this same branch (exporter-trace-otlp-http,
resources) onto one consistent major version — npm had otherwise installed
two incompatible OTel core/resources majors side by side, which also
happened to resolve a moderate DoS advisory in @opentelemetry/core <2.8.0.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
30 tasks across 4 independently-testable user stories plus a shared
foundational phase (ALS request-context store + new OTel dependencies).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Documents the exact hook point chosen for each of the 3 dead observability
primitives (access log, request-duration histogram, tracer provider) and
the 11 named business-health metrics, verified against the real current
code rather than assumed — including a pre-existing SLA-run status data
quality gap surfaced along the way (documented, not fixed here).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Phase 11's second sub-area (full observability), per explicit user
direction. Scopes wiring the already-scaffolded logging/metrics/tracing
into something actually functional, explicitly bounded away from the
separate, not-yet-started reporting/analytics dashboards sub-area.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Closes the two gaps 010-identity-auth explicitly deferred (password reset,
login rate-limiting), plus a shared password-strength validator both the
reset-consume endpoint and admin account creation now depend on.
- Password reset: single-use, paired-Redis-key tokens (never in Postgres),
identical response regardless of account existence, stubbed delivery via
a structured log line (no email infrastructure exists yet).
- Password strength: one validatePasswordStrength() call site, wired into
both POST /admin/users and the reset-consume flow.
- Login rate-limiting: checkRateLimit keyed by submitted email, checked
before any credential verification.
Also fixes tests/helpers/auth.ts's shared loginAs() helper, which reused
two fixed accounts across the whole integration suite via upsert — now
rate-limited per email, that collided across ~30 files sharing one budget.
Each call now gets a unique email; no call sites needed to change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Same class of cross-file test-isolation gap already fixed in
orchestration-flow.test.ts and sla-escalation-flow.test.ts (010's own
regression work): a wildcard (non-product-scoped) SLA policy from
another suite can match this file's own tickets too, leaving a real
sla_run row that RESTRICTs the ticket delete. Also cleaned up several
orphaned wildcard SLA policies that had accumulated in the shared
throwaway test database from earlier runs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Reset tokens live only in Redis as a paired key shape (mirrors 010's own
revocation-denylist pattern) - never in Postgres, never storing the raw
token. Password-strength policy is one shared validator called from both
the new reset-consume endpoint and 010's existing POST /admin/users.
Login rate-limiting reuses the existing checkRateLimit helper from
002's own inbound trust boundary, keyed by submitted email, checked
before any credential verification.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Phase 11's "security hardening pass" (docs/10-implementation-roadmap.md),
first slice, per explicit user direction. Closes the two concrete gaps
010-identity-auth's own Assumptions named as out of its scope. MFA is
intentionally excluded as its own larger follow-up feature. Email
delivery for password-reset is stubbed (server-side log) per explicit
user decision, since this codebase has no email infrastructure at all
today.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Follow-up to 012-admin-list-views, discovered while building
supporthub-web's own escalation admin screen (001-agent-admin-ui User
Story 5): the list endpoint returned bare policies with no way to read
back which rules (trigger type, target node) already existed under
each one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Follow-up to 012-admin-list-views, discovered while building
supporthub-web's own SLA/calendar admin screen (001-agent-admin-ui User
Story 4): holidays could only be added or removed, never read back -
GET /admin/business-calendars/:id returned the bare calendar with no
way to display what holidays were already on file. The repository
already had findByIdWithHolidays; it just wasn't wired to this route.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Follow-up to 012-admin-list-views, discovered while building supporthub-
web's own knowledge-governance screen (001-agent-admin-ui User Story 7):
GET /knowledge/retrieve only ever returns published entries (its own
AI-consumption purpose), so a governance screen that needs to see and
publish a draft entry had no endpoint to list it. Adds a small
admin-list-views-style read query scoped to the knowledge module itself.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds GET /admin/sla-runs (filterable by status), GET /admin/escalation-
events (capped, most-recent-first), and GET /admin/products (with
integration status joined in, never the full ProductIntegration row).
None of these existed as a single query before - only per-ticket or
per-integration-id lookups did.
Discovered while planning supporthub-web's 001-agent-admin-ui User
Stories 6-7 (SLA/escalation monitoring, product catalog), the same way
011-agent-ticket-queue was discovered for User Story 1.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Discovered while planning supporthub-web's 001-agent-admin-ui User
Stories 6-7: no endpoint lists SLA runs or escalation events across
multiple tickets (only per-ticket), and no endpoint returns the product
catalog with integration status joined in.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends PATCH /admin/agents/:agentId with an optional userId to finally
wire Agent.userId (added in 010-identity-auth as schema-only, never
consumed by any workflow), with proactive role/duplicate-link checks
mirroring UsersService.create's own pre-check style.
Adds GET /agents/me/tickets and GET /admin/agents/:agentId/tickets,
sharing one TicketsService.listAssignedTo method, returning a dashboard-
ready summary (product, customer, priority, severity, status, SLA state)
of every ticket currently assigned to an agent — no such query existed
anywhere in the ticketing or orchestration modules before this. Backed by
a new Assignment @@index([agentId, isCurrent]).
Discovered while starting supporthub-web's 001-agent-admin-ui: its agent-
dashboard user story had no backend data source without this.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends the existing PATCH /admin/agents/:agentId with an optional userId
to finish wiring 010's Agent.userId link, and adds GET /agents/me/tickets
+ GET /admin/agents/:agentId/tickets sharing one ticketing/tickets service
method, backed by a new Assignment @@index([agentId, isCurrent]).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Discovered while starting supporthub-web's 001-agent-admin-ui planning:
its agent-dashboard user story needs to list tickets currently assigned
to an agent, and no such query exists anywhere in the ticketing or
orchestration modules. Also finishes wiring Agent.userId (added in
010-identity-auth as schema-only, never consumed by any workflow) so a
logged-in session can resolve to its own agent roster row at all.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replaces the no-op fastify.authenticate stub and the never-implemented
identity/auth login with real bcrypt password verification, JWT session
issuance/verification (reusing the existing JWT_SECRET), and a Redis-backed
revocation denylist for logout. Adds requireRole('ADMIN') to admin-only
configuration writes across 002-009 that previously relied on a decorator
that never actually checked anything. Adds self-identity (GET /auth/me,
re-validated against live account state) and admin-provisioned accounts
(POST /admin/users).
Making the auth check genuinely reject invalid/missing tokens exposed that
~18 pre-existing integration test files called already-gated routes with no
Authorization header (safe against the old no-op stub, broken against a real
one) — fixed via a shared tests/helpers/auth.ts (loginAs/authHeader) and a
file-by-file pass, plus two related SLA-run cleanup races exposed once admin
setup calls in those files' own beforeAll blocks started actually succeeding.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
36 tasks across 8 phases (5 user stories + setup/foundational/polish).
US1 (real login) and US2 (real route/role gating) are the P1 MVP; the
one task that touches code outside identity/* (T020, adding
requireRole('ADMIN') across 002-009's existing admin routes) is called
out explicitly to run each touched module's own test suite immediately
after, not only in the final regression pass.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Phase 0 research resolves the library choices (jsonwebtoken + bcryptjs,
chosen partly to avoid native-build friction on Windows dev
environments), the Redis-backed revocation-denylist shape (reusing
002's own jti-replay-protection pattern exactly), a 4-hour token
lifetime, and why fastify.authenticate populating the already-shared
reqContext.actorId/actorType retroactively makes every audit trail
since 007 accurate for real agent/admin actions instead of always
'unknown'.
Also surfaces and scopes a real gap found along the way: User (login
identity) and Agent (routing/skills profile) have never been linked.
Adds Agent.userId as a nullable FK now (cheap, additive) without
building the actual linking workflow, which belongs in 006's own
identity/agents admin screens as a later, separate piece of work.
Phase 1 adds data-model.md, the login/self-identity/account-creation/
logout contract, and five quickstart scenarios including a specific
requirement to re-verify at least one already-shipped admin route per
module (002-009), not just this feature's own new endpoints.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Not on the original roadmap -- surfaced as a genuine blocking gap while
planning supporthub-web's own agent/admin UI feature: fastify.authenticate
has been a complete no-op stub since 002, and identity/auth's login
endpoint has never taken a password. User/UserRole (two seeded-but-
passwordless demo accounts) and the AuthUser/JwtPayload types were all
already scaffolded and clearly intended for exactly this -- this finishes
that original wiring rather than inventing a new design.
Scope: real login (password hash + JWT), fastify.authenticate actually
rejecting invalid sessions, role-based route gating, a self-identity
endpoint, admin-created accounts, and logout. Password reset, MFA, and
login rate-limiting are explicitly deferred to Phase 11's own security
hardening pass.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
