diff --git a/specs/016-load-concurrency-testing/tasks.md b/specs/016-load-concurrency-testing/tasks.md new file mode 100644 index 0000000..82e7db6 --- /dev/null +++ b/specs/016-load-concurrency-testing/tasks.md @@ -0,0 +1,221 @@ +--- +description: "Task list for 016-load-concurrency-testing" +--- + +# Tasks: Load and Concurrency Testing + +**Input**: Design documents from `specs/016-load-concurrency-testing/` + +**Organization**: Tasks are grouped by user story (US1 = assignment race, US2 = SLA race, US3 = +escalation idempotency, US4 = ticket-status race proof, US5 = load-test tooling). US1-US4 share +one Foundational phase (the schema migration all four rely on); US5 has no schema dependency and +can proceed independently of it. + +## Format: `[ID] [P?] [Story] Description` + +All file paths are relative to `supporthub-api/` (repo root). + +--- + +## Phase 1: Setup + +- [ ] T001 [P] Add `autocannon` as a devDependency (`package.json`) and add + `tests/load/reports/` to `.gitignore` (run artifacts, not fixtures) + +--- + +## Phase 2: Foundational (Blocking Prerequisites for US1-US4) + +**Purpose**: The one shared schema migration US1, US2, and US3's fixes each depend on. US4 (no +schema change, see research.md §4) and US5 (no schema dependency) do not need this phase and can +proceed in parallel with it. + +- [ ] T002 In `prisma/schema.prisma`, add `version Int @default(0)` to `SLARun`; generate one + migration (`npx prisma migrate dev --name concurrency_guards`) that also includes, as raw + SQL, `CREATE UNIQUE INDEX assignments_one_current_per_ticket ON assignments (ticket_id) + WHERE is_current = true;` and `CREATE UNIQUE INDEX escalation_events_ticket_rule_unique ON + escalation_events (ticket_id, rule_id) WHERE rule_id IS NOT NULL;` (data-model.md); apply + to the throwaway test Postgres (`supporthub-test-pg`, port 5433) and the real dev Postgres + (`postgres-development`, port 5434, via `prisma migrate diff` + direct `psql` per this + project's own established non-destructive dev-sync approach); regenerate the Prisma client + +**Checkpoint**: Schema ready — US1, US2, US3 implementation can now begin. + +--- + +## Phase 3: User Story 1 - Assignment never double-assigned under concurrency (Priority: P1) + +**Goal**: Two concurrent assignment attempts on the same ticket always leave exactly one current +assignment. + +**Independent Test**: Run `tests/concurrency/assignment-race.test.ts` alone against the +throwaway Postgres — it creates its own ticket and needs nothing from US2-US5. + +- [ ] T003 [US1] Write `tests/concurrency/assignment-race.test.ts`: create one ticket, fire + >=20 genuinely concurrent assignment attempts at it (via the real assignment + engine/service entry point, not the repository directly), then query `assignments` + directly and assert exactly one row has `is_current = true` for that ticket (depends on + T002) +- [ ] T004 [US1] Fix `AssignmentRepository.createAssignment` in + `src/modules/orchestration/assignments/repository/assignment.repository.ts` to catch the + `assignments_one_current_per_ticket` unique-violation (Prisma `P2002`) and retry the whole + supersede-then-create transaction, bounded to 3 attempts, per research.md §1 (depends on + T002) +- [ ] T005 [US1] Re-run `assignment-race.test.ts` at least 10 times in a row (or extend the test + with its own internal repeat loop) confirming zero failures — SC-001 (depends on T003, T004) + +**Checkpoint**: Quickstart Scenario 1 passes against real infrastructure, consistently. + +--- + +## Phase 4: User Story 2 - SLA clock never corrupted by overlapping pause/resume/sweep (Priority: P1) + +**Goal**: Concurrent pause/resume/breach-sweep activity against the same SLA run always leaves +it in one internally-consistent state. + +**Independent Test**: Run `tests/concurrency/sla-race.test.ts` alone against the throwaway +Postgres — it creates its own ticket + SLA run and needs nothing from US1/US3/US4/US5. + +- [ ] T006 [US2] Replace `SlaRunRepository.update` with `updateWithVersion(id, expectedVersion, + data)` in `src/modules/orchestration/sla/repository/sla-run.repository.ts`, mirroring + `TicketsRepository.updateStatus`'s atomic `updateMany({where:{id, version: + expectedVersion}, data:{...data, version:{increment:1}}})` pattern exactly (depends on T002) +- [ ] T007 [US2] Update `pause`, `resume`, `complete`, and `runBreachDetectionSweep` in + `src/modules/orchestration/sla/service/sla.service.ts` to call `updateWithVersion` with + each run's last-read version, and to re-read + recompute + retry (bounded to 3 attempts) + on a version-conflict `null` result, per research.md §2 (depends on T006) +- [ ] T008 [US2] Write `tests/concurrency/sla-race.test.ts`: create a ticket with an active SLA + run, fire concurrent `pause`/`resume` calls and a `runBreachDetectionSweep()` pass against + it, then query the run directly and assert its final state is internally consistent (never + `paused` with `pausedAt: null`, never a legitimately `breached` run silently reverted to + `running`) (depends on T007) +- [ ] T009 [US2] Re-run `sla-race.test.ts` at least 10 times confirming zero + contradictory-state outcomes — SC-002 (depends on T008) + +**Checkpoint**: Quickstart Scenario 2 passes against real infrastructure, consistently. + +--- + +## Phase 5: User Story 3 - An escalation trigger fired twice never duplicates (Priority: P1) + +**Goal**: The same escalation trigger delivered twice for the same ticket always results in +exactly one escalation event and one reassignment. + +**Independent Test**: Run `tests/concurrency/escalation-idempotency.test.ts` alone against the +throwaway Postgres — it creates its own ticket + escalation rule and needs nothing from +US1/US2/US4/US5 (though it exercises the same `Assignment` table US1 protects, as a +cross-check). + +- [ ] T010 [US3] Fix `EscalationEventRepository.create` in + `src/modules/orchestration/escalation/repository/escalation-event.repository.ts` to catch + the `escalation_events_ticket_rule_unique` unique-violation (Prisma `P2002`) and return + the pre-existing row for that `(ticketId, ruleId)` pair via a `findFirst` fallback instead + of throwing, per research.md §3 (depends on T002) +- [ ] T011 [US3] Confirm `EscalationService.fire` in + `src/modules/orchestration/escalation/service/escalation.service.ts` behaves correctly + when `create` returns a pre-existing event (it must not also re-run + `assignToSpecificNode` for a duplicate trigger) — adjust `fire` if needed so a + duplicate-conflict short-circuits before reassignment (depends on T010) +- [ ] T012 [US3] Write `tests/concurrency/escalation-idempotency.test.ts`: create a ticket + eligible for a specific escalation rule, call the real trigger path (e.g. + `escalationService.handleBreach`) twice concurrently for the identical trigger, then query + `escalation_events` and `assignments` directly and assert exactly one of each resulted + (depends on T011) +- [ ] T013 [US3] Re-run `escalation-idempotency.test.ts` at least 10 times confirming zero + duplicate outcomes — SC-003 (depends on T012) + +**Checkpoint**: Quickstart Scenario 3 passes against real infrastructure, consistently. + +--- + +## Phase 6: User Story 4 - Ticket status optimistic concurrency, proven (Priority: P2) + +**Goal**: Prove the existing version-checked ticket-status update holds under genuine +concurrency. + +**Independent Test**: Run `tests/concurrency/ticket-status-race.test.ts` alone against the +throwaway Postgres — no dependency on T002 or any other user story (research.md §4: no +implementation change expected). + +- [ ] T014 [US4] Write `tests/concurrency/ticket-status-race.test.ts`: create a ticket at a + known status/version, fire >=20 genuinely concurrent `ticketsRepository.updateStatus` + calls all starting from that same version, and assert exactly one returns the updated + ticket while every other call returns `null` — SC-004 + +**Checkpoint**: Quickstart Scenario 4 passes, confirming the existing mechanism (no fix +expected; a failure here would mean research.md's assumption was wrong and needs revisiting). + +--- + +## Phase 7: User Story 5 - Repeatable load/throughput baseline (Priority: P2) + +**Goal**: Repeatable `autocannon`-based load-test tooling and a baseline report for the three +named critical endpoint groups. + +**Independent Test**: Run each `tests/load/*.load.ts` script alone against a real running dev +server — no dependency on T002 or any other user story. + +- [ ] T015 [P] [US5] Create `tests/load/autocannon.config.ts`: a shared runner helper wrapping + `autocannon`'s programmatic API, producing the report shape from data-model.md + (`requestsPerSec`, `latencyP50Ms`/`P90Ms`/`P99Ms`, `non2xxCount`, `rateLimitedCount`), + printing a console summary and writing JSON to `tests/load/reports/` (depends on T001) +- [ ] T016 [P] [US5] Create `tests/load/ticket-creation.load.ts` using the T015 helper against + `POST /v1/support/requests` (depends on T015) +- [ ] T017 [P] [US5] Create `tests/load/ai-support-flow.load.ts` using the T015 helper against + the AI support flow's own endpoints (depends on T015) +- [ ] T018 [P] [US5] Create `tests/load/admin-reporting.load.ts` using the T015 helper, signing + in as the seeded admin first, against the 015-reporting-dashboards endpoints (depends on + T015) +- [ ] T019 [US5] Run all three scripts against a real running dev server, confirm each produces + a report, and run each twice to confirm consistent-shape output for comparison — SC-005 + (depends on T016, T017, T018) + +**Checkpoint**: Quickstart Scenario 5 passes; a baseline report exists for each endpoint group. + +--- + +## Phase 8: Polish & Cross-Cutting Concerns + +- [ ] T020 Update `specs/016-load-concurrency-testing/checklists/requirements.md` Notes with any + implementation-time findings +- [ ] T021 `npx tsc --noEmit` / `npm run lint` / `npx tsx scripts/check-architecture.ts` clean +- [ ] T022 Full existing unit + integration + concurrency suite re-run (throwaway DB), confirming + no regression in 007-orchestration-assignment's, 008-sla-escalation's, + 012-admin-list-views's, and 015-reporting-dashboards's own existing coverage of + `Assignment`/`SLARun`/`EscalationEvent` +- [ ] T023 Mark all of this file's checkboxes complete once verified + +--- + +## Dependencies & Execution Order + +- **Setup (Phase 1)**: No dependencies — can start immediately +- **Foundational (Phase 2)**: No dependencies — BLOCKS User Stories 1, 2, 3 only +- **User Story 4**: No dependency on Phase 2 or any other story — can start immediately +- **User Story 5**: No dependency on Phase 2 or any other story — can start immediately (only + needs Phase 1's `autocannon` devDependency) +- **User Stories 1, 2, 3**: Each depends only on Phase 2 — independent of each other and of + User Stories 4/5 +- **Polish (Phase 8)**: Depends on all five user stories + +## Parallel Example: Foundational-independent stories + +```text +# Once Phase 1 completes, these can start immediately in parallel, without waiting on Phase 2: +Task: "Write tests/concurrency/ticket-status-race.test.ts" (US4, T014) +Task: "Create tests/load/autocannon.config.ts" (US5, T015) +``` + +## Implementation Strategy + +### Suggested order + +1. Phase 1 (Setup) and Phase 2 (Foundational) — Phase 2 unblocks the three highest-severity + real-bug fixes (US1, US2, US3) +2. User Stories 1, 2, 3 (all P1) — each is a real, currently-unguarded race; fix and prove each + in turn, or in parallel across files since they touch different modules +3. User Story 4 (P2) — quick to add, proves existing protection, can be done any time after + Phase 1 +4. User Story 5 (P2) — independent tooling work, can be done any time after Phase 1, in parallel + with 1-4 +5. Phase 8 (Polish) once all five stories are verified