support_backend/specs/016-load-concurrency-testing/tasks.md

description

description
Task list for 016-load-concurrency-testing

Tasks: Load and Concurrency Testing

Input: Design documents from specs/016-load-concurrency-testing/

Organization: Tasks are grouped by user story (US1 = assignment race, US2 = SLA race, US3 = escalation idempotency, US4 = ticket-status race proof, US5 = load-test tooling). US1-US4 share one Foundational phase (the schema migration all four rely on); US5 has no schema dependency and can proceed independently of it.

Format: [ID] [P?] [Story] Description

All file paths are relative to supporthub-api/ (repo root).

---

Phase 1: Setup

• T001 [P] Add autocannon as a devDependency (package.json) and add tests/load/reports/ to .gitignore (run artifacts, not fixtures)

---

Phase 2: Foundational (Blocking Prerequisites for US1-US4)

Purpose: The one shared schema migration US1, US2, and US3's fixes each depend on. US4 (no schema change, see research.md §4) and US5 (no schema dependency) do not need this phase and can proceed in parallel with it.

• T002 In prisma/schema.prisma, add version Int @default(0) to SLARun; generate one migration (npx prisma migrate dev --name concurrency_guards) that also includes, as raw SQL, CREATE UNIQUE INDEX assignments_one_current_per_ticket ON assignments (ticket_id) WHERE is_current = true; and CREATE UNIQUE INDEX escalation_events_ticket_rule_unique ON escalation_events (ticket_id, rule_id) WHERE rule_id IS NOT NULL; (data-model.md); apply to the throwaway test Postgres (supporthub-test-pg, port 5433) and the real dev Postgres (postgres-development, port 5434, via prisma migrate diff + direct psql per this project's own established non-destructive dev-sync approach); regenerate the Prisma client

Checkpoint: Schema ready — US1, US2, US3 implementation can now begin.

---

Phase 3: User Story 1 - Assignment never double-assigned under concurrency (Priority: P1)

Goal: Two concurrent assignment attempts on the same ticket always leave exactly one current assignment.

Independent Test: Run tests/concurrency/assignment-race.test.ts alone against the throwaway Postgres — it creates its own ticket and needs nothing from US2-US5.

• T003 [US1] Write tests/concurrency/assignment-race.test.ts: create one ticket, fire >=20 genuinely concurrent assignment attempts at it (via the real assignment engine/service entry point, not the repository directly), then query assignments directly and assert exactly one row has is_current = true for that ticket (depends on T002)
• T004 [US1] Fix AssignmentRepository.createAssignment in src/modules/orchestration/assignments/repository/assignment.repository.ts to catch the assignments_one_current_per_ticket unique-violation (Prisma P2002) and retry the whole supersede-then-create transaction, bounded to 3 attempts, per research.md §1 (depends on T002)
• T005 [US1] Re-run assignment-race.test.ts at least 10 times in a row (or extend the test with its own internal repeat loop) confirming zero failures — SC-001 (depends on T003, T004)

Checkpoint: Quickstart Scenario 1 passes against real infrastructure, consistently.

---

Phase 4: User Story 2 - SLA clock never corrupted by overlapping pause/resume/sweep (Priority: P1)

Goal: Concurrent pause/resume/breach-sweep activity against the same SLA run always leaves it in one internally-consistent state.

Independent Test: Run tests/concurrency/sla-race.test.ts alone against the throwaway Postgres — it creates its own ticket + SLA run and needs nothing from US1/US3/US4/US5.

• T006 [US2] Replace SlaRunRepository.update with updateWithVersion(id, expectedVersion, data) in src/modules/orchestration/sla/repository/sla-run.repository.ts, mirroring TicketsRepository.updateStatus's atomic updateMany({where:{id, version: expectedVersion}, data:{...data, version:{increment:1}}}) pattern exactly (depends on T002)
• T007 [US2] Update pause, resume, complete, and runBreachDetectionSweep in src/modules/orchestration/sla/service/sla.service.ts to call updateWithVersion with each run's last-read version, and to re-read + recompute + retry (bounded to 3 attempts) on a version-conflict null result, per research.md §2 (depends on T006)
• T008 [US2] Write tests/concurrency/sla-race.test.ts: create a ticket with an active SLA run, fire concurrent pause/resume calls and a runBreachDetectionSweep() pass against it, then query the run directly and assert its final state is internally consistent (never paused with pausedAt: null, never a legitimately breached run silently reverted to running) (depends on T007)
• T009 [US2] Re-run sla-race.test.ts at least 10 times confirming zero contradictory-state outcomes — SC-002 (depends on T008)

Checkpoint: Quickstart Scenario 2 passes against real infrastructure, consistently.

---

Phase 5: User Story 3 - An escalation trigger fired twice never duplicates (Priority: P1)

Goal: The same escalation trigger delivered twice for the same ticket always results in exactly one escalation event and one reassignment.

Independent Test: Run tests/concurrency/escalation-idempotency.test.ts alone against the throwaway Postgres — it creates its own ticket + escalation rule and needs nothing from US1/US2/US4/US5 (though it exercises the same Assignment table US1 protects, as a cross-check).

• T010 [US3] Fix EscalationEventRepository.create in src/modules/orchestration/escalation/repository/escalation-event.repository.ts to catch the escalation_events_ticket_rule_unique unique-violation (Prisma P2002) and return the pre-existing row for that (ticketId, ruleId) pair via a findFirst fallback instead of throwing, per research.md §3 (depends on T002)
• T011 [US3] Confirm EscalationService.fire in src/modules/orchestration/escalation/service/escalation.service.ts behaves correctly when create returns a pre-existing event (it must not also re-run assignToSpecificNode for a duplicate trigger) — adjust fire if needed so a duplicate-conflict short-circuits before reassignment (depends on T010)
• T012 [US3] Write tests/concurrency/escalation-idempotency.test.ts: create a ticket eligible for a specific escalation rule, call the real trigger path (e.g. escalationService.handleBreach) twice concurrently for the identical trigger, then query escalation_events and assignments directly and assert exactly one of each resulted (depends on T011)
• T013 [US3] Re-run escalation-idempotency.test.ts at least 10 times confirming zero duplicate outcomes — SC-003 (depends on T012)

Checkpoint: Quickstart Scenario 3 passes against real infrastructure, consistently.

---

Phase 6: User Story 4 - Ticket status optimistic concurrency, proven (Priority: P2)

Goal: Prove the existing version-checked ticket-status update holds under genuine concurrency.

Independent Test: Run tests/concurrency/ticket-status-race.test.ts alone against the throwaway Postgres — no dependency on T002 or any other user story (research.md §4: no implementation change expected).

• T014 [US4] Write tests/concurrency/ticket-status-race.test.ts: create a ticket at a known status/version, fire >=20 genuinely concurrent ticketsRepository.updateStatus calls all starting from that same version, and assert exactly one returns the updated ticket while every other call returns null — SC-004

Checkpoint: Quickstart Scenario 4 passes, confirming the existing mechanism (no fix expected; a failure here would mean research.md's assumption was wrong and needs revisiting).

---

Phase 7: User Story 5 - Repeatable load/throughput baseline (Priority: P2)

Goal: Repeatable autocannon-based load-test tooling and a baseline report for the three named critical endpoint groups.

Independent Test: Run each tests/load/*.load.ts script alone against a real running dev server — no dependency on T002 or any other user story.

• T015 [P] [US5] Create tests/load/autocannon.config.ts: a shared runner helper wrapping autocannon's programmatic API, producing the report shape from data-model.md (requestsPerSec, latencyP50Ms/P90Ms/P99Ms, non2xxCount, rateLimitedCount), printing a console summary and writing JSON to tests/load/reports/ (depends on T001)
• T016 [P] [US5] Create tests/load/ticket-creation.load.ts using the T015 helper against POST /v1/support/requests (depends on T015)
• T017 [P] [US5] Create tests/load/ai-support-flow.load.ts using the T015 helper against the AI support flow's own endpoints (depends on T015)
• T018 [P] [US5] Create tests/load/admin-reporting.load.ts using the T015 helper, signing in as the seeded admin first, against the 015-reporting-dashboards endpoints (depends on T015)
• T019 [US5] Run all three scripts against a real running dev server, confirm each produces a report, and run each twice to confirm consistent-shape output for comparison — SC-005 (depends on T016, T017, T018)

Checkpoint: Quickstart Scenario 5 passes; a baseline report exists for each endpoint group.

---

Phase 8: Polish & Cross-Cutting Concerns

• T020 Update specs/016-load-concurrency-testing/checklists/requirements.md Notes with any implementation-time findings
• T021 npx tsc --noEmit / npm run lint / npx tsx scripts/check-architecture.ts clean
• T022 Full existing unit + integration + concurrency suite re-run (throwaway DB), confirming no regression in 007-orchestration-assignment's, 008-sla-escalation's, 012-admin-list-views's, and 015-reporting-dashboards's own existing coverage of Assignment/SLARun/EscalationEvent
• T023 Mark all of this file's checkboxes complete once verified

---

Dependencies & Execution Order

• Setup (Phase 1): No dependencies — can start immediately
• Foundational (Phase 2): No dependencies — BLOCKS User Stories 1, 2, 3 only
• User Story 4: No dependency on Phase 2 or any other story — can start immediately
• User Story 5: No dependency on Phase 2 or any other story — can start immediately (only needs Phase 1's autocannon devDependency)
• User Stories 1, 2, 3: Each depends only on Phase 2 — independent of each other and of User Stories 4/5
• Polish (Phase 8): Depends on all five user stories

Parallel Example: Foundational-independent stories

# Once Phase 1 completes, these can start immediately in parallel, without waiting on Phase 2:
Task: "Write tests/concurrency/ticket-status-race.test.ts" (US4, T014)
Task: "Create tests/load/autocannon.config.ts" (US5, T015)

Implementation Strategy

Suggested order

1. Phase 1 (Setup) and Phase 2 (Foundational) — Phase 2 unblocks the three highest-severity real-bug fixes (US1, US2, US3)
2. User Stories 1, 2, 3 (all P1) — each is a real, currently-unguarded race; fix and prove each in turn, or in parallel across files since they touch different modules
3. User Story 4 (P2) — quick to add, proves existing protection, can be done any time after Phase 1
4. User Story 5 (P2) — independent tooling work, can be done any time after Phase 1, in parallel with 1-4
5. Phase 8 (Polish) once all five stories are verified