support_backend/specs/016-load-concurrency-testing/spec.md

Feature Specification: Load and Concurrency Testing

Feature Branch: 016-load-concurrency-testing

Created: 2026-09-09

Status: Draft

Input: User description: "Load and concurrency testing (Phase 11): exercise the concurrency-safety guarantees docs/09-testing-observability-cicd.md's own testing strategy already calls for — assignment race conditions, SLA pause/resume durability, escalation idempotency, ticket optimistic concurrency — under genuinely concurrent requests against real infrastructure, fixing any real race a test reveals; and add real HTTP load/throughput testing against the API's own critical endpoints."

User Scenarios & Testing (mandatory)

User Story 1 - A ticket is never assigned to two agents at once under concurrent escalation (Priority: P1)

An operator needs confidence that when a ticket is escalated to a human (or reassigned) from more than one trigger at nearly the same moment — for example, a manual reassignment landing at the same instant as an automatic escalation-rule firing — the ticket ends up with exactly one current assignment, never two agents both believing they own the same case.

Why this priority: A double-assignment is a customer- and agent-facing correctness failure (two agents work the same ticket, or the SLA/workload dashboards silently double-count it) and undermines every dashboard and workload figure already shipped in this system. This is the most severe class of bug this feature can find.

Independent Test: Can be fully tested by firing many genuinely concurrent assignment requests at the same ticket against a real running instance of the API and a real Postgres database, then confirming exactly one Assignment row is marked current for that ticket afterward — no reliance on timing assumptions or sequential calls.

Acceptance Scenarios:

1. Given a ticket eligible for assignment, When many concurrent assignment attempts are made against it at once, Then exactly one assignment ends up marked as the ticket's current assignment, and the database itself (not just the last response received) confirms this.
2. Given the race in Scenario 1 is exercised repeatedly, When the test is run multiple times, Then the result is consistent every time — the protection does not depend on lucky timing.

---

User Story 2 - An SLA clock is never corrupted by overlapping pause/resume activity (Priority: P1)

An operator needs confidence that when a ticket's SLA clock is paused and resumed by more than one concurrent trigger — for example, a customer-reply webhook resuming the clock at the same moment the scheduled breach-detection sweep is evaluating that same ticket — the SLA run ends up in one coherent, correct state, never a state where the clock is simultaneously "paused" and "counting toward breach," and never a state that silently drops a pause/resume event.

Why this priority: SLA correctness is a contractual promise to customers and already backs the Management and Support dashboards shipped in 015-reporting-dashboards; a corrupted SLA clock produces wrong compliance figures and wrong breach alerts without any visible error.

Independent Test: Can be fully tested by firing concurrent pause and resume operations at the same SLA run against a real running instance of the API and a real Postgres database, then confirming the run's final stored state (paused/active, due-at timestamps) is internally consistent and matches one coherent ordering of the operations — not a mix of both.

Acceptance Scenarios:

1. Given an active SLA run, When a pause and a resume are triggered concurrently, Then the run's final state is exactly one of "paused" or "active" — never a state with contradictory fields (e.g., marked paused with no pause timestamp recorded, or marked active with a stale due-at that never accounted for the pause).
2. Given the breach-detection sweep is evaluating a run at the same moment a resume is requested for it, When both complete, Then the run is not double-processed (no duplicate breach event, no lost resume).

---

User Story 3 - An escalation rule firing twice never creates two escalation events (Priority: P1)

An operator needs confidence that if the same escalation trigger is delivered more than once — for example, a retried background job or a re-processed event — the ticket is escalated exactly once, not reassigned and re-notified redundantly.

Why this priority: Duplicate escalations would double-notify agents, double-count in the Support and Management dashboards, and could re-trigger reassignment away from an agent who has already started work — a direct regression of work already done in this session.

Independent Test: Can be fully tested by firing the same escalation trigger concurrently more than once for the same ticket against a real running instance of the API and a real Postgres database, then confirming only one EscalationEvent row exists for that trigger afterward.

Acceptance Scenarios:

1. Given a ticket eligible for escalation, When the same escalation trigger is delivered twice at nearly the same moment, Then exactly one escalation event is recorded for it.
2. Given Scenario 1's duplicate delivery, When the escalation event is created, Then the ticket is reassigned exactly once, not twice.

---

User Story 4 - A ticket's status can never be corrupted by two simultaneous updates (Priority: P2)

An operator needs confidence that the ticket status-transition safeguard already built for this system actually holds under real concurrent load, not just in isolated sequential tests — this is existing protection, but has never been proven under genuine concurrency.

Why this priority: Lower priority than User Stories 1-3 because a real defensive mechanism already exists here (see Assumptions); this story exists to convert an untested assumption into a proven guarantee, and is valuable but lower-risk than the three unguarded races above.

Independent Test: Can be fully tested by firing multiple concurrent status-update attempts at the same ticket, each based on the same starting version, against a real running API and database, then confirming exactly one update succeeds and every other attempt receives a clear conflict response rather than silently corrupting or skipping the ticket's state.

Acceptance Scenarios:

1. Given a ticket at a known status and version, When multiple concurrent status-update requests are made from that same version, Then exactly one succeeds and the rest are rejected with a conflict response, and the ticket's final status matches the one update that succeeded.

---

User Story 5 - The API's critical endpoints hold up under realistic concurrent traffic (Priority: P2)

An operator needs a documented, repeatable measurement of how the system's most important endpoints — new support requests coming in, the AI support flow, and the admin reporting dashboards — behave under sustained concurrent load, so that a future capacity or performance regression can be caught by comparing against this baseline rather than guessed at.

Why this priority: This is about establishing a measurable baseline and repeatable tooling rather than proving or fixing a specific correctness bug (unlike User Stories 1-4), so it is valuable but not blocking for the correctness guarantees above.

Independent Test: Can be fully tested by running a load-test tool against a real running instance of the API for each of the three named endpoint groups and producing a report of throughput, latency percentiles, and error rate, independent of whether any other user story in this feature has been completed.

Acceptance Scenarios:

1. Given the API is running against real infrastructure, When a defined concurrent load is sent to the ticket-creation endpoint for a sustained period, Then a report is produced showing throughput, latency percentiles, and error rate for that run.
2. Given the same setup, When the same load profile is sent to the AI support flow and to the admin reporting endpoints, Then an equivalent report is produced for each, allowing the three to be compared against each other and against future runs.

Edge Cases

• What happens when a concurrent assignment race includes a ticket that is simultaneously being closed or reopened? The assignment/reassignment safeguard must not be bypassable by a status change racing the same window.
• What happens when a pause and a breach both become due at the exact same instant? The final state must reflect one coherent, auditable outcome, not an unresolvable both-happened state.
• What happens when the load test itself pushes an endpoint into its own rate limiter (e.g. the 013-auth-hardening login rate limit, or the SaaS integration per-minute rate limits)? The report must distinguish "rejected by design (rate limit)" from "failed under load" rather than counting both as the same kind of failure.
• What happens when two of these races are exercised back-to-back against the same throwaway database without cleanup? Each test must use its own uniquely-identified fixtures so repeated runs (and CI re-runs) don't produce false positives or false negatives from leftover state.

Requirements (mandatory)

Functional Requirements

• FR-001: The system MUST guarantee that a ticket never has more than one assignment marked as current, even when multiple assignment operations are attempted concurrently against it.
• FR-002: The system MUST guarantee that an SLA run's paused/active state and its associated timestamps remain internally consistent when pause, resume, and the breach-detection sweep are triggered concurrently against the same run.
• FR-003: The system MUST guarantee that the same escalation trigger delivered more than once for the same ticket produces exactly one escalation event and exactly one resulting reassignment.
• FR-004: The system MUST reject a ticket status update whose expected starting version no longer matches the ticket's actual current version, even when the conflicting updates are concurrent, and MUST leave the ticket in the state produced by whichever single update actually succeeded.
• FR-005: The system's automated test suite MUST include a dedicated concurrency test for each of FR-001 through FR-004, each exercising genuinely concurrent requests against real, live infrastructure (not mocked timers or sequential calls standing in for concurrency).
• FR-006: Where a concurrency test written for this feature reveals that a guarantee in FR-001, FR-002, or FR-003 does not currently hold, the underlying race MUST be fixed as part of this feature, not merely documented.
• FR-007: The system MUST provide repeatable load-test tooling covering, at minimum: new support request submission, the AI support flow, and the admin reporting dashboard endpoints.
• FR-008: Each load test run MUST produce a report including throughput, latency percentiles, and error rate, with rate-limited responses reported separately from failures.
• FR-009: Pass/fail thresholds for the load tests (target throughput, acceptable latency, acceptable error rate) MUST be explicitly marked as OPEN BUSINESS DECISION wherever the business has not already specified a number, per this project's own roadmap convention — never hardcoded as if final.

Key Entities

• Assignment race scenario: A reusable test setup representing "many concurrent attempts to assign or reassign the same ticket," used to exercise FR-001.
• SLA race scenario: A reusable test setup representing "concurrent pause, resume, and sweep activity against the same SLA run," used to exercise FR-002.
• Escalation race scenario: A reusable test setup representing "the same escalation trigger delivered more than once for the same ticket," used to exercise FR-003.
• Load test report: The recorded output of a load-test run against one endpoint group — throughput, latency percentiles, error rate, and rate-limited-response count — kept so a future run can be compared against it.

Success Criteria (mandatory)

Measurable Outcomes

• SC-001: A test run that fires at least 20 genuinely concurrent assignment attempts at the same ticket always results in exactly one current assignment, with zero exceptions across at least 10 repeated runs.
• SC-002: A test run that fires concurrent pause/resume/sweep activity against the same SLA run always leaves that run in one internally-consistent, auditable state, with zero contradictory-state outcomes across at least 10 repeated runs.
• SC-003: A test run that delivers the same escalation trigger twice for the same ticket always results in exactly one escalation event and exactly one reassignment, with zero duplicate outcomes across at least 10 repeated runs.
• SC-004: A test run that fires at least 20 genuinely concurrent status-update attempts from the same starting version against the same ticket always results in exactly one success and the ticket left in that one succeeding state.
• SC-005: A load-test report exists for each of the three named endpoint groups (ticket creation, AI support flow, admin reporting), each independently re-runnable on demand and producing consistent-shape output for comparison across runs.

Assumptions

• Ticket status optimistic concurrency (User Story 4) already has a real defensive mechanism in the codebase (a version-checked atomic update) — this feature's job for that story is to prove it under genuine concurrency with a new test, not to build new protection, unless that test surprises this assumption and reveals a real gap.
• Assignment double-assignment, SLA pause/resume races, and escalation duplicate-event risk (User Stories 1-3) are NOT currently guarded against — this feature's job for those stories is both to prove the gap with a real concurrency test and to implement the fix, per FR-006.
• "Genuinely concurrent" means real parallel requests issued against a real running instance of the API backed by real Postgres/Redis (this project's standing verification discipline throughout every prior feature), not fake-timer or mocked-clock simulations.
• Load testing (User Story 5) targets the existing dev/throwaway infrastructure already used for this project's own manual verification, not a separate staging or production environment — provisioning a dedicated load-test environment is out of scope.
• Specific throughput/latency/error-rate thresholds for "pass" are an OPEN BUSINESS DECISION per FR-009; this feature delivers the tooling and a baseline report, not a final SLA number.
• Round-robin assignment-selection counter safety is already covered by an existing genuine concurrency test and is explicitly out of scope for this feature.