diff --git a/specs/016-load-concurrency-testing/checklists/requirements.md b/specs/016-load-concurrency-testing/checklists/requirements.md new file mode 100644 index 0000000..2272d90 --- /dev/null +++ b/specs/016-load-concurrency-testing/checklists/requirements.md @@ -0,0 +1,41 @@ +# Specification Quality Checklist: Load and Concurrency Testing + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-09-09 +**Feature**: [spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- This feature was scoped from a targeted codebase audit (not guesswork) confirming which + concurrency guarantees already exist untested (ticket optimistic concurrency) versus which + have no protection at all today (assignment double-assignment, SLA pause/resume, escalation + idempotency) — see spec.md's own Assumptions section. +- Per this project's own roadmap convention, exact load-test pass/fail thresholds are left as an + explicit `OPEN BUSINESS DECISION` (FR-009) rather than invented — this is intentional, not a + gap requiring [NEEDS CLARIFICATION]. +- All items pass; no revision iterations were needed. diff --git a/specs/016-load-concurrency-testing/spec.md b/specs/016-load-concurrency-testing/spec.md new file mode 100644 index 0000000..6b5c7c7 --- /dev/null +++ b/specs/016-load-concurrency-testing/spec.md @@ -0,0 +1,240 @@ +# Feature Specification: Load and Concurrency Testing + +**Feature Branch**: `016-load-concurrency-testing` + +**Created**: 2026-09-09 + +**Status**: Draft + +**Input**: User description: "Load and concurrency testing (Phase 11): exercise the concurrency-safety guarantees docs/09-testing-observability-cicd.md's own testing strategy already calls for — assignment race conditions, SLA pause/resume durability, escalation idempotency, ticket optimistic concurrency — under genuinely concurrent requests against real infrastructure, fixing any real race a test reveals; and add real HTTP load/throughput testing against the API's own critical endpoints." + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - A ticket is never assigned to two agents at once under concurrent escalation (Priority: P1) + +An operator needs confidence that when a ticket is escalated to a human (or reassigned) from +more than one trigger at nearly the same moment — for example, a manual reassignment landing at +the same instant as an automatic escalation-rule firing — the ticket ends up with exactly one +current assignment, never two agents both believing they own the same case. + +**Why this priority**: A double-assignment is a customer- and agent-facing correctness failure +(two agents work the same ticket, or the SLA/workload dashboards silently double-count it) and +undermines every dashboard and workload figure already shipped in this system. This is the most +severe class of bug this feature can find. + +**Independent Test**: Can be fully tested by firing many genuinely concurrent assignment +requests at the same ticket against a real running instance of the API and a real Postgres +database, then confirming exactly one `Assignment` row is marked current for that ticket +afterward — no reliance on timing assumptions or sequential calls. + +**Acceptance Scenarios**: + +1. **Given** a ticket eligible for assignment, **When** many concurrent assignment attempts are + made against it at once, **Then** exactly one assignment ends up marked as the ticket's + current assignment, and the database itself (not just the last response received) confirms + this. +2. **Given** the race in Scenario 1 is exercised repeatedly, **When** the test is run multiple + times, **Then** the result is consistent every time — the protection does not depend on + lucky timing. + +--- + +### User Story 2 - An SLA clock is never corrupted by overlapping pause/resume activity (Priority: P1) + +An operator needs confidence that when a ticket's SLA clock is paused and resumed by more than +one concurrent trigger — for example, a customer-reply webhook resuming the clock at the same +moment the scheduled breach-detection sweep is evaluating that same ticket — the SLA run ends up +in one coherent, correct state, never a state where the clock is simultaneously "paused" and +"counting toward breach," and never a state that silently drops a pause/resume event. + +**Why this priority**: SLA correctness is a contractual promise to customers and already backs +the Management and Support dashboards shipped in 015-reporting-dashboards; a corrupted SLA clock +produces wrong compliance figures and wrong breach alerts without any visible error. + +**Independent Test**: Can be fully tested by firing concurrent pause and resume operations at +the same SLA run against a real running instance of the API and a real Postgres database, then +confirming the run's final stored state (paused/active, due-at timestamps) is internally +consistent and matches one coherent ordering of the operations — not a mix of both. + +**Acceptance Scenarios**: + +1. **Given** an active SLA run, **When** a pause and a resume are triggered concurrently, + **Then** the run's final state is exactly one of "paused" or "active" — never a state with + contradictory fields (e.g., marked paused with no pause timestamp recorded, or marked active + with a stale due-at that never accounted for the pause). +2. **Given** the breach-detection sweep is evaluating a run at the same moment a resume is + requested for it, **When** both complete, **Then** the run is not double-processed (no + duplicate breach event, no lost resume). + +--- + +### User Story 3 - An escalation rule firing twice never creates two escalation events (Priority: P1) + +An operator needs confidence that if the same escalation trigger is delivered more than once — +for example, a retried background job or a re-processed event — the ticket is escalated exactly +once, not reassigned and re-notified redundantly. + +**Why this priority**: Duplicate escalations would double-notify agents, double-count in the +Support and Management dashboards, and could re-trigger reassignment away from an agent who has +already started work — a direct regression of work already done in this session. + +**Independent Test**: Can be fully tested by firing the same escalation trigger concurrently +more than once for the same ticket against a real running instance of the API and a real +Postgres database, then confirming only one `EscalationEvent` row exists for that trigger +afterward. + +**Acceptance Scenarios**: + +1. **Given** a ticket eligible for escalation, **When** the same escalation trigger is delivered + twice at nearly the same moment, **Then** exactly one escalation event is recorded for it. +2. **Given** Scenario 1's duplicate delivery, **When** the escalation event is created, + **Then** the ticket is reassigned exactly once, not twice. + +--- + +### User Story 4 - A ticket's status can never be corrupted by two simultaneous updates (Priority: P2) + +An operator needs confidence that the ticket status-transition safeguard already built for this +system actually holds under real concurrent load, not just in isolated sequential tests — this +is existing protection, but has never been proven under genuine concurrency. + +**Why this priority**: Lower priority than User Stories 1-3 because a real defensive mechanism +already exists here (see Assumptions); this story exists to convert an untested assumption into +a proven guarantee, and is valuable but lower-risk than the three unguarded races above. + +**Independent Test**: Can be fully tested by firing multiple concurrent status-update attempts +at the same ticket, each based on the same starting version, against a real running API and +database, then confirming exactly one update succeeds and every other attempt receives a clear +conflict response rather than silently corrupting or skipping the ticket's state. + +**Acceptance Scenarios**: + +1. **Given** a ticket at a known status and version, **When** multiple concurrent status-update + requests are made from that same version, **Then** exactly one succeeds and the rest are + rejected with a conflict response, and the ticket's final status matches the one update that + succeeded. + +--- + +### User Story 5 - The API's critical endpoints hold up under realistic concurrent traffic (Priority: P2) + +An operator needs a documented, repeatable measurement of how the system's most important +endpoints — new support requests coming in, the AI support flow, and the admin reporting +dashboards — behave under sustained concurrent load, so that a future capacity or performance +regression can be caught by comparing against this baseline rather than guessed at. + +**Why this priority**: This is about establishing a measurable baseline and repeatable tooling +rather than proving or fixing a specific correctness bug (unlike User Stories 1-4), so it is +valuable but not blocking for the correctness guarantees above. + +**Independent Test**: Can be fully tested by running a load-test tool against a real running +instance of the API for each of the three named endpoint groups and producing a report of +throughput, latency percentiles, and error rate, independent of whether any other user story in +this feature has been completed. + +**Acceptance Scenarios**: + +1. **Given** the API is running against real infrastructure, **When** a defined concurrent load + is sent to the ticket-creation endpoint for a sustained period, **Then** a report is produced + showing throughput, latency percentiles, and error rate for that run. +2. **Given** the same setup, **When** the same load profile is sent to the AI support flow and + to the admin reporting endpoints, **Then** an equivalent report is produced for each, + allowing the three to be compared against each other and against future runs. + +### Edge Cases + +- What happens when a concurrent assignment race includes a ticket that is simultaneously being + closed or reopened? The assignment/reassignment safeguard must not be bypassable by a + status change racing the same window. +- What happens when a pause and a breach both become due at the exact same instant? The final + state must reflect one coherent, auditable outcome, not an unresolvable both-happened state. +- What happens when the load test itself pushes an endpoint into its own rate limiter (e.g. the + 013-auth-hardening login rate limit, or the SaaS integration per-minute rate limits)? The + report must distinguish "rejected by design (rate limit)" from "failed under load" rather than + counting both as the same kind of failure. +- What happens when two of these races are exercised back-to-back against the same throwaway + database without cleanup? Each test must use its own uniquely-identified fixtures so repeated + runs (and CI re-runs) don't produce false positives or false negatives from leftover state. + +## Requirements *(mandatory)* + +### Functional Requirements + +- **FR-001**: The system MUST guarantee that a ticket never has more than one assignment marked + as current, even when multiple assignment operations are attempted concurrently against it. +- **FR-002**: The system MUST guarantee that an SLA run's paused/active state and its associated + timestamps remain internally consistent when pause, resume, and the breach-detection sweep are + triggered concurrently against the same run. +- **FR-003**: The system MUST guarantee that the same escalation trigger delivered more than + once for the same ticket produces exactly one escalation event and exactly one resulting + reassignment. +- **FR-004**: The system MUST reject a ticket status update whose expected starting version no + longer matches the ticket's actual current version, even when the conflicting updates are + concurrent, and MUST leave the ticket in the state produced by whichever single update + actually succeeded. +- **FR-005**: The system's automated test suite MUST include a dedicated concurrency test for + each of FR-001 through FR-004, each exercising genuinely concurrent requests against real, + live infrastructure (not mocked timers or sequential calls standing in for concurrency). +- **FR-006**: Where a concurrency test written for this feature reveals that a guarantee in + FR-001, FR-002, or FR-003 does not currently hold, the underlying race MUST be fixed as part + of this feature, not merely documented. +- **FR-007**: The system MUST provide repeatable load-test tooling covering, at minimum: new + support request submission, the AI support flow, and the admin reporting dashboard endpoints. +- **FR-008**: Each load test run MUST produce a report including throughput, latency + percentiles, and error rate, with rate-limited responses reported separately from failures. +- **FR-009**: Pass/fail thresholds for the load tests (target throughput, acceptable latency, + acceptable error rate) MUST be explicitly marked as `OPEN BUSINESS DECISION` wherever the + business has not already specified a number, per this project's own roadmap convention — + never hardcoded as if final. + +### Key Entities + +- **Assignment race scenario**: A reusable test setup representing "many concurrent attempts to + assign or reassign the same ticket," used to exercise FR-001. +- **SLA race scenario**: A reusable test setup representing "concurrent pause, resume, and sweep + activity against the same SLA run," used to exercise FR-002. +- **Escalation race scenario**: A reusable test setup representing "the same escalation trigger + delivered more than once for the same ticket," used to exercise FR-003. +- **Load test report**: The recorded output of a load-test run against one endpoint group — + throughput, latency percentiles, error rate, and rate-limited-response count — kept so a + future run can be compared against it. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: A test run that fires at least 20 genuinely concurrent assignment attempts at the + same ticket always results in exactly one current assignment, with zero exceptions across at + least 10 repeated runs. +- **SC-002**: A test run that fires concurrent pause/resume/sweep activity against the same SLA + run always leaves that run in one internally-consistent, auditable state, with zero + contradictory-state outcomes across at least 10 repeated runs. +- **SC-003**: A test run that delivers the same escalation trigger twice for the same ticket + always results in exactly one escalation event and exactly one reassignment, with zero + duplicate outcomes across at least 10 repeated runs. +- **SC-004**: A test run that fires at least 20 genuinely concurrent status-update attempts from + the same starting version against the same ticket always results in exactly one success and + the ticket left in that one succeeding state. +- **SC-005**: A load-test report exists for each of the three named endpoint groups (ticket + creation, AI support flow, admin reporting), each independently re-runnable on demand and + producing consistent-shape output for comparison across runs. + +## Assumptions + +- Ticket status optimistic concurrency (User Story 4) already has a real defensive mechanism in + the codebase (a version-checked atomic update) — this feature's job for that story is to prove + it under genuine concurrency with a new test, not to build new protection, unless that test + surprises this assumption and reveals a real gap. +- Assignment double-assignment, SLA pause/resume races, and escalation duplicate-event risk (User + Stories 1-3) are NOT currently guarded against — this feature's job for those stories is both + to prove the gap with a real concurrency test and to implement the fix, per FR-006. +- "Genuinely concurrent" means real parallel requests issued against a real running instance of + the API backed by real Postgres/Redis (this project's standing verification discipline + throughout every prior feature), not fake-timer or mocked-clock simulations. +- Load testing (User Story 5) targets the existing dev/throwaway infrastructure already used for + this project's own manual verification, not a separate staging or production environment — + provisioning a dedicated load-test environment is out of scope. +- Specific throughput/latency/error-rate thresholds for "pass" are an `OPEN BUSINESS DECISION` + per FR-009; this feature delivers the tooling and a baseline report, not a final SLA number. +- Round-robin assignment-selection counter safety is already covered by an existing genuine + concurrency test and is explicitly out of scope for this feature.