maskantech/support_backend
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

spec(016-load-concurrency-testing): specify concurrency-safety and load testing scope

Browse Source 
Five user stories: assignment double-assignment race, SLA pause/resume race,
escalation idempotency, ticket optimistic-concurrency proof, and HTTP
load/throughput testing tooling. Scoped from a targeted audit of existing
concurrency guarantees rather than guesswork — see spec.md's Assumptions.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
saqib mir
2026-09-09 16:51:00 +05:30
co-authored by Claude Sonnet 5
parent 7106753ed3
commit 9f52d51003
2 changed files with 281 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
specs/016-load-concurrency-testing/checklists/requirements.md
+41
View File
@@ -0,0 +1,41 @@
# Specification Quality Checklist: Load and Concurrency Testing
**Purpose**: Validate specification completeness and quality before proceeding to planning
**Created**: 2026-09-09
**Feature**: [spec.md](../spec.md)
## Content Quality
- [x] No implementation details (languages, frameworks, APIs)
- [x] Focused on user value and business needs
- [x] Written for non-technical stakeholders
- [x] All mandatory sections completed
## Requirement Completeness
- [x] No [NEEDS CLARIFICATION] markers remain
- [x] Requirements are testable and unambiguous
- [x] Success criteria are measurable
- [x] Success criteria are technology-agnostic (no implementation details)
- [x] All acceptance scenarios are defined
- [x] Edge cases are identified
- [x] Scope is clearly bounded
- [x] Dependencies and assumptions identified
## Feature Readiness
- [x] All functional requirements have clear acceptance criteria
- [x] User scenarios cover primary flows
- [x] Feature meets measurable outcomes defined in Success Criteria
- [x] No implementation details leak into specification
## Notes
- This feature was scoped from a targeted codebase audit (not guesswork) confirming which
concurrency guarantees already exist untested (ticket optimistic concurrency) versus which
have no protection at all today (assignment double-assignment, SLA pause/resume, escalation
idempotency) — see spec.md's own Assumptions section.
- Per this project's own roadmap convention, exact load-test pass/fail thresholds are left as an
explicit `OPEN BUSINESS DECISION` (FR-009) rather than invented — this is intentional, not a
gap requiring [NEEDS CLARIFICATION].
- All items pass; no revision iterations were needed.
specs/016-load-concurrency-testing/spec.md
+240
View File
@@ -0,0 +1,240 @@
# Feature Specification: Load and Concurrency Testing
**Feature Branch**: `016-load-concurrency-testing`
**Created**: 2026-09-09
**Status**: Draft
**Input**: User description: "Load and concurrency testing (Phase 11): exercise the concurrency-safety guarantees docs/09-testing-observability-cicd.md's own testing strategy already calls for — assignment race conditions, SLA pause/resume durability, escalation idempotency, ticket optimistic concurrency — under genuinely concurrent requests against real infrastructure, fixing any real race a test reveals; and add real HTTP load/throughput testing against the API's own critical endpoints."
## User Scenarios & Testing *(mandatory)*
### User Story 1 - A ticket is never assigned to two agents at once under concurrent escalation (Priority: P1)
An operator needs confidence that when a ticket is escalated to a human (or reassigned) from
more than one trigger at nearly the same moment — for example, a manual reassignment landing at
the same instant as an automatic escalation-rule firing — the ticket ends up with exactly one
current assignment, never two agents both believing they own the same case.
**Why this priority**: A double-assignment is a customer- and agent-facing correctness failure
(two agents work the same ticket, or the SLA/workload dashboards silently double-count it) and
undermines every dashboard and workload figure already shipped in this system. This is the most
severe class of bug this feature can find.
**Independent Test**: Can be fully tested by firing many genuinely concurrent assignment
requests at the same ticket against a real running instance of the API and a real Postgres
database, then confirming exactly one `Assignment` row is marked current for that ticket
afterward — no reliance on timing assumptions or sequential calls.
**Acceptance Scenarios**:
1. **Given** a ticket eligible for assignment, **When** many concurrent assignment attempts are
made against it at once, **Then** exactly one assignment ends up marked as the ticket's
current assignment, and the database itself (not just the last response received) confirms
this.
2. **Given** the race in Scenario 1 is exercised repeatedly, **When** the test is run multiple
times, **Then** the result is consistent every time — the protection does not depend on
lucky timing.
---
### User Story 2 - An SLA clock is never corrupted by overlapping pause/resume activity (Priority: P1)
An operator needs confidence that when a ticket's SLA clock is paused and resumed by more than
one concurrent trigger — for example, a customer-reply webhook resuming the clock at the same
moment the scheduled breach-detection sweep is evaluating that same ticket — the SLA run ends up
in one coherent, correct state, never a state where the clock is simultaneously "paused" and
"counting toward breach," and never a state that silently drops a pause/resume event.
**Why this priority**: SLA correctness is a contractual promise to customers and already backs
the Management and Support dashboards shipped in 015-reporting-dashboards; a corrupted SLA clock
produces wrong compliance figures and wrong breach alerts without any visible error.
**Independent Test**: Can be fully tested by firing concurrent pause and resume operations at
the same SLA run against a real running instance of the API and a real Postgres database, then
confirming the run's final stored state (paused/active, due-at timestamps) is internally
consistent and matches one coherent ordering of the operations — not a mix of both.
**Acceptance Scenarios**:
1. **Given** an active SLA run, **When** a pause and a resume are triggered concurrently,
**Then** the run's final state is exactly one of "paused" or "active" — never a state with
contradictory fields (e.g., marked paused with no pause timestamp recorded, or marked active
with a stale due-at that never accounted for the pause).
2. **Given** the breach-detection sweep is evaluating a run at the same moment a resume is
requested for it, **When** both complete, **Then** the run is not double-processed (no
duplicate breach event, no lost resume).
---
### User Story 3 - An escalation rule firing twice never creates two escalation events (Priority: P1)
An operator needs confidence that if the same escalation trigger is delivered more than once —
for example, a retried background job or a re-processed event — the ticket is escalated exactly
once, not reassigned and re-notified redundantly.
**Why this priority**: Duplicate escalations would double-notify agents, double-count in the
Support and Management dashboards, and could re-trigger reassignment away from an agent who has
already started work — a direct regression of work already done in this session.
**Independent Test**: Can be fully tested by firing the same escalation trigger concurrently
more than once for the same ticket against a real running instance of the API and a real
Postgres database, then confirming only one `EscalationEvent` row exists for that trigger
afterward.
**Acceptance Scenarios**:
1. **Given** a ticket eligible for escalation, **When** the same escalation trigger is delivered
twice at nearly the same moment, **Then** exactly one escalation event is recorded for it.
2. **Given** Scenario 1's duplicate delivery, **When** the escalation event is created,
**Then** the ticket is reassigned exactly once, not twice.
---
### User Story 4 - A ticket's status can never be corrupted by two simultaneous updates (Priority: P2)
An operator needs confidence that the ticket status-transition safeguard already built for this
system actually holds under real concurrent load, not just in isolated sequential tests — this
is existing protection, but has never been proven under genuine concurrency.
**Why this priority**: Lower priority than User Stories 1-3 because a real defensive mechanism
already exists here (see Assumptions); this story exists to convert an untested assumption into
a proven guarantee, and is valuable but lower-risk than the three unguarded races above.
**Independent Test**: Can be fully tested by firing multiple concurrent status-update attempts
at the same ticket, each based on the same starting version, against a real running API and
database, then confirming exactly one update succeeds and every other attempt receives a clear
conflict response rather than silently corrupting or skipping the ticket's state.
**Acceptance Scenarios**:
1. **Given** a ticket at a known status and version, **When** multiple concurrent status-update
requests are made from that same version, **Then** exactly one succeeds and the rest are
rejected with a conflict response, and the ticket's final status matches the one update that
succeeded.
---
### User Story 5 - The API's critical endpoints hold up under realistic concurrent traffic (Priority: P2)
An operator needs a documented, repeatable measurement of how the system's most important
endpoints — new support requests coming in, the AI support flow, and the admin reporting
dashboards — behave under sustained concurrent load, so that a future capacity or performance
regression can be caught by comparing against this baseline rather than guessed at.
**Why this priority**: This is about establishing a measurable baseline and repeatable tooling
rather than proving or fixing a specific correctness bug (unlike User Stories 1-4), so it is
valuable but not blocking for the correctness guarantees above.
**Independent Test**: Can be fully tested by running a load-test tool against a real running
instance of the API for each of the three named endpoint groups and producing a report of
throughput, latency percentiles, and error rate, independent of whether any other user story in
this feature has been completed.
**Acceptance Scenarios**:
1. **Given** the API is running against real infrastructure, **When** a defined concurrent load
is sent to the ticket-creation endpoint for a sustained period, **Then** a report is produced
showing throughput, latency percentiles, and error rate for that run.
2. **Given** the same setup, **When** the same load profile is sent to the AI support flow and
to the admin reporting endpoints, **Then** an equivalent report is produced for each,
allowing the three to be compared against each other and against future runs.
### Edge Cases
- What happens when a concurrent assignment race includes a ticket that is simultaneously being
closed or reopened? The assignment/reassignment safeguard must not be bypassable by a
status change racing the same window.
- What happens when a pause and a breach both become due at the exact same instant? The final
state must reflect one coherent, auditable outcome, not an unresolvable both-happened state.
- What happens when the load test itself pushes an endpoint into its own rate limiter (e.g. the
013-auth-hardening login rate limit, or the SaaS integration per-minute rate limits)? The
report must distinguish "rejected by design (rate limit)" from "failed under load" rather than
counting both as the same kind of failure.
- What happens when two of these races are exercised back-to-back against the same throwaway
database without cleanup? Each test must use its own uniquely-identified fixtures so repeated
runs (and CI re-runs) don't produce false positives or false negatives from leftover state.
## Requirements *(mandatory)*
### Functional Requirements
- **FR-001**: The system MUST guarantee that a ticket never has more than one assignment marked
as current, even when multiple assignment operations are attempted concurrently against it.
- **FR-002**: The system MUST guarantee that an SLA run's paused/active state and its associated
timestamps remain internally consistent when pause, resume, and the breach-detection sweep are
triggered concurrently against the same run.
- **FR-003**: The system MUST guarantee that the same escalation trigger delivered more than
once for the same ticket produces exactly one escalation event and exactly one resulting
reassignment.
- **FR-004**: The system MUST reject a ticket status update whose expected starting version no
longer matches the ticket's actual current version, even when the conflicting updates are
concurrent, and MUST leave the ticket in the state produced by whichever single update
actually succeeded.
- **FR-005**: The system's automated test suite MUST include a dedicated concurrency test for
each of FR-001 through FR-004, each exercising genuinely concurrent requests against real,
live infrastructure (not mocked timers or sequential calls standing in for concurrency).
- **FR-006**: Where a concurrency test written for this feature reveals that a guarantee in
FR-001, FR-002, or FR-003 does not currently hold, the underlying race MUST be fixed as part
of this feature, not merely documented.
- **FR-007**: The system MUST provide repeatable load-test tooling covering, at minimum: new
support request submission, the AI support flow, and the admin reporting dashboard endpoints.
- **FR-008**: Each load test run MUST produce a report including throughput, latency
percentiles, and error rate, with rate-limited responses reported separately from failures.
- **FR-009**: Pass/fail thresholds for the load tests (target throughput, acceptable latency,
acceptable error rate) MUST be explicitly marked as `OPEN BUSINESS DECISION` wherever the
business has not already specified a number, per this project's own roadmap convention —
never hardcoded as if final.
### Key Entities
- **Assignment race scenario**: A reusable test setup representing "many concurrent attempts to
assign or reassign the same ticket," used to exercise FR-001.
- **SLA race scenario**: A reusable test setup representing "concurrent pause, resume, and sweep
activity against the same SLA run," used to exercise FR-002.
- **Escalation race scenario**: A reusable test setup representing "the same escalation trigger
delivered more than once for the same ticket," used to exercise FR-003.
- **Load test report**: The recorded output of a load-test run against one endpoint group —
throughput, latency percentiles, error rate, and rate-limited-response count — kept so a
future run can be compared against it.
## Success Criteria *(mandatory)*
### Measurable Outcomes
- **SC-001**: A test run that fires at least 20 genuinely concurrent assignment attempts at the
same ticket always results in exactly one current assignment, with zero exceptions across at
least 10 repeated runs.
- **SC-002**: A test run that fires concurrent pause/resume/sweep activity against the same SLA
run always leaves that run in one internally-consistent, auditable state, with zero
contradictory-state outcomes across at least 10 repeated runs.
- **SC-003**: A test run that delivers the same escalation trigger twice for the same ticket
always results in exactly one escalation event and exactly one reassignment, with zero
duplicate outcomes across at least 10 repeated runs.
- **SC-004**: A test run that fires at least 20 genuinely concurrent status-update attempts from
the same starting version against the same ticket always results in exactly one success and
the ticket left in that one succeeding state.
- **SC-005**: A load-test report exists for each of the three named endpoint groups (ticket
creation, AI support flow, admin reporting), each independently re-runnable on demand and
producing consistent-shape output for comparison across runs.
## Assumptions
- Ticket status optimistic concurrency (User Story 4) already has a real defensive mechanism in
the codebase (a version-checked atomic update) — this feature's job for that story is to prove
it under genuine concurrency with a new test, not to build new protection, unless that test
surprises this assumption and reveals a real gap.
- Assignment double-assignment, SLA pause/resume races, and escalation duplicate-event risk (User
Stories 1-3) are NOT currently guarded against — this feature's job for those stories is both
to prove the gap with a real concurrency test and to implement the fix, per FR-006.
- "Genuinely concurrent" means real parallel requests issued against a real running instance of
the API backed by real Postgres/Redis (this project's standing verification discipline
throughout every prior feature), not fake-timer or mocked-clock simulations.
- Load testing (User Story 5) targets the existing dev/throwaway infrastructure already used for
this project's own manual verification, not a separate staging or production environment —
provisioning a dedicated load-test environment is out of scope.
- Specific throughput/latency/error-rate thresholds for "pass" are an `OPEN BUSINESS DECISION`
per FR-009; this feature delivers the tooling and a baseline report, not a final SLA number.
- Round-robin assignment-selection counter safety is already covered by an existing genuine
concurrency test and is explicitly out of scope for this feature.