support_backend/specs/016-load-concurrency-testing/research.md

Research: Load and Concurrency Testing

1. Assignment double-assignment race

Decision: Add a PostgreSQL partial unique index — CREATE UNIQUE INDEX assignments_one_current_per_ticket ON assignments (ticket_id) WHERE is_current = true; — and change AssignmentRepository.createAssignment to catch the resulting unique-violation (Prisma P2002) and retry the whole supersede-then-create transaction (bounded to 3 attempts, matching this codebase's existing small-bounded-retry convention), rather than surfacing a raw 500.

Rationale: createAssignment's existing transaction (updateMany({isCurrent:false}) + create({isCurrent:true})) is correct in isolation but Postgres's default READ COMMITTED isolation lets two concurrent transactions each see "no current row to supersede" and both successfully create their own isCurrent:true row — there is no read-modify-write cycle a version field could guard here (unlike Ticket/SLARun below), because the operation is a create, not an update, and a create can't be conditioned on "no matching row exists" atomically without a DB-level constraint. A partial unique index is the standard, minimal Postgres pattern for "at most one row matching a predicate" and requires no application-level locking. Retrying on conflict (rather than failing the second caller outright) preserves current behavior for the common, non-racing case and correctly resolves the race by making the loser's request apply after the winner's, superseding it — exactly the same "last write wins, but exactly once" semantics createAssignment's own docstring already promises for the non-concurrent case.

Alternatives considered:

• Explicit SERIALIZABLE transaction isolation: would also detect the race (as a serialization failure) but requires the exact same catch-and-retry handling as the unique index approach, adds latency to every assignment (not just racing ones), and does nothing to prevent the row from ever being duplicated if a future code path creates an Assignment outside this transaction — a DB constraint is a stronger, more future-proof guarantee.
• Row-level lock (SELECT ... FOR UPDATE) on a per-ticket lock row: works, but requires inventing a new lock-row concept for a case Postgres's own partial unique index already solves natively.

2. SLA pause/resume/sweep race

Decision: Add version Int @default(0) to SLARun. Replace SlaRunRepository.update(id, data) with updateWithVersion(id, expectedVersion, data), mirroring TicketsRepository.updateStatus's existing atomic updateMany({where:{id, version: expectedVersion}, data:{...data, version:{increment:1}}}) pattern exactly. SlaService.pause, resume, complete, and runBreachDetectionSweep each move to a small read-compute-write-retry loop (bounded to 3 attempts): re-read the run fresh on a version conflict, recompute the operation's own delta (e.g. resume's pausedMs shift) against the fresh state, and retry the versioned write.

Rationale: Every one of pause/resume/complete/the sweep does an unconditional read-then-update(run.id, {...}) with no guard — two of these racing (e.g. resume and the sweep evaluating the same run at once) can silently clobber each other: the sweep's own update(run.id, {status:'breached', breachedAt: now}) could be overwritten moments later by a resume that read the run before the sweep's write and still thinks it's paused, un-breaching a run that was legitimately breached and permanently losing that breach from SLA-compliance figures — a real, silent correctness bug, not a hypothetical one. Ticket already has exactly this problem solved for its own status field with a version counter and an atomic conditional-update; reusing that identical mechanism (rather than inventing a new one) keeps the codebase's concurrency idiom singular and matches Principle III's spirit even though it isn't a cross-module boundary concern.

Alternatives considered:

• Wrap each operation in a Postgres advisory lock keyed by run ID: works but adds a new locking primitive to the codebase for a problem the existing version-counter idiom already solves; rejected for consistency, not because it wouldn't work.
• A single DB transaction spanning the sweep's read+write for all runs at once: would only protect the sweep against itself, not against pause/resume racing it from an unrelated request path — doesn't close the actual gap.

3. Escalation idempotency

Decision: Add a PostgreSQL partial unique index — CREATE UNIQUE INDEX escalation_events_ticket_rule_unique ON escalation_events (ticket_id, rule_id) WHERE rule_id IS NOT NULL; — and change EscalationEventRepository.create (called from EscalationService.fire) to catch the resulting P2002 and return the already-existing event for that (ticketId, ruleId) pair instead of creating a duplicate or throwing.

Rationale: SLARun.ticketId is @unique and "no reopen-cycle support" (existing schema comment) means a given rule can only ever legitimately fire once per ticket's lifetime for a rule-triggered breach (handleBreach's ruleId is always a real rule ID scoped to one specific triggerType; resolution_breach and first_response_breach runs are naturally different rules, so this constraint doesn't conflate the two). Manual escalation (escalateManually/fire(ticketId, ruleId: null, ...)) is deliberately excluded from the constraint (WHERE rule_id IS NOT NULL) because an admin legitimately re-escalating the same ticket manually more than once must keep working exactly as it does today. This directly closes the gap the audit found: runBreachDetectionSweep's findRunningPastResolutionDueAt can return the same still-running row to two overlapping sweep passes (e.g. a slow sweep still finishing when the next scheduled tick fires, or a duplicate BullMQ job delivery calling handleBreach directly) before either pass's own update(run.id, {status:'breached', ...}) commits — without this constraint, both passes independently call fire and each successfully creates its own EscalationEvent plus its own assignToSpecificNode.

Alternatives considered:

• A dedicated idempotency-key column populated by the caller (e.g. a sweep-run ID): more general, but overkill here — the natural, already-unique business key for a rule-triggered escalation genuinely is (ticketId, ruleId) given the "no reopen-cycle" constraint already in place; inventing a separate key would duplicate information the schema already expresses.
• Making the sweep single-flight via a Redis lock around the whole sweep function: would prevent two sweep passes from overlapping, but does not protect against a duplicate BullMQ job calling handleBreach directly for the same trigger outside the sweep's own loop — the DB constraint protects the actual invariant regardless of caller, which is the correct place per Principle VII ("job handlers MUST be idempotent").

4. Ticket optimistic-concurrency proof

Decision: No implementation change. Add tests/concurrency/ticket-status-race.test.ts firing a batch of genuinely concurrent TicketsRepository.updateStatus calls at the same ticket, all from the same starting version, against the real throwaway Postgres, and asserting exactly one succeeds (returns the updated ticket) while every other call returns null (the existing stale-version-mismatch signal) — proving FR-004/SC-004 against the mechanism that already exists (see tests/concurrency/round-robin.test.ts:12's own reference to "003-ticketing's optimistic ticket-status concurrency" as prior art that was never itself concurrency-tested).

Rationale: The existing updateMany({where:{id, version: expectedVersion}, ...}) is a single atomic SQL statement — Postgres itself guarantees only one concurrent UPDATE matching that WHERE clause can succeed before the row's version changes underneath the others. This is sound by construction; the gap is purely "never proven under real concurrency," which this research assumes will simply confirm the existing guarantee (per spec.md's own Assumptions) — but the test is still written to fail loudly if that assumption turns out to be wrong.

5. Load-test tooling choice

Decision: autocannon (npm devDependency), invoked via small TypeScript runner scripts under tests/load/, one per named endpoint group (ticket creation, AI support flow, admin reporting), each producing a JSON report (autocannon's own Result shape: requests/sec, latency p50/p90/p99, non-2xx count) written to tests/load/reports/ (gitignored — these are run artifacts, not fixtures) plus a printed console summary.

Rationale: autocannon is a pure Node.js package (no separate binary to install, unlike k6), is TypeScript-friendly, and its programmatic API (autocannon({url, connections, duration, requests: [...]}, callback)) fits scripting multi-step flows (e.g. sign-in once, then hammer an authenticated endpoint) far more naturally than k6's separate-runtime JS dialect — keeping this feature's new tooling inside the same Node/TS toolchain as the rest of the project (Technical Context), consistent with this project's existing minimal-new-tooling bias.

Alternatives considered:

• k6: the industry-standard load-testing tool with richer scripting and threshold assertions, but ships as a separate Go binary requiring its own install/Docker image outside npm — heavier footprint for a project whose stack is otherwise 100% npm-managed. Reconsider if this project later needs distributed/cloud load generation, which autocannon does not support and k6 does.
• artillery: also npm-native and closer to k6 in scripting richness, but pulls in a much larger dependency tree for YAML-driven scenario files this feature doesn't need — autocannon is a lighter fit for three hand-written TS scripts.

6. Load-test pass/fail thresholds

Decision: Per FR-009, no numeric throughput/latency/error-rate threshold is hardcoded as pass/fail. Each load-test report prints its own measured numbers and the tooling exits 0 regardless of the numbers observed (this is a measurement tool, not a gate) — a comment in each script marks the threshold question as OPEN BUSINESS DECISION and links back to spec.md Assumptions, so a future feature can wire an explicit pass/fail gate into CI once the business sets a real target.

Rationale: Inventing an arbitrary "must handle 500 req/s at p99 < 200ms" number would violate the roadmap's own explicit rule ("Never hardcode a placeholder value for any of the [open business decisions] and ship it as if it were final") — throughput/latency targets are exactly this kind of business-owned number, not an engineering default.