support_backend/specs/014-full-observability/checklists/requirements.md

Specification Quality Checklist: Full Observability

Purpose: Validate specification completeness and quality before proceeding to planning Created: 2026-09-07 Feature: spec.md

Content Quality

• No implementation details (languages, frameworks, APIs)
• Focused on user value and business needs
• Written for non-technical stakeholders
• All mandatory sections completed

Requirement Completeness

• No [NEEDS CLARIFICATION] markers remain
• Requirements are testable and unambiguous
• Success criteria are measurable
• Success criteria are technology-agnostic (no implementation details)
• All acceptance scenarios are defined
• Edge cases are identified
• Scope is clearly bounded
• Dependencies and assumptions identified

Feature Readiness

• All functional requirements have clear acceptance criteria
• User scenarios cover primary flows
• Feature meets measurable outcomes defined in Success Criteria
• No implementation details leak into specification

Notes

• This is docs/10-implementation-roadmap.md's own Phase 11, second sub-area, per explicit user direction (the first was 013-auth-hardening's security pass). The user explicitly chose "Full observability" over "Reporting/analytics dashboards" as a distinct, separately-scoped sub-area — FR-009 and several Assumptions exist specifically to keep this feature from drifting into that adjacent, not-yet-started work.
• The three named infrastructure gaps (no per-request access log, a dead request-duration histogram, a never-initialized tracer) and all eleven "key metrics to track" being completely untracked today were confirmed by direct code inspection before writing this spec, not assumed.
• All items pass; no revision iterations were needed. No [NEEDS CLARIFICATION] markers were required — every open question had a reasonable, documented default (see Assumptions).

Implementation Notes (post-build)

• Registering a real TracerProvider alone was not sufficient to make span nesting work across this feature's own async event-bus subscribers: without also registering an AsyncLocalStorageContextManager (@opentelemetry/context-async-hooks, a third new dependency beyond the two research.md originally named), the OpenTelemetry API's context.active() is a no-op that does not propagate across await boundaries at all — orchestration.assignment came out as its own unrelated root span/trace instead of nesting under ai.escalation. Caught by the tracing integration test's own parent/child assertions actually failing on the first implementation, not assumed correct from reading the SDK's docs.
• Installing @opentelemetry/exporter-trace-otlp-http alongside the already-pinned @opentelemetry/sdk-trace-base@^1.22.0 pulled two incompatible OpenTelemetry core/resources major versions (1.x and 2.x) side by side. Resolved by bumping sdk-trace-base to ^2.11.0 to match — this also happened to close a moderate DoS advisory in @opentelemetry/core <2.8.0 that the 1.x line was pinned to.
• T016 (graceful degradation under an unreachable OTLP endpoint) ended up as its own unit test (tests/unit/observability/tracing-graceful-degradation.test.ts) rather than living in tracing.test.ts as tasks.md originally described. Reason: tracing.ts always uses the in-memory test exporter when NODE_ENV=test, so the integration suite's own running app can't be pointed at a bad OTLP endpoint to exercise this. The unit test instead constructs a real BasicTracerProvider/BatchSpanProcessor/OTLPTraceExporter pointed at a genuinely unreachable address directly, and — importantly — verifies the SDK's background export path (what production actually exercises) never produces an unhandled rejection, rather than calling forceFlush() directly, which is documented OpenTelemetry behavior that does reject on a failed export by design (the first version of this test asserted the wrong thing and failed against real, correct SDK behavior — not a bug in this feature's own code).
• sla.service.ts's pre-existing status-overwrite gap (a 'breached' run's status silently becomes 'completed' if the ticket later resolves — see research.md §5) was worked around for the metric's own correctness (read run.status before the overwrite) but left unfixed in the underlying data, consistent with how 013-auth-hardening documented a pre-existing bug it found without fixing it.
• Found and fixed one genuine cross-file test-isolation bug this feature's own new test caused: business-metrics.test.ts's "human resolution" case originally drove a ticket through a real HUMAN_ESCALATION transition via ticketsService.updateStatus, which — same as any other escalation in this codebase — triggers the real orchestration subscriber's default ROUND_ROBIN auto-assignment against every agent in the shared throwaway database, including other concurrently-running test files' own dedicated agents (reproduced deterministically against agent-ticket-queue.test.ts). Fixed by driving the intermediate state-machine transitions directly through ticketsRepository.updateStatus (no domain-event publish) instead, reserving the real, event-publishing ticketsService.updateStatus call for only the final RESOLVED transition the metric subscriber actually needs to observe.
• Separately, found (not caused by this feature — confirmed via git checkout to the clean pre-014 commit and reproducing the identical failure) a pre-existing systemic collision risk in ticket-code generation: ticket-code.ts's deriveProductCode keeps only the first 4 alphabetic characters of externalProductId, so essentially every integration test file in this codebase (nearly all of which name their test products TEST_<SOMETHING>) collapses to the identical "TEST" code prefix. Running enough TEST_*-prefixed files concurrently (as vitest does by default across worker threads/processes) makes independent files race for the same TEST-<year>-<sequence> numbering space, occasionally exceeding tickets.service.ts's fixed MAX_CODE_RETRIES = 5 and surfacing as a real 500 (Unique constraint failed on the fields: (code)) instead of the retry silently absorbing it. Confirmed independent of this feature (reproduces on 79bc2ef, 013-auth-hardening's tip, with none of this feature's code present) and left unfixed here — a ticket-code-generation concurrency fix belongs to 003-ticketing's own module, out of scope for an observability feature. Worth a dedicated future fix (e.g. a longer/hash-based product code, or a database-level sequence rather than a COUNT-then-retry scheme).