support_backend/specs/014-full-observability/plan.md

Implementation Plan: Full Observability

Branch: 014-full-observability | Date: 2026-09-07 | Spec: spec.md

Input: Feature specification from specs/014-full-observability/spec.md

Summary

Wires three already-scaffolded-but-inert observability primitives into something real: a per-request structured access log (none exists today — Fastify's own request logging is fully disabled), the existing-but-never-observed request-duration histogram, and a real OpenTelemetry tracer provider behind the existing-but-never-called getTracer() helper. Adds eleven live Prometheus counters/histograms for the business-health metrics docs/09-testing-observability- cicd.md names, each wired at one existing choke point per metric (an event-bus subscriber where one already exists for the transition, a single already-existing method otherwise) rather than scattered across every call site. No new endpoints, no schema changes, no supporthub-web work — see research.md for the exact hook point chosen for each of the fourteen instrumentation targets (3 infra + 11 named metrics) and why.

Technical Context

Language/Version: TypeScript 5.4 / Node.js 20+ (unchanged).

Primary Dependencies: New — @opentelemetry/exporter-trace-otlp-http (OTLP/HTTP span export), @opentelemetry/resources (service-name resource attribute). Reused, already installed — @opentelemetry/api, @opentelemetry/sdk-trace-base (provider, processors, and both the console and in-memory exporters used here all come from this one package), prom-client, pino. Reused Node built-in — async_hooks' AsyncLocalStorage.

Storage: No schema change. All new state is either in-process (Prometheus metric registry, the ALS request-context store, the tracer provider) or exported to wherever tracing is configured to send it — no new Postgres/Redis reads or writes beyond a handful of existing-table lookups already needed to label a metric correctly (e.g. resolutionRepository.findByTicketId to distinguish AI vs. human resolution).

Testing: Vitest — unit tests for the ALS-based logger mixin (a log call inside a request context carries requestId/correlationId; one outside carries neither) and for the SLA-compliance metric's "don't double-count an already-breached run as met" guard. Integration tests against real Postgres/Redis for: the access-log line's presence/shape (captured via a logger.info spy, same technique as 013's password-reset test), /metrics scraped before/after real traffic showing the duration histogram and each of the eleven business counters/histograms change by the expected amount when their real underlying event is driven through the real API, and a real multi-span trace (read back from the test-environment InMemorySpanExporter) for the two named cross-module paths.

Target Platform: Same Fastify modular monolith. Modifies infrastructure/observability/* (logger, metrics, tracing, a new request-context store) and plugins/request-context.plugin.ts (the new onResponse hook); adds small, single-call-site instrumentation lines inside ai-support/sessions, ai-support/knowledge, ai-support/tools, ticketing/tickets, ticketing/messages, orchestration/sla, and a handful of new subscribers in src/events/handlers/index.ts. No module gains a new public export surface beyond what getTracer() already exposed.

Project Type: Backend service — single project.

Performance Goals: The onResponse hook adds one Pino log call and one histogram .observe per request — both already-paid-for infrastructure (the logger and the metric object already exist), no new I/O on the request hot path. Trace export runs via BatchSpanProcessor (out of the request's own async chain) so span export latency never adds to response time. Metric increments at the eleven business hook points are in-memory counter operations, not database writes — the handful of read lookups needed for correct labeling (e.g. the resolution lookup for #3/#4) are single-row, already-indexed reads on tables these modules already query routinely.

Constraints: FR-007 — tracing must degrade gracefully; the API must start and serve traffic normally with no collector configured or reachable. FR-009 — no new human-facing endpoint, dashboard, or aggregation logic; every FR-008 metric is a raw counter/histogram for an external scraper, full stop. FR-010 — /health* and the existing histogram's shape on /metrics must not change for any existing consumer (only new metrics are added, nothing existing is renamed or removed).

Scale/Scope: Zero new routes. Three modified observability infrastructure files plus one new request-context store. Eleven new metric definitions plus their one-choke-point instrumentation call each. Two new dependencies. No schema migration, no new module.

Constitution Check

GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.

Principle / Section	Check	Result
I. SaaS Is the Sole Identity & Access Authority	Not applicable — no identity/access surface touched.	PASS — N/A
II. Configuration Over Hardcoding	The tracing exporter destination (OTEL_EXPORTER_OTLP_ENDPOINT) is env-driven, not hardcoded per environment; no business policy value is introduced by this feature (no SLA/routing/threshold numbers).	PASS
III. Layered Architecture With Enforced Module Boundaries	No new module; existing module boundaries unchanged (each metric's instrumentation call lives inside the module that already owns the event, per research.md's per-metric table). The two repository-layer instrumentation calls (#1/#2, AI session status) are a deliberate, disclosed exception — see research.md §5's justification: observability calls are already a cross-cutting concern used from any layer in this codebase (e.g. logger.error inside tool-executor.ts), not the kind of business-logic leakage this principle targets.	PASS
IV. AI Recommends, Deterministic Policy Decides	Not applicable — no AI decision logic changed, only observation of its outcomes.	PASS — N/A
V. Evidence-Based Verification	Not applicable.	PASS — N/A
VI. Durable Audit & History	Directly implements this principle's own stated requirement — "every log line MUST carry a request ID/correlation ID" is written in the constitution today but not actually true until this feature (FR-001/FR-002).	PASS — this feature closes a pre-existing constitutional gap
VII. Concurrency-Safe, Durable Job Handling	The first-response-time metric (#5) has a benign, disclosed race (two concurrent first AGENT_MESSAGEs could both read "zero prior messages" and both observe) — acceptable because it is a best-effort observability metric, not the assignment/SLA correctness this principle is protecting; no persisted state or business decision depends on it.	PASS
VIII. Problem and Ticket Are Separate, Related Entities	Not applicable — no model change.	PASS — N/A
Technology & Platform Constraints	Two new dependencies (both OpenTelemetry, both already in the stack's declared technology list — "OpenAPI" aside, tracing itself was always part of the stated stack via the pre-existing @opentelemetry/api/sdk-trace-base dependencies) — no new infrastructure category introduced.	PASS

No violations requiring Complexity Tracking justification.

Project Structure

Documentation (this feature)

specs/014-full-observability/
├── plan.md
├── research.md
├── data-model.md
├── quickstart.md
├── contracts/
│   └── metrics-contract.md
└── tasks.md

Source Code (repository root)

supporthub-api/
├── src/
│   ├── infrastructure/
│   │   └── observability/
│   │       ├── logger.ts                    # MODIFIED — mixin reads the new ALS store
│   │       ├── metrics.ts                   # MODIFIED — 11 new Counter/Histogram definitions
│   │       ├── tracing.ts                   # MODIFIED — real provider init, exporter selection
│   │       └── request-context.store.ts     # NEW — AsyncLocalStorage<RequestContext>
│   ├── plugins/
│   │   └── request-context.plugin.ts        # MODIFIED — onResponse access-log + histogram hook,
│   │                                            onRequest now runs the rest of the request
│   │                                            inside the ALS store
│   ├── events/
│   │   └── handlers/index.ts                # MODIFIED — 3 new subscribers (human-resolution +
│   │                                            resolution-time on TICKET_UPDATED/RESOLVED,
│   │                                            escalation-rate on ESCALATION_TRIGGERED)
│   └── modules/
│       ├── ai-support/
│       │   ├── sessions/repository/session.repository.ts   # MODIFIED — AI resolution/escalation
│       │   ├── knowledge/service/error-codes.service.ts     # MODIFIED — most-common-errors
│       │   └── tools/service/tools.service.ts                # MODIFIED — tool-failure + knowledge-
│       │                                                        effectiveness
│       ├── ticketing/
│       │   ├── tickets/service/tickets.service.ts            # MODIFIED — recurring-problems, plus
│       │   │                                                    the two named trace spans
│       │   └── messages/service/messages.service.ts           # MODIFIED — first-response-time
│       └── orchestration/
│           └── sla/service/sla.service.ts                     # MODIFIED — SLA-compliance
└── tests/
    ├── unit/observability/                  # ALS mixin, SLA-compliance double-count guard
    └── integration/observability/           # access log, /metrics scrape assertions (11 metrics
                                                 + duration histogram), cross-module trace

Structure Decision: Single project, no new module. All changes are surgical additions inside infrastructure/observability (the module that already owns this concern) plus one small, justified instrumentation line inside each of six existing business modules, following the per-metric hook points research.md already identified against the real, current code.

Complexity Tracking

No constitution violations — table intentionally omitted.