support_backend/specs/014-full-observability/spec.md

Feature Specification: Full Observability

Feature Branch: 014-full-observability

Created: 2026-09-07

Status: Draft

Input: User description: "Full observability: wire the already-scaffolded logging, metrics, and tracing infrastructure into an actually working end-to-end observability layer — structured per-request access logs, a working request-duration histogram, real OpenTelemetry tracing with exported spans across critical request paths, and live Prometheus counters for the key operational metrics named in docs/09-testing-observability-cicd.md."

User Scenarios & Testing (mandatory)

User Story 1 - Trace one request end to end from its logs (Priority: P1)

An engineer investigating a production incident (a customer's ticket got stuck, an API call failed) needs to reconstruct exactly what the system did for that one request: which route was hit, how long it took, what it returned, and — because a support case touches many internal calls (AI session → tool calls → escalation → assignment → SLA events) — which of those internal log lines belong to the same originating request.

Why this priority: Without a per-request access log, there is currently no record that a given request even happened unless it errored. This is the minimum viable observability floor everything else builds on.

Independent Test: Can be fully tested by sending a request to any route and confirming exactly one structured access-log line is emitted for it, carrying the same request ID as any other log line produced while handling that request.

Acceptance Scenarios:

1. Given the API is running, When any HTTP request completes (success or failure), Then exactly one structured log line is emitted recording its method, route, status code, and duration.
2. Given a request carries an inbound correlation ID header (or one is generated for it), When that request triggers further log lines anywhere in the codebase during its handling, Then every one of those log lines carries the same request ID and correlation ID as the access-log line for that request.
3. Given a request fails with an unhandled error, When the access log line is emitted, Then it is distinguishable (by log level) from a successful request without needing to duplicate the existing error-handler logging.

---

User Story 2 - See live request-health metrics (Priority: P1)

An engineer wants to know, right now, whether the API is healthy under current traffic — request volume, latency distribution, and error rate by route — without needing to grep logs.

Why this priority: A request-duration metric already exists in code but is never recorded, so /metrics currently reports nothing useful about request health. This is the second half of the observability floor (logs tell you what happened to one request; metrics tell you the shape of all of them).

Independent Test: Can be fully tested by sending a mix of successful and failing requests, then scraping /metrics and confirming the request-duration histogram and a request-count-by-status metric both reflect that traffic.

Acceptance Scenarios:

1. Given the API has served requests since it started, When /metrics is scraped, Then the request-duration histogram has observations labeled by method, route, and status code matching that traffic.
2. Given some requests succeeded and others returned 4xx/5xx, When /metrics is scraped, Then a request-count metric lets an operator compute error rate by route and status class.

---

User Story 3 - Trace a single incident's cross-module path (Priority: P2)

An engineer debugging why a specific ticket took an unexpectedly long or unexpected path (e.g., AI failed to resolve it, escalation didn't fire when expected) wants to see the causal chain of operations across modules for that one ticket — not just isolated log lines, but a connected trace showing how long each step took relative to the others.

Why this priority: Distributed tracing infrastructure already exists in the dependency list and a getTracer() helper is exported, but no tracer provider is ever initialized and no code ever calls it — today it silently does nothing. This is more valuable than plain logs for understanding why a multi-step flow behaved the way it did, but the system is usable without it (User Stories 1-2 already restore basic visibility), so it is P2.

Independent Test: Can be fully tested by triggering a request that flows through at least two instrumented modules (e.g., an AI escalation that results in orchestration/assignment) and confirming a trace is produced whose spans are parented correctly and whose combined duration accounts for the modules involved.

Acceptance Scenarios:

1. Given tracing is enabled, When the API starts, Then a real tracer provider is active (not the OpenTelemetry no-op default) and spans created via the existing getTracer() helper are actually exported somewhere inspectable.
2. Given a request flows through multiple instrumented operations (e.g., AI diagnosis triggers an escalation which triggers orchestration/assignment), When that request completes, Then the resulting trace shows each operation as a distinct, correctly-nested span under one root.
3. Given tracing is not configured with an external collector in a given environment, When the API starts, Then it still starts successfully (tracing degrades gracefully, it never blocks startup or request handling).

---

User Story 4 - See the business-health metrics this project committed to tracking (Priority: P2)

An engineer or team lead wants live visibility (via the same /metrics endpoint, for consumption by whatever monitoring stack is deployed) into the operational health metrics this project's own design doc names as important: AI resolution rate, AI escalation rate, human resolution rate, average resolution time, first response time, SLA compliance, escalation rate, recurring problems, most common errors, knowledge effectiveness, and tool failure rate.

Why this priority: These are real, currently-invisible gaps — none of them are tracked anywhere today, live or otherwise. They are P2 (not P1) because they instrument business outcomes that already have a durable system of record (the ticket/problem/SLA/escalation tables) — a missing counter is a visibility gap, not a data-loss risk, unlike User Stories 1-2's request-level blind spot.

Why this scope boundary: This story is about each metric existing and being live-updated correctly at the point the underlying event occurs, exposed as raw counters/histograms on /metrics for an external monitoring stack to graph and alert on. It explicitly does NOT include building any dashboard, chart, or human-facing report — that is a separate, not-yet-started project phase (reporting/analytics dashboards).

Independent Test: Can be fully tested, metric by metric, by driving the real underlying event (resolve a ticket via AI, resolve one via a human agent, breach an SLA, trigger an escalation, log a known error, etc.) against a running instance and confirming the corresponding value on /metrics changed by exactly the expected amount.

Acceptance Scenarios:

1. Given an AI session resolves a ticket without escalating, When /metrics is scraped, Then the AI-resolution counter has incremented and the AI-escalation counter has not.
2. Given an AI session escalates to a human and that human later resolves the ticket, When /metrics is scraped, Then the AI-escalation counter and the human-resolution counter have both incremented.
3. Given a ticket is resolved, When /metrics is scraped, Then the resolution-time histogram has a new observation reflecting that ticket's actual open-to-resolved duration.
4. Given an agent sends the first reply on a ticket, When /metrics is scraped, Then the first-response-time histogram has a new observation.
5. Given an SLA run resolves as either met or breached, When /metrics is scraped, Then the SLA-compliance counter reflects that outcome.
6. Given an escalation event fires, When /metrics is scraped, Then the escalation-rate counter increments, labeled by trigger reason.
7. Given an AI tool invocation succeeds or fails, When /metrics is scraped, Then the tool-failure-rate counter reflects the outcome, labeled by tool name.
8. Given a known error code is surfaced to a customer, When /metrics is scraped, Then a counter labeled by that error code has incremented (supports both "most common errors" and, via repeated occurrence on the same product/category, "recurring problems").
9. Given the AI's knowledge retrieval step either does or does not find a usable match for the customer's problem, When /metrics is scraped, Then a knowledge-effectiveness counter reflects that outcome.

---

Edge Cases

• What happens when the configured tracing exporter/collector is unreachable? The API must still start and continue serving requests; span export failures must be logged but never surface to the request/response cycle.
• What happens to in-flight metrics/traces if the process crashes before a scrape/export completes? Acceptable data loss for that window — this feature does not need to guarantee zero metric loss across a crash, only correctness of what is recorded and exported during normal operation.
• What happens when a request has no matching route (404) or is rejected before reaching a handler (e.g., by a global rate limiter)? It must still produce exactly one access-log line and one metrics observation, so operators can see rejected traffic, not just successfully-routed traffic.
• What happens when two requests share the same client-supplied correlation ID (e.g., a retried request)? Each still gets its own request ID and its own access-log line; only the correlation ID is shared, by design (that is what lets an operator group retries together).
• How does the system behave for a route that legitimately never touches any of the business-event counters (e.g., a health check)? No business-metric line is expected for it — only the generic request-count/duration metrics from User Story 2 apply.

Requirements (mandatory)

Functional Requirements

• FR-001: System MUST emit exactly one structured access-log line per completed HTTP request (including requests that error, 404, or are rejected by a global hook before reaching a route handler), containing at minimum: HTTP method, route/path, response status code, duration, request ID, and correlation ID.
• FR-002: System MUST attach the request ID and correlation ID already established by the existing request-context mechanism to every log line produced while handling that request, not only the access-log line.
• FR-003: System MUST record every completed HTTP request's duration into the existing request-duration metric, labeled at minimum by method, route, and status code.
• FR-004: System MUST expose a request-count metric (or equivalent derivable from FR-003's histogram) sufficient to compute error rate per route and status class.
• FR-005: System MUST initialize a real distributed-tracing pipeline at startup so that spans created via the existing getTracer() helper are captured and exported to an inspectable destination, rather than discarded by the OpenTelemetry no-op default.
• FR-006: System MUST create spans for the AI diagnosis → escalation → orchestration/assignment path and for the ticket-creation → orchestration path, correctly nested under one root span per originating request, so a single incident's cross-module timing is visible in one trace.
• FR-007: System MUST continue to start up and serve requests normally if the configured tracing export destination is unreachable; export failures MUST be logged, never raised to the request/response cycle.
• FR-008: System MUST expose live counters/histograms on the existing /metrics endpoint for each of: AI resolution rate, AI escalation rate, human resolution rate, average resolution time, first response time, SLA compliance, escalation rate, recurring problems, most common errors, knowledge effectiveness, and tool failure rate — each updated at the moment its underlying real event occurs (not computed by a batch job or exposed through any new endpoint).
• FR-009: System MUST NOT introduce any new human-facing dashboard, chart, or reporting API as part of this feature — every metric from FR-008 is a raw, unaggregated-by-this-system counter/histogram intended for an external monitoring stack to graph, in keeping with the explicit scope boundary against the separate reporting/analytics dashboards work.
• FR-010: Existing /health, /health/live, /health/ready, and /metrics endpoints MUST continue to function unchanged in shape for any existing consumer.

Key Entities

• Access log line: A structured log record emitted once per completed HTTP request; not a persisted database entity — it exists only in the log stream.
• Request-duration metric: A histogram, keyed by method/route/status, recording how long each request took.
• Trace / span: A record of one operation's start/end time and its parent-child relationship to other operations within the same originating request, exported to wherever tracing is configured to send it.
• Business-event counter: One of the eleven named live metrics in FR-008/User Story 4, each incremented (or observed, for the two duration-based ones) at the exact point its real-world event already occurs elsewhere in the system (ticket resolution, SLA run completion, escalation firing, tool invocation, etc.) — this feature adds the instrumentation call at each of those existing points, it does not change what those points do.

Success Criteria (mandatory)

Measurable Outcomes

• SC-001: Given any request made to the running API, an operator can identify, from logs alone, its method, route, outcome, duration, and every other log line produced while handling it, within seconds of it happening.
• SC-002: An operator watching /metrics can determine current request error rate and latency distribution per route without needing to read application logs.
• SC-003: An operator can find and inspect the complete cross-module trace for a specific incident that touched at least two instrumented modules, showing correctly-attributed timing per module.
• SC-004: All eleven named business-health metrics are visible on /metrics and each one's value changes correctly and immediately in response to its real underlying event, verified against real (non-mocked) system behavior.
• SC-005: Enabling this feature's tracing pipeline introduces no observable request-handling failure, and the API starts and serves traffic normally even when the tracing destination is unreachable.

Assumptions

• "Exported to an inspectable destination" (FR-005) means a destination this project's own test/dev environment can actually verify against — an OTLP-compatible collector endpoint in production-like environments, and an in-process/console exporter for local development and automated tests, both driven by configuration rather than hardcoded per environment. No specific commercial tracing backend (e.g., Jaeger, Honeycomb, Datadog) is mandated by this feature; wiring a specific backend in a given deployment is an operations concern outside this spec.
• The existing Prometheus (prom-client) and Pino stack are the metrics/logging technology already chosen for this project (confirmed by existing code) and are reused rather than replaced.
• "Knowledge effectiveness" is scoped to whether the AI's knowledge-retrieval step found and used a matching entry for a given diagnosis attempt (a binary outcome per attempt), not a more elaborate relevance-scoring scheme — no such scoring exists elsewhere in the system to build on.
• "Recurring problems" and "most common errors" (FR-008) are satisfied by labeled counters an operator's monitoring stack can rank/aggregate over any time window (e.g., topk in PromQL) — this feature does not need to compute or store a "top N" itself, consistent with FR-009's boundary against building reporting logic.
• This feature is backend-only (supporthub-api); no supporthub-web changes are in scope, since nothing here is presented to any human through a UI.
• Existing RequestContext (requestId/correlationId), already populated by both the customer and staff auth paths (010-identity-auth), is reused as the identifier scheme for FR-001/FR-002 rather than introducing a second identifier scheme.