support_backend/specs/014-full-observability/tasks.md

description

description
Task list for 014-full-observability

Tasks: Full Observability

Input: Design documents from specs/014-full-observability/

Prerequisites: plan.md, spec.md, research.md, data-model.md, contracts/metrics-contract.md, quickstart.md

Organization: Tasks are grouped by user story (US1 = P1 access log, US2 = P1 request-health metrics, US3 = P2 tracing, US4 = P2 business-health metrics). US2 shares its hook point with US1 (both live in the same onResponse hook) so US2 depends on US1's hook existing, not on its own separate one. US3 and US4 are each independent of US1/US2 and of each other.

Format: [ID] [P?] [Story] Description

All file paths are relative to supporthub-api/ (repo root).

---

Phase 1: Foundational (Blocking Prerequisites)

• T001 Add @opentelemetry/exporter-trace-otlp-http and @opentelemetry/resources to package.json (npm install)
• T002 Add src/infrastructure/observability/request-context.store.ts — a module-level AsyncLocalStorage<{requestId: string; correlationId: string}> with a run() passthrough and a getStore() re-export
• T003 [P] Wire logger.ts's Pino options with a mixin function reading from T002's store (returns {} when no store is active — a log call outside any request, e.g. at startup, must not throw) (depends on T002)

Checkpoint: Every subsequent log call through the shared logger singleton is request-correlated automatically, once a request actually runs inside the store (US1 wires that part next).

---

Phase 2: User Story 1 - Trace one request end to end from its logs (Priority: P1)

Goal: One structured access-log line per request; every other log line produced during that request's handling shares its request ID.

Independent Test: Quickstart Scenario 1.

Tests for User Story 1

• T004 [P] [US1] Unit test: a logger.info(...) call made inside T002's store.run(...) carries requestId/correlationId in its output; one made outside carries neither, in tests/unit/observability/request-context-mixin.test.ts (depends on T003)

Implementation for User Story 1

• T005 [US1] In plugins/request-context.plugin.ts's existing onRequest hook, after building request.reqContext, call the T002 store's run() wrapping the remainder of the request's handling (Fastify's onRequest hooks accept a done callback / return a promise — the run wraps whichever style this hook currently uses) so every subsequent hook/handler for this request executes inside the ALS context (depends on T002)
• T006 [US1] Add an onResponse hook (same plugin) that logs one line via the shared logger: {event: "http_request_completed", method, route: request.routeOptions.url, statusCode: reply.statusCode, durationMs: reply.elapsedTime}, at info/warn/error level by status class (depends on T005)
• T007 [US1] Integration test covering Quickstart Scenario 1 (one access-log line per request incl. 404; shared requestId across the access-log line and an internal log line from the same request) in tests/integration/observability/access-log.test.ts, using a logger.info/logger.warn spy the same way password-reset-flow.test.ts (013) already does (depends on T006)

Checkpoint: Quickstart Scenario 1 passes. Every request is now visible in logs even when it never errors.

---

Phase 3: User Story 2 - See live request-health metrics (Priority: P1)

Goal: The existing (previously dead) request-duration histogram actually has observations; error rate per route/status is computable from /metrics alone.

Independent Test: Quickstart Scenario 2.

Implementation for User Story 2

• T008 [US2] In the same onResponse hook added by T006, call httpRequestDurationHistogram.observe({method, route: request.routeOptions.url, status_code: String(reply.statusCode)}, reply.elapsedTime / 1000) (depends on T006)
• T009 [US2] Integration test covering Quickstart Scenario 2 (send a mix of successful/ failing requests to the same route, scrape /metrics, assert both status-code label values are present with the expected counts) in tests/integration/observability/request-metrics.test.ts (depends on T008)

Checkpoint: Quickstart Scenario 2 passes. /metrics now reflects real request traffic.

---

Phase 4: User Story 3 - Trace a single incident's cross-module path (Priority: P2)

Goal: A real TracerProvider is active; getTracer() produces spans that are actually exported; two named cross-module paths are instrumented.

Independent Test: Quickstart Scenario 3.

Implementation for User Story 3

• T010 [P] [US3] Rewrite infrastructure/observability/tracing.ts to initialize a BasicTracerProvider at module load with a Resource (service.name: "supporthub-api") and register it via trace.setGlobalTracerProvider(...); exporter/processor chosen by NODE_ENV/OTEL_EXPORTER_OTLP_ENDPOINT per research.md §4 (InMemorySpanExporter + SimpleSpanProcessor in test, OTLPTraceExporter + BatchSpanProcessor when the env var is set, ConsoleSpanExporter + SimpleSpanProcessor otherwise); export a getTestSpanExporter() accessor (test env only) for T015 to read exported spans back; getTracer()'s own exported signature is unchanged (depends on T001)
• T011 [US3] Wire the OpenTelemetry SDK's internal diagnostic logger (diag.setLogger(...)) to the shared logger.warn, so span-export failures land in this project's own log stream instead of stderr or nowhere (depends on T010)
• T012 [P] [US3] Add a ticket.create span (getTracer().startActiveSpan(...)) around tickets/service/tickets.service.ts's ticket-creation method, with ticket.id and product.externalProductId attributes, ended in a finally (depends on T010)
• T013 [P] [US3] Add an ai.escalation span around ai-support/sessions/service/ session.service.ts's escalation branch(es), with ticket.id/session.id attributes (depends on T010)
• T014 [US3] Add an orchestration.assignment span wrapping the existing orchestrationService.handleHumanEscalation call in the TICKET_UPDATED/ HUMAN_ESCALATION subscriber (src/events/handlers/index.ts), with ticket.id/ strategy attributes, so it nests under T013's span when both occur in the same request (depends on T010, T013)
• T015 [US3] Integration test covering Quickstart Scenario 3 steps 1-2: drive a real ticket-creation → escalation → orchestration/assignment flow, read spans back via T010's getTestSpanExporter(), assert ticket.create/ai.escalation/orchestration.assignment all share one trace ID with correct parent/child spanId relationships, in tests/integration/observability/tracing.test.ts (depends on T012, T013, T014)
• T016 [US3] Integration test covering Quickstart Scenario 3 steps 3-4: point OTEL_EXPORTER_OTLP_ENDPOINT at an unreachable address, confirm buildApp() still resolves and a request still completes successfully, in the same test file (depends on T010)

Checkpoint: Quickstart Scenario 3 passes. A real, inspectable trace exists for the first time; tracing failure never blocks the app.

---

Phase 5: User Story 4 - See the business-health metrics this project committed to tracking (Priority: P2)

Goal: All eleven named metrics (data-model.md) are live on /metrics, each updated at the exact real event research.md identified.

Independent Test: Quickstart Scenario 4.

Implementation for User Story 4

• T017 [US4] Define all eleven new Counter/Histogram objects in infrastructure/observability/metrics.ts per data-model.md's table, exported individually (depends on T001)
• T018 [P] [US4] Increment supporthub_ai_session_outcomes_total in ai-support/sessions/ repository/session.repository.ts's updateStatus, labeled outcome when status is 'resolved'/'escalated' (depends on T017)
• T019 [P] [US4] Add a TICKET_UPDATED/newStatus === 'RESOLVED' subscriber in src/events/handlers/index.ts that looks up resolutionRepository.findByTicketId, increments supporthub_ticket_resolutions_total{resolved_by} (ai vs. any other value), fetches the ticket for createdAt, and observes supporthub_ticket_resolution_duration_seconds (depends on T017)
• T020 [P] [US4] In ticketing/messages/service/messages.service.ts's post, when type === 'AGENT_MESSAGE', check for a prior AGENT_MESSAGE on the ticket and — only for the first one — observe supporthub_ticket_first_response_duration_seconds against the ticket's createdAt (depends on T017)
• T021 [P] [US4] In orchestration/sla/service/sla.service.ts: in complete(), read run.status before updating and increment supporthub_sla_run_outcomes_total{outcome: "met"} only if it was not already 'breached'; in runBreachDetectionSweep(), increment {outcome: "breached"} for each newly-flagged run (depends on T017)
• T022 [P] [US4] Add an ESCALATION_TRIGGERED subscriber in src/events/handlers/index.ts that increments supporthub_escalations_total{reason} from the event payload's reason (depends on T017)
• T023 [P] [US4] In ticketing/tickets/service/tickets.service.ts's ticket-creation method, increment supporthub_problems_created_total{category_id} right after problemsRepo.create succeeds (categoryId ?? 'uncategorized') (depends on T017)
• T024 [P] [US4] In ai-support/knowledge/service/error-codes.service.ts's findKnownIssuesByErrorCode, increment supporthub_known_error_lookups_total{code} once the error code is confirmed to exist (after the NotFoundError branch, not before) (depends on T017)
• T025 [P] [US4] In ai-support/tools/service/tools.service.ts's single executeTool(...) call site: increment supporthub_tool_invocations_total{tool, outcome} for every call, and — only when block.name === 'searchProductKnowledge' — increment supporthub_knowledge_retrieval_outcomes_total{matched} from whether result.output is a non-empty array (depends on T017)
• T026 [US4] Unit test: sla.service.ts's complete() does not increment the met outcome for a run already 'breached' (a fake repo returning status: 'breached') in tests/unit/observability/sla-compliance-metric.test.ts (depends on T021)
• T027 [US4] Integration test covering Quickstart Scenario 4 (all ten sub-scenarios — the eleventh, tool-failure, is covered by the same test file's tool-invocation case) — scrape /metrics before/after driving each real event through the real API, in tests/integration/observability/business-metrics.test.ts (depends on T018, T019, T020, T021, T022, T023, T024, T025)

Checkpoint: Quickstart Scenario 4 passes. All eleven named metrics are live and correct against real infrastructure.

---

Phase 6: Polish & Cross-Cutting Concerns

• T028 [P] Update specs/014-full-observability/checklists/requirements.md Notes with any implementation-time findings (including the pre-existing SLA-run status data-quality gap research.md §5 already surfaced)
• T029 Run npx tsx scripts/check-architecture.ts and npm run lint/npm run typecheck
• T030 Full regression: npm run test:unit then the full integration suite against real Docker-provisioned Postgres/Redis, confirming nothing outside this feature regressed (particularly every module touched by a single-line instrumentation addition: ai-support sessions/knowledge/tools, ticketing tickets/messages, orchestration/sla, and the event-bus handlers)

---

Dependencies & Execution Order

• Foundational (Phase 1): No dependencies — BLOCKS User Story 1 (and transitively 2)
• User Story 1 (Phase 2): Depends on Foundational — BLOCKS User Story 2 (shares its hook)
• User Story 2 (Phase 3): Depends on User Story 1
• User Story 3 (Phase 4): Depends only on Foundational (T001) — independent of US1/US2/US4
• User Story 4 (Phase 5): Depends only on Foundational (T001/T017) — independent of US1/US2/US3
• Polish (Phase 5): Depends on all four user stories