support_backend/specs/014-full-observability/tasks.md

---
description: 'Task list for 014-full-observability'
---

# Tasks: Full Observability

**Input**: Design documents from `specs/014-full-observability/`

**Prerequisites**: [plan.md](./plan.md), [spec.md](./spec.md), [research.md](./research.md),
[data-model.md](./data-model.md),
[contracts/metrics-contract.md](./contracts/metrics-contract.md), [quickstart.md](./quickstart.md)

**Organization**: Tasks are grouped by user story (US1 = P1 access log, US2 = P1 request-health
metrics, US3 = P2 tracing, US4 = P2 business-health metrics). US2 shares its hook point with
US1 (both live in the same `onResponse` hook) so US2 depends on US1's hook existing, not on its
own separate one. US3 and US4 are each independent of US1/US2 and of each other.

## Format: `[ID] [P?] [Story] Description`

All file paths are relative to `supporthub-api/` (repo root).

---

## Phase 1: Foundational (Blocking Prerequisites)

- [x] T001 Add `@opentelemetry/exporter-trace-otlp-http` and `@opentelemetry/resources` to
      `package.json` (`npm install`)
- [x] T002 Add `src/infrastructure/observability/request-context.store.ts` — a module-level
      `AsyncLocalStorage<{requestId: string; correlationId: string}>` with a `run()` passthrough
      and a `getStore()` re-export
- [x] T003 [P] Wire `logger.ts`'s Pino options with a `mixin` function reading from T002's store
      (returns `{}` when no store is active — a log call outside any request, e.g. at startup,
      must not throw) (depends on T002)

**Checkpoint**: Every subsequent log call through the shared `logger` singleton is
request-correlated automatically, once a request actually runs inside the store (US1 wires that
part next).

---

## Phase 2: User Story 1 - Trace one request end to end from its logs (Priority: P1)

**Goal**: One structured access-log line per request; every other log line produced during that
request's handling shares its request ID.

**Independent Test**: Quickstart Scenario 1.

### Tests for User Story 1

- [x] T004 [P] [US1] Unit test: a `logger.info(...)` call made inside T002's `store.run(...)`
      carries `requestId`/`correlationId` in its output; one made outside carries neither, in
      `tests/unit/observability/request-context-mixin.test.ts` (depends on T003)

### Implementation for User Story 1

- [x] T005 [US1] In `plugins/request-context.plugin.ts`'s existing `onRequest` hook, after
      building `request.reqContext`, call the T002 store's `run()` wrapping the remainder of the
      request's handling (Fastify's `onRequest` hooks accept a `done` callback / return a
      promise — the run wraps whichever style this hook currently uses) so every subsequent
      hook/handler for this request executes inside the ALS context (depends on T002)
- [x] T006 [US1] Add an `onResponse` hook (same plugin) that logs one line via the shared
      `logger`: `{event: "http_request_completed", method, route: request.routeOptions.url,
    statusCode: reply.statusCode, durationMs: reply.elapsedTime}`, at `info`/`warn`/`error`
      level by status class (depends on T005)
- [x] T007 [US1] Integration test covering Quickstart Scenario 1 (one access-log line per
      request incl. 404; shared requestId across the access-log line and an internal log line
      from the same request) in `tests/integration/observability/access-log.test.ts`, using a
      `logger.info`/`logger.warn` spy the same way `password-reset-flow.test.ts` (013) already
      does (depends on T006)

**Checkpoint**: Quickstart Scenario 1 passes. Every request is now visible in logs even when it
never errors.

---

## Phase 3: User Story 2 - See live request-health metrics (Priority: P1)

**Goal**: The existing (previously dead) request-duration histogram actually has observations;
error rate per route/status is computable from `/metrics` alone.

**Independent Test**: Quickstart Scenario 2.

### Implementation for User Story 2

- [x] T008 [US2] In the same `onResponse` hook added by T006, call
      `httpRequestDurationHistogram.observe({method, route: request.routeOptions.url, status_code:
    String(reply.statusCode)}, reply.elapsedTime / 1000)` (depends on T006)
- [x] T009 [US2] Integration test covering Quickstart Scenario 2 (send a mix of successful/
      failing requests to the same route, scrape `/metrics`, assert both status-code label
      values are present with the expected counts) in
      `tests/integration/observability/request-metrics.test.ts` (depends on T008)

**Checkpoint**: Quickstart Scenario 2 passes. `/metrics` now reflects real request traffic.

---

## Phase 4: User Story 3 - Trace a single incident's cross-module path (Priority: P2)

**Goal**: A real `TracerProvider` is active; `getTracer()` produces spans that are actually
exported; two named cross-module paths are instrumented.

**Independent Test**: Quickstart Scenario 3.

### Implementation for User Story 3

- [x] T010 [P] [US3] Rewrite `infrastructure/observability/tracing.ts` to initialize a
      `BasicTracerProvider` at module load with a `Resource` (`service.name: "supporthub-api"`)
      and register it via `trace.setGlobalTracerProvider(...)`; exporter/processor chosen by
      `NODE_ENV`/`OTEL_EXPORTER_OTLP_ENDPOINT` per research.md §4 (`InMemorySpanExporter` +
      `SimpleSpanProcessor` in test, `OTLPTraceExporter` + `BatchSpanProcessor` when the env var
      is set, `ConsoleSpanExporter` + `SimpleSpanProcessor` otherwise); export a
      `getTestSpanExporter()` accessor (test env only) for T015 to read exported spans back;
      `getTracer()`'s own exported signature is unchanged (depends on T001)
- [x] T011 [US3] Wire the OpenTelemetry SDK's internal diagnostic logger
      (`diag.setLogger(...)`) to the shared `logger.warn`, so span-export failures land in this
      project's own log stream instead of stderr or nowhere (depends on T010)
- [x] T012 [P] [US3] Add a `ticket.create` span (`getTracer().startActiveSpan(...)`) around
      `tickets/service/tickets.service.ts`'s ticket-creation method, with `ticket.id` and
      `product.externalProductId` attributes, ended in a `finally` (depends on T010)
- [x] T013 [P] [US3] Add an `ai.escalation` span around `ai-support/sessions/service/
    session.service.ts`'s escalation branch(es), with `ticket.id`/`session.id` attributes
      (depends on T010)
- [x] T014 [US3] Add an `orchestration.assignment` span wrapping the existing
      `orchestrationService.handleHumanEscalation` call in the `TICKET_UPDATED`/
      `HUMAN_ESCALATION` subscriber (`src/events/handlers/index.ts`), with `ticket.id`/
      `strategy` attributes, so it nests under T013's span when both occur in the same request
      (depends on T010, T013)
- [x] T015 [US3] Integration test covering Quickstart Scenario 3 steps 1-2: drive a real
      ticket-creation → escalation → orchestration/assignment flow, read spans back via T010's
      `getTestSpanExporter()`, assert `ticket.create`/`ai.escalation`/`orchestration.assignment`
      all share one trace ID with correct parent/child `spanId` relationships, in
      `tests/integration/observability/tracing.test.ts` (depends on T012, T013, T014)
- [x] T016 [US3] Integration test covering Quickstart Scenario 3 steps 3-4: point
      `OTEL_EXPORTER_OTLP_ENDPOINT` at an unreachable address, confirm `buildApp()` still
      resolves and a request still completes successfully, in the same test file (depends on
      T010)

**Checkpoint**: Quickstart Scenario 3 passes. A real, inspectable trace exists for the first
time; tracing failure never blocks the app.

---

## Phase 5: User Story 4 - See the business-health metrics this project committed to tracking (Priority: P2)

**Goal**: All eleven named metrics (data-model.md) are live on `/metrics`, each updated at the
exact real event research.md identified.

**Independent Test**: Quickstart Scenario 4.

### Implementation for User Story 4

- [x] T017 [US4] Define all eleven new `Counter`/`Histogram` objects in
      `infrastructure/observability/metrics.ts` per data-model.md's table, exported individually
      (depends on T001)
- [x] T018 [P] [US4] Increment `supporthub_ai_session_outcomes_total` in `ai-support/sessions/
    repository/session.repository.ts`'s `updateStatus`, labeled `outcome` when `status` is
      `'resolved'`/`'escalated'` (depends on T017)
- [x] T019 [P] [US4] Add a `TICKET_UPDATED`/`newStatus === 'RESOLVED'` subscriber in
      `src/events/handlers/index.ts` that looks up `resolutionRepository.findByTicketId`,
      increments `supporthub_ticket_resolutions_total{resolved_by}` (`ai` vs. any other value),
      fetches the ticket for `createdAt`, and observes
      `supporthub_ticket_resolution_duration_seconds` (depends on T017)
- [x] T020 [P] [US4] In `ticketing/messages/service/messages.service.ts`'s `post`, when
      `type === 'AGENT_MESSAGE'`, check for a prior `AGENT_MESSAGE` on the ticket and — only for
      the first one — observe `supporthub_ticket_first_response_duration_seconds` against the
      ticket's `createdAt` (depends on T017)
- [x] T021 [P] [US4] In `orchestration/sla/service/sla.service.ts`: in `complete()`, read
      `run.status` before updating and increment `supporthub_sla_run_outcomes_total{outcome:
    "met"}` only if it was not already `'breached'`; in `runBreachDetectionSweep()`, increment
      `{outcome: "breached"}` for each newly-flagged run (depends on T017)
- [x] T022 [P] [US4] Add an `ESCALATION_TRIGGERED` subscriber in `src/events/handlers/index.ts`
      that increments `supporthub_escalations_total{reason}` from the event payload's `reason`
      (depends on T017)
- [x] T023 [P] [US4] In `ticketing/tickets/service/tickets.service.ts`'s ticket-creation method,
      increment `supporthub_problems_created_total{category_id}` right after `problemsRepo.create`
      succeeds (`categoryId ?? 'uncategorized'`) (depends on T017)
- [x] T024 [P] [US4] In `ai-support/knowledge/service/error-codes.service.ts`'s
      `findKnownIssuesByErrorCode`, increment `supporthub_known_error_lookups_total{code}` once
      the error code is confirmed to exist (after the `NotFoundError` branch, not before)
      (depends on T017)
- [x] T025 [P] [US4] In `ai-support/tools/service/tools.service.ts`'s single `executeTool(...)`
      call site: increment `supporthub_tool_invocations_total{tool, outcome}` for every call, and
      — only when `block.name === 'searchProductKnowledge'` — increment
      `supporthub_knowledge_retrieval_outcomes_total{matched}` from whether `result.output` is a
      non-empty array (depends on T017)
- [x] T026 [US4] Unit test: `sla.service.ts`'s `complete()` does not increment the `met` outcome
      for a run already `'breached'` (a fake repo returning `status: 'breached'`) in
      `tests/unit/observability/sla-compliance-metric.test.ts` (depends on T021)
- [x] T027 [US4] Integration test covering Quickstart Scenario 4 (all ten sub-scenarios — the
      eleventh, tool-failure, is covered by the same test file's tool-invocation case) —
      scrape `/metrics` before/after driving each real event through the real API, in
      `tests/integration/observability/business-metrics.test.ts` (depends on T018, T019, T020,
      T021, T022, T023, T024, T025)

**Checkpoint**: Quickstart Scenario 4 passes. All eleven named metrics are live and correct
against real infrastructure.

---

## Phase 6: Polish & Cross-Cutting Concerns

- [x] T028 [P] Update `specs/014-full-observability/checklists/requirements.md` Notes with any
      implementation-time findings (including the pre-existing SLA-run status data-quality gap
      research.md §5 already surfaced)
- [x] T029 Run `npx tsx scripts/check-architecture.ts` and `npm run lint`/`npm run typecheck`
- [x] T030 Full regression: `npm run test:unit` then the full integration suite against real
      Docker-provisioned Postgres/Redis, confirming nothing outside this feature regressed
      (particularly every module touched by a single-line instrumentation addition: ai-support
      sessions/knowledge/tools, ticketing tickets/messages, orchestration/sla, and the event-bus
      handlers)

---

## Dependencies & Execution Order

- **Foundational (Phase 1)**: No dependencies — BLOCKS User Story 1 (and transitively 2)
- **User Story 1 (Phase 2)**: Depends on Foundational — BLOCKS User Story 2 (shares its hook)
- **User Story 2 (Phase 3)**: Depends on User Story 1
- **User Story 3 (Phase 4)**: Depends only on Foundational (T001) — independent of US1/US2/US4
- **User Story 4 (Phase 5)**: Depends only on Foundational (T001/T017) — independent of
  US1/US2/US3
- **Polish (Phase 5)**: Depends on all four user stories