diff --git a/specs/014-full-observability/tasks.md b/specs/014-full-observability/tasks.md new file mode 100644 index 0000000..0146e06 --- /dev/null +++ b/specs/014-full-observability/tasks.md @@ -0,0 +1,222 @@ +--- +description: "Task list for 014-full-observability" +--- + +# Tasks: Full Observability + +**Input**: Design documents from `specs/014-full-observability/` + +**Prerequisites**: [plan.md](./plan.md), [spec.md](./spec.md), [research.md](./research.md), +[data-model.md](./data-model.md), +[contracts/metrics-contract.md](./contracts/metrics-contract.md), [quickstart.md](./quickstart.md) + +**Organization**: Tasks are grouped by user story (US1 = P1 access log, US2 = P1 request-health +metrics, US3 = P2 tracing, US4 = P2 business-health metrics). US2 shares its hook point with +US1 (both live in the same `onResponse` hook) so US2 depends on US1's hook existing, not on its +own separate one. US3 and US4 are each independent of US1/US2 and of each other. + +## Format: `[ID] [P?] [Story] Description` + +All file paths are relative to `supporthub-api/` (repo root). + +--- + +## Phase 1: Foundational (Blocking Prerequisites) + +- [ ] T001 Add `@opentelemetry/exporter-trace-otlp-http` and `@opentelemetry/resources` to + `package.json` (`npm install`) +- [ ] T002 Add `src/infrastructure/observability/request-context.store.ts` — a module-level + `AsyncLocalStorage<{requestId: string; correlationId: string}>` with a `run()` passthrough + and a `getStore()` re-export +- [ ] T003 [P] Wire `logger.ts`'s Pino options with a `mixin` function reading from T002's store + (returns `{}` when no store is active — a log call outside any request, e.g. at startup, + must not throw) (depends on T002) + +**Checkpoint**: Every subsequent log call through the shared `logger` singleton is +request-correlated automatically, once a request actually runs inside the store (US1 wires that +part next). + +--- + +## Phase 2: User Story 1 - Trace one request end to end from its logs (Priority: P1) + +**Goal**: One structured access-log line per request; every other log line produced during that +request's handling shares its request ID. + +**Independent Test**: Quickstart Scenario 1. + +### Tests for User Story 1 + +- [ ] T004 [P] [US1] Unit test: a `logger.info(...)` call made inside T002's `store.run(...)` + carries `requestId`/`correlationId` in its output; one made outside carries neither, in + `tests/unit/observability/request-context-mixin.test.ts` (depends on T003) + +### Implementation for User Story 1 + +- [ ] T005 [US1] In `plugins/request-context.plugin.ts`'s existing `onRequest` hook, after + building `request.reqContext`, call the T002 store's `run()` wrapping the remainder of the + request's handling (Fastify's `onRequest` hooks accept a `done` callback / return a + promise — the run wraps whichever style this hook currently uses) so every subsequent + hook/handler for this request executes inside the ALS context (depends on T002) +- [ ] T006 [US1] Add an `onResponse` hook (same plugin) that logs one line via the shared + `logger`: `{event: "http_request_completed", method, route: request.routeOptions.url, + statusCode: reply.statusCode, durationMs: reply.elapsedTime}`, at `info`/`warn`/`error` + level by status class (depends on T005) +- [ ] T007 [US1] Integration test covering Quickstart Scenario 1 (one access-log line per + request incl. 404; shared requestId across the access-log line and an internal log line + from the same request) in `tests/integration/observability/access-log.test.ts`, using a + `logger.info`/`logger.warn` spy the same way `password-reset-flow.test.ts` (013) already + does (depends on T006) + +**Checkpoint**: Quickstart Scenario 1 passes. Every request is now visible in logs even when it +never errors. + +--- + +## Phase 3: User Story 2 - See live request-health metrics (Priority: P1) + +**Goal**: The existing (previously dead) request-duration histogram actually has observations; +error rate per route/status is computable from `/metrics` alone. + +**Independent Test**: Quickstart Scenario 2. + +### Implementation for User Story 2 + +- [ ] T008 [US2] In the same `onResponse` hook added by T006, call + `httpRequestDurationHistogram.observe({method, route: request.routeOptions.url, status_code: + String(reply.statusCode)}, reply.elapsedTime / 1000)` (depends on T006) +- [ ] T009 [US2] Integration test covering Quickstart Scenario 2 (send a mix of successful/ + failing requests to the same route, scrape `/metrics`, assert both status-code label + values are present with the expected counts) in + `tests/integration/observability/request-metrics.test.ts` (depends on T008) + +**Checkpoint**: Quickstart Scenario 2 passes. `/metrics` now reflects real request traffic. + +--- + +## Phase 4: User Story 3 - Trace a single incident's cross-module path (Priority: P2) + +**Goal**: A real `TracerProvider` is active; `getTracer()` produces spans that are actually +exported; two named cross-module paths are instrumented. + +**Independent Test**: Quickstart Scenario 3. + +### Implementation for User Story 3 + +- [ ] T010 [P] [US3] Rewrite `infrastructure/observability/tracing.ts` to initialize a + `BasicTracerProvider` at module load with a `Resource` (`service.name: "supporthub-api"`) + and register it via `trace.setGlobalTracerProvider(...)`; exporter/processor chosen by + `NODE_ENV`/`OTEL_EXPORTER_OTLP_ENDPOINT` per research.md §4 (`InMemorySpanExporter` + + `SimpleSpanProcessor` in test, `OTLPTraceExporter` + `BatchSpanProcessor` when the env var + is set, `ConsoleSpanExporter` + `SimpleSpanProcessor` otherwise); export a + `getTestSpanExporter()` accessor (test env only) for T015 to read exported spans back; + `getTracer()`'s own exported signature is unchanged (depends on T001) +- [ ] T011 [US3] Wire the OpenTelemetry SDK's internal diagnostic logger + (`diag.setLogger(...)`) to the shared `logger.warn`, so span-export failures land in this + project's own log stream instead of stderr or nowhere (depends on T010) +- [ ] T012 [P] [US3] Add a `ticket.create` span (`getTracer().startActiveSpan(...)`) around + `tickets/service/tickets.service.ts`'s ticket-creation method, with `ticket.id` and + `product.externalProductId` attributes, ended in a `finally` (depends on T010) +- [ ] T013 [P] [US3] Add an `ai.escalation` span around `ai-support/sessions/service/ + session.service.ts`'s escalation branch(es), with `ticket.id`/`session.id` attributes + (depends on T010) +- [ ] T014 [US3] Add an `orchestration.assignment` span wrapping the existing + `orchestrationService.handleHumanEscalation` call in the `TICKET_UPDATED`/ + `HUMAN_ESCALATION` subscriber (`src/events/handlers/index.ts`), with `ticket.id`/ + `strategy` attributes, so it nests under T013's span when both occur in the same request + (depends on T010, T013) +- [ ] T015 [US3] Integration test covering Quickstart Scenario 3 steps 1-2: drive a real + ticket-creation → escalation → orchestration/assignment flow, read spans back via T010's + `getTestSpanExporter()`, assert `ticket.create`/`ai.escalation`/`orchestration.assignment` + all share one trace ID with correct parent/child `spanId` relationships, in + `tests/integration/observability/tracing.test.ts` (depends on T012, T013, T014) +- [ ] T016 [US3] Integration test covering Quickstart Scenario 3 steps 3-4: point + `OTEL_EXPORTER_OTLP_ENDPOINT` at an unreachable address, confirm `buildApp()` still + resolves and a request still completes successfully, in the same test file (depends on + T010) + +**Checkpoint**: Quickstart Scenario 3 passes. A real, inspectable trace exists for the first +time; tracing failure never blocks the app. + +--- + +## Phase 5: User Story 4 - See the business-health metrics this project committed to tracking (Priority: P2) + +**Goal**: All eleven named metrics (data-model.md) are live on `/metrics`, each updated at the +exact real event research.md identified. + +**Independent Test**: Quickstart Scenario 4. + +### Implementation for User Story 4 + +- [ ] T017 [US4] Define all eleven new `Counter`/`Histogram` objects in + `infrastructure/observability/metrics.ts` per data-model.md's table, exported individually + (depends on T001) +- [ ] T018 [P] [US4] Increment `supporthub_ai_session_outcomes_total` in `ai-support/sessions/ + repository/session.repository.ts`'s `updateStatus`, labeled `outcome` when `status` is + `'resolved'`/`'escalated'` (depends on T017) +- [ ] T019 [P] [US4] Add a `TICKET_UPDATED`/`newStatus === 'RESOLVED'` subscriber in + `src/events/handlers/index.ts` that looks up `resolutionRepository.findByTicketId`, + increments `supporthub_ticket_resolutions_total{resolved_by}` (`ai` vs. any other value), + fetches the ticket for `createdAt`, and observes + `supporthub_ticket_resolution_duration_seconds` (depends on T017) +- [ ] T020 [P] [US4] In `ticketing/messages/service/messages.service.ts`'s `post`, when + `type === 'AGENT_MESSAGE'`, check for a prior `AGENT_MESSAGE` on the ticket and — only for + the first one — observe `supporthub_ticket_first_response_duration_seconds` against the + ticket's `createdAt` (depends on T017) +- [ ] T021 [P] [US4] In `orchestration/sla/service/sla.service.ts`: in `complete()`, read + `run.status` before updating and increment `supporthub_sla_run_outcomes_total{outcome: + "met"}` only if it was not already `'breached'`; in `runBreachDetectionSweep()`, increment + `{outcome: "breached"}` for each newly-flagged run (depends on T017) +- [ ] T022 [P] [US4] Add an `ESCALATION_TRIGGERED` subscriber in `src/events/handlers/index.ts` + that increments `supporthub_escalations_total{reason}` from the event payload's `reason` + (depends on T017) +- [ ] T023 [P] [US4] In `ticketing/tickets/service/tickets.service.ts`'s ticket-creation method, + increment `supporthub_problems_created_total{category_id}` right after `problemsRepo.create` + succeeds (`categoryId ?? 'uncategorized'`) (depends on T017) +- [ ] T024 [P] [US4] In `ai-support/knowledge/service/error-codes.service.ts`'s + `findKnownIssuesByErrorCode`, increment `supporthub_known_error_lookups_total{code}` once + the error code is confirmed to exist (after the `NotFoundError` branch, not before) + (depends on T017) +- [ ] T025 [P] [US4] In `ai-support/tools/service/tools.service.ts`'s single `executeTool(...)` + call site: increment `supporthub_tool_invocations_total{tool, outcome}` for every call, and + — only when `block.name === 'searchProductKnowledge'` — increment + `supporthub_knowledge_retrieval_outcomes_total{matched}` from whether `result.output` is a + non-empty array (depends on T017) +- [ ] T026 [US4] Unit test: `sla.service.ts`'s `complete()` does not increment the `met` outcome + for a run already `'breached'` (a fake repo returning `status: 'breached'`) in + `tests/unit/observability/sla-compliance-metric.test.ts` (depends on T021) +- [ ] T027 [US4] Integration test covering Quickstart Scenario 4 (all ten sub-scenarios — the + eleventh, tool-failure, is covered by the same test file's tool-invocation case) — + scrape `/metrics` before/after driving each real event through the real API, in + `tests/integration/observability/business-metrics.test.ts` (depends on T018, T019, T020, + T021, T022, T023, T024, T025) + +**Checkpoint**: Quickstart Scenario 4 passes. All eleven named metrics are live and correct +against real infrastructure. + +--- + +## Phase 6: Polish & Cross-Cutting Concerns + +- [ ] T028 [P] Update `specs/014-full-observability/checklists/requirements.md` Notes with any + implementation-time findings (including the pre-existing SLA-run status data-quality gap + research.md §5 already surfaced) +- [ ] T029 Run `npx tsx scripts/check-architecture.ts` and `npm run lint`/`npm run typecheck` +- [ ] T030 Full regression: `npm run test:unit` then the full integration suite against real + Docker-provisioned Postgres/Redis, confirming nothing outside this feature regressed + (particularly every module touched by a single-line instrumentation addition: ai-support + sessions/knowledge/tools, ticketing tickets/messages, orchestration/sla, and the event-bus + handlers) + +--- + +## Dependencies & Execution Order + +- **Foundational (Phase 1)**: No dependencies — BLOCKS User Story 1 (and transitively 2) +- **User Story 1 (Phase 2)**: Depends on Foundational — BLOCKS User Story 2 (shares its hook) +- **User Story 2 (Phase 3)**: Depends on User Story 1 +- **User Story 3 (Phase 4)**: Depends only on Foundational (T001) — independent of US1/US2/US4 +- **User Story 4 (Phase 5)**: Depends only on Foundational (T001/T017) — independent of + US1/US2/US3 +- **Polish (Phase 5)**: Depends on all four user stories