support_backend/specs/014-full-observability/quickstart.md

Quickstart: Full Observability

Manual verification steps for each user story, against a running instance backed by real Postgres/Redis (the throwaway Docker containers already used throughout this project's test suite work equally well for a manual run).

Scenario 1 — Per-request access log (User Story 1)

1. Start the API. Send any request (e.g. GET /health).
2. Expected: exactly one log line appears with event: "http_request_completed", the request's method, route, status code, and a requestId.
3. Send a request to a route that triggers additional internal logging (e.g. a login attempt).
4. Expected: every log line produced while handling that request — the access-log line and any domain log lines — carries the same requestId/correlationId.
5. Send a request to a route that doesn't exist.
6. Expected: a 404 access-log line is still emitted (not silently dropped).

Scenario 2 — Live request-health metrics (User Story 2)

1. Send a mix of successful and failing requests (e.g. a valid login, then three wrong-password logins).
2. Scrape GET /metrics.
3. Expected: supporthub_http_request_duration_seconds_count has observations labeled route="/auth/login" with both status_code="200" and status_code="401" present, letting an operator compute the error rate for that route from these two series alone.

Scenario 3 — Cross-module trace (User Story 3)

1. With the API running in a mode where tracing exports to the console (no OTEL_EXPORTER_OTLP_ENDPOINT configured), drive a request that escalates a ticket to a human and triggers automatic orchestration/assignment.
2. Expected: console output shows a ticket.create-or-ai.escalation root span and an orchestration.assignment child span sharing the same trace ID, with the child's start time at or after the parent's.
3. Stop the (nonexistent) collector / leave OTEL_EXPORTER_OTLP_ENDPOINT pointed at an unreachable address.
4. Expected: the API still starts and serves requests normally; only a logged export-failure warning appears, nothing surfaces to any HTTP response.

Scenario 4 — Business-health metrics (User Story 4)

For each metric, scrape /metrics, note the current value, drive the real event, scrape again, and confirm the expected series moved by exactly one (or by the expected duration observation):

1. Complete an AI session without escalating → supporthub_ai_session_outcomes_total{outcome="resolved"} +1.
2. Complete an AI session that escalates, then have a human agent resolve the ticket → supporthub_ai_session_outcomes_total{outcome="escalated"} +1, and once resolved, supporthub_ticket_resolutions_total{resolved_by="human"} +1.
3. Resolve any ticket → supporthub_ticket_resolution_duration_seconds gains one new observation.
4. Post the first agent reply on a ticket → supporthub_ticket_first_response_duration_seconds gains one new observation.
5. Let an SLA run complete on time, and separately let one breach (via the existing breach-sweep test helper) → supporthub_sla_run_outcomes_total{outcome="met"} and {outcome="breached"} each +1 respectively.
6. Trigger an escalation → supporthub_escalations_total{reason="<the actual reason>"} +1.
7. Create a ticket for a categorized problem → supporthub_problems_created_total{category_id="<id>"} +1.
8. Look up a valid error code's known issues → supporthub_known_error_lookups_total{code="<code>"} +1.
9. Have the AI's searchProductKnowledge tool return zero results, then results → supporthub_knowledge_retrieval_outcomes_total{matched="false"} then {matched="true"}, each +1 in turn.
10. Have any AI tool invocation fail → supporthub_tool_invocations_total{tool="<name>", outcome="failed"} +1.

What "done" looks like

All four scenarios pass against a real Postgres/Redis, /health* and the existing supporthub_http_request_duration_seconds metric's shape are unchanged for any existing consumer, and the API starts and serves traffic normally with no tracing collector configured.