71 lines
4.0 KiB
Markdown
71 lines
4.0 KiB
Markdown
|
# Quickstart: Full Observability
|
||
|
|
|
||
|
|
Manual verification steps for each user story, against a running instance backed by real
|
||
|
|
Postgres/Redis (the throwaway Docker containers already used throughout this project's test
|
||
|
|
suite work equally well for a manual run).
|
||
|
|
|
||
|
|
## Scenario 1 — Per-request access log (User Story 1)
|
||
|
|
|
||
|
|
1. Start the API. Send any request (e.g. `GET /health`).
|
||
|
|
2. **Expected**: exactly one log line appears with `event: "http_request_completed"`, the
|
||
|
|
request's method, route, status code, and a `requestId`.
|
||
|
|
3. Send a request to a route that triggers additional internal logging (e.g. a login attempt).
|
||
|
|
4. **Expected**: every log line produced while handling that request — the access-log line and
|
||
|
|
any domain log lines — carries the same `requestId`/`correlationId`.
|
||
|
|
5. Send a request to a route that doesn't exist.
|
||
|
|
6. **Expected**: a 404 access-log line is still emitted (not silently dropped).
|
||
|
|
|
||
|
|
## Scenario 2 — Live request-health metrics (User Story 2)
|
||
|
|
|
||
|
|
1. Send a mix of successful and failing requests (e.g. a valid login, then three wrong-password
|
||
|
|
logins).
|
||
|
|
2. Scrape `GET /metrics`.
|
||
|
|
3. **Expected**: `supporthub_http_request_duration_seconds_count` has observations labeled
|
||
|
|
`route="/auth/login"` with both `status_code="200"` and `status_code="401"` present, letting
|
||
|
|
an operator compute the error rate for that route from these two series alone.
|
||
|
|
|
||
|
|
## Scenario 3 — Cross-module trace (User Story 3)
|
||
|
|
|
||
|
|
1. With the API running in a mode where tracing exports to the console (no
|
||
|
|
`OTEL_EXPORTER_OTLP_ENDPOINT` configured), drive a request that escalates a ticket to a human
|
||
|
|
and triggers automatic orchestration/assignment.
|
||
|
|
2. **Expected**: console output shows a `ticket.create`-or-`ai.escalation` root span and an
|
||
|
|
`orchestration.assignment` child span sharing the same trace ID, with the child's start time
|
||
|
|
at or after the parent's.
|
||
|
|
3. Stop the (nonexistent) collector / leave `OTEL_EXPORTER_OTLP_ENDPOINT` pointed at an
|
||
|
|
unreachable address.
|
||
|
|
4. **Expected**: the API still starts and serves requests normally; only a logged export-failure
|
||
|
|
warning appears, nothing surfaces to any HTTP response.
|
||
|
|
|
||
|
|
## Scenario 4 — Business-health metrics (User Story 4)
|
||
|
|
|
||
|
|
For each metric, scrape `/metrics`, note the current value, drive the real event, scrape again,
|
||
|
|
and confirm the expected series moved by exactly one (or by the expected duration observation):
|
||
|
|
|
||
|
|
1. Complete an AI session without escalating → `supporthub_ai_session_outcomes_total{outcome="resolved"}` +1.
|
||
|
|
2. Complete an AI session that escalates, then have a human agent resolve the ticket →
|
||
|
|
`supporthub_ai_session_outcomes_total{outcome="escalated"}` +1, and once resolved,
|
||
|
|
`supporthub_ticket_resolutions_total{resolved_by="human"}` +1.
|
||
|
|
3. Resolve any ticket → `supporthub_ticket_resolution_duration_seconds` gains one new observation.
|
||
|
|
4. Post the first agent reply on a ticket → `supporthub_ticket_first_response_duration_seconds`
|
||
|
|
gains one new observation.
|
||
|
|
5. Let an SLA run complete on time, and separately let one breach (via the existing breach-sweep
|
||
|
|
test helper) → `supporthub_sla_run_outcomes_total{outcome="met"}` and
|
||
|
|
`{outcome="breached"}` each +1 respectively.
|
||
|
|
6. Trigger an escalation → `supporthub_escalations_total{reason="<the actual reason>"}` +1.
|
||
|
|
7. Create a ticket for a categorized problem →
|
||
|
|
`supporthub_problems_created_total{category_id="<id>"}` +1.
|
||
|
|
8. Look up a valid error code's known issues →
|
||
|
|
`supporthub_known_error_lookups_total{code="<code>"}` +1.
|
||
|
|
9. Have the AI's `searchProductKnowledge` tool return zero results, then results →
|
||
|
|
`supporthub_knowledge_retrieval_outcomes_total{matched="false"}` then `{matched="true"}`,
|
||
|
|
each +1 in turn.
|
||
|
|
10. Have any AI tool invocation fail → `supporthub_tool_invocations_total{tool="<name>",
|
||
|
|
outcome="failed"}` +1.
|
||
|
|
|
||
|
|
## What "done" looks like
|
||
|
|
|
||
|
|
All four scenarios pass against a real Postgres/Redis, `/health*` and the existing
|
||
|
|
`supporthub_http_request_duration_seconds` metric's shape are unchanged for any existing
|
||
|
|
consumer, and the API starts and serves traffic normally with no tracing collector configured.
|