# 09 — Testing, Observability & CI/CD

## 1. Testing strategy

### Backend
- Unit tests
- Integration tests
- E2E tests
- Concurrency tests (assignment race conditions — two tickets assigned simultaneously must never corrupt round-robin state or double-assign)
- SLA tests (pause/resume correctness, business-calendar math, durability across a simulated process restart)
- Escalation idempotency tests (a rule firing twice must not create duplicate escalation events)
- Orchestration tests (capability matching, hierarchy traversal, strategy selection)
- AI tool permission tests (the AI must never be able to invoke a tool it isn't scoped for; high-risk tools must require policy/approval regardless of AI confidence)

### Frontend
- Unit tests
- Integration tests
- Playwright E2E, covering:
  - AI support flow
  - Customer escalation flow
  - Agent flow
  - Admin configuration flow

### Critical end-to-end scenarios (must both exist as automated tests)

**Scenario A — AI resolves directly:**
```
Customer → Product → Support → Problem → AI → Knowledge
        → Guided troubleshooting → Verification → AI resolved
```

**Scenario B — AI escalates to human:**
```
Customer → Problem → AI → troubleshooting failed → human escalation
        → orchestration → assignment → SLA → investigation → solution
        → verification → resolution → closure
```

## 2. Observability

### Logging
Pino structured logging across the backend, with a request ID and correlation ID attached to every log line so a single ticket's full journey (AI session → tool calls → escalation → assignment → SLA events) can be traced end to end.

### Metrics & tracing
Wire metrics and tracing in from the start, not retrofitted. Expose:

```
GET /health
GET /health/live
GET /health/ready
GET /metrics
```

### Key metrics to track

- AI resolution rate
- AI escalation rate
- Human resolution rate
- Average resolution time
- First response time
- SLA compliance
- Escalation rate
- Recurring problems
- Most common errors
- Knowledge effectiveness
- Tool failure rate

### Reporting dashboards

| Dashboard | Contents |
|---|---|
| **Management** | Total cases, AI resolved, human escalated, resolved, open, SLA compliance, SLA breaches, escalation count, average response, average resolution |
| **Product** | Support volume by product, problem types, recurring problems, AI resolution rate, human escalation rate, top errors |
| **Support** | Workload, agent assignments, SLA risk, escalations, response performance, resolution performance |
| **AI** | AI resolution rate, failed troubleshooting, knowledge match rate, confidence distribution, tool success/failure, human handoff rate |

## 3. CI/CD (Jenkins)

Repository includes a `Jenkinsfile` implementing:

```
Checkout
→ Install
→ Environment validation
→ Typecheck
→ Lint
→ Format check
→ Unit test
→ Integration test
→ E2E test
→ Build
→ Docker build
→ Publish
→ Deploy
```

Production deployments use **protected Jenkins credentials/environment variables** — real secrets are never committed to the repository (see [02](./02-integration-and-security.md#7-environment--secrets-handling)).