98 lines
3.1 KiB
Markdown
98 lines
3.1 KiB
Markdown
|
# 09 — Testing, Observability & CI/CD
|
||
|
|
|
||
|
|
## 1. Testing strategy
|
||
|
|
|
||
|
|
### Backend
|
||
|
|
- Unit tests
|
||
|
|
- Integration tests
|
||
|
|
- E2E tests
|
||
|
|
- Concurrency tests (assignment race conditions — two tickets assigned simultaneously must never corrupt round-robin state or double-assign)
|
||
|
|
- SLA tests (pause/resume correctness, business-calendar math, durability across a simulated process restart)
|
||
|
|
- Escalation idempotency tests (a rule firing twice must not create duplicate escalation events)
|
||
|
|
- Orchestration tests (capability matching, hierarchy traversal, strategy selection)
|
||
|
|
- AI tool permission tests (the AI must never be able to invoke a tool it isn't scoped for; high-risk tools must require policy/approval regardless of AI confidence)
|
||
|
|
|
||
|
|
### Frontend
|
||
|
|
- Unit tests
|
||
|
|
- Integration tests
|
||
|
|
- Playwright E2E, covering:
|
||
|
|
- AI support flow
|
||
|
|
- Customer escalation flow
|
||
|
|
- Agent flow
|
||
|
|
- Admin configuration flow
|
||
|
|
|
||
|
|
### Critical end-to-end scenarios (must both exist as automated tests)
|
||
|
|
|
||
|
|
**Scenario A — AI resolves directly:**
|
||
|
|
```
|
||
|
|
Customer → Product → Support → Problem → AI → Knowledge
|
||
|
|
→ Guided troubleshooting → Verification → AI resolved
|
||
|
|
```
|
||
|
|
|
||
|
|
**Scenario B — AI escalates to human:**
|
||
|
|
```
|
||
|
|
Customer → Problem → AI → troubleshooting failed → human escalation
|
||
|
|
→ orchestration → assignment → SLA → investigation → solution
|
||
|
|
→ verification → resolution → closure
|
||
|
|
```
|
||
|
|
|
||
|
|
## 2. Observability
|
||
|
|
|
||
|
|
### Logging
|
||
|
|
Pino structured logging across the backend, with a request ID and correlation ID attached to every log line so a single ticket's full journey (AI session → tool calls → escalation → assignment → SLA events) can be traced end to end.
|
||
|
|
|
||
|
|
### Metrics & tracing
|
||
|
|
Wire metrics and tracing in from the start, not retrofitted. Expose:
|
||
|
|
|
||
|
|
```
|
||
|
|
GET /health
|
||
|
|
GET /health/live
|
||
|
|
GET /health/ready
|
||
|
|
GET /metrics
|
||
|
|
```
|
||
|
|
|
||
|
|
### Key metrics to track
|
||
|
|
|
||
|
|
- AI resolution rate
|
||
|
|
- AI escalation rate
|
||
|
|
- Human resolution rate
|
||
|
|
- Average resolution time
|
||
|
|
- First response time
|
||
|
|
- SLA compliance
|
||
|
|
- Escalation rate
|
||
|
|
- Recurring problems
|
||
|
|
- Most common errors
|
||
|
|
- Knowledge effectiveness
|
||
|
|
- Tool failure rate
|
||
|
|
|
||
|
|
### Reporting dashboards
|
||
|
|
|
||
|
|
| Dashboard | Contents |
|
||
|
|
|---|---|
|
||
|
|
| **Management** | Total cases, AI resolved, human escalated, resolved, open, SLA compliance, SLA breaches, escalation count, average response, average resolution |
|
||
|
|
| **Product** | Support volume by product, problem types, recurring problems, AI resolution rate, human escalation rate, top errors |
|
||
|
|
| **Support** | Workload, agent assignments, SLA risk, escalations, response performance, resolution performance |
|
||
|
|
| **AI** | AI resolution rate, failed troubleshooting, knowledge match rate, confidence distribution, tool success/failure, human handoff rate |
|
||
|
|
|
||
|
|
## 3. CI/CD (Jenkins)
|
||
|
|
|
||
|
|
Repository includes a `Jenkinsfile` implementing:
|
||
|
|
|
||
|
|
```
|
||
|
|
Checkout
|
||
|
|
→ Install
|
||
|
|
→ Environment validation
|
||
|
|
→ Typecheck
|
||
|
|
→ Lint
|
||
|
|
→ Format check
|
||
|
|
→ Unit test
|
||
|
|
→ Integration test
|
||
|
|
→ E2E test
|
||
|
|
→ Build
|
||
|
|
→ Docker build
|
||
|
|
→ Publish
|
||
|
|
→ Deploy
|
||
|
|
```
|
||
|
|
|
||
|
|
Production deployments use **protected Jenkins credentials/environment variables** — real secrets are never committed to the repository (see [02](./02-integration-and-security.md#7-environment--secrets-handling)).
|