Files

T

saqib mirandClaude Sonnet 5 f475a55a53 docs: add product and engineering specification

Full system blueprint (docs 01-10): product vision, integration &
security, AI support architecture, ticketing & problem management,
orchestration/SLA/escalation, database schema, backend/frontend
architecture, testing/observability/CI-CD, and the implementation
roadmap. This is the pre-implementation design reference the codebase
is being built against.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

2026-08-21 16:22:21 +05:30

3.1 KiB

Raw Blame History

09 — Testing, Observability & CI/CD

1. Testing strategy

Backend

Unit tests
Integration tests
E2E tests
Concurrency tests (assignment race conditions — two tickets assigned simultaneously must never corrupt round-robin state or double-assign)
SLA tests (pause/resume correctness, business-calendar math, durability across a simulated process restart)
Escalation idempotency tests (a rule firing twice must not create duplicate escalation events)
Orchestration tests (capability matching, hierarchy traversal, strategy selection)
AI tool permission tests (the AI must never be able to invoke a tool it isn't scoped for; high-risk tools must require policy/approval regardless of AI confidence)

Frontend

Unit tests
Integration tests
Playwright E2E, covering:
- AI support flow
- Customer escalation flow
- Agent flow
- Admin configuration flow

Critical end-to-end scenarios (must both exist as automated tests)

Scenario A — AI resolves directly:

Customer → Product → Support → Problem → AI → Knowledge
        → Guided troubleshooting → Verification → AI resolved

Scenario B — AI escalates to human:

Customer → Problem → AI → troubleshooting failed → human escalation
        → orchestration → assignment → SLA → investigation → solution
        → verification → resolution → closure

2. Observability

Logging

Pino structured logging across the backend, with a request ID and correlation ID attached to every log line so a single ticket's full journey (AI session → tool calls → escalation → assignment → SLA events) can be traced end to end.

Metrics & tracing

Wire metrics and tracing in from the start, not retrofitted. Expose:

GET /health
GET /health/live
GET /health/ready
GET /metrics

Key metrics to track

AI resolution rate
AI escalation rate
Human resolution rate
Average resolution time
First response time
SLA compliance
Escalation rate
Recurring problems
Most common errors
Knowledge effectiveness
Tool failure rate

Reporting dashboards

Dashboard	Contents
Management	Total cases, AI resolved, human escalated, resolved, open, SLA compliance, SLA breaches, escalation count, average response, average resolution
Product	Support volume by product, problem types, recurring problems, AI resolution rate, human escalation rate, top errors
Support	Workload, agent assignments, SLA risk, escalations, response performance, resolution performance
AI	AI resolution rate, failed troubleshooting, knowledge match rate, confidence distribution, tool success/failure, human handoff rate

3. CI/CD (Jenkins)

Repository includes a Jenkinsfile implementing:

Checkout
→ Install
→ Environment validation
→ Typecheck
→ Lint
→ Format check
→ Unit test
→ Integration test
→ E2E test
→ Build
→ Docker build
→ Publish
→ Deploy

Production deployments use protected Jenkins credentials/environment variables — real secrets are never committed to the repository (see 02).

3.1 KiB Raw Blame History