thePlug/docs/backend-testing-strategy.md

# Backend Testing Strategy

## Goal
Replace the current mostly mocked backend suite with a professional layered strategy that:
- keeps fast unit tests
- adds realistic integration coverage with real app wiring
- adds an external performance/load harness
- minimizes monkeypatch use in behavior-level tests

This strategy is backend-only. Frontend automation is not part of this phase.

## Current State
- Tooling:
  - pytest only
  - no async test helpers
  - no realistic HTTP/WebSocket client beyond `TestClient`
  - no containerized test infra
  - no benchmark/load tooling
- Problems:
  - custom `FastAPI()` test app in `backend/tests/conftest.py`
  - heavy monkeypatching of router dependencies
  - almost no real lifecycle/integration confidence
  - no system perf or throughput measurements

## Target Test Pyramid

### 1. Unit tests
Purpose:
- pure logic only
- parser/validator/serialization helpers
- deterministic error mapping

Rules:
- monkeypatch allowed only at narrow technical seams
- no network, no Mongo, no Docker, no filesystem mutation beyond temp dirs

Examples:
- proxy code validation
- mode/protocol validation
- template metadata normalization
- health result shaping helpers

### 2. Integration tests
Purpose:
- real FastAPI app
- real startup/shutdown
- real Mongo-backed repositories
- realistic temp runtime directories
- real HTTP and WebSocket behavior

Recommended tooling:
- `pytest-asyncio`
- `httpx`
- `asgi-lifespan`
- `testcontainers[mongodb]`

Rules:
- use the actual app from `app.main`, not a bespoke test app
- override as little as possible
- prefer seeded fixtures over monkeypatching repositories
- temp runtime dirs are acceptable; fake repositories are not

Coverage to add:
- auth login/logout with real repository behavior
- proxy create/update/delete against real Mongo
- health history endpoints with persisted samples
- logs and health websocket auth and streaming
- startup reconciliation with persisted runtime state
- failure-path integration for runtime validation and cleanup

### 3. System tests
Purpose:
- realistic end-to-end control plane on a running stack
- optional and slower than integration tests

Scope:
- backend process
- Mongo container
- optional playground/docker service fixture
- real HTTP calls and websocket clients

Use for:
- reverse-mode service remap
- mitmdump orphan recovery
- runtime artifact cleanup
- template apply/save behavior at the API level

## Fixture Architecture

### Core fixtures
- `mongo_container`
  - real Mongo container started once per session
- `app_settings`
  - isolated env/config for test DB, auth credentials, runtime dirs
- `real_app`
  - imported from `app.main`
- `lifespan_app`
  - app with startup/shutdown active
- `http_client`
  - async `httpx.AsyncClient`
- `auth_token`
  - produced through real login endpoint
- `runtime_dir`
  - temp dir for scripts/config/logs
- `seed_proxy`
  - repository-backed fixture that inserts a real proxy document

### Markers
- `unit`
- `integration`
- `api`
- `websocket`
- `slow`
- `requires_docker`
- `perf`

These should drive selective local and CI runs.

## Monkeypatch Policy

Allowed:
- forcing subprocess failure
- forcing clock/timeouts
- fault injection into low-level boundaries
- temp path redirection where isolation is needed

Not the default:
- patching repository functions in router tests
- patching service functions in API behavior tests
- bypassing authentication for general API coverage

Rule of thumb:
- if the test is about endpoint behavior, use the real endpoint stack
- if the test is about pure logic, isolate it completely

## Performance and Load Harness

### Tool choice
Use `k6` as the primary external load tool.

Reason:
- good HTTP load support
- scriptable scenarios
- usable for websocket-related smoke scenarios
- clearer reporting than ad-hoc shell loops

### Perf layout
Add a dedicated non-default perf layer, for example under:
- `backend/perf/`

Include:
- startup script or instructions
- scenario scripts
- seeded data setup
- report capture format

### Baseline scenarios
1. Read-heavy API latency
- `GET /proxy`
- `GET /proxy/{port}/health/history`
- measure latency percentiles and RPS

2. Control-plane mutation loop
- create/update/start/stop/delete proxy in repeated cycles
- measure failure rate and latency

3. WebSocket fan-out
- concurrent clients on logs and health streams
- verify stability under multiple listeners

4. Health monitor cost
- N running proxies with scheduled checks
- measure API latency degradation and monitor lag

### Perf outputs
For each scenario record:
- concurrency
- duration
- warmup
- RPS
- p50/p95/p99 latency
- error rate
- environment notes

These should not be strict CI gates initially; they should establish baselines first.

## Migration Plan

### Phase 1
- Add pytest markers and split directories by layer.
- Introduce async test dependencies and real app client fixtures.
- Keep existing suite running.

### Phase 2
- Convert router tests away from broad monkeypatching.
- Replace bespoke app fixture with real lifespan-backed app in integration tests.
- Add Mongo testcontainer fixture.

### Phase 3
- Add websocket integration tests.
- Add startup/recovery/runtime integration tests.
- Shrink old unit/API tests where they duplicate new integration coverage.

### Phase 4
- Introduce `k6` perf scripts and documentation.
- Add baseline measurement workflow for local runs and CI nightly/optional jobs.

## Acceptance Criteria
- Fast local unit suite remains available.
- Integration suite uses the real app and real Mongo.
- WebSocket paths are tested without bypassing core auth behavior.
- Router-level monkeypatching is no longer the dominant pattern.
- Reverse-mode lifecycle and recovery get realistic coverage.
- Perf/load scripts exist and produce repeatable latency/RPS baselines.