thePlug/docs/backend-testing-strategy.md

Backend Testing Strategy

Goal

Replace the current mostly mocked backend suite with a professional layered strategy that:

• keeps fast unit tests
• adds realistic integration coverage with real app wiring
• adds an external performance/load harness
• minimizes monkeypatch use in behavior-level tests

This strategy is backend-only. Frontend automation is not part of this phase.

Current State

• Tooling:
  • pytest only
  • no async test helpers
  • no realistic HTTP/WebSocket client beyond TestClient
  • no containerized test infra
  • no benchmark/load tooling
• Problems:
  • custom FastAPI() test app in backend/tests/conftest.py
  • heavy monkeypatching of router dependencies
  • almost no real lifecycle/integration confidence
  • no system perf or throughput measurements

Target Test Pyramid

1. Unit tests

Purpose:

• pure logic only
• parser/validator/serialization helpers
• deterministic error mapping

Rules:

• monkeypatch allowed only at narrow technical seams
• no network, no Mongo, no Docker, no filesystem mutation beyond temp dirs

Examples:

• proxy code validation
• mode/protocol validation
• template metadata normalization
• health result shaping helpers

2. Integration tests

Purpose:

• real FastAPI app
• real startup/shutdown
• real Mongo-backed repositories
• realistic temp runtime directories
• real HTTP and WebSocket behavior

Recommended tooling:

• pytest-asyncio
• httpx
• asgi-lifespan
• testcontainers[mongodb]

Rules:

• use the actual app from app.main, not a bespoke test app
• override as little as possible
• prefer seeded fixtures over monkeypatching repositories
• temp runtime dirs are acceptable; fake repositories are not

Coverage to add:

• auth login/logout with real repository behavior
• proxy create/update/delete against real Mongo
• health history endpoints with persisted samples
• logs and health websocket auth and streaming
• startup reconciliation with persisted runtime state
• failure-path integration for runtime validation and cleanup

3. System tests

Purpose:

• realistic end-to-end control plane on a running stack
• optional and slower than integration tests

Scope:

• backend process
• Mongo container
• optional playground/docker service fixture
• real HTTP calls and websocket clients

Use for:

• reverse-mode service remap
• mitmdump orphan recovery
• runtime artifact cleanup
• template apply/save behavior at the API level

Fixture Architecture

Core fixtures

• mongo_container
  • real Mongo container started once per session
• app_settings
  • isolated env/config for test DB, auth credentials, runtime dirs
• real_app
  • imported from app.main
• lifespan_app
  • app with startup/shutdown active
• http_client
  • async httpx.AsyncClient
• auth_token
  • produced through real login endpoint
• runtime_dir
  • temp dir for scripts/config/logs
• seed_proxy
  • repository-backed fixture that inserts a real proxy document

Markers

• unit
• integration
• api
• websocket
• slow
• requires_docker
• perf

These should drive selective local and CI runs.

Monkeypatch Policy

Allowed:

• forcing subprocess failure
• forcing clock/timeouts
• fault injection into low-level boundaries
• temp path redirection where isolation is needed

Not the default:

• patching repository functions in router tests
• patching service functions in API behavior tests
• bypassing authentication for general API coverage

Rule of thumb:

• if the test is about endpoint behavior, use the real endpoint stack
• if the test is about pure logic, isolate it completely

Performance and Load Harness

Tool choice

Use k6 as the primary external load tool.

Reason:

• good HTTP load support
• scriptable scenarios
• usable for websocket-related smoke scenarios
• clearer reporting than ad-hoc shell loops

Perf layout

Add a dedicated non-default perf layer, for example under:

• backend/perf/

Include:

• startup script or instructions
• scenario scripts
• seeded data setup
• report capture format

Baseline scenarios

1. Read-heavy API latency

• GET /proxy
• GET /proxy/{port}/health/history
• measure latency percentiles and RPS

2. Control-plane mutation loop

• create/update/start/stop/delete proxy in repeated cycles
• measure failure rate and latency

3. WebSocket fan-out

• concurrent clients on logs and health streams
• verify stability under multiple listeners

4. Health monitor cost

• N running proxies with scheduled checks
• measure API latency degradation and monitor lag

Perf outputs

For each scenario record:

• concurrency
• duration
• warmup
• RPS
• p50/p95/p99 latency
• error rate
• environment notes

These should not be strict CI gates initially; they should establish baselines first.

Migration Plan

Phase 1

• Add pytest markers and split directories by layer.
• Introduce async test dependencies and real app client fixtures.
• Keep existing suite running.

Phase 2

• Convert router tests away from broad monkeypatching.
• Replace bespoke app fixture with real lifespan-backed app in integration tests.
• Add Mongo testcontainer fixture.

Phase 3

• Add websocket integration tests.
• Add startup/recovery/runtime integration tests.
• Shrink old unit/API tests where they duplicate new integration coverage.

Phase 4

• Introduce k6 perf scripts and documentation.
• Add baseline measurement workflow for local runs and CI nightly/optional jobs.

Acceptance Criteria

• Fast local unit suite remains available.
• Integration suite uses the real app and real Mongo.
• WebSocket paths are tested without bypassing core auth behavior.
• Router-level monkeypatching is no longer the dominant pattern.
• Reverse-mode lifecycle and recovery get realistic coverage.
• Perf/load scripts exist and produce repeatable latency/RPS baselines.