Development and Testing

Relevant source files

• .github/workflows/docs.yaml
• .github/workflows/release.yaml
• .github/workflows/repo-maintenance.yaml
• .github/workflows/tests.yaml
• Jenkinsfile
• SConstruct
• conftest.py
• pyproject.toml
• tools/setup_dependencies.sh

This page provides an architectural overview of openpilot's development and testing infrastructure. It covers the build system, testing philosophy, multi-layered validation frameworks (unit tests, process replay, integration tests), and the hybrid CI/CD pipelines that ensure code quality across PC environments and physical device hardware.

For specific details on these topics, see the following child pages:

• Build System and Environment Setup — SCons configuration, C++/Cython compilation in SConstruct SConstruct1-168 and platform-specific logic for TICI (comma 3/3X) and PC.
• Testing Framework Overview — Pytest-based infrastructure, OpenpilotPrefix for IPC isolation conftest.py43-52 and parallel execution using pytest-xdist pyproject.toml128-142
• Hardware Testing on Device — System integration testing via test_onroad.py Jenkinsfile205-211 power draw measurements, and real-time timing requirements on TICI hardware.
• Process Replay Testing — Deterministic regression testing that compares current execution outputs against reference logs pyproject.toml114-130
• Model Testing and Validation — Machine learning model validation, checking execution timing and output consistency.
• CI/CD Pipeline — Orchestration between GitHub Actions for software tasks .github/workflows/tests.yaml1-112 and Jenkins for hardware-in-the-loop (HIL) testing Jenkinsfile161-214
• Translation Management — Pipeline for managing internationalization, including extraction and validation.
• Simulation and Car Porting Tools — MetaDrive simulation bridge pyproject.toml98 car fingerprinting, and longitudinal maneuver testing.

Testing Philosophy

openpilot employs a multi-layered testing strategy that prioritizes determinism and reproducibility. The core of this strategy is the ability to replay real-world driving data through individual processes in isolation. This allows for regression detection and fast iteration on PC hardware without requiring physical vehicles.

The testing pyramid consists of:

• Unit tests: Fast, isolated tests of individual functions using pytest with custom fixtures in conftest.py conftest.py39-66
• Process replay tests: Deterministic validation of control and model processes using recorded logs to detect logic regressions pyproject.toml114-130
• Integration tests: Full-system validation measuring CPU, memory, and timing on device hardware via test_onroad.py Jenkinsfile209
• Hardware-in-the-loop (HIL) tests: On-device testing on actual comma hardware orchestrated by Jenkins to ensure performance requirements like power draw are met Jenkinsfile92-121

Sources: conftest.py39-66 Jenkinsfile12-74 pyproject.toml126-142 .github/workflows/tests.yaml92-113

Development Infrastructure

The development environment is managed through a unified setup process that handles system and Python dependencies via uv pyproject.toml208-223

Environment Management Diagram

Sources: tools/setup_dependencies.sh92-113 SConstruct121-168 pyproject.toml152-195

Build System: openpilot uses SCons for compiling C++ and Cython modules SConstruct163-168 The build system enforces a whitelist of allowed system libraries in allowed_system_libs to prevent hidden dependencies SConstruct88-103 It manages complex dependencies like acados for MPC solvers and ffmpeg for video processing SConstruct58-81

Environment Isolation: To allow parallel testing without Inter-Process Communication (IPC) interference, openpilot uses the OpenpilotPrefix context manager conftest.py48-55 This creates isolated messaging namespaces for each test session by modifying the OPENPILOT_PREFIX environment variable.

Sources: SConstruct1-168 conftest.py44-63 pyproject.toml1-103 tools/setup_dependencies.sh117-132

CI/CD Pipeline

The CI/CD infrastructure is split between GitHub Actions for lightweight tasks and Jenkins for hardware-specific testing.

Hardware and Performance Metrics

The system monitors key performance indicators during on-device testing. Failure to meet these thresholds results in a CI failure.

Metric Category	Purpose	Code Reference
Build Integrity	Ensure the release branch builds and is not "dirty"	Jenkinsfile207-208
Onroad Integration	Validate system-wide process interaction on hardware	Jenkinsfile209
Unit Testing	Execute hardware-specific unit tests (TICI markers)	conftest.py83-91
Static Analysis	Run ruff and scripts/lint/lint.sh	.github/workflows/tests.yaml77-90

CI Device Orchestration Diagram

Sources: Jenkinsfile12-76 Jenkinsfile92-121 conftest.py72-81

Jenkins Integration: Jenkins manages a pool of physical devices. It uses a lock() mechanism to acquire a device and ssh to execute commands via the device() helper function Jenkinsfile12-76 It provisions the environment using setup_device_ci.sh before running tests like test_onroad.py Jenkinsfile98-113

GitHub Actions: Handles repo maintenance such as updating Python packages via uv lock --upgrade .github/workflows/repo-maintenance.yaml25-26 building documentation .github/workflows/docs.yaml19-34 and running software unit tests .github/workflows/tests.yaml92-113

Sources: Jenkinsfile12-157 .github/workflows/tests.yaml1-113 .github/workflows/repo-maintenance.yaml14-78