Testing Framework Overview

Relevant source files

• Jenkinsfile
• conftest.py
• openpilot/cereal/messaging/tests/test_services.py
• openpilot/common/hardware/tici/agnos.py
• openpilot/selfdrive/monitoring/policy.py
• openpilot/selfdrive/monitoring/test_monitoring.py

Purpose and Scope

This document describes the core testing infrastructure that underpins all openpilot tests. It covers the pytest-based framework, test isolation mechanisms using OpenpilotPrefix, fixture configurations in conftest.py, and the test environment setup used in both local development and CI/CD pipelines.

For specific testing subsystems, see:

• Hardware-in-the-loop testing on TICI devices: 5.3 Hardware Testing on Device
• Deterministic log replay testing: 5.4 Process Replay Testing
• Machine learning model validation: 5.5 Model Testing and Validation
• CI/CD pipeline orchestration: 5.6 CI/CD Pipeline

Core Testing Architecture

The openpilot testing architecture is designed for strict isolation to support parallel execution and deterministic results. It leverages pytest fixtures to manage a unique environment for every test case.

Natural Language to Code Entity Mapping: Testing Core

System Concept	Code Entity	File Path
Test Orchestrator	pytest	N/A (External)
Global Hooks	pytest_runtest_call	conftest.py25-33
IPC Isolation	OpenpilotPrefix	openpilot/common/prefix.py12-62
Environment Reset	clean_env	conftest.py36-41
Process Cleanup	manager_cleanup	openpilot/system/manager/manager.py
Hardware Init	initialize_hardware	openpilot/common/hardware/__init__.py

Architecture Diagram: Test Isolation Lifecycle

Sources: conftest.py44-96 openpilot/common/prefix.py12-62

pytest Framework Configuration

The testing framework is built on pytest with custom configurations and fixtures defined in conftest.py. The framework provides:

1. Test isolation through unique IPC prefixes per test.
2. Automatic fixture injection based on test markers.
3. Environment cleanup between tests.
4. Hardware-specific test routing (PC vs. TICI/Comma Three).

Global Configuration

Test collection filters exclude certain paths from automatic discovery, such as heavy CI-only process replay tests and simulation tools:

conftest.py11-16

The pytest_runtest_call hook ensures that tests marked with nocapture disable global output capture, allowing real-time logging to the console:

conftest.py25-33

Sources: conftest.py11-33

Test Isolation with OpenpilotPrefix

The OpenpilotPrefix class is the cornerstone of test isolation, providing unique namespaces for inter-process communication (IPC) and data storage.

Implementation Details

OpenpilotPrefix is a context manager that ensures every test has a private sandbox for its processes.

Key behaviors:

• Prefix generation: A random 15-character hex string is generated openpilot/common/prefix.py14
• Message queue path: Sets the directory for msgq IPC sockets to /dev/shm/msgq_{prefix} (Linux) or /tmp/msgq_{prefix} (macOS) openpilot/common/prefix.py15-16
• Environment variable: Exports OPENPILOT_PREFIX which is inherited by any processes spawned during the test openpilot/common/prefix.py23
• Shared cache support: If shared_download_cache is enabled, it points the download cache to the default root instead of a per-test directory openpilot/common/prefix.py28-29

The cleanup process removes all created resources upon exit, including shared memory segments and log directories:

openpilot/common/prefix.py51-62

Sources: openpilot/common/prefix.py12-62

Test Fixtures and Hooks

Core Fixtures

The conftest.py defines several autouse fixtures that run automatically for all tests:

openpilot_function_fixture

Applied to every test function, this fixture orchestrates the lifecycle:

1. Creates a clean environment via clean_env() conftest.py46
2. Instantiates OpenpilotPrefix conftest.py48
3. Yields control to the test code conftest.py51
4. Validates the prefix hasn't changed conftest.py54
5. Cleans up processes via manager.manager_cleanup() conftest.py57
6. Re-enables garbage collection (GC) if it was disabled by a process conftest.py60-62

conftest.py44-63

tici_setup_fixture

Automatically injected for tests marked with tici. It prepares the Comma Three (TICI) hardware:

• Initializes hardware state via HARDWARE.initialize_hardware() conftest.py77
• Disables power save mode conftest.py78
• Kills conflicting background daemons like athena conftest.py79

conftest.py72-80

Sources: conftest.py44-80

Test Markers and Parallel Execution

Test Marker System

Marker	Purpose	Effect
tici	Mark test as device-only	Automatically skipped on PC, adds tici_setup_fixture on device conftest.py86-90
nocapture	Disable output capture	Allows real-time output during test conftest.py28-31
shared_download_cache	Share download cache	Allows OpenpilotPrefix to reuse existing data conftest.py48
xdist_group_class_property	Group tests for parallel execution	Groups tests by a specific class property value for pytest-xdist conftest.py92-95

Sources: conftest.py25-95

Jenkins Device Testing

Testing on actual hardware is orchestrated by Jenkins. The Jenkinsfile defines how tests are deployed to a pool of devices.

Device Stage Execution Flow

Key Execution Parameters:

• Isolation: Tests on device use PYTEST_ADDOPTS="-n0 -s" to ensure single-threaded execution for Hardware-In-The-Loop (HIL) stability Jenkinsfile34
• Cleanup: Devices are prepared by clearing /dev/shm/* and /dev/tmp/tmp* before each stage Jenkinsfile45-46
• Selective Testing: Jenkins uses git diff to determine which tests to run based on changed file paths Jenkinsfile90-116

Sources: Jenkinsfile12-75 Jenkinsfile78-121

Example: Monitoring Policy Testing

The testing framework is used to validate complex state machines like DriverMonitoring.

Monitoring Test Flow: Code Entity Mapping

Key Implementation:

• Message Mocking: make_msg creates log.DriverStateV2 Cap'n Proto messages with specific probabilities for face detection and distraction openpilot/selfdrive/monitoring/test_monitoring.py14-27
• Sequence Simulation: _run_seq iterates through these messages, updating the DriverMonitoring state and verifying alert levels openpilot/selfdrive/monitoring/test_monitoring.py50-61

Sources: openpilot/selfdrive/monitoring/test_monitoring.py14-61 openpilot/selfdrive/monitoring/policy.py130-170