CI/CD Pipeline

This document describes openpilot's continuous integration and continuous deployment (CI/CD) pipeline, covering the automated testing, building, and validation processes that ensure code quality and system reliability. The pipeline combines GitHub Actions for software testing with Jenkins for hardware-in-the-loop (HIL) validation on physical devices.

For information about the testing framework itself, see Testing Framework Overview. For details about model validation specifically, see Model Testing and Validation.

Overview

openpilot's CI/CD pipeline consists of two primary systems working in tandem: GitHub Actions handles software-based testing and builds, while Jenkins manages hardware testing on physical devices. This dual approach ensures that changes are validated both in simulated environments and on actual hardware before integration.

High-Level Orchestration

Sources: .github/workflows/tests.yaml1-137 Jenkinsfile161-215 .github/workflows/jenkins-pr-trigger.yaml58-70

GitHub Actions Workflows

The GitHub Actions infrastructure handles lightweight validation, repository maintenance, and software-only testing.

Core Testing Workflow

The main tests.yaml workflow triggers on pushes to master and pull requests .github/workflows/tests.yaml3-9 It executes parallel jobs including:

• Static Analysis: Runs scripts/lint/lint.sh to ensure code style and catch common errors .github/workflows/tests.yaml77-91
• Unit Tests: Executes pytest with RAYLIB_BACKEND=headless for UI-related components .github/workflows/tests.yaml92-113
• Process Replay: Runs test_processes.py to ensure deterministic behavior of core daemons against reference logs, uploading diff reports to the ci-artifacts repository for visualization .github/workflows/tests.yaml114-177
• Build Validation: Verifies that openpilot compiles on Linux and macOS .github/workflows/tests.yaml26-75

Specialized Reviews and Maintenance

• UI Preview: Generates video replays of the UI for PRs affecting assets or UI code, comparing them against master and posting a diff report .github/workflows/ui_preview.yaml1-133
• Model Review: Automatically triggers when .onnx model files are modified, running scripts/reporter.py to compare performance against master .github/workflows/model_review.yaml1-42
• Repo Maintenance: Schedules weekly tasks to upgrade Python packages via uv lock, update car documentation via openpilot/selfdrive/car/docs.py, and clean up closed PR branches .github/workflows/repo-maintenance.yaml1-118
• Release: Manages the creation of __nightly and master-ci branches, waiting for green CI status before pushing stripped builds .github/workflows/release.yaml1-32

For details, see GitHub Actions Workflows.

Sources: .github/workflows/tests.yaml1-183 .github/workflows/ui_preview.yaml1-133 .github/workflows/repo-maintenance.yaml1-118 .github/workflows/model_review.yaml1-42 .github/workflows/release.yaml1-32

Jenkins Device Testing

Jenkins is used for Hardware-in-the-Loop (HIL) testing on physical comma devices, coordinated via a Groovy-based Jenkinsfile.

Device Orchestration

The pipeline uses a lock mechanism to ensure exclusive access to a device IP from the pool .Jenkinsfile92 The device() function establishes an SSH connection to the device (running as comma@${ip}) to execute commands .Jenkinsfile12-75 Before running tests, the system executes setup_device_ci.sh to provision the repository and clean the environment (e.g., clearing /dev/shm/* and restarting systemd-resolved) .Jenkinsfile40-60

Hardware-Specific Validation

The pipeline executes tests that require physical hardware sensors or specific SoC performance:

• Onroad Tests: Validates the full system stack on device hardware, running pytest openpilot/selfdrive/test/test_onroad.py to check process health and timing .Jenkinsfile205-211
• Conditional Execution: To optimize the device pool, the hasPathChanged() function checks if modified files in a PR overlap with defined diffPaths for a test stage .Jenkinsfile108-130
• Release Builds: Jenkins handles building specialized release branches like release-tizi-staging and release-mici-staging using tools/release/build_release.sh .Jenkinsfile182-186

Code-to-System Mapping

For details, see Jenkins Device Testing.

Sources: Jenkinsfile12-130 Jenkinsfile182-211 conftest.py44-60 .github/workflows/jenkins-pr-trigger.yaml1-97