Main UI System

Relevant source files

• openpilot/selfdrive/controls/lib/longitudinal_mpc_lib/long_mpc.py
• openpilot/selfdrive/controls/radard.py
• openpilot/selfdrive/test/longitudinal_maneuvers/plant.py
• openpilot/selfdrive/ui/layouts/settings/software.py
• openpilot/selfdrive/ui/mici/layouts/settings/software.py
• openpilot/selfdrive/ui/mici/onroad/model_renderer.py
• openpilot/selfdrive/ui/onroad/model_renderer.py
• openpilot/tools/replay/lib/ui_helpers.py

This document covers openpilot's main user interface system, which provides a pyray (raylib) based graphical interface for both onroad driving and offroad settings management. The system handles rendering, device interaction, and state management across different UI modes.

For information about network and settings management, see Network and Settings Management. For details about the camera hardware layer, see Camera System.

Purpose and Scope

The Main UI System encompasses:

• GuiApplication framework: The gui_app singleton for window and rendering management openpilot/system/ui/lib/application.py196-254
• Touch input handling: A dedicated MouseState thread polling at 140Hz openpilot/system/ui/lib/application.py137-194
• Layout Orchestration: MainLayout for comma 3/3X and MiciMainLayout for comma 4 hardware.
• UIState and Device management: Centralized system state and hardware control openpilot/selfdrive/ui/ui_state.py26-205
• Onroad Visualization: Real-time rendering of model paths and lead vehicles via ModelRenderer.
• Software Management: Handling updates and branch switching via SoftwareLayout openpilot/selfdrive/ui/layouts/settings/software.py51-80 and SoftwareInfoLayoutMici openpilot/selfdrive/ui/mici/layouts/settings/software.py36-74

The UI runs at 60 FPS on TICI hardware and uses raylib for cross-platform graphics rendering openpilot/system/ui/lib/application.py25

System Architecture

The UI system operates as a pyray/raylib application that bridges openpilot's messaging system with user interaction. The architecture is built around a singleton GuiApplication instance that manages the rendering loop.

Core Entity Mapping

Sources: openpilot/system/ui/lib/application.py196-254 openpilot/selfdrive/ui/ui_state.py26-62 openpilot/selfdrive/ui/mici/onroad/model_renderer.py52-62 openpilot/selfdrive/ui/layouts/settings/software.py89-106

GuiApplication and Rendering Loop

The GuiApplication class manages window initialization, the rendering generator loop, and input handling.

• Window Setup: Configures raylib with MSAA 4x and platform-specific GL versions openpilot/system/ui/lib/application.py256-314
• Rendering Loop: A generator-based frame loop that handles frame timing and coordinate scaling openpilot/system/ui/lib/application.py455-537
• Asset Caching: Caches textures and fonts to prevent redundant I/O openpilot/system/ui/lib/application.py420-445
• Input System: MouseState tracks up to 2 touch slots openpilot/system/ui/lib/application.py30 It polls rl.get_touch_position(slot) and generates MouseEvent objects openpilot/system/ui/lib/application.py176-194

Input Processing Logic

Sources: openpilot/system/ui/lib/application.py137-194

ModelRenderer and Visualization

The ModelRenderer draws the predicted driving path, lane lines, and road edges projected from the model output into screen space.

• Data Sources: Subscribes to modelV2 for path geometry and radarState for lead vehicle tracking openpilot/selfdrive/ui/mici/onroad/model_renderer.py121-124
• Projection Logic: 3D points from the model are transformed into screen space via _car_space_transform openpilot/selfdrive/ui/onroad/model_renderer.py80-82
• Dynamic Path Coloring: In Experimental Mode, the path color is determined by the model's predicted acceleration openpilot/selfdrive/ui/mici/onroad/model_renderer.py161
• Lead Vehicles: LeadVehicle objects store glow and chevron data for rendering indicators over detected cars openpilot/selfdrive/ui/onroad/model_renderer.py39-42
• Calibration Integration: Uses liveCalibration to adjust the _path_offset_z based on camera height openpilot/selfdrive/ui/onroad/model_renderer.py101

Model Visualization Components

Component	Logic	File
ModelPoints	Stores raw 3D and projected 2D points	openpilot/selfdrive/ui/onroad/model_renderer.py33-36
LeadVehicle	Manages glow/chevron rendering state	openpilot/selfdrive/ui/onroad/model_renderer.py39-42
_update_raw_points	Extracts position/lane data from modelV2	openpilot/selfdrive/ui/mici/onroad/model_renderer.py149-161
_draw_path	Renders the driving path with gradients	openpilot/selfdrive/ui/onroad/model_renderer.py128

Sources: openpilot/selfdrive/ui/onroad/model_renderer.py45-132 openpilot/selfdrive/ui/mici/onroad/model_renderer.py52-148

Software and Updates

The UI provides a comprehensive interface for managing system updates and software configuration.

• Update Polling: SoftwareLayout monitors UpdaterState, UpdateAvailable, and UpdaterFetchAvailable params openpilot/selfdrive/ui/layouts/settings/software.py102-105
• Mici Info: SoftwareInfoLayoutMici splits the UpdaterCurrentDescription to display version, branch, commit, and date openpilot/selfdrive/ui/mici/layouts/settings/software.py21-27
• Update Triggers: The UI signals the updated daemon via Unix signals: SIGUSR1 for checking and SIGHUP for downloading openpilot/selfdrive/ui/layouts/settings/software.py161-166
• Time Validation: Update buttons are disabled or prompt for Wi-Fi if system time is invalid, as the updater requires valid TLS certificates openpilot/selfdrive/ui/mici/layouts/settings/software.py95-98

Sources: openpilot/selfdrive/ui/layouts/settings/software.py51-166 openpilot/selfdrive/ui/mici/layouts/settings/software.py76-184

UIState and Device Management

State management is centralized in the UIState and Device singletons.

• Messaging: UIState uses SubMaster to subscribe to core messages like carState, selfdriveState, and modelV2 openpilot/selfdrive/ui/ui_state.py37-62
• Transition Callbacks: Supports add_offroad_transition_callback to allow widgets to react when the car turns on/off openpilot/selfdrive/ui/mici/layouts/settings/software.py86-88
• Ignition Tracking: Monitors panda states to determine if the vehicle is started openpilot/selfdrive/ui/ui_state.py131-153

Sources: openpilot/selfdrive/ui/ui_state.py26-207