Log Analysis and Debugging Tools

Relevant source files

• .gitignore
• launch_chffrplus.sh
• openpilot/selfdrive/test/setup_device_ci.sh
• openpilot/tools/cabana/SConscript
• openpilot/tools/jotpluggler/layout.cc
• openpilot/tools/jotpluggler/sketch_layout.cc
• openpilot/tools/plotjuggler/juggle.py
• openpilot/tools/replay/ui.py
• tools/scripts/count_events.py
• tools/scripts/dump.py
• tools/scripts/filter_log_message.py
• tools/scripts/live_cpu_and_temp.py
• tools/scripts/mem_usage.py

This document describes the tools and frameworks used to analyze, debug, and validate openpilot behavior through log inspection and replay. These tools enable developers to investigate issues, validate changes, and perform regression testing using recorded driving data.

For information about the Cabana CAN analysis GUI tool, see Cabana CAN Analysis Tool. For information about the broader testing framework, see Testing Framework Overview and Process Replay Testing.

---

Overview

The openpilot debugging toolkit provides several complementary capabilities for inspecting both real-time and recorded data:

Tool	Purpose	Use Case
dump.py	Real-time message inspection	Live debugging of running processes and sockets
LogReader	Log access API	Python interface for reading rlog/qlog files
replay	C++ route playback	Simulating driving sessions with full UI/video
juggle.py	Time-series visualization	Plotting signals from logs using PlotJuggler
jotpluggler	Next-gen visualization	High-performance C++ based time-series plotting
filter_log_message.py	Log filtering	Isolating specific logMessage or errorLogMessage entries

Sources: tools/scripts/dump.py28-37 tools/plotjuggler/juggle.py132-143 tools/scripts/filter_log_message.py48-53 openpilot/tools/jotpluggler/sketch_layout.cc34-43

---

Real-Time Inspection Tools

dump.py Utility

The dump.py script provides real-time inspection of cereal messages flowing through the messaging infrastructure. It uses raw_live_logreader to subscribe to ZMQ/MSGQ sockets and can output data in various formats. tools/scripts/dump.py9-10 tools/scripts/dump.py39-45

• Hexdump: View raw message bytes via hexdump(msg). tools/scripts/dump.py14-23
• JSON Output: Use --dump-json to export the full Cap'n Proto event as a dictionary via evt.to_dict(). tools/scripts/dump.py55-56
• Value Filtering: The --values flag allows monitoring specific nested fields (e.g., carState.vEgo) instead of the entire event. tools/scripts/dump.py57-64

filter_log_message.py

This utility specializes in extracting human-readable logs produced by SwagLogger. It filters logMessage, errorLogMessage, and operatingSystemLog services. tools/scripts/filter_log_message.py64-69

• Level Filtering: Supports filtering by severity (DEBUG, INFO, WARNING, ERROR, CRITICAL). tools/scripts/filter_log_message.py8-14
• Offline Support: Can read directly from a route or segment using LogReader. tools/scripts/filter_log_message.py57-60
• OS Logs: Decodes operatingSystemLog to identify the source (MAIN, KERNEL, SYSTEM, etc.). tools/scripts/filter_log_message.py16-23

Sources: tools/scripts/dump.py1-75 tools/scripts/filter_log_message.py1-80

---

Visualization and Analysis

PlotJuggler and JotPluggler

juggle.py is the current wrapper for PlotJuggler, including an automated installation script that downloads prebuilt binaries for Linux and macOS. tools/plotjuggler/juggle.py43-62

• DBC Inference: Automatically detects the car platform from carParams and loads the corresponding DBC via generate_dbc_dict. tools/plotjuggler/juggle.py117-124
• Log Migration: Ensures compatibility with the current cereal schema by running migrate_all on log data before loading. tools/plotjuggler/juggle.py112-113

JotPluggler is the high-performance C++ successor. It features:

• Route Loading: Uses LogReader and py_downloader.h to fetch segments. openpilot/tools/jotpluggler/sketch_layout.cc27-28
• Dynamic Extractors: Uses generated_event_extractors.h for efficient Cap'n Proto field extraction. openpilot/tools/jotpluggler/sketch_layout.cc94
• Live Streaming: Supports CerealLocal (MSGQ) and CerealRemote (ZMQ) sources. openpilot/tools/jotpluggler/layout.cc26-36

Title: JotPluggler Architecture

Sources: tools/plotjuggler/juggle.py1-170 openpilot/tools/jotpluggler/sketch_layout.cc34-106 openpilot/tools/jotpluggler/layout.cc118-153

---

Route Replay and Debug UI

The replay system is complemented by a Python-based ui.py for debugging specific control signals and model outputs.

Debug UI (replay/ui.py)

This tool provides a multi-pane visualization using pyray (Raylib) to display:

• Camera View: Renders the road camera stream via CameraView. openpilot/tools/replay/ui.py60
• Top-Down View: Displays radar points and model paths in a bird's-eye perspective. openpilot/tools/replay/ui.py68-71
• Time-Series Plots: Real-time plotting of v_ego, steeringAngleDeg, gas, and accel actuators. openpilot/tools/replay/ui.py96-127

Implementation Details

The UI thread maintains a SubMaster to synchronize messages including carState, longitudinalPlan, carControl, and modelV2. openpilot/tools/replay/ui.py73-88 It uses a Calibration class to project 3D model paths onto the 2D camera frame. openpilot/tools/replay/ui.py200-202

Sources: openpilot/tools/replay/ui.py1-208 openpilot/tools/replay/SConscript1-108

---

System Health Monitoring

live_cpu_and_temp.py

Monitors hardware performance by subscribing to deviceState and procLog. tools/scripts/live_cpu_and_temp.py37

• CPU Tracking: Calculates per-core utilization by comparing cpuTimes across updates. tools/scripts/live_cpu_and_temp.py58-75
• Process Analysis: When run with --cpu or --mem, it identifies the top 10 processes by resource consumption, resolving names via proc_name. tools/scripts/live_cpu_and_temp.py77-103

count_events.py

Analyzes a qlog to provide a summary of a drive's events.

• Event Statistics: Counts occurrences of onroadEvents (e.g., lane departures, FCW). tools/scripts/count_events.py29-35
• Camera Health: Validates camera frame rates by comparing received frames against expected frequencies in SERVICE_LIST. tools/scripts/count_events.py62-66
• Alert Timeline: Logs the timestamp and type of every selfdriveState alert. tools/scripts/count_events.py37-41

Title: Debug Script Data Sources

Sources: tools/scripts/live_cpu_and_temp.py1-107 tools/scripts/count_events.py1-78 tools/scripts/mem_usage.py