Log Upload and Data Pipeline

Relevant source files

• openpilot/system/loggerd/SConscript
• openpilot/system/loggerd/encoder/jpeg_encoder.cc
• openpilot/system/loggerd/encoder/jpeg_encoder.h
• openpilot/tools/jotpluggler/SConscript
• openpilot/tools/replay/SConscript
• tools/release/build_stripped.sh

The openpilot data pipeline is responsible for capturing high-bandwidth sensor data, segmenting it for reliability, and delivering it to comma's cloud infrastructure. This system ensures that driving data is preserved for model training and debugging while managing limited on-device storage and varying network conditions.

Loggerd Segment Rotation

loggerd is the primary C++ daemon responsible for writing cereal messages and encoded video to the device's storage. It organizes data into segments, which are typically 60-second chunks of a drive system/loggerd/loggerd.cc24

Segment Management and Rotation

Rotation is synchronized across multiple encoders to ensure that logs (rlog), low-resolution video (qcamera.ts), and high-resolution video (fcamera.hevc, ecamera.hevc, dcamera.hevc) all align to the same time boundaries.

• Triggering Rotation: Rotation occurs when all active encoders report they are ready for a new segment. The LoggerdState tracks this via ready_to_rotate system/loggerd/loggerd.cc32-54
• Fallback Logic: If an encoder or camera malfunctions, loggerd will force a rotation after a timeout or if the segment length exceeds standard limits system/loggerd/loggerd.cc42-50
• Initialization: Every new segment begins with initData, containing device metadata, kernel arguments, and system parameters system/loggerd/logger.cc15-94
• Sentinels: Segments are bookended by Sentinel messages indicating the start and end of routes or segments system/loggerd/logger.cc156-162
• Thumbnail Generation: loggerd utilizes JpegEncoder to generate MJPEG thumbnails from camera frames, which are published for the UI and uploaded as part of the drive summary system/loggerd/encoder/jpeg_encoder.cc11-62

Loggerd Data Flow

The following diagram illustrates how loggerd.cc receives encoded data and manages segment transitions.

Diagram: Loggerd Segment Orchestration

Sources: system/loggerd/loggerd.cc24-54 system/loggerd/loggerd.cc115-186 system/loggerd/logger.cc177-197 system/loggerd/encoder/jpeg_encoder.cc52-62 system/loggerd/SConscript1-20

---

Cloud Delivery with Uploader.py

The uploader.py daemon manages the background delivery of logged segments to the comma cloud via Azure Blob Storage.

Selection and Priority

The uploader selects files based on a priority queue to ensure critical small files are delivered before large video files.

• Immediate Folders: Files in crash/ or boot/ are prioritized.
• File Priority: qlog and qcamera have higher priority (lower integer value) than full-resolution .hevc files.
• Metered Connections: If the device is on a metered connection (cellular), the uploader restricts large file transfers and prioritizes recently viewed routes.

Upload Process

1. Request URL: The uploader calls the comma API to get a signed Azure Blob Storage URL.
2. Stream Data: It reads the file, potentially compressing it with zstd on the fly.
3. Tagging: Upon successful upload, the file is tagged using the extended attribute (xattr) user.upload set to b'1'. This signals to deleter.py that the file is safe to remove.

Table: Upload Priority and Constraints

File Type	Priority	Metered Policy	Max Size
qlog	0	12h delay	25 MB
qcamera.ts	1	Only if recently viewed	5 MB
*.hevc	1000	WiFi Only (default)	No Limit

---

Storage Management: Deleter.py

deleter.py ensures the device does not run out of storage by deleting old segments.

• Deletion Criteria: Files are eligible for deletion if they have the user.upload xattr set.
• Preservation: Files tagged with user.preserve are skipped by the deleter to allow users to save specific drives.
• Safety: The deleter avoids deleting active segments by checking for .lock files created by loggerd.

---

Content-Addressable Sync (casync)

For Over-The-Air (OTA) updates and release distribution, openpilot utilizes a casync inspired system. This provides efficient delta updates by breaking the system image into variable-sized chunks indexed by their SHA-256 hash.

• Release Preparation: The build_stripped.sh script prepares the repository for release by removing submodules, cleaning the environment, and chunking large model files (e.g., .onnx files > 95MB) to stay within GitHub limits tools/release/build_stripped.sh29-48
• Chunking: Large files are split using file_chunker.py to facilitate efficient synchronization and storage tools/release/build_stripped.sh48
• Content-Addressable Storage: The system ensures that only changed chunks are downloaded during an update, significantly reducing OTA bandwidth.

Sources: tools/release/build_stripped.sh29-87

---

Log Analysis and Replay Tools

The data pipeline extends to development tools that consume the logs produced by loggerd.

• Replay System: The replay tool (C++) allows developers to stream logged data back through the openpilot process stack. It uses LogReader, FrameReader, and SegmentManager to reconstruct the driving environment openpilot/tools/replay/SConscript9-16
• JotPluggler: A specialized visualization tool that extracts events from logs using generated extractors openpilot/tools/jotpluggler/SConscript83-115 It maps car fingerprints to specific DBC files to decode CAN traffic from logs openpilot/tools/jotpluggler/SConscript40-79

Diagram: Log Consumption Pipeline

Sources: openpilot/tools/replay/SConscript1-16 openpilot/tools/jotpluggler/SConscript40-115

---

Athena Integration

While uploader.py handles bulk data, athenad.py provides a WebSocket-based JSON-RPC interface for real-time cloud interaction and user-requested uploads.

• Upload Queue: athenad maintains its own upload_queue for specific files requested via the comma app.
• Priority Management: UploadItem objects are stored in a PriorityQueue, where a lower priority number indicates higher urgency.
• Persistence: The queue is cached in system parameters to survive reboots.
• Network Quality of Service: Athena uses specific Type of Service (ToS) bits for uploads and SSH to prioritize interactive traffic over background data delivery.

Diagram: Athena Upload Pipeline