WebRTC and Body Teleop

Relevant source files

• openpilot/system/webrtc/webrtcd.py

This page documents the WebRTC infrastructure in openpilot, specifically the webrtcd daemon and its integration with the "comma body" robot for teleoperation. The system enables low-latency, bidirectional streaming of video, audio, and cereal messages between an openpilot device and a web browser.

Overview of webrtcd

webrtcd is an asyncio-based daemon that serves as a WebRTC gateway. It bridges openpilot's internal msgq messaging system (cereal) and hardware video encoders to a standard WebRTC PeerConnection.

The daemon handles:

1. SDP Negotiation: Processing Session Description Protocol (SDP) offers from clients to establish connections system/webrtc/webrtcd.py270-285
2. Video Streaming: Wrapping hardware-encoded video streams (H.264) into WebRTC video tracks system/webrtc/device/video.py11-46
3. Data Bridging: Proxying cereal messages between the device's PubMaster/SubMaster and WebRTC DataChannels system/webrtc/webrtcd.py84-154
4. Bitrate Control: Dynamically adjusting the video encoder bitrate based on network packet loss system/webrtc/webrtcd.py168-218

Data Flow Architecture

The following diagram illustrates the flow of video data from the cameras and control messages from the browser to the robot's actuators.

WebRTC Teleop Data Flow

Sources: system/webrtc/webrtcd.py84-218 system/webrtc/device/video.py11-46 tools/bodyteleop/static/js/webrtc.js111-115

Cereal Message Proxying

To allow the browser to interact with the vehicle/robot, webrtcd provides bidirectional proxying of cereal messages.

Outgoing Proxy (Device to Browser)

The CerealOutgoingMessageProxy class monitors specific cereal services using a messaging.SubMaster system/webrtc/webrtcd.py84-88 When new messages arrive, it converts the Cap'n Proto structures to JSON and broadcasts them over the WebRTC RTCDataChannel system/webrtc/webrtcd.py110-121

• Key Function: to_json() recursively converts capnp._DynamicStructReader and bytes into JSON-serializable formats system/webrtc/webrtcd.py98-108
• Execution: Managed as an AsyncTaskRunner, running an async loop calling update() every 10ms system/webrtc/webrtcd.py123-138

Incoming Proxy (Browser to Device)

The CerealIncomingMessageProxy receives JSON strings from the browser, parses them, and publishes them to the local msgq bus using a messaging.PubMaster system/webrtc/webrtcd.py140-154

• Dynamic Publishing: Uses DynamicPubMaster to lazily create publisher sockets for services requested by the client session system/webrtc/webrtcd.py156-166

Sources: system/webrtc/webrtcd.py84-166

Video Streaming and Bitrate Control

Video Tracks

Video tracks are implemented in LiveStreamVideoStreamTrack, which inherits from teleoprtc.tracks.TiciVideoStreamTrack system/webrtc/device/video.py11 It maps camera types (driver, wideRoad, road) to corresponding "livestream" cereal sockets system/webrtc/device/video.py12-16 The recv() method extracts the header and data from the cereal message to create an av.Packet system/webrtc/device/video.py35-42

Bitrate Controller

The LivestreamBitrateController monitors network conditions via peer_connection.getStats() system/webrtc/webrtcd.py220 It implements a three-tier bitrate system (Low: 500kbps, Med: 1.5Mbps, High: 5Mbps) system/webrtc/webrtcd.py169-170

• Downscaling: If loss_rate exceeds high_level (0.1) or stays above med_level (0.05) for 1 second, it drops the bitrate tier and doubles the recovery time (exponential backoff) system/webrtc/webrtcd.py205-211
• Upscaling: If loss_rate is 0 for a sufficient interval, it increases the bitrate tier system/webrtc/webrtcd.py212-217
• Persistence: Changes are written to the LivestreamEncoderBitrate param system/webrtc/webrtcd.py176

Sources: system/webrtc/webrtcd.py168-243 system/webrtc/device/video.py11-46

Body Teleop Interface

The bodyteleop tool provides a web interface for remote control of a comma body robot.

Backend (web.py)

The backend uses aiohttp to serve the static UI and act as a signaling proxy for webrtcd tools/bodyteleop/web.py8-12

• Offer Endpoint: Receives an SDP offer from the browser and forwards it to webrtcd with requested services: driver camera, testJoystick (incoming), and carState (outgoing) tools/bodyteleop/web.py57-67
• SSL: Automatically generates self-signed certificates via openssl command, as WebRTC requires HTTPS in browsers tools/bodyteleop/web.py23-44

Frontend (JavaScript)

• Negotiation: Uses RTCPeerConnection to create an offer, sends it to the /offer endpoint, and sets the remote description from the returned answer tools/bodyteleop/static/js/webrtc.js42-71
• Controls: Captures WASD key states in controls.js tools/bodyteleop/static/js/controls.js1-20 and sends them as testJoystick messages over the RTCDataChannel every 50ms tools/bodyteleop/static/js/webrtc.js111-135
• Telemetry: Receives carState messages to update battery level and ping time visualizations using Chart.js tools/bodyteleop/static/js/webrtc.js139-154

Sources: tools/bodyteleop/web.py1-82 tools/bodyteleop/static/js/webrtc.js1-176 tools/bodyteleop/static/js/controls.js1-20

Code Entity Map

System Entity Mapping

Sources: system/webrtc/webrtcd.py245-285 system/webrtc/device/video.py11 tools/bodyteleop/web.py57-67 tools/bodyteleop/static/js/webrtc.js42-71