Streams API

Relevant source files

• ext/fetch/22_body.js
• ext/web/00_url.js
• ext/web/06_streams.js
• ext/webidl/00_webidl.js
• ext/webidl/internal.d.ts
• tests/unit/request_test.ts
• tests/unit/streams_test.ts
• tests/unit/structured_clone_test.ts
• tests/wpt/runner/expectations/FileAPI.json
• tests/wpt/runner/expectations/compression.json
• tests/wpt/runner/expectations/dom.json
• tests/wpt/runner/expectations/encoding.json
• tests/wpt/runner/expectations/fetch.json
• tests/wpt/runner/expectations/html.json
• tests/wpt/runner/expectations/mimesniff.json
• tests/wpt/runner/expectations/streams.json
• tests/wpt/runner/expectations/url.json
• tests/wpt/runner/expectations/urlpattern.json
• tests/wpt/runner/expectations/user-timing.json
• tests/wpt/runner/expectations/wasm.json
• tests/wpt/runner/expectations/workers.json
• tests/wpt/runner/expectations/xhr.json

This page documents Deno's implementation of the Web Streams API standard, which provides interfaces for creating, composing, and consuming streams of data. Streams enable efficient handling of large data sets by processing data incrementally rather than loading everything into memory.

For information about HTTP-specific streaming (fetch requests/responses), see HTTP, Fetch, and WebSocket. For file-based streaming operations, see File System Operations.

---

Overview and Purpose

The Streams API implementation in Deno provides three primary stream types aligned with the WHATWG Streams Standard:

• ReadableStream: Represents a source of streaming data that can be read sequentially.
• WritableStream: Represents a destination for streaming data that can be written sequentially.
• TransformStream: Provides a readable/writable pair for transforming data as it flows through.

Deno extends the standard with resource-backed streams that bridge JavaScript streams to Rust resources (file handles, network sockets, etc.) for efficient I/O operations.

Sources: ext/web/06_streams.js1-6564

---

Core Architecture

Diagram: Streams API Component Architecture

The architecture separates concerns into distinct layers:

1. User-facing classes (ReadableStream, WritableStream, TransformStream).
2. Access interfaces (readers and writers).
3. Internal controllers that manage state and queuing.
4. Resource integration for efficient native I/O.

Sources: ext/web/06_streams.js3747-6564 ext/web/06_streams.js896-1212

---

ReadableStream Implementation

Class Structure

The ReadableStream class provides methods for reading data incrementally. Key internal symbols:

• [_controller] - References the underlying controller ext/web/06_streams.js347
• [_reader] - References the currently locked reader (if any) ext/web/06_streams.js358
• [_state] - Stream state: "readable", "closed", or "errored" ext/web/06_streams.js361
• [_disturbed] - Tracks whether the stream has been consumed ext/web/06_streams.js348
• [_storedError] - Stores error if stream errored ext/web/06_streams.js362

ext/web/06_streams.js3747-4110

Stream Readers

Diagram: ReadableStream Reader Access Patterns

• ReadableStreamDefaultReader: Used for standard chunk-by-chunk reading ext/web/06_streams.js4112-4312
• ReadableStreamBYOBReader: Allows reading directly into user-provided buffers (Bring Your Own Buffer) for zero-copy operations ext/web/06_streams.js4315-4590

Sources: ext/web/06_streams.js4112-4590

Controllers

ReadableStreamDefaultController

Manages streams with arbitrary chunk values:

Key methods:

• enqueue(chunk) - Add chunk to internal queue ext/web/06_streams.js4645
• close() - Signal no more data ext/web/06_streams.js4630
• error(e) - Error the stream ext/web/06_streams.js4660

ext/web/06_streams.js4593-4896

ReadableByteStreamController

Optimized for byte streams with BYOB reader support:

This controller enables efficient zero-copy reading by allowing the underlying source to write directly into buffers provided by readers.

ext/web/06_streams.js4899-5382

Sources: ext/web/06_streams.js4593-5382

---

WritableStream Implementation

Class Structure

The WritableStream class represents a destination for writing data:

• [_state] - One of: "writable", "closed", "erroring", "errored" ext/web/06_streams.js361
• [_writer] - Currently locked writer (if any) ext/web/06_streams.js364
• [_controller] - WritableStreamDefaultController instance ext/web/06_streams.js347
• [_backpressure] - Boolean indicating if writes should slow down ext/web/06_streams.js344
• [_writeRequests] - Queue of pending write promises ext/web/06_streams.js365

ext/web/06_streams.js5385-5635

Writer and Operations

Diagram: WritableStream Backpressure Mechanism

The ready promise on WritableStreamDefaultWriter provides backpressure signaling. It remains pending when the stream's queue exceeds the high water mark and resolves when space is available ext/web/06_streams.js5710

Sources: ext/web/06_streams.js5638-5856 ext/web/06_streams.js5859-6172

---

TransformStream Implementation

Structure and Data Flow

Diagram: TransformStream Data Pipeline

A TransformStream consists of:

• A writable side that accepts input ext/web/06_streams.js6214
• A readable side that produces output ext/web/06_streams.js6213
• A transformer that processes chunks from writable to readable.
• Backpressure propagation between both sides ext/web/06_streams.js6233

ext/web/06_streams.js6175-6564

Transformer Interface

When creating a TransformStream, you provide a transformer object:

The TransformStreamDefaultController provides methods:

• enqueue(chunk) - Send transformed chunk to readable side ext/web/06_streams.js6400
• terminate() - Close both sides ext/web/06_streams.js6418
• error(reason) - Error both sides ext/web/06_streams.js6410

ext/web/06_streams.js6361-6564

Sources: ext/web/06_streams.js6175-6564

---

Resource-Backed Streams

Deno extends the standard Streams API with resource-backed implementations for efficient I/O with native resources.

Creating Resource-Backed Streams

Diagram: Resource-Backed Stream Integration

Sources: ext/web/06_streams.js896-1212 ext/web/06_streams.js19-29

readableStreamForRid

Creates a ReadableStream backed by a Deno resource:

• rid - Resource ID from Deno's resource table.
• autoClose - Whether to close the resource when stream closes.
• Uses byte-oriented reading with BYOB support ext/web/06_streams.js947
• Implements fast-path optimizations for readAll() operations ext/web/06_streams.js965

The underlying source uses core.read() to pull data from the resource into provided buffers.

ext/web/06_streams.js946-1006

writableStreamForRid

Creates a WritableStream backed by a Deno resource:

The underlying sink uses core.writeAll() to write chunks to the resource ext/web/06_streams.js1186

ext/web/06_streams.js1173-1212

resourceForReadableStream

Converts a ReadableStream into a Deno resource:

This enables passing JavaScript streams to Rust ops that expect resource IDs. The implementation:

1. Acquires a reader from the stream ext/web/06_streams.js900
2. Allocates a resource with op_readable_stream_resource_allocate ext/web/06_streams.js904
3. Continuously reads from the stream and writes to the resource sink ext/web/06_streams.js810-860
4. Handles backpressure and errors.

ext/web/06_streams.js896-926

Sources: ext/web/06_streams.js896-926 ext/web/06_streams.js946-1006 ext/web/06_streams.js1173-1212

Fast-Path Optimizations

Resource-backed streams include optimizations for common patterns:

readableStreamCollectIntoUint8Array

When collecting a stream's entire contents into memory:

• Fast path: If stream is resource-backed and undisturbed, uses op_read_all() for single operation ext/web/06_streams.js1109
• Slow path: Reads chunks iteratively and concatenates into final buffer ext/web/06_streams.js1120-1158

ext/web/06_streams.js1095-1160

Sources: ext/web/06_streams.js1095-1160

---

Integration with Fetch API

Streams are central to the Fetch API implementation for handling request and response bodies.

InnerBody Class

The InnerBody class in the fetch implementation manages the body's stream state:

This abstraction allows bodies to be static (Uint8Array/string) or streaming (ReadableStream).

ext/fetch/22_body.js89-255

Sources: ext/fetch/22_body.js89-255

---

Queuing and Backpressure

Queue Management

Streams maintain internal queues to buffer chunks:

Operations:

• enqueueValueWithSize(container, value, size) - Add item to queue ext/web/06_streams.js505-541
• dequeueValue(container) - Remove and return item from queue ext/web/06_streams.js543-560

ext/web/06_streams.js505-560

Queuing Strategies

Two built-in strategies control buffering:

• ByteLengthQueuingStrategy - Buffers based on byte length ext/web/06_streams.js3603-3639
• CountQueuingStrategy - Buffers based on chunk count ext/web/06_streams.js3642-3676

ext/web/06_streams.js3603-3676

Sources: ext/web/06_streams.js505-560 ext/web/06_streams.js3603-3676

---

Helper Utilities

Stream Teeing

The readableStreamTee() function creates two independent branches from a single stream ext/web/06_streams.js2715 Chunks are cloned for the second branch if requested ext/web/06_streams.js2718

ext/web/06_streams.js2715-2946

Stream Piping

The pipeTo() method connects a readable stream to a writable stream ext/web/06_streams.js3901 It handles chunk transfer with backpressure and cleanup.

ext/web/06_streams.js3901-4110

Sources: ext/web/06_streams.js2715-2946 ext/web/06_streams.js3901-4110

---

Key Implementation Details

Symbol-Based Private State

The implementation uses symbols for private state mapping to specification internal slots:

ext/web/06_streams.js343-394

Resource Lifecycle and Cleanup

Resource-backed streams use SafeFinalizationRegistry to clean up resources if streams are garbage collected:

ext/web/06_streams.js931-933

Sources: ext/web/06_streams.js343-394 ext/web/06_streams.js931-933