Skip to content

Commit e094691

Browse files
committed
Revise backpressure examples in streams documentation
Discussed here: https://stackoverflow.com/questions/79821132/node-js-back-pressure-documentation-whats-the-relevance-of-their-example Original backpressure example contains following oddities: 1) Incorrect zip example 2) Incorrect assumption that zip console command loads full file into memory. Updated examples to demonstrate file compression using Node.js streams with backpressure handling. Signed-off-by: Dmitry Baskakov <dmitry@bask.ws>
1 parent f0a969b commit e094691

1 file changed

Lines changed: 52 additions & 10 deletions

File tree

pages/modules/backpressuring-in-streams.md

Lines changed: 52 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -67,16 +67,62 @@ A good example of why the backpressure mechanism implemented through streams is
6767
a great optimization can be demonstrated by comparing the internal system tools
6868
from Node.js' [`Stream`][] implementation.
6969

70-
In one scenario, we will take a large file (approximately ~9 GB) and compress it
71-
using the familiar [`zip(1)`][] tool.
70+
In one scenario, we will read a large file (approximately ~9 GB) using `fs.readFileSync()`: and compress it
71+
using the module [`zlib`][], that wraps around another compression tool, [`gzip(1)`][].
7272

73+
```cjs
74+
const fs = require('node:fs');
75+
const zlib = require('node:zlib');
76+
77+
const data = fs.readFileSync('The.Matrix.1080p.mkv');
78+
const compressed = zlib.gzipSync(data);
79+
fs.writeFileSync('The.Matrix.1080p.mkv.gz', compressed);
80+
```
81+
82+
```mjs
83+
import { readFileSync, writeFileSync } from 'node:fs';
84+
import { gzipSync } from 'node:zlib';
85+
86+
const data = readFileSync('The.Matrix.1080p.mkv');
87+
const compressed = gzipSync(data);
88+
writeFileSync('The.Matrix.1080p.mkv.gz', compressed);
89+
```
90+
91+
This fails on two separate limits, whichever you hit first: the buffer size cap or heap exhaustion. Let’s rewrite it another way, using Node.js' [`Stream`][] but without backpressure:
92+
93+
```cjs
94+
const { createReadStream, createWriteStream } = require('node:fs');
95+
const { createGzip } = require('node:zlib');
96+
97+
const gzip = createGzip();
98+
99+
const inp = createReadStream('The.Matrix.1080p.mkv');
100+
const out = createWriteStream('The.Matrix.1080p.mkv.gz');
101+
102+
inp.on('data', (chunk) => gzip.write(chunk));
103+
inp.on('end', () => gzip.end());
104+
gzip.on('data', (chunk) => out.write(chunk));
105+
gzip.on('end', () => out.end());
73106
```
74-
zip The.Matrix.1080p.mkv
107+
108+
```mjs
109+
import { createReadStream, createWriteStream } from 'node:fs';
110+
import { createGzip } from 'node:zlib';
111+
112+
const gzip = createGzip();
113+
114+
const inp = createReadStream('The.Matrix.1080p.mkv');
115+
const out = createWriteStream('The.Matrix.1080p.mkv.gz');
116+
117+
inp.on('data', (chunk) => gzip.write(chunk));
118+
inp.on('end', () => gzip.end());
119+
gzip.on('data', (chunk) => out.write(chunk));
120+
gzip.on('end', () => out.end());
75121
```
76122

77-
While that will take a few minutes to complete, in another shell we may run
78-
a script that takes Node.js' module [`zlib`][], that wraps around another
79-
compression tool, [`gzip(1)`][].
123+
Neither write respects backpressure. Stage 1 keeps pushing into gzip even after `gzip.write()` returns false, and stage 2 keeps pushing into out even after `out.write()` returns false. Both internal buffers can grow without bound, so this is very prone to running out of memory. On a large compressible file both numbers climb fast and it heads for `JavaScript heap out of memory`.
124+
125+
To resolve this, we may use pipe, which pauses the read when `write()` returns false. When `gzip.write()` returns `false`, `pipe` calls `pause()` on the read stream, halting disk reads. Once gzip works through its backlog and the buffer empties, it emits a `'drain'` event, and `pipe` calls `resume()` to start reading again.
80126

81127
```cjs
82128
const fs = require('node:fs');
@@ -100,10 +146,6 @@ const out = createWriteStream('The.Matrix.1080p.mkv.gz');
100146
inp.pipe(gzip).pipe(out);
101147
```
102148

103-
To test the results, try opening each compressed file. The file compressed by
104-
the [`zip(1)`][] tool will notify you the file is corrupt, whereas the
105-
compression finished by [`Stream`][] will decompress without error.
106-
107149
> In this example, we use `.pipe()` to get the data source from one end
108150
> to the other. However, notice there are no proper error handlers attached. If
109151
> a chunk of data were to fail to be properly received, the `Readable` source or

0 commit comments

Comments
 (0)