You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This fails on two separate limits, whichever you hit first: the buffer size cap or heap exhaustion. Let’s rewrite it another way, using Node.js' [`Stream`][] but without backpressure:
While that will take a few minutes to complete, in another shell we may run
78
-
a script that takes Node.js' module [`zlib`][], that wraps around another
79
-
compression tool, [`gzip(1)`][].
123
+
Neither write respects backpressure. Stage 1 keeps pushing into gzip even after `gzip.write()` returns false, and stage 2 keeps pushing into out even after `out.write()` returns false. Both internal buffers can grow without bound, so this is very prone to running out of memory. On a large compressible file both numbers climb fast and it heads for `JavaScript heap out of memory`.
124
+
125
+
To resolve this, we may use pipe, which pauses the read when `write()` returns false. When `gzip.write()` returns `false`, `pipe` calls `pause()` on the read stream, halting disk reads. Once gzip works through its backlog and the buffer empties, it emits a `'drain'` event, and `pipe` calls `resume()` to start reading again.
80
126
81
127
```cjs
82
128
constfs=require('node:fs');
@@ -100,10 +146,6 @@ const out = createWriteStream('The.Matrix.1080p.mkv.gz');
100
146
inp.pipe(gzip).pipe(out);
101
147
```
102
148
103
-
To test the results, try opening each compressed file. The file compressed by
104
-
the [`zip(1)`][] tool will notify you the file is corrupt, whereas the
105
-
compression finished by [`Stream`][] will decompress without error.
106
-
107
149
> In this example, we use `.pipe()` to get the data source from one end
108
150
> to the other. However, notice there are no proper error handlers attached. If
109
151
> a chunk of data were to fail to be properly received, the `Readable` source or
0 commit comments