When should you use streams instead of reading whole files or buffering data?
2 minintermediatenodejsstreamsmemoryperformancescalability
Quick Answer
Use streams when data is large or unbounded, or when you want to start processing before all data arrives. Streaming keeps memory usage roughly constant and reduces latency, whereas reading everything into a Buffer/string scales memory with payload size and can exhaust RAM under load.
Detailed Answer
Answer:
Buffering (read it all at once):
const data = await fs.promises.readFile('report.csv'); // whole file in memory
Simple and fine for small, bounded data. But memory usage = file size × concurrent requests, so a 1 GB file (or many medium ones) can OOM the process.
Streaming (process chunk by chunk):
fs.createReadStream('report.csv')
.pipe(csvParser())
.pipe(transformRows())
.pipe(res); // send to the client as you go
Use streams when:
- Large or unknown-size data — big files, uploads/downloads, DB exports, log processing.
- Constant memory matters — memory stays around the buffer size regardless of total volume.
- Lower latency / TTFB — you can start sending/processing before all input is read (e.g., piping a file straight to an HTTP response).
- Composable pipelines — chain read → decompress → parse → transform → write.
Stick with buffering when:
- Data is small and you need the whole thing at once (e.g., parse a small JSON config).
- The processing genuinely requires random access to all the data.
Real-world example: serving a large file download should createReadStream(...).pipe(res), not readFile then res.send, so one big download doesn't spike memory for every concurrent client.