Stream piping is one of the features of Node which I especially like. It exists to simplify data transfer between two or more streams.
For instance, with pipes, you can read HTTP POST request and write the received data to terminal or a file straight away. If you need to process or transform the transferred data, there are tools that you can integrate through pipes too.
I guess you think you know how to pipe one stream to another because it looks pretty straightforward at least in the official docs.
Like this:
readable.pipe(writable);
But what if the source stream fails because of an error, or the target stream interrupts the connection before the operation finishes properly? Let's answer these questions together.
First, create server.js with this particular example of stream piping:
const fs = require('fs');
const http = require('http');
http.createServer((request, response) => {
const readStream = fs.createReadStream('./source.txt');
readStream.pipe(response);
}).listen(3000);
The code above creates a simple HTTP server which listens for connections on http://localhost:3000 and responds with the content of the source.txt file.
Now create the source.txt file in the code folder with a "Hello World" string and run the server:
node server.js
Open http://localhost:3000 or run the following curl command in terminal to test that it works:
curl http://localhost:3000/
Then add some console output to track the open and close events of the source stream.
// ...
const readStream = fs.createReadStream('./source.txt');
readStream.on('open', () => console.log('open'));
readStream.on('close', () => console.log('close'));
readStream.pipe(response);
// ...
Now restart the server and download the page through curl in a separate terminal window.
curl http://localhost:3000/
You should see two messages in the console:
open
close
This output indicates that the source file stream has been opened and closed properly.
However, the situation when the source file is large, or the connection speed of the client is not sufficient, is quite possible.
To simulate such situation, we need to generate a long text fragment through loremipsum.io for example. About 10000 paragraphs of dummy text should be enough, which is about 8MB of disk space. Once generated, copy and paste the text into the source.txt file.
We can use curl with bandwidth throttling enabled. For this purpose, make sure your server is active and execute the following command in a separate terminal window:
curl --limit-rate 1K http://localhost:3000/
Don't wait until the command finishes, and, instead, interrupt it earlier. This way we simulate the situation when user closes the browser tab.
Time to look at the server terminal. We can see the "open" message there, but where is the "close" message? The "close" event hasn't still happened. It means that the source file stream remains open. And what would happen if thousands of people on slow connections quit before their browsers would receive the content completely? You would probably have thousands of file streams open, which is not perfect, because it can cause memory leaks, or you can reach the limit of simultaneously opened files set by the OS.
Luckily, we can track the event of closing client connection by subscribing to the "close" event of the response stream, and destroy the source file stream manually.
// ...
const readStream = fs.createReadStream('./source.txt');
readStream.on('open', () => console.log('open'));
readStream.on('close', () => console.log('close'));
readStream.pipe(response);
// Handle closing connection by the client.
response.on('close', () => {
readStream.destroy();
})
// ...
You may now restart your server, run curl with throttling and interrupt the connection again. The server terminal should now have both, "open" and "close" messages.
Alright, our server has become better but there is one more thing which wouldn't let me sleep at night if I would deploy this code to production.
What if the file system fails someway, or the source file is busy, and it's not possible to read from it? Guess what? The server will fall down with an exception. Just one request can break the server because we don't handle file system errors.
Let's fix that by adding the read stream error handler:
// ...
const readStream = fs.createReadStream('./source.txt');
// Handle read stream errors.
readStream.on('error', (error) => {
console.error(error);
response.writeHead(500, 'Server error');
response.end('Server error');
});
readStream.pipe(response);
// ...
To test the fix, change the filename in fs.createReadStream() to a wrong one, restart the server and load the page again. The server should respond to current client with the error message but stay operational to process other clients requests, which is much better.
To finish our simple server, I would also set the response content type to let the browser know that our server responds with plain text.
// ...
// Instruct the client on what it's going to receive.
response.setHeader('Content-type', 'text/plain; charset=UTF-8');
readStream.pipe(response);
// ...
Here is how our final code looks:
const fs = require('fs');
const http = require('http');
http.createServer((request, response) => {
const readStream = fs.createReadStream('./source.txt');
// Handle read stream errors.
readStream.on('error', (error) => {
console.error(error);
response.writeHead(500, 'Server error');
response.end('Server error');
});
// Instruct the client on what it's going to receive.
response.setHeader('Content-type', 'text/plain; charset=UTF-8');
readStream.pipe(response);
// Handle closing connection by the client.
response.on('close', () => {
readStream.destroy();
})
}).listen(3000);
I have to admit that it's not that simple as readable.pipe(writable) but if you develop the discipline of checking
for such kind of nuances, it will soon become a no-brainer for you.
I'd like to thank Ilya Kantor for covering this and many other topics in his educational materials.