NodeJS Http Requests stuck at five
Whilst working on our monitoring system we encountered some rather odd behaviour sending data into Elasticsearch over HTTP. We would get 5 responses, and then it would abruptly stop.
We were calling request.end(), so as far as we knew we were ending the connection, after some googling and a rather helpful stackoverflow post we discovered the issue was we were not consuming the data. Once we added in:
response.on('data', function() {});
All the data started flowing in. The NodeJS API guide for the Http Request class seems to be notably missing this rather core piece of information, so if you hit upon a limit of five requests, or massive memory leaks, then make sure you’re calling on .on(‘data’... even if you don’t want to do anything with it!
If it helps, our code to call elasticsearch over HTTP looked like
var params = {
host: this.host,
port: this.port,
method: 'POST',
path: elasticSearchPath
};
var json = JSON.stringify(data);
var req = http.request(params, function(res) {
if (res.statusCode != 201) {
this.error_buffer.emit('error', new Error('Wrong ElasticSearch return code ' + res.statusCode + ', should be 201. Data: ' + json));
}
else {
this.error_buffer.emit('ok');
}
res.on('data', function() {});
}.bind(this));
req.on('error', function(err) {
this.error_buffer.emit('error', err);
}.bind(this));
req.end(json);
Update:
After searching through the node documentation, it does actual reference the need to read the data, it’s just extremely easy to miss. “Also, until the data is read it will consume memory that can eventually lead to a 'process out of memory' error.”