Node.js Multicore

1. Multi-core Node.js
Node.js servers on today’s multi-core systems

2. Node.js is single-threaded
● One thread per process
● Deliberate design decision to avoid semaphore/mutex and other locking mechanisms that
introduce complexity and hard-to-debug bugs into the code
● Drawback: Massively under-utilizes raw parallelism available with today’s 2, 4, 8, 16 [, …] core
systems

3. Approaches to Node.js execution on Multi-core
systems
1. Single threaded execution
2. Execution in single-core virtual machines with multiple virtual machines each behind a load
balancer
3. Clustering Node.js
a. Run as many processes as cores on a single node

4. Approaches to Clustering Node.js processes
● [obsolete] Learnboost cluster module
○ older solution
○ no longer maintained
● Node.js core module cluster
○ official clustering solution

5. What is Node.js Clustering?
● Node.js server consists of multiple processes
● We start the main process called the master process
● Master process starts and manages other processes called worker processes
● Worker processes do the actual work
● TCP connections are shared and a default load balancer is included transparently

6. Multi-core and Multi-process
● Multi-core with cluster is basically multi-process on a single machine using Node’s built in load
balancer which defaults to round robin scheduling
● Multi-process on multiple machines requires the use of an external load balancer such as nginx

7. Sample Project to test multi-core performance
● A simple app was generated using express generator
● A processing intensive root route was created to test performance under heavy load conditions
● Multi-core code using Node.js cluster module was added

8. Why Multi-core?
● throughput ~= single-threaded throughput x cores
● test command:
○ siege -c100 -t1M http://localhost:3000/
■ 1 million total hits with 100 concurrent at a time

9. Single-core Results using Siege
Transactions: 1237 hits
Availability: 100.00 %
Elapsed time: 59.70 secs
Data transferred: 0.03 MB
Response time: 4.14 secs
Transaction rate: 20.72 trans/sec
Throughput: 0.00 MB/sec
Concurrency: 85.68
Successful transactions: 1237
Failed transactions: 0
Longest transaction: 4.58
Shortest transaction: 0.11

10. Multi-core Results using Siege
Transactions: 2523 hits
Availability: 100.00 %
Elapsed time: 59.92 secs
Data transferred: 0.06 MB
Response time: 1.82 secs
Transaction rate: 42.11 trans/sec
Throughput: 0.00 MB/sec
Concurrency: 76.67
Successful transactions: 2523
Failed transactions: 0
Longest transaction: 2.86
Shortest transaction: 0.12

11. Why Multi-core?
● Decrease in latency
● Test using node siege benchmarking
var siege = require('siege');
// run cpu bound service benchmark
siege()
.on(3000)
.for(1000).times
.concurrent(100)
.get('/')
.attack();

12. Siege Single-core Latency Results
[22:46:17 multicore-demo] (master)*$ node test/benchmark.js
GET:/
done:1000
200 OK: 1000
rps: 20
response: 220ms(min) 9797ms(max) 4635ms(avg)

13. Siege Multi-core Latency Results
[22:47:18 multicore-demo] (master)*$ node test/benchmark.js
GET:/
done:1000
200 OK: 1000
rps: 35
response: 251ms(min) 9200ms(max) 2659ms(avg)

14. Cluster Semantics
● Very simple to use
● Reap multi-core performance benefits with only a few extra lines of code
● Code divided into two sections:
○ One for master process
○ One for worker processes
● Various events are available to communicate between master/workers

15. Cluster Example Code
var cluster = require('cluster');
var numCPUs = require('os').cpus().length;
if (cluster.isMaster) { // true for master process
cluster.setupMaster({
exec: 'bin/www'
});
...

16. Cluster Example Code (contd.)
// Fork workers.
for (var i = 0; i < numCPUs; i++) {
cluster.fork();
}
cluster.on('listening', function (worker, address) {
console.log('Worker id: ' + worker.id + ' listening at: ' +
JSON.stringify(address));
});

17. Cluster Example Code (contd.)
Object.keys(cluster.workers).forEach(function (id) {
console.log('Worker id: ' + id + ' with pid: ' +
cluster.workers[id].process.pid);
});
cluster.on('exit', function (worker, code, signal) {
console.log('worker ' + worker.process.pid + ' died: Respawning...');
cluster.fork();
});
}

18. Cluster Considerations
● Place your Node.js behind a connection throttling proxy such as apache or nginx so that a
high load does not down the server completely
● Restart worker processes that die
● Refresh worker processes to avoid memory leaks
● Place a monitor such as forever to restart the master Node.js process in case it dies

19. PM2
● Utility based clustering solution, runs as daemon
● No need to write clustering code
● Just write single-core code and invoke it with PM2 indicating number of processes to create
○ pm2 start app.js -i 4
● Other useful commands such as runtime scaling available

20. Links
● Cluster links
○ https://nodejs.org/api/cluster.html
○ https://keymetrics.io/2015/03/26/pm2-clustering-made-easy/