I am running a very simple RESTful API on AWS using Node.js. The API takes a request in the form of '/rest/users/jdoe' and returns the following (it's all done in memory, no database involved):
{
username: 'jdoe',
firstName: 'John',
lastName: 'Doe'
}
The performance of this API on Node.js + AWS is horrible compared to the local network - only 9 requests/sec vs. 2,214 requests/sec on a local network. AWS is running a m1.medium instance whereas the local Node server is a desktop machine with an Intel i7-950 processor. Trying to figure out why such a huge difference in performance.
Benchmarks using Apache Bench are as follows:
Local Network
10,000 requests with concurrency of 100/group
> ab -n 10000 -c 100 http://192.168.1.100:8080/rest/users/jdoe
Document Path: /rest/users/jdoe
Document Length: 70 bytes
Concurrency Level: 100
Time taken for tests: 4.516 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 2350000 bytes
HTML transferred: 700000 bytes
Requests per second: 2214.22 [#/sec] (mean)
Time per request: 45.163 [ms] (mean)
Time per request: 0.452 [ms] (mean, across all concurrent requests)
Transfer rate: 508.15 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.4 0 2
Processing: 28 45 7.2 44 74
Waiting: 22 43 7.5 42 74
Total: 28 45 7.2 44 74
Percentage of the requests served within a certain time (ms)
50% 44
66% 46
75% 49
80% 51
90% 54
95% 59
98% 65
99% 67
100% 74 (longest request)
AWS
1,000 requests with concurrency of 100/group (10,000 requests would have taken too long)
C:\apps\apache-2.2.21\bin>ab -n 1000 -c 100 http://54.200.x.xxx:8080/rest/users/jdoe
Document Path: /rest/users/jdoe
Document Length: 70 bytes
Concurrency Level: 100
Time taken for tests: 105.693 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 235000 bytes
HTML transferred: 70000 bytes
Requests per second: 9.46 [#/sec] (mean)
Time per request: 10569.305 [ms] (mean)
Time per request: 105.693 [ms] (mean, across all concurrent requests)
Transfer rate: 2.17 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 98 105 3.8 106 122
Processing: 103 9934 1844.8 10434 10633
Waiting: 103 5252 3026.5 5253 10606
Total: 204 10040 1844.9 10540 10736
Percentage of the requests served within a certain time (ms)
50% 10540
66% 10564
75% 10588
80% 10596
90% 10659
95% 10691
98% 10710
99% 10726
100% 10736 (longest request)
Questions:
- Connect time for AWS is 105 ms (avg) compared to 0 ms on local network. I assume that this is because it takes a lot more time to open a socket to AWS then to a server on a local network. Is there anything to be done here for better performance under load assuming requests are coming in from multiple machines across the globe.
- More serious is the server processing time - 45 ms for local server compared to 9.9 seconds for AWS! I can't figure out what's going on in here. The server is only pressing 9.46 requests/sec. which is peanuts!
Any insight into these issues much appreciated. I am nervous about putting a serious application on Node+AWS if it can't perform super fast on such a simple application.
For reference here's my server code:
var express = require('express');
var app = express();
app.get('/rest/users/:id', function(req, res) {
var user = {
username: req.params.id,
firstName: 'John',
lastName: 'Doe'
};
res.json(user);
});
app.listen(8080);
console.log('Listening on port 8080');
Edit
Single request sent in isolation (-n 1 -c 1)
Requests per second: 4.67 [#/sec] (mean)
Time per request: 214.013 [ms] (mean)
Time per request: 214.013 [ms] (mean, across all concurrent requests)
Transfer rate: 1.07 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 104 104 0.0 104 104
Processing: 110 110 0.0 110 110
Waiting: 110 110 0.0 110 110
Total: 214 214 0.0 214 214
10 request all sent concurrently (-n 10 -c 10)
Requests per second: 8.81 [#/sec] (mean)
Time per request: 1135.066 [ms] (mean)
Time per request: 113.507 [ms] (mean, across all concurrent requests)
Transfer rate: 2.02 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 98 103 3.4 102 110
Processing: 102 477 296.0 520 928
Waiting: 102 477 295.9 520 928
Total: 205 580 295.6 621 1033
Results using wrk
As suggested by Andrey Sidorov. The results are MUCH better - 2821 requests per second:
Running 30s test @ http://54.200.x.xxx:8080/rest/users/jdoe
12 threads and 400 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 137.04ms 48.12ms 2.66s 98.89%
Req/Sec 238.11 27.97 303.00 88.91%
84659 requests in 30.01s, 19.38MB read
Socket errors: connect 0, read 0, write 0, timeout 53
Requests/sec: 2821.41
Transfer/sec: 661.27KB
So it certainly looks like the culprit is ApacheBench! Unbelievable!
