# wrk over internet, no encryption
$ time wrk -t12 -c100 --latency -d120s http://example.org:5984/
Running 2m test @ http://example.org:5984/
12 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 5.25ms 2.79ms 417.48ms 91.33%
Req/Sec 1.54k 165.15 2.49k 71.89%
Latency Distribution
50% 5.20ms
75% 6.23ms
90% 7.20ms
99% 9.75ms
2206417 requests in 2.00m, 715.43MB read
Requests/sec: 18380.55
Transfer/sec: 5.96MB
120.05 real 13.01 user 40.98 sys
# wrk over internet, via spiped localhost->example.org
$ time wrk -t12 -c100 --latency -d120s http://127.0.0.1:5984/
Running 2m test @ http://127.0.0.1:5984/
12 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 7.22ms 7.82ms 315.05ms 99.46%
Req/Sec 1.17k 108.75 2.67k 74.26%
Latency Distribution
50% 6.80ms
75% 7.60ms
90% 8.40ms
99% 11.40ms
1669142 requests in 2.00m, 541.22MB read
Socket errors: connect 0, read 5298, write 24, timeout 0
Requests/sec: 13897.76
Transfer/sec: 4.51MB
120.10 real 5.76 user 15.58 sys
$
# wrk with ssh tunnel
$ time wrk -t12 -c100 --latency -d120s http://127.0.0.1:5988/
Running 2m test @ http://127.0.0.1:5988/
12 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 6.72ms 2.45ms 40.54ms 82.96%
Req/Sec 1.21k 111.67 2.91k 74.04%
Latency Distribution
50% 6.49ms
75% 7.64ms
90% 8.87ms
99% 16.28ms
1739884 requests in 2.00m, 564.16MB read
Requests/sec: 14485.13
Transfer/sec: 4.70MB
120.12 real 6.95 user 18.03 sys
# wrk with spiped tunnel
$ time wrk -t12 -c100 --latency -d120s http://127.0.0.1:5989/
Running 2m test @ http://127.0.0.1:5989/
12 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 7.14ms 4.90ms 264.43ms 98.57%
Req/Sec 1.16k 114.78 1.60k 73.02%
Latency Distribution
50% 6.81ms
75% 7.60ms
90% 8.44ms
99% 14.58ms
1657756 requests in 2.00m, 537.53MB read
Socket errors: connect 0, read 6966, write 1, timeout 0
Requests/sec: 13803.33
Transfer/sec: 4.48MB
120.10 real 6.73 user 15.40 sys
<7>0 2016-12-19T21:53:20.304319+00:00 example.org kernel - - sonewconn: pcb 0xfffff8009f5a11d0: Listen queue overflow: 16 already in queue awaiting acceptance (7817 occurrences)
# wrk with more threads and connections to provoke the error
# via normal http
# 40 threads and 400 concurrent connections
$ time wrk -t40 -c400 --latency -d120s http://example.org:5984/
Running 2m test @ http://example.org:5984/
40 threads and 400 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 22.64ms 9.89ms 441.42ms 83.79%
Req/Sec 447.90 62.00 6.22k 81.18%
Latency Distribution
50% 23.03ms
75% 26.70ms
90% 30.60ms
99% 40.58ms
2139522 requests in 2.00m, 693.74MB read
Socket errors: connect 0, read 11861, write 2059, timeout 0
Requests/sec: 17810.62
Transfer/sec: 5.78MB
120.14 real 11.89 user 37.29 sys
# wrk with more threads and connections to provoke the error
# via spiped
$ time wrk -t40 -c400 --latency -d120s http://127.0.0.1:5989/
Running 2m test @ http://127.0.0.1:5989/
40 threads and 400 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 7.27ms 3.82ms 278.52ms 98.82%
Req/Sec 481.32 387.22 1.93k 89.48%
Latency Distribution
50% 7.10ms
75% 7.92ms
90% 8.74ms
99% 11.34ms
1662769 requests in 2.00m, 539.15MB read
Socket errors: connect 284, read 130452, write 4341, timeout 1
Requests/sec: 13842.55
Transfer/sec: 4.49MB
120.13 real 6.81 user 20.64 sys
<7>Dec 16 16:35:42 0 2016-12-16T16: 35:40.500908+00:00 beatrix kernel - - [72288] sonewconn: pcb 0xfffff80d68996cb0: Listen queue overflow: 16 already in queue awaiting acceptance (64492 occurrences)
<5>Dec 16 16:35:42 0 2016-12-16T16: 35:42.555992+00:00 beatrix kernel - - [72290] Limiting open port RST response from 11938 to 200 packets/sec
<5>Dec 16 16:35:43 0 2016-12-16T16: 35:43.561039+00:00 beatrix kernel - - [72291] Limiting open port RST response from 26544 to 200 packets/sec
<6>Dec 16 16:35:44 0 2016-12-16T16: 35:44.170158+00:00 beatrix kernel - - [72291] [zone: pf states] PF states limit reached
# netstat -ALan (beatrix)
Current listen queue sizes (qlen/incqlen/maxqlen)
Tcpcb Proto Listen Local Address
fffff80ce6368820 tcp4 0/0/256 127.0.0.1.5984
fffff80c02c8b000 tcp4 0/0/128 *.4370
fffff80cba76e000 tcp4 0/0/10 127.0.0.1.14514
fffff80ce4be8000 tcp4 0/0/10 127.0.0.1.5555
fffff80cba91a410 tcp4 0/0/10 127.0.0.1.5673
fffff80ce404a410 tcp4 0/0/10 127.0.0.1.5672
fffff80ce4706410 tcp4 0/0/10 127.0.0.1.1979
fffff80cc052d410 tcp4 0/0/10 127.0.0.1.5983
fffff80cc0495820 tcp4 0/0/10 127.0.0.1.5985
fffff80cca843000 tcp4 16/0/10 <---BAD ---- *.15984
fffff8012cf47410 tcp6 0/0/128 *.4369
fffff8012cf47820 tcp4 0/0/128 *.4369
fffff8012cdfe410 tcp4 0/0/128 *.2200
fffff8012cdfe820 tcp6 0/0/128 *.2200
fffff8012cf1b820 tcp4 0/0/128 127.0.0.1.1978
fffff8012ccdc820 tcp4 0/0/5 127.0.0.1.5335
fffff8012c636820 tcp4 0/0/25 127.0.0.1.514
fffff8012c446410 tcp4 0/0/256 127.0.0.1.53
fffff8012c446820 tcp6 0/0/256 ::1.53
unix 0/0/100 /var/run/mdnsd
unix 0/0/256 /var/run/local_unbound.ctl
unix 0/0/4 /var/run/devd.pipe
unix 0/0/4 /var/run/devd.seqpacket.pipe
- https://wiki.freebsd.org/SystemTuning
- https://wiki.freebsd.org/NetworkPerformanceTuning
- https://www.psc.edu/index.php/using-joomla/extensions/templates/atomic/641-tcp-tune#FreeBSD
- https://blog.whatsapp.com/196/1-million-is-so-2011
- https://forums.freebsd.org/threads/53381/
- https://fasterdata.es.net/host-tuning/freebsd/
- https://calomel.org/freebsd_network_tuning.html
- http://tilt.lib.tsinghua.edu.cn/docs/books/squid/TheDefinitiveGuideChinese/0596001622/squid-CHP-3-SECT-3.html
- https://blog.box.com/blog/ephemeral-port-exhaustion-and-web-services-at-scale/
- https://en.wikipedia.org/wiki/Ephemeral_port
- http://aleccolocco.blogspot.co.at/2008/11/ephemeral-ports-problem-and-solution.html
- https://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-3
- https://idea.popcount.org/2014-04-03-bind-before-connect/
- http://www.ncftp.com/ncftpd/doc/misc/ephemeral_ports.html#FreeBSD
- https://pleiades.ucsc.edu/hyades/FreeBSD_Network_Tuning