编写Go程序对Nginx服务器进行性能测试的方法
目前有很多提供go语言http应用服务的方法,但其中最好的选择取决于每个应用的实际情况。目前,nginx看起来是每个新项目的标准web服务器,即使在有其他许多不错web服务器的情况下。然而,在nginx上提供go应用服务的开销是多少呢?我们需要一些nginx的特性参数(vhosts,负载均衡,缓存,等等)或者直接使用go提供服务?如果你需要nginx,最快的连接机制是什么?这就是在这我试图回答的问题。该基准测试的目的不是要验证go比nginx的快或慢。那将会很愚蠢。
下面是我们要比较不同的设置:
- go http standalone (as the control group)
- nginx proxy to go http
- nginx fastcgi to go tcp fastcgi
- nginx fastcgi to go unix socket fastcgi
硬件
因为我们将在相同的硬件下比较所有设置,硬件选择的是廉价的一个。这不应该是一个大问题。
- samsung 笔记本 np550p5c-ad1br
- intel core i7 3630qm @2.4ghz (quad core, 8 threads)
- cpu caches: (l1: 256kib, l2: 1mib, l3: 6mib)
- ram 8gib ddr3 1600mhz
软件
- ubuntu 13.10 amd64 saucy salamander (updated)
- nginx 1.4.4 (1.4.4-1~saucy0 amd64)
- go 1.2 (linux/amd64)
- wrk 3.0.4
设置
内核
只需很小的一点调整,将内核的limits调高。如果你对这一变量有更好的想法,请在写在下面评论处:
fs.nr_open 9999999
net.core.netdev_max_backlog 4096
net.core.rmem_max 16777216
net.core.somaxconn 65535
net.core.wmem_max 16777216
net.ipv4.ip_forward 0
net.ipv4.ip_local_port_range 1025 65535
net.ipv4.tcp_fin_timeout 30
net.ipv4.tcp_keepalive_time 30
net.ipv4.tcp_max_syn_backlog 20480
net.ipv4.tcp_max_tw_buckets 400000
net.ipv4.tcp_no_metrics_save 1
net.ipv4.tcp_syn_retries 2
net.ipv4.tcp_synack_retries 2
net.ipv4.tcp_tw_recycle 1
net.ipv4.tcp_tw_reuse 1
vm.min_free_kbytes 65536
vm.overcommit_memory 1
limits
供root和www-data打开的最大文件数限制被配置为200000。
nginx
有几个必需得nginx调整。有人跟我说过,我禁用了gzip以保证比较公平。下面是它的配置文件/etc/nginx/nginx.conf:
user www-data;
worker_processes auto;
worker_rlimit_nofile 200000;
pid /var/run/nginx.pid;
events {
worker_connections 10000;
use epoll;
multi_accept on;
}
http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 300;
keepalive_requests 10000;
types_hash_max_size 2048;
open_file_cache max=200000 inactive=300s;
open_file_cache_valid 300s;
open_file_cache_min_uses 2;
open_file_cache_errors on;
server_tokens off;
dav_methods off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
access_log /var/log/nginx/access.log combined;
error_log /var/log/nginx/error.log warn;
gzip off;
gzip_vary off;
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*.conf;
}
nginx vhosts
upstream go_http {
server 127.0.0.1:8080;
keepalive 300;
}
server {
listen 80;
server_name go.http;
access_log off;
error_log /dev/null crit;
location / {
proxy_pass http://go_http;
proxy_http_version 1.1;
proxy_set_header connection "";
}
}
upstream go_fcgi_tcp {
server 127.0.0.1:9001;
keepalive 300;
}
server {
listen 80;
server_name go.fcgi.tcp;
access_log off;
error_log /dev/null crit;
location / {
include fastcgi_params;
fastcgi_keep_conn on;
fastcgi_pass go_fcgi_tcp;
}
}
upstream go_fcgi_unix {
server unix:/tmp/go.sock;
keepalive 300;
}
server {
listen 80;
server_name go.fcgi.unix;
access_log off;
error_log /dev/null crit;
location / {
include fastcgi_params;
fastcgi_keep_conn on;
fastcgi_pass go_fcgi_unix;
}
}
go源码
package main
import (
"fmt"
"log"
"net"
"net/http"
"net/http/fcgi"
"os"
"os/signal"
"syscall"
)
var (
abort bool
)
const (
sock = "/tmp/go.sock"
)
type server struct {
}
func (s server) servehttp(w http.responsewriter, r *http.request) {
body := "hello world\n"
// try to keep the same amount of headers
w.header().set("server", "gophr")
w.header().set("connection", "keep-alive")
w.header().set("content-type", "text/plain")
w.header().set("content-length", fmt.sprint(len(body)))
fmt.fprint(w, body)
}
func main() {
sigchan := make(chan os.signal, 1)
signal.notify(sigchan, os.interrupt)
signal.notify(sigchan, syscall.sigterm)
server := server{}
go func() {
http.handle("/", server)
if err := http.listenandserve(":8080", nil); err != nil {
log.fatal(err)
}
}()
go func() {
tcp, err := net.listen("tcp", ":9001")
if err != nil {
log.fatal(err)
}
fcgi.serve(tcp, server)
}()
go func() {
unix, err := net.listen("unix", sock)
if err != nil {
log.fatal(err)
}
fcgi.serve(unix, server)
}()
<-sigchan
if err := os.remove(sock); err != nil {
log.fatal(err)
}
}
检查http header
为公平起见,所有的请求必需大小相同。
http/1.1 200 ok
connection: keep-alive
content-length: 12
content-type: text/plain
server: gophr
date: sun, 15 dec 2013 14:59:14 gmt
$ curl -si http://127.0.0.1:8080/ | wc -c
141
$ curl -si http://go.http/
http/1.1 200 ok
server: nginx
date: sun, 15 dec 2013 14:59:31 gmt
content-type: text/plain
content-length: 12
connection: keep-alive
$ curl -si http://go.http/ | wc -c
141
$ curl -si http://go.fcgi.tcp/
http/1.1 200 ok
content-type: text/plain
content-length: 12
connection: keep-alive
date: sun, 15 dec 2013 14:59:40 gmt
server: gophr
$ curl -si http://go.fcgi.tcp/ | wc -c
141
$ curl -si http://go.fcgi.unix/
http/1.1 200 ok
content-type: text/plain
content-length: 12
connection: keep-alive
date: sun, 15 dec 2013 15:00:15 gmt
server: gophr
$ curl -si http://go.fcgi.unix/ | wc -c
141
启动引擎
- 使用sysctl配置内核
- 配置nginx
- 配置nginx vhosts
- 用www-data启动服务
- 运行基准测试
基准测试
gomaxprocs = 1
go standalone
# wrk -t100 -c5000 -d30s http://127.0.0.1:8080/
running 30s test @ http://127.0.0.1:8080/
100 threads and 5000 connections
thread stats avg stdev max +/- stdev
latency 116.96ms 17.76ms 173.96ms 85.31%
req/sec 429.16 49.20 589.00 69.44%
1281567 requests in 29.98s, 215.11mb read
requests/sec: 42745.15
transfer/sec: 7.17mb
nginx + go through http
# wrk -t100 -c5000 -d30s http://go.http/
running 30s test @ http://go.http/
100 threads and 5000 connections
thread stats avg stdev max +/- stdev
latency 124.57ms 18.26ms 209.70ms 80.17%
req/sec 406.29 56.94 0.87k 89.41%
1198450 requests in 29.97s, 201.16mb read
requests/sec: 39991.57
transfer/sec: 6.71mb
nginx + go through fastcgi tcp
# wrk -t100 -c5000 -d30s http://go.fcgi.tcp/
running 30s test @ http://go.fcgi.tcp/
100 threads and 5000 connections
thread stats avg stdev max +/- stdev
latency 514.57ms 119.80ms 1.21s 71.85%
req/sec 97.18 22.56 263.00 79.59%
287416 requests in 30.00s, 48.24mb read
socket errors: connect 0, read 0, write 0, timeout 661
requests/sec: 9580.75
transfer/sec: 1.61mb
nginx + go through fastcgi unix socket
# wrk -t100 -c5000 -d30s http://go.fcgi.unix/
running 30s test @ http://go.fcgi.unix/
100 threads and 5000 connections
thread stats avg stdev max +/- stdev
latency 425.64ms 80.53ms 925.03ms 76.88%
req/sec 117.03 22.13 255.00 81.30%
350162 requests in 30.00s, 58.77mb read
socket errors: connect 0, read 0, write 0, timeout 210
requests/sec: 11670.72
transfer/sec: 1.96mb
gomaxprocs = 8
go standalone
# wrk -t100 -c5000 -d30s http://127.0.0.1:8080/
running 30s test @ http://127.0.0.1:8080/
100 threads and 5000 connections
thread stats avg stdev max +/- stdev
latency 39.25ms 8.49ms 86.45ms 81.39%
req/sec 1.29k 129.27 1.79k 69.23%
3837995 requests in 29.89s, 644.19mb read
requests/sec: 128402.88
transfer/sec: 21.55mb
nginx + go through http
running 30s test @ http://go.http/
100 threads and 5000 connections
thread stats avg stdev max +/- stdev
latency 336.77ms 297.88ms 632.52ms 60.16%
req/sec 2.36k 2.99k 19.11k 84.83%
2232068 requests in 29.98s, 374.64mb read
requests/sec: 74442.91
transfer/sec: 12.49mb
nginx + go through fastcgi tcp
# wrk -t100 -c5000 -d30s http://go.fcgi.tcp/
running 30s test @ http://go.fcgi.tcp/
100 threads and 5000 connections
thread stats avg stdev max +/- stdev
latency 217.69ms 121.22ms 1.80s 75.14%
req/sec 263.09 102.78 629.00 62.54%
721027 requests in 30.01s, 121.02mb read
socket errors: connect 0, read 0, write 176, timeout 1343
requests/sec: 24026.50
transfer/sec: 4.03mb
nginx + go through fastcgi unix socket
# wrk -t100 -c5000 -d30s http://go.fcgi.unix/
running 30s test @ http://go.fcgi.unix/
100 threads and 5000 connections
thread stats avg stdev max +/- stdev
latency 694.32ms 332.27ms 1.79s 62.13%
req/sec 646.86 669.65 6.11k 87.80%
909836 requests in 30.00s, 152.71mb read
requests/sec: 30324.77
transfer/sec: 5.09mb
结论
第一组基准测试时一些nginx的设置还没有很好的优化(启用gzip,go的后端没有使用keep-alive连接)。当改为wrk以及按建议优化nginx后结果有较大差异。
当gomaxprocs=1时,nginx的开销不是那么大,但当omaxprocs=8时差异就很大了。以后可能会再试一下其他设置。如果你需要使用nginx像虚拟主机,负载均衡,缓存等特性,使用http proxy,别使用fastcgi。有些人说go的fastcgi还没有被很好优化,这也许就是测试结果中巨大差异的原因。