worker cpu balance
Aleksandar Lazic
al-nginx at none.at
Sun Apr 13 05:04:20 MSD 2008
Hi all,
during the testing with the donated 10GB network cards of Myricom to the
haproxy project http://haproxy.1wt.eu/ I have asked the author of this
nice peace of SW if he will be so pleasent to run a test with nginx
instead of tux. He was ;-))
Here the description of his test with haproxy
http://haproxy.1wt.eu/10g.html and now what he have send me back from
his tests with nginx.
---
It works fast. Since it uses sendfile, it is as fast as Tux on large
files (>= 1MB), and saturates 10 Gbps with 10% of CPU with 1MB files.
However, it does not scale on multiple CPUs, whatever the number of
worker_processes. I've tried 1, 2, 8, ... The processes are quite there,
but something's preventing them from sharing a resource since the
machine never goes beyond 50% CPU used (it's a dual core). Sometimes,
"top" looks like this :
Tasks: 189 total, 3 running, 186 sleeping, 0 stopped, 0 zombie
Cpu0 : 40.3% user, 55.2% system, 0.0% nice, 4.5% idle, 0.0% IO-wait
Cpu1 : 2.7% user, 1.3% system, 0.0% nice, 96.0% idle, 0.0% IO-wait
Mem: 2072968k total, 92576k used, 1980392k free, 11604k buffers
Swap: 0k total, 0k used, 0k free, 25656k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ Command
1984 nobody 20 0 2980 996 492 S 34.7 0.0 0:49.85 nginx.bin
1986 nobody 20 0 2980 992 488 S 34.7 0.0 0:51.91 nginx.bin
1980 nobody 20 0 2980 996 492 S 25.8 0.0 0:47.29 nginx.bin
1983 nobody 20 0 2980 996 492 S 2.0 0.0 0:48.07 nginx.bin
1988 nobody 20 0 2980 996 492 R 2.0 0.0 0:45.75 nginx.bin
Sometime it looks like this :
Tasks: 188 total, 2 running, 186 sleeping, 0 stopped, 0 zombie
Cpu0 : 12.7% user, 12.7% system, 0.0% nice, 74.6% idle, 0.0% IO-wait
Cpu1 : 32.4% user, 39.4% system, 0.0% nice, 28.2% idle, 0.0% IO-wait
Mem: 2072968k total, 92820k used, 1980148k free, 11604k buffers
Swap: 0k total, 0k used, 0k free, 25660k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ Command
1985 nobody 20 0 2980 996 492 R 53.7 0.0 0:48.14 nginx.bin
1982 nobody 20 0 2980 992 488 S 31.8 0.0 0:39.40 nginx.bin
1986 nobody 20 0 2980 992 488 S 8.0 0.0 0:54.71 nginx.bin
1988 nobody 20 0 2980 996 492 S 5.0 0.0 0:48.79 nginx.bin
1983 nobody 20 0 2980 996 492 S 2.0 0.0 0:52.20 nginx.bin
Rather strange.
---
I have seen the same behaviour.
Here the description of my setup:
### cat /proc/cpuinfo of both mashines
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz
stepping : 6
cpu MHz : 2400.075
cache size : 4096 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts pni
monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr lahf_lm
bogomips : 4802.73
clflush size : 64
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz
stepping : 6
cpu MHz : 2400.075
cache size : 4096 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr lahf_lm
bogomips : 4800.13
clflush size : 64
###
free -m ( no swap usage )
total used free shared buffers cached
Mem: 2027 1132 895 0 155 828
###
When I run ab I get the follwoing:
close:
ab -n 40000 -c 2500 http://192.168.1.17:8080/10k
Document Length: 10240 bytes
Concurrency Level: 2500
Complete requests: 40000
Server Software: lighttpd/1.5.0
Time taken for tests: 7.807093 seconds
Requests per second: 5123.55 [#/sec] (mean)
Server Software: nginx/0.6.29
Time taken for tests: 8.96004 seconds
Requests per second: 4940.71 [#/sec] (mean)
lighttpd use both workers with similar usage.
nginx use both workers but one is used ~70-80 and the other only ~20-30
percent.
keep alive:
ab -n 40000 -c 2500 -k http://192.168.1.17:8080/10k
Server Software: lighttpd/1.5.0
Time taken for tests: 6.625588 seconds
Requests per second: 6037.20 [#/sec] (mean)
Server Software: nginx/0.6.29
Time taken for tests: 6.732870 seconds
Requests per second: 5941.00 [#/sec] (mean)
lighttpd use both workers but not similar.
nginx use only one worker.
Well with a 10k file it's easy but what is the behaviour with a 1M file?
ab -n 4000 -c 250 -k http://192.168.1.17:8080/1M
Document Length: 1048576 bytes
Concurrency Level: 250
Complete requests: 4000
Server Software: lighttpd/1.5.0
Time taken for tests: 59.870157 seconds
Keep-Alive requests: 3909
Requests per second: 66.81 [#/sec] (mean)
Server Software: nginx/0.6.29
Time taken for tests: 59.899784 seconds
Keep-Alive requests: 4000
Requests per second: 66.78 [#/sec] (mean)
lighttpd use both workers with similar usage.
nginx use again only one the first worker.
A different picture is shown when I use inject
http://1wt.eu/tools/inject/
---nginx
Clients : 5499
Hits : 178202 + 0 abortés
Octets : 3863054678
Duree : 61014 ms
Debit : 63314 kB/s
Reponse : 2920 hits/s
Erreurs : 0
Timeouts: 0
Temps moyen de hit: 1729965.8 ms
Temps moyen d'une page complete: 9067.0 ms
Date de demarrage: 1208046930 (13 Avr 2008 - 2:35:30)
Ligne de commande : ./inject29 -l -n 40000 -p 2 -o 8 -u 2500 -s 20 -G 192.168.1.17:8080/1M -d 60
---lighty
Clients : 5000
Hits : 57310 + 0 abortés
Octets : 4022907712
Duree : 61010 ms
Debit : 65938 kB/s
Reponse : 939 hits/s
Erreurs : 0
Timeouts: 0
Temps moyen de hit: 0.0 ms
Temps moyen d'une page complete: 0.0 ms
Date de demarrage: 1208047028 (13 Avr 2008 - 2:37:08)
Ligne de commande : ./inject29 -l -n 40000 -p 2 -o 8 -u 2500 -s 20 -G 192.168.1.17:8080/1M -d 60
with this testing tool both servers distribute the workers is similar
manner.
The first worker get always the most 'load'.
Have anybody seen the same behaviour in the real world or happen this
only at test time?
You can get both config files from
http://none.at/lighttpd.conf
http://none.at/nginx.conf
BR
Aleks
More information about the nginx
mailing list