high bandwidth configuration help

[Igor Sysoev]

On Sun, Feb 24, 2008 at 09:47:02AM +0000, nginx.mailinglist wrote:

> Hi
> 
> thanks for all the replies! i should have provided more details but I was in
> dispair and it was late at night, have clear head now after good sleep.
> 
> Ok heres info requested, the servers (i have another 6 pushing 120~mbit
> average with lighttpd 1.4 compiled now, all servers same config)
> 
> 
> 
> OS: Suse 10.3 (minimal, i have barely anything installed and allmost
> everythign compiled by me)  http://pastebin.com/m41b5a046
> 
> Applications Running:
> *nginx_0.5.35 (compiled http://pastebin.com/m6462475a )  used for file
> downloads
> *php_5.2.5 (compiled  http://pastebin.com/m3c02ea6e , ~20 threads started
> with lighttps spawn fcgi), very short php scripts are run that use
> X-Accell-Redirect to pass control to nginx of file serving
> *lighttpd_1.5r2048 (compiled) used for file uploads, due to the progress
> meter, fairly stable in production tho its from the svn
> *custom php_cli socket deamons for inter server RPC, these are very light
> 
> 
> *top http://pastebin.com/m2587b666*
> *dstat http://pastebin.com/m2055efd4*
> *df http://pastebin.com/m724e52a7*
> *vmstat http://pastebin.com/m5cdc2f0b*
> **
> heres *lighttpd 1.4.18 config* from other servers on network (they are all
> pretty much same just differing hosts) http://pastebin.com/m7b1af1e6
> 
> 
> now the only thing that changed on this server is lighttpd 1.4.18 was
> replaced by nginx, and now the request go thru nginx X-AcellRedirect not
> lighttpd's mod_secdownload
> 
> the php5 fcgi scripts are very small it check file exists etc, then just
> adds a downlaod id into headers for nginx to pick up and passes control onto
> nginx the relevant bit is here > http://pastebin.com/m2bcbe7fd
> 
> basically the server is capable of pushing above 100mbit easily before, now
> the load is high, but this could be due to me setting high number of worker
> threads like Igor said and more php-cgi processes floating around
> 
> so to summarise I need to figure out how to tweak this so i can move all the
> servers from lighttpd to nginx for file downloads

You should try to set
worker_processes 1;

LA in Linux is sum of processes that run or ready to run and processes that
wait for disk I/O. LA 6 and "70.5%wa" means that all 6 nginx workers wait
for disk I/O. Here is your bottleneck.


-- 
Igor Sysoev
http://sysoev.ru/en/