very poor performance for serving static files

[asdasd77]

zls Wrote:
-------------------------------------------------------
> any system information like io stats?
> 
> btw, "worker_connections 768", is this enough ?
> maybe you can try a bigger limit


i dont know how can i provide that io stats information? what is the
command? 

we increased worker_connections to 10240. but nothings changed.






Maxim Dounin Wrote:
-------------------------------------------------------
> Hello!
> 
> On Mon, Jul 18, 2011 at 11:46:41AM -0400, asdasd77
> wrote:
> 
> > we are a video hosting company and storing files
> in mp4 format. we have
> > another server, its litespeed and handle nearly
> 1gbit connection
> > succesfully. slow but it is responsing requests
> at least. but this new
> > server (nginx) cant handle requests. if bw usage
> hit the 150-200mbit its
> > going down. actually not down, just doesnt
> response any http request. we
> > tried and also hired someones and they tried but
> no success. this is the
> > last chance. is there anybody knows what the
> problem?
> > 
> > here is the nginx.conf
> >
> **************************************************
> **************************************************
> **************************************
> > #user  nobody;
> > worker_processes  8;
> > worker_rlimit_nofile 20480;
> > 
> > error_log  /var/log/nginx/error.log info;
> > 
> > #pid        logs/nginx.pid;
> > 
> > 
> > events {
> > worker_connections  768;
> 
> You are using really low number for
> worker_connections, with 8 
> workers you'll be only able to serve about 6k
> connections in 
> total.  If you see nginx not responding to http
> requests - you are 
> probably hitting this limit.  Try looking at
> error_log and 
> stub_status output to see if it's true.

i looked those files but couldnt see relative errors. and we increased
worker_connections to 10240. but nothings changed.


> 
> > use epoll;
> > }
> 
> [...]
> 
> >     sendfile        on;
> 
> You rely on OS to do actual IO, and this may not
> be a good idea if 
> you are serving large files.  Your OS will likely
> use something 
> about 16k read requests and this will trash your
> disks with IOPS 
> and seeks.
> 
> Try either AIO or at least normal reading with big
> buffers (and 
> without sendfile), i.e.
> 
>       sendfile off;
>       output_buffers 2 512k;
> 
> or something like.
> 

we applied this settings too, but nothings changed.

> [...]
> 
> > our server has 1tb hdd and 12gb ram and 8core
> cpu and 1gbit line
> 
> Just 1 spindle isn't really good, but you should
> be able to get 
> something about 600 Mbit/s (raw disk speed on
> sequentional 
> reading, test your disk to see more correct
> numbers) with large 
> files and proper tuning even if your working set
> is much bigger 
> than memory and effective caching isn't possible. 
> 
> Maxim Dounin
> 
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://nginx.org/mailman/listinfo/nginx

i am looking the server simultaneously with "top -c -d 1", here is the
output
**************************************************************************************************
[root at srv-46 ~]# top -c -d 1
top - 12:33:04 up 13 min,  1 user,  load average: 13.82, 10.34, 5.30
Tasks: 158 total,  17 running, 137 sleeping,   0 stopped,   4 zombie
Cpu(s):  1.9%us, 45.7%sy,  0.0%ni, 11.4%id, 37.1%wa,  0.0%hi,  3.8%si, 
0.0%st
Mem:  12299044k total,  8913512k used,  3385532k free,    33100k
buffers
Swap:  6094840k total,        0k used,  6094840k free,  5757568k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 3180 root      20  -5     0    0    0 R  5.6  0.0   0:53.27 [vmmemctl]
 4478 nginx     25   0  103m  76m  800 R  4.9  0.6   0:24.45 nginx:
worker process
 4481 nginx     23   0  102m  75m  800 R  4.9  0.6   0:25.05 nginx:
worker process
 4483 nginx     25   0  111m  83m  808 R  4.2  0.7   0:25.92 nginx:
worker process
 4476 nginx     25   0  102m  75m  812 R  2.8  0.6   0:26.27 nginx:
worker process
 4480 nginx     25   0  102m  75m  800 R  2.8  0.6   0:26.22 nginx:
worker process
 4488 nginx     25   0  102m  75m  800 R  2.8  0.6   0:25.68 nginx:
worker process
 4485 nginx     25   0  101m  73m  800 R  2.1  0.6   0:25.19 nginx:
worker process
  893 root      10  -5     0    0    0 D  1.4  0.0   0:01.23
[kjournald]
 4477 nginx     25   0  106m  78m  808 R  1.4  0.7   0:24.86 nginx:
worker process
   24 root      10  -5     0    0    0 S  0.7  0.0   0:00.04 [events/6]
 4337 root      15   0 90168 3420 2660 R  0.7  0.0   0:00.28 sshd:
root at pts/0
 5657 root      15   0 12764 1168  832 R  0.7  0.0   0:00.29 top -c -d
1
    1 root      15   0 10372  696  580 S  0.0  0.0   0:00.46 init [3]
    2 root      RT  -5     0    0    0 R  0.0  0.0   0:00.00
[migration/0]
    3 root      34  19     0    0    0 S  0.0  0.0   0:00.00
[ksoftirqd/0]
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00
[migration/1]
**************************************************************************************************

any other suggestions?


actually i dont get it. other server with the same hardware (but
litespeed) is doing its job, but this one dont? i thought nginx created
for especially these situations. why is setting up too difficult and
manual doesnt giving enough information.

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,212555,212607#msg-212607