nginx slow at streaming flv or strange behavior

[vnjug]

Hi all,

I have installed nginx 0.8.35 (flv streaming and h264 module enabled) on my production video streaming server to see if it can perform better than lighttpd.

However, lighttpd outperforms nginx

     + speed
     + throughput

Here is my settings 

+ CentOS release 5.3 (Final) with Lustre 1.8 patch
+ Kernel 2.6.18-92.1.17.el5_lustre.1.8.0smp x64
+ lighttpd-1.5.0 

     server.network-backend = "gthread-aio"
     server.max-read-threads = 120
     server.stat-cache-engine = "fam"
     server.event-handler = "linux-sysepoll"
     server.max-worker = 25

+ nginx 0.8.35 

     worker_processes 50; # I have tested with 4, 8, 20 processes
     worker_connections  2048;
     sendfile     on;
     tcp_nopush   on;
     tcp_nodelay  off;

+ iptraf is used to monitor network activity in real time

All the video files is LustreFS based, network mounted. The real data is stored on 6 storage servers (Lustre OSS)

I have configured 2 1-GB network cards: one for outgoing data (streaming via port 80) and the another for Lustre client, reading data from storage servers

nginx
=====

When I turned nginx on, I saw strange things on network activity ( I used iptraf to see network activity per network cards)

  + Incoming data: data read from storage servers: 250 - 290 MBit/s (better than lighttpd - see below)
  + Outgoing data: data sent to clients (flash players, VLC ...): 90 - 140 Mbit/s (much worse than lighttpd - see below)

It means that nginx can read more data from disk than the throughput it can send to clients. 

At 1300 - 1400 concurrent connections, it is expected to wait 42 - 51 s before my VLC player can create a new streaming connection.

lighttpd
=======

After that I turned off nginx and switched on lighttpd. The network activity seemed to be much more logical.

  + Incoming data: data read from storage servers: 180 - 220 MBit/s
  + Outgoing data: data sent to clients (flash players, VLC ...): 190 - 230 Mbit/s 

It means that lighttpd can send everything it read from the disk to clients.

At 1300 - 1400 concurrent connections, it is expected to wait 16 - 23 s before my VLC player can create a new streaming connection. Much faster than in nginx.

I can not use sendfile on lighttpd because it is extremely slow.

Conclusion
==========
25 lighttpd worker processes outperforms 50 nginx processes in term of 

  + response time
  + throughput

==========

What I really want to replace lighttpd with nginx is that lighttpd 
  + crashed many time a day 
  + caused very high load average.
  + unable to killall because it can become zombie after that. Every lighttpd processes switches to S status while streaming. nginx switches to D status.

But I still don't know why nginx is slower than lighttpd on sending data to clients while it can read more data from disk than the counterpart.

Any thought?

Regards,

Hoang

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,77934,77934#msg-77934