nginx slow at streaming flv or strange behavior

[Maxim Dounin]

Hello!

On Thu, Apr 22, 2010 at 04:54:05AM -0400, vnjug wrote:

> I have installed nginx 0.8.35 (flv streaming and h264 module 
> enabled) on my production video streaming server to see if it 
> can perform better than lighttpd.
> 
> However, lighttpd outperforms nginx
> 
>      + speed
>      + throughput
> 
> Here is my settings 
> 
> + CentOS release 5.3 (Final) with Lustre 1.8 patch
> + Kernel 2.6.18-92.1.17.el5_lustre.1.8.0smp x64
> + lighttpd-1.5.0 
> 
>      server.network-backend = "gthread-aio"
>      server.max-read-threads = 120
>      server.stat-cache-engine = "fam"
>      server.event-handler = "linux-sysepoll"
>      server.max-worker = 25
> 
> + nginx 0.8.35 
> 
>      worker_processes 50; # I have tested with 4, 8, 20 
>      processes
>      worker_connections  2048;
>      sendfile     on;
>      tcp_nopush   on;
>      tcp_nodelay  off;
> 
> + iptraf is used to monitor network activity in real time
> 
> All the video files is LustreFS based, network mounted. The real 
> data is stored on 6 storage servers (Lustre OSS)
> 
> I have configured 2 1-GB network cards: one for outgoing data 
> (streaming via port 80) and the another for Lustre client, 
> reading data from storage servers

You may want to use proxy instead of network filesystem (as nginx 
normally blocks on file IO requests, which likely to block the 
whole worker process for a relatively long time in case of network 
filesystem).

Alternatively you may try to use aio in nginx, but

1. This doesn't help with mp4 streaming much, see below.

2. It has some known not-yet-fixed bugs which may cause socket 
leak.

3. It requires O_DIRECT under Linux and this may not be best 
choise.

> 
> nginx
> =====
> 
> When I turned nginx on, I saw strange things on network activity 
> ( I used iptraf to see network activity per network cards)
> 
>   + Incoming data: data read from storage servers: 250 - 290 
>   MBit/s (better than lighttpd - see below)
>   + Outgoing data: data sent to clients (flash players, VLC 
>   ...): 90 - 140 Mbit/s (much worse than lighttpd - see below)
> 
> It means that nginx can read more data from disk than the 
> throughput it can send to clients. 

Mp4 streaming module is known to have non-optimal working model: 
it get time position from client and then scans mp4 file for 
coresponding file position.  This may cause very strange effects.  
Could you please confirm that you are seeing the same picture with 
flv streaming only?

[..]

> But I still don't know why nginx is slower than lighttpd on 
> sending data to clients while it can read more data from disk 
> than the counterpart.
> 
> Any thought?

You may try to play with:

1. Using sendfile_max_chunk (if not yet).

2. Switching sendfile off and tuning output_buffers instead (i.e. 
use small number of big buffers, something like "output_buffers 1 
512k;").  This is known to perform better when serving large files 
as it reduces number of requests to disks.  Alternative aproach 
would be to tune your os sendfile() to do reads of appropriate 
sizes, but I'm not sure it's possible under Linux.

3. Using aio, see above.  Though I suppose it don't help with mp4 
streaming as it uses blocking read()'s by itself.

4. Using directio and read_ahead directives.

5. Using proxy instead of network filesystem (and tuning proxy 
module).

Maxim Dounin