D state when high load on same static file.

[Maxim Dounin]

Hello!

On Wed, Dec 21, 2011 at 11:22:58PM -0500, atrus wrote:

> I have some file hosting servers running nginx, serving static files
> which size average from 500MB to 6GB.
> The servers use Lustre1.8 as cluster filesystem.
> Files stay on some raid6 array with stripe size = 512KB.
> 
> At normal situation, nginx works very well. 

[...]

> But, when in hot situation (when a new hot file, may be a porn dvd or
> JAV recently uploaded), there are a lot of clients (more than 800)
> download that file at the same time, and nginx processes are going to D
> state. 
> At last, nearly all processes are D state ! This makes the download
> speed very slow :(

[...]

> I have tried many time tuning the number of process worker, but I didn't
> work !
> How can I fix this ? I think when a lot off client access the same file,
> it suppose to be better because of caching ?!

No, filesystem cache isn't supposed to work as you specifically 
requested directio:

[...]

>         directio        1m;

[...]

You may try removing the "directio" directive to see if it helps, 
but with large working set it make make things worse.

Also it's a good idea to try using AIO, see
http://nginx.org/en/docs/http/ngx_http_core_module.html#aio

Maxim Dounin