Image Hosting

[Maxim Dounin]

Hello!

On Thu, Oct 14, 2010 at 12:26:59PM +0000, iptablez at yahoo.com wrote:

> Actually the image size is arround 100KB each. The server is running in 250Mbps traffic. 
> I already described the disk I'm using is scsi 15K RPM in raid 0

Basic tunings you have to apply when serving static which doesn't 
fit into memory are:

If you use sendfile:

- Make sure your OS uses appropriate read-ahead for sendfile to 
avoid trashing disks with small requests (and seeks).  For FreeBSD 
8.1+ it should be enough to set read_ahead directive in nginx 
config (0.8.18+).

http://sysoev.ru/nginx/docs/http/ngx_http_core_module.html#read_ahead
(in Russian)

Using bigger socket buffers (listen ... sndbuf=... in nginx 
config) may help too.

- If serving large files - make sure you use appropriate 
sendfile_max_chunk to avoid blocking nginx worker on disk for too 
long.

- Consider switching sendfile off if you can't persuade it to read 
large blocks from disk.

If not using sendfile:

- Tune output_buffers (again, to avoid trashing disks with small 
requests and seeks).  Default is 2x32k, which will result in 4 
disk requests for 100k file.  Changing it to 1x128k would result 
in 2x memory usage but 4x less disk requests, this is probably 
good thing to do if you are disk-bound.

In both cases:

- Using aio may help a lot ("aio sendfile" is only available under 
FreeBSD) by adding more concurency to disk load and generally 
improving nginx interactivity.  Though right now it requires 
patches to avoid socket leaks, see here:

http://nginx.org/pipermail/nginx-devel/2010-October/000498.html

- Using directio may help to improve disk cache effictiveness by 
excluding large files (if you have some) from cache.   Though keep 
in mind that it disables sendfile so if you generally tune for 
sendfile - you may have to apply output_buffers tunings as well.

It's hard to say anything more than this without knowing lots of 
details.

Maxim Dounin