Image serving via nginx are too slow, why ?

[tristanb]

Hello,

Before posting, i've done a lot of google search, and i found other similar
issue without any solutions.
So, Why nginx is slow serving files in my case ?

Part 1. The hardware

Intel Xeon E5-1620v2 4c/8t 3,7 GHz
32 Go DDR3 ECC 1600MHz
3xSSD Intel 160Go
1 Gbps

The load average is 0.13
I've been reproducing the "slow loading image effect" with all kind of
traffic situation from 0 to a lot.
I got 2 new servers load balanced, I disabled the 2nd server, and it had no
effect at all.

Part2. The software

Nginx 1.6.1
PHP-FPM 5.5.16
Varnish 4, but the cache is disabled for all requests in the .vcl

My nginx configuration can be found on this gist :
https://gist.github.com/tristanbes/ade26d83ce154f7d2630
My nginx vhost can be found there :
https://gist.github.com/tristanbes/f15e9269cf63b5a52340

You can "feel" the problem for example here:  http://bentm.webfactional.com/

It's not my website, but I took it from a post describing the same problem.

A 2 Mo PDF takes 8 sec to load on any connection internet (from 20Mbps @work
to my personal fiber 1Gbps) 
http://i.imgur.com/cxYRm8c.png

I'm sure I did tweaked too much without really knowing which side effect it
could have. Can you help me on this please ?

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,252816,252816#msg-252816