Optimize nginx for uploading/download huge files..

[ronjpark]

Hello everyone,

Firstly, I'm sorry for posting the same question here and there (How to
forum).
It is urgent issue to me.

Could you give me a tip for which directives and values I have to set
for the optimization?

I installed nginx on virtual machine as reverse proxy server.
It checks the authentication and bypasses the client request to the
backend servers.
The most transactions are to upload/download files, from 1M to 1G...

The CPU usage hits 100% when the 1M file download request reaches around
25 TPS..
around 20 TPS for upload request..

I set client_body_buffer_size, proxy_buffer_size, proxy_buffers and so
on..
But not so good yet.. :(

client_body_buffer_size is set to 2M and it helps to increase TPS for
uploading, but not enough.
proxy_buffer_size = 1M and proxy_buffers = 4 1M help to increase TPS for
downloading little bit, but not so much..
Other directives are set by its default.

The below is the result of profiling with google_proftools_module.
As you can see, the most time is spent on readv() and writev().

Please give me any advice or your experience.
Thank you for your time.

Total: 22244 samples
11928 53.6% 53.6% 11930 53.6% readv
8212 36.9% 90.5% 8212 36.9% writev
907 4.1% 94.6% 907 4.1% recv
359 1.6% 96.2% 359 1.6% __connect_nocancel
248 1.1% 97.3% 248 1.1% __close_nocancel
35 0.2% 97.5% 46 0.2% _IO_str_pbackfail
34 0.2% 97.7% 34 0.2% __write_nocancel
31 0.1% 97.8% 31 0.1% socket
28 0.1% 97.9% 28 0.1% memset
and
Total: 64475 samples
34117 52.9% 52.9% 34117 52.9% writev
14173 22.0% 74.9% 14181 22.0% readv
11981 18.6% 93.5% 11981 18.6% recv
1046 1.6% 95.1% 1049 1.6% __connect_nocancel
878 1.4% 96.5% 878 1.4% __close_nocancel
343 0.5% 97.0% 343 0.5% epoll_wait
160 0.2% 97.2% 160 0.2% __write_nocancel
89 0.1% 97.4% 89 0.1% __xstat64
81 0.1% 97.5% 115 0.2% _IO_str_pbackfail


Here is the information of my VM:
ronjpark at PSEProxy:~/release/REL.0.13$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 2
model name : QEMU Virtual CPU version 0.14.1
stepping : 3
cpu MHz : 2393.998
cache size : 4096 KB
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36
clflush mmx fxsr sse sse2 syscall nx lm up rep_good nopl pni cx16 popcnt
hypervisor lahf_lm
bogomips : 4787.99
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

ronjpark at PSEProxy:~/release/REL.0.13$ cat /proc/meminfo
MemTotal: 4057052 kB
MemFree: 2651832 kB
Buffers: 80984 kB
Cached: 1100692 kB
SwapCached: 3856 kB
Active: 325744 kB
Inactive: 910176 kB
Active(anon): 48884 kB
Inactive(anon): 5684 kB
Active(file): 276860 kB
Inactive(file): 904492 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 2095100 kB
SwapFree: 2082688 kB
Dirty: 4 kB
Writeback: 0 kB
AnonPages: 51356 kB
Mapped: 15740 kB
Shmem: 324 kB
Slab: 66252 kB
SReclaimable: 56796 kB
SUnreclaim: 9456 kB
KernelStack: 1160 kB
PageTables: 4124 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 4123624 kB
Committed_AS: 176112 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 28816 kB
VmallocChunk: 34359707236 kB
HardwareCorrupted: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 42996 kB
DirectMap2M: 4151296 kB
ronjpark at PSEProxy:~/release/REL.0.13$ uname -a
Linux PSEProxy 2.6.38-8-server #42-Ubuntu SMP Mon Apr 11 03:49:04 UTC
2011 x86_64 x86_64 x86_64 GNU/Linux

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,220778,220778#msg-220778