bad performance with nginx cache

[Igor Sysoev]

On Mon, Aug 30, 2010 at 10:56:45PM +0400, Franchoze Eric wrote:

> 
> 
> 30.08.10, 19:40, "Igor Sysoev" <igor at sysoev.ru>:
> 
> > On Mon, Aug 30, 2010 at 02:15:29PM +0400, Franchoze Eric wrote:
> >  
> >  > Interesing may email got to the list, but I did not get your answer.
> >  
> >  Because you has not subscribed to the list. The email has been
> >  bounced to me and I have added your address to the write only list.
> >  
> >  > > What OS do you use and how much memory do you have on the host?
> >  > 
> >  > Linux 2.6.32+, CentOS 5.2, 15GB of RAM. - so 3 GB of ram should be enought.
> >  
> >  What do
> >  cat /proc/sys/fs/inode-max
> >  cat /proc/sys/fs/inode-nr
> >  find /var/www/cache2 | wc -l
> >  show ?
> >  
> >  
> >  
> 
> #cat /proc/sys/fs/file-max 
> 1646126
> #cat /proc/sys/fs/inode-nr 
> 230866	0
> #mount |grep cache2
> tmpfs on /var/www/cache2 type tmpfs (rw)
> #find /var/www/cache2 | wc -l
> 164865
> 
> Igor, I found that with my first settings (cache settings from first email) nginx god segfaults each 15 minutes at different random places. I changed invalidate period to 15 minutes,
> cache time to 10 minutes, removed my ugly proxy_cache_key and limited to cache only one destination. Load avarage is rapidly decreased. No segfaults. I will do some tests to verify impact on turning on cache with high load and email results.
> 
> Got segfault in that places:
> Core1:
> [New process 24925]
> #0  0x0000000000427c49 in ngx_http_request_handler ()
> (gdb) bt
> #0  0x0000000000427c49 in ngx_http_request_handler ()
> #1  0x000000000041e777 in ngx_epoll_process_events ()
> #2  0x0000000000417978 in ngx_process_events_and_timers ()
> #3  0x000000000041d3b0 in ngx_worker_process_cycle ()
> #4  0x000000000041bcf7 in ngx_spawn_process ()
> #5  0x000000000041db99 in ngx_master_process_cycle ()
> #6  0x00000000004044da in main ()
> (gdb)
> 
> 
> Core2:
> (gdb) bt
> #0  0x000000348d47bf0b in memcpy () from /lib64/libc.so.6
> #1  0x0000000000408851 in ngx_vslprintf ()
> #2  0x0000000000409445 in ngx_snprintf ()
> #3  0x00000000004285e8 in ngx_http_log_error_handler ()
> #4  0x00000000004286fc in ngx_http_log_error ()
> #5  0x0000000000404a3d in ngx_log_error_core ()
> #6  0x000000000040dd6a in ngx_connection_error ()
> #7  0x000000000041a681 in ngx_unix_recv ()
> #8  0x000000000043840d in ngx_http_upstream_process_header ()
> #9  0x00000000004368e2 in ngx_http_upstream_handler ()
> #10 0x000000000041e716 in ngx_epoll_process_events ()
> #11 0x0000000000417978 in ngx_process_events_and_timers ()
> #12 0x000000000041d3b0 in ngx_worker_process_cycle ()
> #13 0x000000000041bcf7 in ngx_spawn_process ()
> #14 0x000000000041db99 in ngx_master_process_cycle ()
> #15 0x00000000004044da in main ()
> (gdb)
> 
> Core3:
> (gdb) bt
> #0  0x000000348d430265 in raise () from /lib64/libc.so.6
> #1  0x000000348d431d10 in abort () from /lib64/libc.so.6
> #2  0x000000348d46a84b in __libc_message () from /lib64/libc.so.6
> #3  0x000000348d4722ef in _int_free () from /lib64/libc.so.6
> #4  0x000000348d47273b in free () from /lib64/libc.so.6
> #5  0x0000000000405038 in ngx_destroy_pool ()
> #6  0x0000000000427e8a in ngx_http_close_connection ()
> #7  0x0000000000428bf7 in ngx_http_keepalive_handler ()
> #8  0x0000000000417a99 in ngx_event_expire_timers ()
> #9  0x00000000004179c5 in ngx_process_events_and_timers ()
> #10 0x000000000041d3b0 in ngx_worker_process_cycle ()
> #11 0x000000000041bcf7 in ngx_spawn_process ()
> #12 0x000000000041ca49 in ngx_start_worker_processes ()
> #13 0x000000000041d914 in ngx_master_process_cycle ()
> #14 0x00000000004044da in main ()
> (gdb)
> 
> Core4:
> (gdb) bt
> #0  0x000000000043bf23 in ngx_http_range_body_filter ()
> #1  0x0000000000421e89 in ngx_http_output_filter ()
> #2  0x0000000000437270 in ngx_http_upstream_process_non_buffered_request ()
> #3  0x000000000043754a in ngx_http_upstream_process_non_buffered_upstream ()
> #4  0x00000000004368e2 in ngx_http_upstream_handler ()
> #5  0x000000000041e716 in ngx_epoll_process_events ()
> #6  0x0000000000417978 in ngx_process_events_and_timers ()
> #7  0x000000000041d3b0 in ngx_worker_process_cycle ()
> #8  0x000000000041bcf7 in ngx_spawn_process ()
> #9  0x000000000041ca49 in ngx_start_worker_processes ()
> #10 0x000000000041d914 in ngx_master_process_cycle ()
> #11 0x00000000004044da in main ()
> (gdb) 

Unfortunatly, these backtraces have not debug information.
What does "nginx -V" show ?


-- 
Igor Sysoev
http://sysoev.ru/en/