slow response times

[alexey ptushkin]

  We are currently using nginx/1.7.10 @Ubuntu12.04 for serving high load 
webapp with about 12-15k QPS mostly tiny requests / HTTP/1.1 keep-alive 
mode.  Nginx is used as proxy for http backends. From time to time we 
are noticing traffic gaps and during this time nginx answers very slowly 
(up to 20-25s) across all  virtual hosts. There are no any error 
messages in general error log and also server has much of resources 
available. The nature of service is that http backends have very low 
read/connect proxy timeout value (40ms) and nginx should be able to send 
static file during error interception, those timeout happen very often 
(2-3k during a second) But unfortunately we cant make it stable enough 
to provide fast static answer during backend slowness periods since it 
hangs and answers very slow

Location configuration:

             access_log off;

             error_page 500 501 502 503 504 408 404 =200 /file.dat;

             include proxy_params;
             proxy_intercept_errors on;
             proxy_cache off;
             proxy_redirect off;
             proxy_pass_request_body on;
             proxy_pass_request_headers on;
             proxy_next_upstream off;
             proxy_read_timeout    40ms;
             proxy_connect_timeout 40ms;
             proxy_send_timeout    40ms;

             set $args src=g2;
             proxy_pass http://upstream/;
             proxy_http_version 1.1;
             proxy_set_header Connection "";
         }




Stub_status during normal operation:
Active connections: 831
server accepts handled requests
  919758 919758 945641607
Reading: 0 Writing: 44 Waiting: 787

I can share any needed debug/stats info for sure.

Thank you for your cooperation