Monthly Gateway Timeout

[Marc Kramis]

Hi Maxim, Hi Piotr

nginx -V:

nginx version: nginx/0.8.54
built by Sun C 5.10 SunOS_sparc Patch 141861-06 2010/07/28
TLS SNI support disabled
configure arguments: --with-cc=/opt/sunstudio12.1/bin/cc
--with-cpp=/opt/sunstudio12.1/bin/cc --with-cc-opt='-xtarget=ultraT2plus
-xO5 -I /usr/sfw/include' --with-ld-opt='-R/usr/sfw/lib -L/usr/sfw/lib'
--prefix=/nginx --user=daemon --group=daemon --with-http_ssl_module
--with-pcre=../pcre-8.12 --with-zlib=../zlib-1.2.5

Note that the bug also appeared with optimization level O3.


nginx.conf:

# --- Basic Configuration
-----------------------------------------------------

user                        daemon daemon;
error_log                   /nginx/logs/error.log warn;
ssl_engine                  pkcs11;
worker_processes            16;

events {
  worker_connections        256;
}

# --- HTTP Configuration
------------------------------------------------------

http {

  log_format                LOG '$remote_addr - $remote_user
[$time_local] "$request" $status $body_bytes_sent "$http_referer"
"$http_user_agent"';
  access_log                /nginx/logs/$host.access.log LOG;

  server_tokens             off;

  gzip                      on;
  gzip_vary                 on;
  gzip_proxied              any;
  gzip_types                text/plain text/xml text/css text/javascript
image/svg+xml application/xhtml+xml application/xml application/rss+xml
application/atom+xml application/x-javascript application/json;

  client_body_buffer_size   128k;
  client_max_body_size      256m;
  client_body_temp_path     /nginx/client_body_temp 1 2;

  proxy_read_timeout        3600;
  proxy_redirect            off;
  proxy_pass_header         Set-Cookie;
  proxy_temp_path           /nginx/proxy_temp;

  # --- https://foo -------------------------------------------

  server {

    listen                  446;
    server_name             foo;

    ssl                     on;
    ssl_certificate         /nginx/ssl/foo.crt;
    ssl_certificate_key     /nginx/ssl/foo.key;
    ssl_session_cache       shared:SSL:8m;

    location /bar {
      rewrite               ^/(.*)$ https://foo/bar/ permanent;
    }

    location /bar/ {
      proxy_pass            http://10.10.10.1:8080/bar/;
    }

    location / {
      rewrite               ^/(.*)$ https://foo permanent;
    }

  }

  server {

    listen                  80 default;
    server_name             _;
    server_name_in_redirect off;

    location / {
      rewrite               ^/(.*)$ http://foo permanent;
    }

  }

}


The error log is full of the following error (only during the
problematic hour):

2011/03/04 08:40:28 [error] 20062#0: *507995 upstream timed out (145:
Connection timed out) while reading response header from upstream,
client: ***IP***, server: ***SERVER***, request: "GET ***URL***
HTTP/1.1", upstream: "***UPSTREAM***", host: "***HOST***", referrer:
"***REFERER"


I just realized that only during this hour, the firewall lists blocked
outgoing traffic exactly to the client IPs of the error log at random
ports, i.e., I assume that during this hour, nginx mistakenly sends the
proxied request back to the client instead of the internal server.

Regards,
Marc

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,180289,180623#msg-180623