Monthly Gateway Timeout

[Maxim Dounin]

Hello!

On Sat, Mar 05, 2011 at 06:18:00AM -0500, Marc Kramis wrote:

[...]

>   log_format                LOG '$remote_addr - $remote_user
> [$time_local] "$request" $status $body_bytes_sent "$http_referer"
> "$http_user_agent"';
>   access_log                /nginx/logs/$host.access.log LOG;

Just a side note: this, along with no root specified in server{} 
blocks, makes your system vulnerable to inode exhaustion attack 
(i.e. attacker may create arbitrary number of log files on your 
system, eventually bringing it down).

[...]

>   proxy_read_timeout        3600;

Uhm... Its really big one.  It's unlikely that clients will wait 
for so long for an answer.

[...]

>     location /bar/ {
>       proxy_pass            http://10.10.10.1:8080/bar/;

With this configuration nginx will convert ip/port to binary form 
once (while reading config), and will use it (binary) while doing 
connect()'s.  It is very unlikely that something bad will happen 
in nginx at this point.

[...]

> The error log is full of the following error (only during the
> problematic hour):
> 
> 2011/03/04 08:40:28 [error] 20062#0: *507995 upstream timed out (145:
> Connection timed out) while reading response header from upstream,
> client: ***IP***, server: ***SERVER***, request: "GET ***URL***
> HTTP/1.1", upstream: "***UPSTREAM***", host: "***HOST***", referrer:
> "***REFERER"

Words "while reading response header" in error_log suggest that 
connection to upstream was established (and request was sent), but 
upstream failed to generate an answer in time.

> I just realized that only during this hour, the firewall lists blocked
> outgoing traffic exactly to the client IPs of the error log at random
> ports, i.e., I assume that during this hour, nginx mistakenly sends the
> proxied request back to the client instead of the internal server.

With your proxy_read_timeout it's very likely that appropriate 
firewall states for connections with clients were already expired, 
and that's why your firewall complains about nginx trying to 
reply "504 Gateway timeout" back to clients once it detects 
timeout.

>From here it looks like it's your backend problem, not nginx.  And 
strange firwall complains you see are due to timeouts 
misconfiguration (you have to configure timeouts on your firewall 
at least as big as ones in nginx).

Maxim Dounin