Monthly Gateway Timeout
mdounin at mdounin.ru
Fri Mar 4 13:02:56 MSK 2011
On Fri, Mar 04, 2011 at 04:12:59AM -0500, Marc Kramis wrote:
> Hi All
> I'm experiencing a very strange network problem that occurs every four
> to six weeks and lasts for approximately one hour. I can not manually
> provoke the problem and, during this very hour, I can not resolve it
> even with rebooting the server. Even stranger, two physically separated
> servers suffer from the same problem at the same time. Both servers use
> nginx as SSL reverse proxy. Each server has disjunct domains to handle.
> During one hour, we just see: Nginx - Gateway Timout. After one hour, it
> suddenly works again. It started around four months ago. Note that all
> other network traffic is not affected, only nginx HTTPS and HTTP.
> So, what's in common with both servers:
> 1) Hardware (UltraSPARC T2 Plus).
Some endianness issue?...
> 2) OS (Solaris 10 U9 latest patch level).
> 3) Time (both servers use the exact same NTP-controlled time).
> 4) Switch.
> 5) Firewall (I replaced the firewall four weeks ago, the error still
> just appeared).
> 6) Nginx 0.8.46-0.8.54, same configuration but for different domains
> internally hosted at different servers, compiled with Solaris OpenSSL.
> I observed that nginx, during this mysterious hour, mistakenly proxies
> the requests back to the original IP on random ports instead of the
> proxy IP and that these requests are blocked by the firewall.
> Because two different machines are affected at the same time and it
> cannot be resolved by a restart of nginx or a reboot of the whole
> server, and it resolves itself after approximately one hour, my guess is
> that some time-dependent error occurs in nginx.
> I will replace nginx with apache to verify the problem actually is nginx
> and not the OS, switch are whatever and then wait and hope :-)
> Does anyone have an idea how to locate or investigate this problem?
You may want to follow http://wiki.nginx.org/Debugging and provide
config and debug log (if you are able to obtain one), as well as
nginx -V output. This may help to investigate the problem if the
issue is actually in nginx.
More information about the nginx