<div><br></div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jan 4, 2023 at 12:33 PM Eduardo Kortright <<a href="mailto:ekortright@ewtn.com">ekortright@ewtn.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div name="messageBodySection">
<div dir="auto">I have several servers hosting multiple Rails sites, using nginx as a reverse proxy. All sites have unique host names and, at least at first, nginx returns the content for each site correctly (dynamic content from Rails as well as static assets
such as images, javascript and CSS).<br>
<br>
Both Rails and nginx are running in Docker containers. I am using nginx 1.23.1 running in a Docker image built from the official Debian Docker image (I only added certbot for TLS certificate processing). nginx connects to each proxy via HTTP using the internal
service name defined in the Docker network by each docker-compose.yml file.<br>
<br>
For some reason, some time after nginx starts nginx gets confused and begins serving content from the wrong proxy. For instance, requesting a page from <a href="http://aaa.com" target="_blank">aaa.com</a> returns the Rails content for <a href="http://bbb.com" target="_blank">bbb.com</a>;
requesting <a href="http://bbb.com" target="_blank">bbb.com</a> returns content from <a href="http://ccc.com" target="_blank">ccc.com</a>; and requesting <a href="http://ccc.com" target="_blank">ccc.com</a> returns content from <a href="http://aaa.com" target="_blank">aaa.com</a>.
This problem only affects the proxy content; the static assets for <a href="http://aaa.com" target="_blank">aaa.com</a> are returned as expected (and so on for all the other sites).<br>
<br>
This is not a problem of nginx simply returning content from the wrong site. If you request a page from <a href="http://aaa.com" target="_blank">aaa.com</a>, since the dynamic content (HTML markup) comes from <a href="http://bbb.com" target="_blank">bbb.com</a>,
the markup will contain references to assets from <a href="http://bbb.com" target="_blank">bbb.com</a>, but since the URLs are relative they will be requested from <a href="http://aaa.com" target="_blank">aaa.com</a>; those assets return 404 Not found because
they do not exist in <a href="http://aaa.com" target="_blank">aaa.com</a>. If you change the URL manually and request them from <a href="http://bbb.com" target="_blank">bbb.com</a>, they are returned with no problem, so it’s not that nginx can’t resolve the
host name, just that the proxy content is being routed incorrectly.<br>
<br>
Also, I don’t believe this is a problem with Docker. When nginx gets confused, I can run a shell in any Docker container and connect directly to Rails (by running a cURL command pointing to the proxy URL configured for each site in nginx), and I get the correct
content every time. It is only when I request it through nginx that the content comes from a different site than the one requested.<br>
<br>
I have no idea what triggers this behavior. Once it happens, the only thing that can be done to correct it is to restart nginx. After that (could be minutes, hours, or days), the server will function as expected once again. Since I am using this setup in several
production servers, at first I created a cron job to restart nginx every day, then every hour, and finally I decided to poll the sites on each server every five minutes, so that if the responses don’t look right I can restart nginx without having users experience
a lengthy interruption.<br>
<br>
I have confirmed that this problem occurs on more than one server (although on one server I have only observed it once). I have also set up a staging server that is as close to one of the production servers as possible, but so far the problem has not occurred
there (since this staging server does not get any traffic, the problem may never surface if it is triggered by a particular kind of incoming request).<br>
<br>
I have asked a question at Server Fault (<a href="https://serverfault.com/questions/1117412/nginx-serving-content-from-wrong-proxy" target="_blank">https://serverfault.com/questions/1117412/nginx-serving-content-from-wrong-proxy</a>), where I have posted sanitized
versions of the nginx configuration for two sample sites. I can post here the same or any other configuration that might help diagnose this. I have never seen nginx behave like this before (I have used it to host multiple Rails sites for years without any
problems—only not in combination with Docker).<br>
<br>
Any suggestions as to what to look for or what to try would be most appreciated.</div>
</div>
<div name="messageSignatureSection"><br>
<div>—
</div></div></div><div><div name="messageSignatureSection"><div><div dir="auto">Eduardo Kortright</div>
<div dir="auto">EWTN Online Services</div>
<div dir="auto">(205) 271-2900</div>
<div dir="auto">(205) 332-4835 (cell)</div>
</div>
</div>
</div>
_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a><br>
<a href="https://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">https://mailman.nginx.org/mailman/listinfo/nginx</a><br>
</blockquote></div></div><div dir="auto">Sounds like you are either having hash collisions or incorrect dns resolution.</div><div dir="auto"><br></div><div dir="auto">- Are you caching? Set a larger key size</div><div dir="auto">- Have you looked at dns and host resolution for the impacted requests?</div><div dir="auto"><br></div><div dir="auto">—</div><div dir="auto">Payam</div><div dir="auto"><br></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Payam Tarverdyan Chychi<br></div></div>