nginx serving wrong proxy content, static assets not affected

[Payam Chychi]

On Wed, Jan 4, 2023 at 12:33 PM Eduardo Kortright <ekortright at ewtn.com>
wrote:

> I have several servers hosting multiple Rails sites, using nginx as a
> reverse proxy. All sites have unique host names and, at least at first,
> nginx returns the content for each site correctly (dynamic content from
> Rails as well as static assets such as images, javascript and CSS).
>
> Both Rails and nginx are running in Docker containers. I am using nginx
> 1.23.1 running in a Docker image built from the official Debian Docker
> image (I only added certbot for TLS certificate processing). nginx connects
> to each proxy via HTTP using the internal service name defined in the
> Docker network by each docker-compose.yml file.
>
> For some reason, some time after nginx starts nginx gets confused and
> begins serving content from the wrong proxy. For instance, requesting a
> page from aaa.com returns the Rails content for bbb.com; requesting
> bbb.com returns content from ccc.com; and requesting ccc.com returns
> content from aaa.com. This problem only affects the proxy content; the
> static assets for aaa.com are returned as expected (and so on for all the
> other sites).
>
> This is not a problem of nginx simply returning content from the wrong
> site. If you request a page from aaa.com, since the dynamic content (HTML
> markup) comes from bbb.com, the markup will contain references to assets
> from bbb.com, but since the URLs are relative they will be requested from
> aaa.com; those assets return 404 Not found because they do not exist in
> aaa.com. If you change the URL manually and request them from bbb.com,
> they are returned with no problem, so it’s not that nginx can’t resolve the
> host name, just that the proxy content is being routed incorrectly.
>
> Also, I don’t believe this is a problem with Docker. When nginx gets
> confused, I can run a shell in any Docker container and connect directly to
> Rails (by running a cURL command pointing to the proxy URL configured for
> each site in nginx), and I get the correct content every time. It is only
> when I request it through nginx that the content comes from a different
> site than the one requested.
>
> I have no idea what triggers this behavior. Once it happens, the only
> thing that can be done to correct it is to restart nginx. After that (could
> be minutes, hours, or days), the server will function as expected once
> again. Since I am using this setup in several production servers, at first
> I created a cron job to restart nginx every day, then every hour, and
> finally I decided to poll the sites on each server every five minutes, so
> that if the responses don’t look right I can restart nginx without having
> users experience a lengthy interruption.
>
> I have confirmed that this problem occurs on more than one server
> (although on one server I have only observed it once). I have also set up a
> staging server that is as close to one of the production servers as
> possible, but so far the problem has not occurred there (since this staging
> server does not get any traffic, the problem may never surface if it is
> triggered by a particular kind of incoming request).
>
> I have asked a question at Server Fault (
> https://serverfault.com/questions/1117412/nginx-serving-content-from-wrong-proxy),
> where I have posted sanitized versions of the nginx configuration for two
> sample sites. I can post here the same or any other configuration that
> might help diagnose this. I have never seen nginx behave like this before
> (I have used it to host multiple Rails sites for years without any
> problems—only not in combination with Docker).
>
> Any suggestions as to what to look for or what to try would be most
> appreciated.
>
> —
> Eduardo Kortright
> EWTN Online Services
> (205) 271-2900
> (205) 332-4835 (cell)
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> https://mailman.nginx.org/mailman/listinfo/nginx
>
Sounds like you are either having hash collisions or incorrect dns
resolution.

- Are you caching? Set a larger key size
- Have you looked at dns and host resolution for the impacted requests?

—
Payam

-- 
Payam Tarverdyan Chychi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20230104/c2ec4847/attachment-0001.htm>