Debugging Nginx Memory Spikes on Production Servers

[Manuel]

Hello,

apparently you could look into dmesg. There should be a stacktrace of the process.

Also you could somehow start nginx with gdb.

You could also log all request and then when the server crashed try to replay them to be confident that the crash is reproducible.

What does ChatGpt says? 😅

Do you run the latest nginx version?

Any obscure modules / extensions?

Kind regards,
Manuel


> Am 20.09.2023 um 18:56 schrieb Lance Dockins <lance at wordkeeper.com>:
> 
> 
> Are there any best practices or processes for debugging sudden memory spikes in Nginx on production servers?  We have a few very high-traffic servers that are encountering events where the Nginx process memory suddenly spikes from around 300mb to 12gb of memory before being shut down by an out-of-memory termination script.  We don't have Nginx compiled with debug mode and even if we did, I'm not sure that we could enable that without overly taxing the server due to the constant high traffic load that the server is under.  Since it's a server with public websites on it, I don't know that we could filter the debug log to a single IP either.
> 
> Access, error, and info logs all seem to be pretty normal.  Internal monitoring of the Nginx process doesn't suggest that there are major connection spikes either.  Theoretically, it is possible that there is just a very large sudden burst of traffic coming in that is hitting our rate limits very hard and bumping the memory that Nginx is using until the OOM termination process closes Nginx (which would prevent Nginx from logging the traffic).  We just don't have a good way to see where the memory in Nginx is being allocated when these sorts of spikes occur and are looking for any good insight into how to go about debugging that sort of thing on a production server.
> 
> Any insights into how to go about troubleshooting it?
> 
> -- 
> Lance Dockins
> 
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> https://mailman.nginx.org/mailman/listinfo/nginx
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20230920/e6bd1954/attachment.htm>