Nginx request processing is slow when logging disabled

[Maxim Dounin]

Hello!

On Sat, Jul 13, 2019 at 09:50:50AM -0400, heythisisom wrote:

> Hi Maxim,
> 
> The nginx reverse proxy and uWSGI runs on the same host. Each nginx reverse
> proxies are connected to only one single Instance of the uWSGI backend. 
> 
> But in the uWSGI backend, I'm running 4 workers in total based on the
> configuration 2 workers can be handled by 1 VCPU. Essentially the Instance I
> run has 2 VCPUs hence It translates to 4 workers.

4 uWSGI workers means that only 4 requests can be handled in 
parallel.  As long as your code uses external resources, such as 
databases or external requests, this may be way too low.

> The listen queue length of
> my backend is sufficiently high i.e. 4096 and I have set my somaxconn
> parameter to 32768. So I think everything with respect to the backend seems
> fine.

To large listen queue length might be a reason which actually 
causes the 499 errors you've observed.  For example, assuming 
processing each request requires 1 second on your backend, and 
given you have 4 backend workers, listen queue length of 4096 
translates to 1000 seconds delay due to queueing.  You may want to 
monitor actual number of queued connection requests in the 
listening socket queue when the issue happens.  Assuming you are 
using Linux, try "ss -nlt" see queue sizes and numbers of 
currently queue connection requests.

> Only when I disable logging in nginx I could see this Issue happen. Once I
> enable it, my hosts never raised Timeout error at all. Also, note that this
> Issue happens more often when the server is in an Idle state and not when
> the server is in peak.

Disabling/enabled logging implies slightly different load pattern, 
might change various timings and/or OS scheduler behaviour, and 
hence change the observed results - either by triggering bugs in 
various places (including kernel, nginx, and your backend) or 
simply by making things less efficient.  What exactly happens requires 
further investigation.  You may start with looking at the "ss 
-nlt" numbers as suggested above.

-- 
Maxim Dounin
http://mdounin.ru/