Buffering issues with nginx

[Dan34]

Hello Valentin,

> 1. Write socket buffer in kernel on node.js side where node.js writes
data.

we can throw this out from equation, as I measure my end time by the event
when socket is closed on nodejs side, (I use http1.0 from nginx to node to
make it simple for this case).

> 2. Read socket buffer in kernel for node.js connection from what nginx
reads data.

SO_RCVBUF shouldn't be over 64KB by default. What does nginx use, is there a
config that controls it?.. still this shouldn't be a big issue, I'm fine if
there is such a constant buf.


> 3. Heap memory buffer to that nginx reads data from kernel socket buffer
(controlled by proxy_buffers
> and proxy_buffer_size directives).
> 
> No buffering here means that nginx doesn't keep that data in buffers
> for some time, but writes it immediately to write socket buffer in kernel
> for client connection.

I'm trying to configure these to be skipped or used to minimum. E.g. I don't
wan any data to be held in these.

> 4. Write socket buffer in kernel for client connection where nginx writes
data.

SO_SNDBUF shouldn't be over 64KB by default, perhaps nginx changes it as
well. What's the value that nginx uses and is there a config that controls
it?

> 5. Read socket buffer in kernel for client connection from what wget reads
data.

We can throw this out from equation, we may assume these aren't used, as for
my test I use final time when wget finishes and prints stats. There is
obviously highly unlikely chance that wget actually reads data twice faster
from network, but shows slower speed in it's cli results and "waits" for
data even though it's already received and is in local buffers. This is
totally dumb and I don't think this might happen, but I could check with
wireshark just in case.


In short, these could affect my case: SO_RCVBUF, SO_SNDBUF on nginx side and
whatever buffering nginx uses for handling data. I run that same test with
25MB data and I got totally identical result: 12.5MB was buffered on nginx
side. That stuff that could affect my case cannot really add up to 12.5MB
and 10 minute of time.
There is a wild possibility that tcp window scaling resulted in some huge
window on node->nginx side and ended up storing that 12MB in tcp window
itself but i'm not sure if TCP window should be accounted into these
SO_RCVBUF or that RCVBUF is extra data on top of internals of TCP.

So,.. any ideas how come nginx ends up buffering 12.5MB data?

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,275526,275608#msg-275608