Disable proxy buffering for websockets

[Maxim Dounin]

Hello!

On Tue, Dec 18, 2018 at 11:58:49AM -0500, thehunmonkgroup wrote:

> I could deal with *some* inaccuracy, but the results are completely out of
> whack. Downloading 256KB of data via the websocket over a poor DSL
> connection happens near instantaneously from the websocket server's point of
> view, which to me indicates that Nginx is consuming all that data in a
> buffer instead of passing it along to the client without buffering.

Well, 256KB is likely several times smaller than socket buffers 
used, and you are going to see problems if you are testing with 
such small sizes without also tuning socket buffers.

E.g., on Linux default socket buffers sizes are autoscaled 
depending on the connection speed, and can be up to several 
megabytes between nginx and the backend, as these are on a fast 
connection.

> You mentioned that there's a proxy buffer within Nginx in the case of
> websockets, is there a setting to disable that? The ' proxy_buffering off;'
> setting I mentioned previously didn't seem to do it.

No.  To copy data from one socket to another you need a buffer.  
You can control size of the buffer nginx use internally via the 
proxy_buffer_size directive (see http://nginx.org/r/proxy_buffer_size).
But the default size is pretty low - 4k - so this is unlikely the 
source of your problems, unless you've tuned it to a larger value 
yourself.

Most likely, you have to tune socket buffers to be smaller to get 
more accurate results.  On Linux, socket buffers can be tuned 
using the net.ipv4.tcp_rmem and net.ipv4.tcp_wmem on Linux.  Also, 
in nginx itself you can control socket buffers towards the client 
using the "sndbuf" parameter of the "listen" directive 
(http://nginx.org/r/listen), but this is unlikely to be enough in 
such a setup.

Note well that measuring connection speed on the server side might 
not be a good idea, as this will inevitably lead to inacurate 
results.

-- 
Maxim Dounin
http://mdounin.ru/