Proxy buffering and slow clients related issues

[rnmx18]

Hi,

We are trying to use NGINX for caching service in low bandwidth, high
latency mobile networks. The service is to stream 10-sec video segments of
different types ranging from 2MB to 50MB.

NGINX proxy_buffering configuration is as follows:

    proxy_buffering on;
    proxy_buffer_size          4k;
    proxy_buffers              64 4k;
    proxy_busy_buffers_size    128k;
    proxy_temp_file_write_size 64k;

The slow clients results in NGINX buffering of the response and writing data
to disk as part of temporary buffering.  The disk IO is causing higher TTFB
and higher load time for video download.

We have tried to configure the proxy_max_temp_file_size to 0 to disable the
buffering.  This change results in interrupts not being balanced as shown
below in the top command output  - core0 and core15 is using 100%.

top - 13:53:04 up 6 days,  2:31,  5 users,  load average: 6.42, 4.35, 3.84
Tasks: 370 total,   6 running, 364 sleeping,   0 stopped,   0 zombie
%Cpu0  :  0.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,100.0 si,  0.0
st
%Cpu1  :  7.9 us, 10.3 sy,  0.0 ni, 79.7 id,  0.0 wa,  0.0 hi,  2.1 si,  0.0
st
%Cpu2  : 18.7 us, 14.8 sy,  0.0 ni, 49.0 id,  0.0 wa,  0.0 hi, 17.5 si,  0.0
st
%Cpu3  : 18.7 us, 11.5 sy,  0.0 ni, 52.7 id,  0.0 wa,  0.0 hi, 17.2 si,  0.0
st
%Cpu4  : 20.1 us, 12.4 sy,  0.0 ni, 47.1 id,  0.0 wa,  0.0 hi, 20.5 si,  0.0
st
%Cpu5  : 22.2 us, 12.3 sy,  0.0 ni, 45.6 id,  0.0 wa,  0.0 hi, 19.9 si,  0.0
st
%Cpu6  : 15.1 us, 13.0 sy,  0.0 ni, 68.5 id,  0.0 wa,  0.0 hi,  3.4 si,  0.0
st
%Cpu7  :  9.6 us, 10.7 sy,  0.0 ni, 77.7 id,  0.3 wa,  0.0 hi,  1.7 si,  0.0
st
%Cpu8  :  9.9 us, 12.0 sy,  0.0 ni, 75.3 id,  0.0 wa,  0.0 hi,  2.7 si,  0.0
st
%Cpu9  : 11.7 us, 11.7 sy,  0.0 ni, 73.5 id,  0.0 wa,  0.0 hi,  3.1 si,  0.0
st
%Cpu10 : 10.0 us, 11.0 sy,  0.0 ni, 74.9 id,  0.0 wa,  0.0 hi,  4.1 si,  0.0
st
%Cpu11 :  5.5 us, 12.3 sy,  0.0 ni, 80.9 id,  0.0 wa,  0.0 hi,  1.4 si,  0.0
st
%Cpu12 :  1.7 us,  3.7 sy,  0.0 ni, 93.6 id,  0.0 wa,  0.0 hi,  1.0 si,  0.0
st
%Cpu13 :  2.0 us,  3.7 sy,  0.0 ni, 93.6 id,  0.0 wa,  0.0 hi,  0.7 si,  0.0
st
%Cpu14 :  1.4 us,  2.4 sy,  0.0 ni, 95.3 id,  0.7 wa,  0.0 hi,  0.3 si,  0.0
st
%Cpu15 :  0.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,100.0 si,  0.0
st
%Cpu16 : 18.0 us, 10.9 sy,  0.0 ni, 55.5 id,  0.0 wa,  0.0 hi, 15.6 si,  0.0
st
%Cpu17 : 18.8 us, 11.9 sy,  0.0 ni, 48.8 id,  0.0 wa,  0.0 hi, 20.4 si,  0.0
st
%Cpu18 :  6.4 us,  6.8 sy,  0.0 ni, 85.1 id,  0.0 wa,  0.0 hi,  1.7 si,  0.0
st
%Cpu19 :  3.8 us,  6.5 sy,  0.0 ni, 86.3 id,  0.0 wa,  0.0 hi,  3.4 si,  0.0
st
%Cpu20 :  1.7 us,  5.1 sy,  0.0 ni, 92.2 id,  0.3 wa,  0.0 hi,  0.7 si,  0.0
st
%Cpu21 :  2.4 us,  3.0 sy,  0.0 ni, 93.9 id,  0.0 wa,  0.0 hi,  0.7 si,  0.0
st
%Cpu22 :  1.0 us,  4.0 sy,  0.0 ni, 93.9 id,  0.0 wa,  0.0 hi,  1.0 si,  0.0
st
%Cpu23 :  1.3 us,  3.7 sy,  0.0 ni, 94.6 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0
st
KiB Mem : 11523494+total, 27725092 free, 10556008 used, 76953848 buff/cache
KiB Swap: 16777212 total, 16675408 free,   101804 used. 83445760 avail Mem 

Couple of queries:
a) Why do we get unbalanced interrupts when buffering is disabled?
b) How to configure NGINX to throttle the upstream read and avoid temp
buffering?

Thanks
Rajesh

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,276656,276656#msg-276656