upgrading binary failed - execve - too long argument list

[Charlie Kilo]

Thanks Maxim for further explaining!

We do listen to a huge number of IPs with tons of traffic and huge spikes
on them. We really need to avoid any type
of congestion, therefore the reuseport.

While many of the ip:port combos are simply there for failover purposes and
actually aren't "in use", I see right now
no feasible way to reduce the number of listening sockets before the
upgrade and restore them afterwards. That would be hugely complex,
error-prone and would still leave us with a window where the instant
failover wouldn't work as expected.

As we run a custom kernel anyways, i'll still try to look into adjusting
MAX_ARG_STRLEN as a workaround.
Having a solution that wouldn't make this necessary would be simply a blast
though and greatly appreciated :-)


On Fri, Apr 30, 2021 at 11:14 AM Charlie Kilo <krikkiteer at gmail.com> wrote:

> correction.. if i count with
> ss -l -t |grep http | wc -l
> we have around 58340 listening sockets.. at least on that machine..
>
>
> On Fri, Apr 30, 2021 at 8:27 AM Charlie Kilo <krikkiteer at gmail.com> wrote:
>
>> Thanks a lot for the hints so far.. to provide the further info and
>> answer the questions..
>>
>> getconf ARG_MAX shows 2097152
>> ulimit -s shows 8192
>> setting it to unlimited, doesn't change anything (also not with prlimit)
>> wc -c /proc/<pid>/environ shows 1949
>>
>> it seems on a regular machine we have around 1200 listening sockets and do indeed use "reuseport"
>>
>> nginx -V shows
>> nginx version: nginx/1.18.0
>> built with OpenSSL 1.1.1j  16 Feb 2021
>> TLS SNI support enabled
>> configure arguments: --with-ipv6 --with-file-aio --with-http_ssl_module --with-http_v2_module --with-http_realip_module --with-http_stub_status_module --with-http_gzip_static_module --with-http_sub_module --without-http_scgi_module --with-stream --with-stream_ssl_module --with-stream_realip_module --with-stream_geoip_module --with-stream_ssl_preread_module --with-http_slice_module --add-module=../lua-nginx-module-0.10.15 --add-module=../headers-more-v0.25 --add-module=../ngx-devel-kit-master --add-module=../dist/nginx-rtmp-module --add-module=../build/nginx_upstream_check_module --conf-path=/etc/nginx.conf --with-mail --with-mail_ssl_module --without-mail_pop3_module --with-cc-opt='-D_FORTIFY_SOURCE=1 -fstack-protector -fstack-clash-protection -pipe -march=westmere -mtune=intel -O3 -I/build/nginx/boringssl/.compat/include -g' --with-ld-opt='-Wl,-z,relro,-z,now,-lgmp -ldl'
>>
>>
>>
>> On Tue, Apr 27, 2021 at 12:35 PM Charlie Kilo <krikkiteer at gmail.com>
>> wrote:
>>
>>> Hi everyone,
>>> i'm trying to upgrade an nginx-binary while running.
>>> When i do kill -s USR2 <pid>, i get the following error in the logs..
>>>
>>> 11:40:38 [alert] 52701#0: execve() failed while executing new binary
>>> process "/opt/sbin/nginx" (7: Argument list too long)
>>>
>>> Anybody knows what exactly is in those arguments ? We have ~ 20-55
>>> worker processes if that might be related..
>>> nginx-version is 1.18.0, os: debian buster
>>>
>>> thanks everyone in advance!
>>> chris
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20210508/94560edf/attachment.htm>