upgrading binary failed - execve - too long argument list

[Maxim Dounin]

Hello!

On Fri, Apr 30, 2021 at 11:14:48AM +0200, Charlie Kilo wrote:

> correction.. if i count with
> ss -l -t |grep http | wc -l
> we have around 58340 listening sockets.. at least on that machine..

Since "listen ... reuseport;" implies a separate listening socket 
for each worker process, that's expected with 50 worker processes 
and 1200 "listen ... reuseport" in the configuration.

> On Fri, Apr 30, 2021 at 8:27 AM Charlie Kilo <krikkiteer at gmail.com> wrote:
> 
> > Thanks a lot for the hints so far.. to provide the further info and answer
> > the questions..
> >
> > getconf ARG_MAX shows 2097152
> > ulimit -s shows 8192
> > setting it to unlimited, doesn't change anything (also not with prlimit)
> > wc -c /proc/<pid>/environ shows 1949
> >
> > it seems on a regular machine we have around 1200 listening sockets and do indeed use "reuseport"

Thanks, so all seem to be using the defaults.

Looking more closely at Linux limits, it seems that you are 
hitting the MAX_ARG_STRLEN limit.  It limits single argument 
length (and a single environment variable length), and hardcoded 
to be 32 pages, that is, almost always 128k.  

With 128k limit nginx should be able to pass about 20k listening 
sockets, so with 58340 sockets you are well above the limit.

An obvious workaround would be to reduce the amount of listening 
sockets - either by dropping the "reuseport", or by reducing the 
the number of listening sockets (normally just one on the wildcard 
address is enough), or by reducing the number of worker processes.

It would be interesting to know why you are using such a large 
number of listening sockets with reuseport enabled.  If the use 
case is reasonable, we probably should consider implementing some 
workaround for the 128k limit.

-- 
Maxim Dounin
http://mdounin.ru/