SO_REUSEPORT

[Valentin V. Bartenev]

On Friday 01 February 2019 11:04:50 Mathew Heard wrote:
> Hi All,
> 
> Hit a rather strange issue today on a production service where during a
> configuration reload (evident by the worker processes in the process of
> being shutdown). During this reload a percentage of connections were not
> getting accepted (and hence not processed). I was able to confirm that none
> of the processes were accepting the connections.
> 
> Our configuration includes the reuseport option so my theory was that for
> some reason connections were still being routed to the shutting down
> worker, which was not accepting new connections.
> 
> This has a striking resemblance to the HAProxy issues in previous years
> with connections received during live configuration reloads.

HAProxy case is irrelevant here, as nginx uses SO_REUSEPORT for another
purpose and in a different way.

> 
> Software Versions:
> nginx version: nginx/1.15.8
> linux kernel: 4.9.2
> 
> Does anyone have an idea of where to look for this issue? Or have you see
> anything like it before?
> 
> I am aware the kernel is a bit on the older side, anyone seen any relevant
> issues / patches?
> 
[..]

Have you changed the number of worker processes?

If you're using SO_REUSEPORT and has decreased the number of workers,
then it's expected that some connections will be lost on Linux during
reload.

  wbr, Valentin V. Bartenev