Strange Unit failure

[Andrew Clayton]

On Wed, 21 Jun 2023 13:48:19 +0000
Peter TKATCHENKO <p.tkatchenko at bimp.fr> wrote:

> Thanks for your answer, Andrew.

NP.

> >> External access to the application is proxied by HAProxy with sticky
> >> sessions configured. Direct access to NGINX servers is possible too
> >> (with another URLs).  
> > You mention proxies. Are you using the "proxy" setting in your config?  
> 
> I have
> 
> proxy_pass http://unit_backend$request_uri;
> 
> in NGINX config, that's all. Nothing in Unit config.

OK.
 
> >> Yesterday we began to (randomly) get 503 errors on different pages of
> >> the application. The same page could be OK for one user, but failed
> >> with 503 error for other user. The source of these errors was Unit. We  
> > Are the error cases using WebSockets?  
> 
> Non, we don't use WebSockets.

OK.
 
> > Do you see process exited on signal 11 (or maybe 6) messages in the unit
> > log for example or core dumps being generated?  
> 
> Yes, I see many logs with signal 11 just after several writev failed:

Right...
 
> 2023/06/19 16:52:06 [info] 56492#102962 *39777 writev(21, 3) failed (32: 
> Broken pipe)
> 2023/06/19 16:52:22 [info] 56492#102962 *39738 writev(22, 3) failed (32: 
> Broken pipe)
> 2023/06/19 16:52:46 [info] 56492#102968 *39753 writev(30, 3) failed (32: 
> Broken pipe)
> 2023/06/19 16:53:10 [alert] 56493#101705 app process 11701 exited on 
> signal 11
> 2023/06/19 16:53:11 [alert] 56493#101705 app process 84631 exited on 
> signal 11
> 2023/06/19 16:53:11 [alert] 56493#101705 app process 98305 exited on 
> signal 11
> 
> I don't think Unitd worker can write his dumps somewhere as it is 
> started as www user, so hi has no write permissions.

So this will be hard to diagnose without either a backtrace from a core
dump or a reliable way to reproduce it.

I'm not sure how to do it on FreeBSD (on Linux it's just a matter of
editing /proc/sys/kernel/core_pattern, and these days coredumps
get piped through to systemd by default), but perhaps you can have the
coredumps go to /var/tmp or something?

Or if you have an easy way to reproduce it?

Cheers,
Andrew