From nginx-forum at forum.nginx.org Wed Aug 1 08:50:27 2018 From: nginx-forum at forum.nginx.org (shiver25) Date: Wed, 01 Aug 2018 04:50:27 -0400 Subject: Nginx mail proxy LDAP iRedMail Message-ID: <06115d20dc20b7c25dd7ea7c54989c3f.NginxMailingListEnglish@forum.nginx.org> Hi there, I try configure a little mail infrastructure but i have problem with this. So i have exacly three servers. One is MX (frontend) there is nginx with configuration: user nginx; worker_processes 2; error_log /var/log/nginx/error.log info; pid /var/run/nginx.pid; load_module /usr/lib64/nginx/modules/ngx_http_perl_module.so; load_module /usr/lib64/nginx/modules/ngx_mail_module.so; events { worker_connections 1024; multi_accept on; } http { perl_modules perl/lib; perl_require mailauth.pm; server { location /auth { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; perl mailauth::handler; } } } mail { auth_http 127.0.0.1:80/auth; pop3_capabilities "TOP" "USER"; imap_capabilities "IMAP4rev1" "UIDPLUS"; server { listen 110; protocol pop3; proxy on; } server { listen 143; protocol imap; proxy on; } server { listen 25; protocol smtp; proxy on; } } And i try write auth script in perl, look like: package mailauth; use strict; use warnings; use nginx; use Net::LDAP; my $mail_server1 = "10.12.1.109"; my $mail_server2 = "10.12.1.109"; our $mail_server_ip={}; our $protocol_ports={}; $mail_server-ip->{'mailhost01'}="10.12.1.109"; $mail_server_ip->{'mailhost02'}="192.168.1.33"; $protocol_ports->{'pop3'}=110; $protocol_ports->{'imap'}=143; my $ldapconnect = Net::LDAP->new( "10.12.1.109", version => 3, port => 389 ) or die $@; my $bind = $ldapconnect->bind( "cn=vmail,dc=poczta,dc=coml", password => "PPkRSNeYtIDm7QXAq7Dr" ); if ( $bind->code ) { LDAPerror( "Bind: ", $bind); } sub handler { my $r = shift; our $mail_server; my $auth_user->execute($r->header_in("Auth-User")); if ($auth_user =~ m/^[abcdefghijklmp]/) { $mail_server = $mail_server1; } else { $mail_server = $mail_server2; } my $search = $ldapconnect->search( base => "o=domains,dc=poczta,dc=com", filter => '(&(mail=' . $r->header_in("Auth-User") . '))' ); my $goto = $search->entry(0)->get_value('mail'); $r->header_out( "Auth-Status", "OK" ); $r->header_out( "Auth-Server", $mail_server); $r->header_out( "Auth-Port", $protocol_ports->{$r->header_in("Auth-Protocol")}); $r->send_http_header("text/html"); return OK; } 1; $ldapconnect->unbind; __END__ Two backend servers installed with LDAP form iRedMail package. I want have two servers backend with half and half users. So i add to script logic like: our $mail_server; my $auth_user->execute($r->header_in("Auth-User")); if ($auth_user =~ m/^[abcdefghijklmp]/) { $mail_server = $mail_server1; } else { $mail_server = $mail_server2; } Check with curl: curl -i -H 'Auth-User: postmaster at com' -H 'Auth-Pass: supersecret' -H 'Auth-Protocol: imap' 10.12.1.128:80/auth and ive got: HTTP/1.0 200 OK Server: nginx/1.12.2 Date: Wed, 01 Aug 2018 08:40:49 GMT Content-Type: text/html Auth-Status: OK Auth-Server: Auth-Port: 143 telnet 10.12.1.128 143 Trying 10.12.1.128... Connected to 10.12.1.128. Escape character is '^]'. * OK IMAP4 ready LOGIN postmaster at com supersecret LOGIN BAD invalid command Connection closed by foreign host. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,280741,280741#msg-280741 From mdounin at mdounin.ru Wed Aug 1 13:06:43 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 1 Aug 2018 16:06:43 +0300 Subject: Nginx mail proxy LDAP iRedMail In-Reply-To: <06115d20dc20b7c25dd7ea7c54989c3f.NginxMailingListEnglish@forum.nginx.org> References: <06115d20dc20b7c25dd7ea7c54989c3f.NginxMailingListEnglish@forum.nginx.org> Message-ID: <20180801130643.GL56558@mdounin.ru> Hello! On Wed, Aug 01, 2018 at 04:50:27AM -0400, shiver25 wrote: > Hi there, > > I try configure a little mail infrastructure but i have problem with this. > So i have exacly three servers. One is MX (frontend) there is nginx with > configuration: > > user nginx; > worker_processes 2; > error_log /var/log/nginx/error.log info; > pid /var/run/nginx.pid; > load_module /usr/lib64/nginx/modules/ngx_http_perl_module.so; > load_module /usr/lib64/nginx/modules/ngx_mail_module.so; > > > events { > worker_connections 1024; > multi_accept on; > } > > http { > perl_modules perl/lib; > perl_require mailauth.pm; > > server { > location /auth { > proxy_set_header Host $host; > proxy_set_header X-Real-IP $remote_addr; > proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; Just a side note: these proxy_set_header directives are useless. > perl mailauth::handler; > } > } > } [...] > And i try write auth script in perl, look like: > > package mailauth; > > use strict; > use warnings; > use nginx; > use Net::LDAP; > > my $mail_server1 = "10.12.1.109"; > my $mail_server2 = "10.12.1.109"; > > our $mail_server_ip={}; > our $protocol_ports={}; > $mail_server-ip->{'mailhost01'}="10.12.1.109"; With "-" here, loading the configuration is expected to fail with an error like: nginx: [emerg] require_pv("mailauth.pm") failed: "Can't use bareword ("ip") as a HASH ref while "strict refs" in use at /path/to/mailauth.pm line 13. If the code provided exactly as used, likely you are testing with some older version which does not have this bug. In no particular order: - make sure to reload nginx configuration after changing the perl module; - try looking into nginx error logs, it might have helpful information. Note well that using embedded perl for potentially blocking lookups in the LDAP database might not be a good idea. Quoting http://nginx.org/en/docs/http/ngx_http_perl_module.html: : While the Perl module is performing a long-running operation, such : as resolving a domain name, connecting to another server, or : querying a database, other requests assigned to the current worker : process will not be processed. It is thus recommended to perform : only such operations that have predictable and short execution : time, such as accessing the local file system. [...] -- Maxim Dounin http://mdounin.ru/ From anoopalias01 at gmail.com Thu Aug 2 05:15:22 2018 From: anoopalias01 at gmail.com (Anoop Alias) Date: Thu, 2 Aug 2018 10:45:22 +0530 Subject: posix_memalign error In-Reply-To: <20180731133813.GG56558@mdounin.ru> References: <20180731133813.GG56558@mdounin.ru> Message-ID: Hi Maxim, I enabled debug and the memalign call is happening on nginx reloads and the ENOMEM happen sometimes on the reload(not on all reloads) 2018/08/02 05:59:08 [notice] 872052#872052: signal process started 2018/08/02 05:59:23 [notice] 871570#871570: signal 1 (SIGHUP) received from 872052, reconfiguring 2018/08/02 05:59:23 [debug] 871570#871570: wake up, sigio 0 2018/08/02 05:59:23 [notice] 871570#871570: reconfiguring 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 0000000002B0DA00:16384 @16 === > the memalign call on reload 2018/08/02 05:59:23 [debug] 871570#871570: malloc: 00000000087924D0:4560 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 000000000E442E00:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: malloc: 0000000005650850:4096 20 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #71 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #72 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #73 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #74 2018/08/02 05:48:49 [debug] 871275#871275: add cleanup: 000000005340D728 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000024D3260:4096 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000517BAF10:4096 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053854FC0:4096 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053855FD0:4096 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053856FE0:4096 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053857FF0:4096 2018/08/02 05:48:49 [debug] 871275#871275: posix_memalign: 0000000053859000:16384 @16 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385D010:4096 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385E020:4096 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385F030:4096 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CD160:4096 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CE170:4096 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CF180:4096 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D0190:4096 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D11A0:4096 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D21B0:4096 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D31C0:4096 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D41D0:4096 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D51E0:4096 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D61F0:4096 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D7200:4096 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D8210:4096 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D9220:4096 Infact there are lot of such calls during a reload 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA17ED00:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA1B0FF0:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA1E12C0:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA211590:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA243880:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA271B30:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA2A3E20:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA2D20D0:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA3063E0:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA334690:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA366980:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA396C50:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA3C8F40:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA3F9210:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA4294E0:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA45B7D0:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA489A80:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA4BBD70:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA4EA020:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA51E330:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA54C5E0:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA57E8D0:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA5AEBA0:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA5DEE70:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA611160:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA641430:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA671700:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA6A29E0:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA6D5CE0:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA707FD0:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA736280:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA768570:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA796820:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA7CAB30:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA7F8DE0:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA82B0D0:16384 @16 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: 00000000BA85B3A0:16384 @16 What is perplexing is that the system has enough free (available RAM) ############# # free -g total used free shared buff/cache available Mem: 125 54 24 8 46 58 Swap: 0 0 0 ############# # ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 514579 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 514579 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited ######################################### There is no other thing limiting memory allocation Any way to prevent this or probably identify/prevent this On Tue, Jul 31, 2018 at 7:08 PM Maxim Dounin wrote: > Hello! > > On Tue, Jul 31, 2018 at 09:52:29AM +0530, Anoop Alias wrote: > > > I am repeatedly seeing errors like > > > > ###################### > > 2018/07/31 03:46:33 [emerg] 2854560#2854560: posix_memalign(16, 16384) > > failed (12: Cannot allocate memory) > > 2018/07/31 03:54:09 [emerg] 2890190#2890190: posix_memalign(16, 16384) > > failed (12: Cannot allocate memory) > > 2018/07/31 04:08:36 [emerg] 2939230#2939230: posix_memalign(16, 16384) > > failed (12: Cannot allocate memory) > > 2018/07/31 04:24:48 [emerg] 2992650#2992650: posix_memalign(16, 16384) > > failed (12: Cannot allocate memory) > > 2018/07/31 04:42:09 [emerg] 3053092#3053092: posix_memalign(16, 16384) > > failed (12: Cannot allocate memory) > > 2018/07/31 04:42:17 [emerg] 3053335#3053335: posix_memalign(16, 16384) > > failed (12: Cannot allocate memory) > > 2018/07/31 04:42:28 [emerg] 3053937#3053937: posix_memalign(16, 16384) > > failed (12: Cannot allocate memory) > > 2018/07/31 04:47:54 [emerg] 3070638#3070638: posix_memalign(16, 16384) > > failed (12: Cannot allocate memory) > > #################### > > > > on a few servers > > > > The servers have enough memory free and the swap usage is 0, yet somehow > > the kernel denies the posix_memalign with ENOMEM ( this is what I think > is > > happening!) > > > > The numbers requested are always 16, 16k . This makes me suspicious > > > > I have no setting in nginx.conf that reference a 16k > > > > Is there any chance of finding out what requests this and why this is not > > fulfilled > > There are at least some buffers which default to 16k - for > example, ssl_buffer_size (http://nginx.org/r/ssl_buffer_size). > > You may try debugging log to futher find out where the particular > allocation happens, see here for details: > > http://nginx.org/en/docs/debugging_log.html > > But I don't really think it worth the effort. The error is pretty > clear, and it's better to focus on why these allocations are > denied. Likely you are hitting some limit. > > -- > Maxim Dounin > http://mdounin.ru/ > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx > -- *Anoop P Alias* -------------- next part -------------- An HTML attachment was scrubbed... URL: From iippolitov at nginx.com Thu Aug 2 07:07:16 2018 From: iippolitov at nginx.com (Igor A. Ippolitov) Date: Thu, 2 Aug 2018 10:07:16 +0300 Subject: posix_memalign error In-Reply-To: References: <20180731133813.GG56558@mdounin.ru> Message-ID: <4fa47f31-9c32-5386-7c2d-a4e9347b3dfc@nginx.com> Anoop, I doubt this will be the solution, but may we have a look at /proc/buddyinfo and /proc/slabinfo the moment when nginx can't allocate memory? On 02.08.2018 08:15, Anoop Alias wrote: > Hi Maxim, > > I enabled debug and the memalign call is happening on nginx reloads > and the ENOMEM happen sometimes on the reload(not on all reloads) > > 2018/08/02 05:59:08 [notice] 872052#872052: signal process started > 2018/08/02 05:59:23 [notice] 871570#871570: signal 1 (SIGHUP) received > from 872052, reconfiguring > 2018/08/02 05:59:23 [debug] 871570#871570: wake up, sigio 0 > 2018/08/02 05:59:23 [notice] 871570#871570: reconfiguring > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 0000000002B0DA00:16384 @16? ? ? === > the memalign call on reload > 2018/08/02 05:59:23 [debug] 871570#871570: malloc: 00000000087924D0:4560 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 000000000E442E00:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: malloc: 0000000005650850:4096 > 20 > > > > > 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #71 > 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #72 > 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #73 > 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #74 > 2018/08/02 05:48:49 [debug] 871275#871275: add cleanup: 000000005340D728 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000024D3260:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000517BAF10:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053854FC0:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053855FD0:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053856FE0:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053857FF0:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: posix_memalign: > 0000000053859000:16384 @16 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385D010:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385E020:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385F030:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CD160:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CE170:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CF180:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D0190:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D11A0:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D21B0:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D31C0:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D41D0:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D51E0:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D61F0:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D7200:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D8210:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D9220:4096 > > > Infact there are lot of such calls during a reload > > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA17ED00:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA1B0FF0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA1E12C0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA211590:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA243880:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA271B30:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA2A3E20:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA2D20D0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA3063E0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA334690:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA366980:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA396C50:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA3C8F40:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA3F9210:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA4294E0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA45B7D0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA489A80:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA4BBD70:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA4EA020:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA51E330:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA54C5E0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA57E8D0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA5AEBA0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA5DEE70:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA611160:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA641430:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA671700:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA6A29E0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA6D5CE0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA707FD0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA736280:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA768570:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA796820:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA7CAB30:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA7F8DE0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA82B0D0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA85B3A0:16384 @16 > > > > What is perplexing is that the system has enough free (available RAM) > ############# > # free -g > ? ? ? ? ? ? ? total? ? ? ? used? ? ? ? free? ? ? shared buff/cache? > ?available > Mem:? ? ? ? ? ? 125? ? ? ? ? 54? ? ? ? ? 24? ? ? ? ? ?8 ? ? ? ? 46? ? > ? ? ? 58 > Swap:? ? ? ? ? ? ?0? ? ? ? ? ?0? ? ? ? ? ?0 > ############# > > # ulimit -a > core file size? ? ? ? ? (blocks, -c) 0 > data seg size? ? ? ? ? ?(kbytes, -d) unlimited > scheduling priority? ? ? ? ? ? ?(-e) 0 > file size? ? ? ? ? ? ? ?(blocks, -f) unlimited > pending signals? ? ? ? ? ? ? ? ?(-i) 514579 > max locked memory? ? ? ?(kbytes, -l) 64 > max memory size? ? ? ? ?(kbytes, -m) unlimited > open files? ? ? ? ? ? ? ? ? ? ? (-n) 1024 > pipe size? ? ? ? ? ? (512 bytes, -p) 8 > POSIX message queues? ? ?(bytes, -q) 819200 > real-time priority? ? ? ? ? ? ? (-r) 0 > stack size? ? ? ? ? ? ? (kbytes, -s) 8192 > cpu time? ? ? ? ? ? ? ?(seconds, -t) unlimited > max user processes? ? ? ? ? ? ? (-u) 514579 > virtual memory? ? ? ? ? (kbytes, -v) unlimited > file locks? ? ? ? ? ? ? ? ? ? ? (-x) unlimited > > ######################################### > > There is no other thing limiting memory allocation > > Any way to prevent this or probably identify/prevent this > > > On Tue, Jul 31, 2018 at 7:08 PM Maxim Dounin > wrote: > > Hello! > > On Tue, Jul 31, 2018 at 09:52:29AM +0530, Anoop Alias wrote: > > > I am repeatedly seeing errors like > > > > ###################### > > 2018/07/31 03:46:33 [emerg] 2854560#2854560: posix_memalign(16, > 16384) > > failed (12: Cannot allocate memory) > > 2018/07/31 03:54:09 [emerg] 2890190#2890190: posix_memalign(16, > 16384) > > failed (12: Cannot allocate memory) > > 2018/07/31 04:08:36 [emerg] 2939230#2939230: posix_memalign(16, > 16384) > > failed (12: Cannot allocate memory) > > 2018/07/31 04:24:48 [emerg] 2992650#2992650: posix_memalign(16, > 16384) > > failed (12: Cannot allocate memory) > > 2018/07/31 04:42:09 [emerg] 3053092#3053092: posix_memalign(16, > 16384) > > failed (12: Cannot allocate memory) > > 2018/07/31 04:42:17 [emerg] 3053335#3053335: posix_memalign(16, > 16384) > > failed (12: Cannot allocate memory) > > 2018/07/31 04:42:28 [emerg] 3053937#3053937: posix_memalign(16, > 16384) > > failed (12: Cannot allocate memory) > > 2018/07/31 04:47:54 [emerg] 3070638#3070638: posix_memalign(16, > 16384) > > failed (12: Cannot allocate memory) > > #################### > > > > on a few servers > > > > The servers have enough memory free and the swap usage is 0, yet > somehow > > the kernel denies the posix_memalign with ENOMEM ( this is what > I think is > > happening!) > > > > The numbers requested are always 16, 16k . This makes me suspicious > > > > I have no setting in nginx.conf that reference a 16k > > > > Is there any chance of finding out what requests this and why > this is not > > fulfilled > > There are at least some buffers which default to 16k - for > example, ssl_buffer_size (http://nginx.org/r/ssl_buffer_size). > > You may try debugging log to futher find out where the particular > allocation happens, see here for details: > > http://nginx.org/en/docs/debugging_log.html > > But I don't really think it worth the effort.? The error is pretty > clear, and it's better to focus on why these allocations are > denied.? Likely you are hitting some limit. > > -- > Maxim Dounin > http://mdounin.ru/ > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx > > > > -- > *Anoop P Alias* > > > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx -------------- next part -------------- An HTML attachment was scrubbed... URL: From anoopalias01 at gmail.com Thu Aug 2 10:06:33 2018 From: anoopalias01 at gmail.com (Anoop Alias) Date: Thu, 2 Aug 2018 15:36:33 +0530 Subject: posix_memalign error In-Reply-To: <4fa47f31-9c32-5386-7c2d-a4e9347b3dfc@nginx.com> References: <20180731133813.GG56558@mdounin.ru> <4fa47f31-9c32-5386-7c2d-a4e9347b3dfc@nginx.com> Message-ID: Hi Igor, The error happens randomly 2018/08/02 06:52:42 [emerg] 874514#874514: posix_memalign(16, 16384) failed (12: Cannot allocate memory) 2018/08/02 09:42:53 [emerg] 872996#872996: posix_memalign(16, 16384) failed (12: Cannot allocate memory) 2018/08/02 10:16:14 [emerg] 877611#877611: posix_memalign(16, 16384) failed (12: Cannot allocate memory) 2018/08/02 10:16:48 [emerg] 879410#879410: posix_memalign(16, 16384) failed (12: Cannot allocate memory) 2018/08/02 10:17:55 [emerg] 876563#876563: posix_memalign(16, 16384) failed (12: Cannot allocate memory) 2018/08/02 10:20:21 [emerg] 879263#879263: posix_memalign(16, 16384) failed (12: Cannot allocate memory) 2018/08/02 10:20:51 [emerg] 878991#878991: posix_memalign(16, 16384) failed (12: Cannot allocate memory) # date Thu Aug 2 10:58:48 BST 2018 ------------------------------------------ # cat /proc/buddyinfo Node 0, zone DMA 0 0 1 0 2 1 1 0 1 1 3 Node 0, zone DMA32 11722 11057 4663 1647 609 72 10 7 1 0 0 Node 0, zone Normal 755026 710760 398136 21462 1114 18 1 0 0 0 0 Node 1, zone Normal 341295 801810 179604 256 0 0 0 0 0 0 0 ----------------------------------------- slabinfo - version: 2.1 # name : tunables : slabdata SCTPv6 21 21 1536 21 8 : tunables 0 0 0 : slabdata 1 1 0 SCTP 0 0 1408 23 8 : tunables 0 0 0 : slabdata 0 0 0 kcopyd_job 0 0 3312 9 8 : tunables 0 0 0 : slabdata 0 0 0 dm_uevent 0 0 2608 12 8 : tunables 0 0 0 : slabdata 0 0 0 nf_conntrack_ffffffff81acbb00 14054 14892 320 51 4 : tunables 0 0 0 : slabdata 292 292 0 lvp_cache 36 36 224 36 2 : tunables 0 0 0 : slabdata 1 1 0 lve_struct 4140 4140 352 46 4 : tunables 0 0 0 : slabdata 90 90 0 fat_inode_cache 0 0 744 44 8 : tunables 0 0 0 : slabdata 0 0 0 fat_cache 0 0 40 102 1 : tunables 0 0 0 : slabdata 0 0 0 isofs_inode_cache 0 0 664 49 8 : tunables 0 0 0 : slabdata 0 0 0 ext4_inode_cache 30 30 1088 30 8 : tunables 0 0 0 : slabdata 1 1 0 ext4_xattr 0 0 88 46 1 : tunables 0 0 0 : slabdata 0 0 0 ext4_free_data 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 ext4_allocation_context 32 32 128 32 1 : tunables 0 0 0 : slabdata 1 1 0 ext4_io_end 0 0 72 56 1 : tunables 0 0 0 : slabdata 0 0 0 ext4_extent_status 102 102 40 102 1 : tunables 0 0 0 : slabdata 1 1 0 jbd2_journal_handle 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0 jbd2_journal_head 0 0 112 36 1 : tunables 0 0 0 : slabdata 0 0 0 jbd2_revoke_table_s 256 256 16 256 1 : tunables 0 0 0 : slabdata 1 1 0 jbd2_revoke_record_s 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0 kvm_async_pf 0 0 136 30 1 : tunables 0 0 0 : slabdata 0 0 0 kvm_vcpu 0 0 18560 1 8 : tunables 0 0 0 : slabdata 0 0 0 xfs_dqtrx 992 992 528 31 4 : tunables 0 0 0 : slabdata 32 32 0 xfs_dquot 3264 3264 472 34 4 : tunables 0 0 0 : slabdata 96 96 0 xfs_ili 4342175 4774399 152 53 2 : tunables 0 0 0 : slabdata 90083 90083 0 xfs_inode 4915588 5486076 1088 30 8 : tunables 0 0 0 : slabdata 182871 182871 0 xfs_efd_item 2680 2760 400 40 4 : tunables 0 0 0 : slabdata 69 69 0 xfs_da_state 1088 1088 480 34 4 : tunables 0 0 0 : slabdata 32 32 0 xfs_btree_cur 1248 1248 208 39 2 : tunables 0 0 0 : slabdata 32 32 0 xfs_log_ticket 14874 15048 184 44 2 : tunables 0 0 0 : slabdata 342 342 0 xfs_ioend 12909 13104 104 39 1 : tunables 0 0 0 : slabdata 336 336 0 scsi_cmd_cache 5400 5652 448 36 4 : tunables 0 0 0 : slabdata 157 157 0 ve_struct 0 0 848 38 8 : tunables 0 0 0 : slabdata 0 0 0 ip6_dst_cache 1152 1152 448 36 4 : tunables 0 0 0 : slabdata 32 32 0 RAWv6 910 910 1216 26 8 : tunables 0 0 0 : slabdata 35 35 0 UDPLITEv6 0 0 1216 26 8 : tunables 0 0 0 : slabdata 0 0 0 UDPv6 832 832 1216 26 8 : tunables 0 0 0 : slabdata 32 32 0 tw_sock_TCPv6 1152 1376 256 32 2 : tunables 0 0 0 : slabdata 43 43 0 TCPv6 510 510 2176 15 8 : tunables 0 0 0 : slabdata 34 34 0 cfq_queue 3698 5145 232 35 2 : tunables 0 0 0 : slabdata 147 147 0 bsg_cmd 0 0 312 52 4 : tunables 0 0 0 : slabdata 0 0 0 mqueue_inode_cache 136 136 960 34 8 : tunables 0 0 0 : slabdata 4 4 0 hugetlbfs_inode_cache 1632 1632 632 51 8 : tunables 0 0 0 : slabdata 32 32 0 configfs_dir_cache 1472 1472 88 46 1 : tunables 0 0 0 : slabdata 32 32 0 dquot 0 0 256 32 2 : tunables 0 0 0 : slabdata 0 0 0 userfaultfd_ctx_cache 32 32 128 32 1 : tunables 0 0 0 : slabdata 1 1 0 fanotify_event_info 2336 2336 56 73 1 : tunables 0 0 0 : slabdata 32 32 0 dio 6171 6222 640 51 8 : tunables 0 0 0 : slabdata 122 122 0 pid_namespace 42 42 2192 14 8 : tunables 0 0 0 : slabdata 3 3 0 posix_timers_cache 1056 1056 248 33 2 : tunables 0 0 0 : slabdata 32 32 0 UDP-Lite 0 0 1088 30 8 : tunables 0 0 0 : slabdata 0 0 0 flow_cache 2268 2296 144 28 1 : tunables 0 0 0 : slabdata 82 82 0 xfrm_dst_cache 896 896 576 28 4 : tunables 0 0 0 : slabdata 32 32 0 ip_fib_alias 2720 2720 48 85 1 : tunables 0 0 0 : slabdata 32 32 0 RAW 3977 4224 1024 32 8 : tunables 0 0 0 : slabdata 132 132 0 UDP 4110 4110 1088 30 8 : tunables 0 0 0 : slabdata 137 137 0 tw_sock_TCP 4756 5216 256 32 2 : tunables 0 0 0 : slabdata 163 163 0 TCP 2705 2768 1984 16 8 : tunables 0 0 0 : slabdata 173 173 0 scsi_data_buffer 5440 5440 24 170 1 : tunables 0 0 0 : slabdata 32 32 0 blkdev_queue 154 154 2208 14 8 : tunables 0 0 0 : slabdata 11 11 0 blkdev_requests 4397688 4405884 384 42 4 : tunables 0 0 0 : slabdata 104902 104902 0 blkdev_ioc 11232 11232 112 36 1 : tunables 0 0 0 : slabdata 312 312 0 user_namespace 0 0 1304 25 8 : tunables 0 0 0 : slabdata 0 0 0 sock_inode_cache 12282 12282 704 46 8 : tunables 0 0 0 : slabdata 267 267 0 file_lock_cache 20056 20960 200 40 2 : tunables 0 0 0 : slabdata 524 524 0 net_namespace 6 6 5056 6 8 : tunables 0 0 0 : slabdata 1 1 0 shmem_inode_cache 16970 18952 712 46 8 : tunables 0 0 0 : slabdata 412 412 0 Acpi-ParseExt 39491 40432 72 56 1 : tunables 0 0 0 : slabdata 722 722 0 Acpi-State 1683 1683 80 51 1 : tunables 0 0 0 : slabdata 33 33 0 Acpi-Namespace 11424 11424 40 102 1 : tunables 0 0 0 : slabdata 112 112 0 task_delay_info 15336 15336 112 36 1 : tunables 0 0 0 : slabdata 426 426 0 taskstats 1568 1568 328 49 4 : tunables 0 0 0 : slabdata 32 32 0 proc_inode_cache 169897 190608 680 48 8 : tunables 0 0 0 : slabdata 3971 3971 0 sigqueue 2208 2208 168 48 2 : tunables 0 0 0 : slabdata 46 46 0 bdev_cache 792 792 896 36 8 : tunables 0 0 0 : slabdata 22 22 0 sysfs_dir_cache 74698 74698 120 34 1 : tunables 0 0 0 : slabdata 2197 2197 0 mnt_cache 163197 163424 256 32 2 : tunables 0 0 0 : slabdata 5107 5107 0 filp 64607 97257 320 51 4 : tunables 0 0 0 : slabdata 1907 1907 0 inode_cache 370744 370947 616 53 8 : tunables 0 0 0 : slabdata 6999 6999 0 dentry 1316262 2139228 192 42 2 : tunables 0 0 0 : slabdata 50934 50934 0 iint_cache 0 0 80 51 1 : tunables 0 0 0 : slabdata 0 0 0 buffer_head 1441470 2890290 104 39 1 : tunables 0 0 0 : slabdata 74110 74110 0 vm_area_struct 194998 196840 216 37 2 : tunables 0 0 0 : slabdata 5320 5320 0 mm_struct 2679 2760 1600 20 8 : tunables 0 0 0 : slabdata 138 138 0 files_cache 8680 8925 640 51 8 : tunables 0 0 0 : slabdata 175 175 0 signal_cache 3691 3780 1152 28 8 : tunables 0 0 0 : slabdata 135 135 0 sighand_cache 1950 2160 2112 15 8 : tunables 0 0 0 : slabdata 144 144 0 task_xstate 8070 8658 832 39 8 : tunables 0 0 0 : slabdata 222 222 0 task_struct 1913 2088 4080 8 8 : tunables 0 0 0 : slabdata 261 261 0 cred_jar 31699 33936 192 42 2 : tunables 0 0 0 : slabdata 808 808 0 anon_vma_chain 164026 168704 64 64 1 : tunables 0 0 0 : slabdata 2636 2636 0 anon_vma 84104 84594 88 46 1 : tunables 0 0 0 : slabdata 1839 1839 0 pid 11127 12576 128 32 1 : tunables 0 0 0 : slabdata 393 393 0 shared_policy_node 9350 9350 48 85 1 : tunables 0 0 0 : slabdata 110 110 0 numa_policy 62 62 264 31 2 : tunables 0 0 0 : slabdata 2 2 0 radix_tree_node 771778 1194312 584 28 4 : tunables 0 0 0 : slabdata 42654 42654 0 idr_layer_cache 2538 2565 2112 15 8 : tunables 0 0 0 : slabdata 171 171 0 dma-kmalloc-8192 0 0 8192 4 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-4096 0 0 4096 8 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-2048 0 0 2048 16 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-1024 0 0 1024 32 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-512 0 0 512 32 4 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-256 0 0 256 32 2 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-128 0 0 128 32 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-64 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-32 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-16 0 0 16 256 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-8 0 0 8 512 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-192 0 0 192 42 2 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-96 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-8192 385 388 8192 4 8 : tunables 0 0 0 : slabdata 97 97 0 kmalloc-4096 9296 10088 4096 8 8 : tunables 0 0 0 : slabdata 1261 1261 0 kmalloc-2048 65061 133536 2048 16 8 : tunables 0 0 0 : slabdata 8346 8346 0 kmalloc-1024 11987 21120 1024 32 8 : tunables 0 0 0 : slabdata 660 660 0 kmalloc-512 107510 187072 512 32 4 : tunables 0 0 0 : slabdata 5846 5846 0 kmalloc-256 160498 199104 256 32 2 : tunables 0 0 0 : slabdata 6222 6222 0 kmalloc-192 144975 237426 192 42 2 : tunables 0 0 0 : slabdata 5653 5653 0 kmalloc-128 36799 108096 128 32 1 : tunables 0 0 0 : slabdata 3378 3378 0 kmalloc-96 99510 238896 96 42 1 : tunables 0 0 0 : slabdata 5688 5688 0 kmalloc-64 7978152 8593280 64 64 1 : tunables 0 0 0 : slabdata 134270 134270 0 kmalloc-32 2939882 3089664 32 128 1 : tunables 0 0 0 : slabdata 24138 24138 0 kmalloc-16 172057 172288 16 256 1 : tunables 0 0 0 : slabdata 673 673 0 kmalloc-8 109568 109568 8 512 1 : tunables 0 0 0 : slabdata 214 214 0 kmem_cache_node 893 896 64 64 1 : tunables 0 0 0 : slabdata 14 14 0 kmem_cache 612 612 320 51 4 : tunables 0 0 0 : slabdata 12 12 0 ------------------------------------------------- # uname -r 3.10.0-714.10.2.lve1.5.17.1.el7.x86_64 -------------------------------------------------------- Core part of glances http://i.imgur.com/La5JbQn.png ----------------------------------------------------------- Thank you very much for looking into this On Thu, Aug 2, 2018 at 12:37 PM Igor A. Ippolitov wrote: > Anoop, > > I doubt this will be the solution, but may we have a look at > /proc/buddyinfo and /proc/slabinfo the moment when nginx can't allocate > memory? > > On 02.08.2018 08:15, Anoop Alias wrote: > > Hi Maxim, > > I enabled debug and the memalign call is happening on nginx reloads and > the ENOMEM happen sometimes on the reload(not on all reloads) > > 2018/08/02 05:59:08 [notice] 872052#872052: signal process started > 2018/08/02 05:59:23 [notice] 871570#871570: signal 1 (SIGHUP) received > from 872052, reconfiguring > 2018/08/02 05:59:23 [debug] 871570#871570: wake up, sigio 0 > 2018/08/02 05:59:23 [notice] 871570#871570: reconfiguring > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 0000000002B0DA00:16384 @16 === > the memalign call on reload > 2018/08/02 05:59:23 [debug] 871570#871570: malloc: 00000000087924D0:4560 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 000000000E442E00:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: malloc: 0000000005650850:4096 > 20 > > > > > 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #71 > 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #72 > 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #73 > 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #74 > 2018/08/02 05:48:49 [debug] 871275#871275: add cleanup: 000000005340D728 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000024D3260:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000517BAF10:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053854FC0:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053855FD0:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053856FE0:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053857FF0:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: posix_memalign: > 0000000053859000:16384 @16 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385D010:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385E020:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385F030:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CD160:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CE170:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CF180:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D0190:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D11A0:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D21B0:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D31C0:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D41D0:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D51E0:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D61F0:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D7200:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D8210:4096 > 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D9220:4096 > > > Infact there are lot of such calls during a reload > > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA17ED00:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA1B0FF0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA1E12C0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA211590:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA243880:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA271B30:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA2A3E20:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA2D20D0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA3063E0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA334690:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA366980:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA396C50:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA3C8F40:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA3F9210:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA4294E0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA45B7D0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA489A80:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA4BBD70:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA4EA020:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA51E330:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA54C5E0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA57E8D0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA5AEBA0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA5DEE70:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA611160:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA641430:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA671700:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA6A29E0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA6D5CE0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA707FD0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA736280:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA768570:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA796820:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA7CAB30:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA7F8DE0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA82B0D0:16384 @16 > 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: > 00000000BA85B3A0:16384 @16 > > > > What is perplexing is that the system has enough free (available RAM) > ############# > # free -g > total used free shared buff/cache > available > Mem: 125 54 24 8 46 > 58 > Swap: 0 0 0 > ############# > > # ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 514579 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 1024 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 514579 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > > ######################################### > > There is no other thing limiting memory allocation > > Any way to prevent this or probably identify/prevent this > > > On Tue, Jul 31, 2018 at 7:08 PM Maxim Dounin wrote: > >> Hello! >> >> On Tue, Jul 31, 2018 at 09:52:29AM +0530, Anoop Alias wrote: >> >> > I am repeatedly seeing errors like >> > >> > ###################### >> > 2018/07/31 03:46:33 [emerg] 2854560#2854560: posix_memalign(16, 16384) >> > failed (12: Cannot allocate memory) >> > 2018/07/31 03:54:09 [emerg] 2890190#2890190: posix_memalign(16, 16384) >> > failed (12: Cannot allocate memory) >> > 2018/07/31 04:08:36 [emerg] 2939230#2939230: posix_memalign(16, 16384) >> > failed (12: Cannot allocate memory) >> > 2018/07/31 04:24:48 [emerg] 2992650#2992650: posix_memalign(16, 16384) >> > failed (12: Cannot allocate memory) >> > 2018/07/31 04:42:09 [emerg] 3053092#3053092: posix_memalign(16, 16384) >> > failed (12: Cannot allocate memory) >> > 2018/07/31 04:42:17 [emerg] 3053335#3053335: posix_memalign(16, 16384) >> > failed (12: Cannot allocate memory) >> > 2018/07/31 04:42:28 [emerg] 3053937#3053937: posix_memalign(16, 16384) >> > failed (12: Cannot allocate memory) >> > 2018/07/31 04:47:54 [emerg] 3070638#3070638: posix_memalign(16, 16384) >> > failed (12: Cannot allocate memory) >> > #################### >> > >> > on a few servers >> > >> > The servers have enough memory free and the swap usage is 0, yet somehow >> > the kernel denies the posix_memalign with ENOMEM ( this is what I think >> is >> > happening!) >> > >> > The numbers requested are always 16, 16k . This makes me suspicious >> > >> > I have no setting in nginx.conf that reference a 16k >> > >> > Is there any chance of finding out what requests this and why this is >> not >> > fulfilled >> >> There are at least some buffers which default to 16k - for >> example, ssl_buffer_size (http://nginx.org/r/ssl_buffer_size). >> >> You may try debugging log to futher find out where the particular >> allocation happens, see here for details: >> >> http://nginx.org/en/docs/debugging_log.html >> >> But I don't really think it worth the effort. The error is pretty >> clear, and it's better to focus on why these allocations are >> denied. Likely you are hitting some limit. >> >> -- >> Maxim Dounin >> http://mdounin.ru/ >> _______________________________________________ >> nginx mailing list >> nginx at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx >> > > > -- > *Anoop P Alias* > > > > _______________________________________________ > nginx mailing listnginx at nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx > > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx -- *Anoop P Alias* -------------- next part -------------- An HTML attachment was scrubbed... URL: From iippolitov at nginx.com Thu Aug 2 12:32:20 2018 From: iippolitov at nginx.com (Igor A. Ippolitov) Date: Thu, 2 Aug 2018 15:32:20 +0300 Subject: posix_memalign error In-Reply-To: References: <20180731133813.GG56558@mdounin.ru> <4fa47f31-9c32-5386-7c2d-a4e9347b3dfc@nginx.com> Message-ID: Anoop, There are two guesses: either mmap allocations limit is hit or memory is? way too fragmented. Could you please track amount of mapped regions for a worker with pmap and amount of 16k areas in Normal zones (it is the third number)? You can also set vm.max_map_count to a higher number (like 20 times higher than default) and look if the error is gone. Please, let me know if increasing vm.max_map_count helps you. On 02.08.2018 13:06, Anoop Alias wrote: > Hi Igor, > > The error happens randomly > > 2018/08/02 06:52:42 [emerg] 874514#874514: posix_memalign(16, 16384) > failed (12: Cannot allocate memory) > 2018/08/02 09:42:53 [emerg] 872996#872996: posix_memalign(16, 16384) > failed (12: Cannot allocate memory) > 2018/08/02 10:16:14 [emerg] 877611#877611: posix_memalign(16, 16384) > failed (12: Cannot allocate memory) > 2018/08/02 10:16:48 [emerg] 879410#879410: posix_memalign(16, 16384) > failed (12: Cannot allocate memory) > 2018/08/02 10:17:55 [emerg] 876563#876563: posix_memalign(16, 16384) > failed (12: Cannot allocate memory) > 2018/08/02 10:20:21 [emerg] 879263#879263: posix_memalign(16, 16384) > failed (12: Cannot allocate memory) > 2018/08/02 10:20:51 [emerg] 878991#878991: posix_memalign(16, 16384) > failed (12: Cannot allocate memory) > > # date > Thu Aug? 2 10:58:48 BST 2018 > > ------------------------------------------ > # cat /proc/buddyinfo > Node 0, zone? ? ? DMA? ? ? 0? ? ? 0? ? ? 1? ? ? 0 2? ? ? 1? ? ? 1? ? ? > 0? ? ? 1? ? ? 1? ? ? 3 > Node 0, zone? ? DMA32? 11722? 11057? ?4663? ?1647 609? ? ?72? ? ?10? ? > ? 7? ? ? 1? ? ? 0? ? ? 0 > Node 0, zone? ?Normal 755026 710760 398136? 21462 ?1114? ? ?18? ? ? 1? > ? ? 0? ? ? 0? ? ? 0? ? ? 0 > Node 1, zone? ?Normal 341295 801810 179604? ? 256 0? ? ? 0? ? ? 0? ? ? > 0? ? ? 0? ? ? 0? ? ? 0 > ----------------------------------------- > > > slabinfo - version: 2.1 > # name? ? ? ? ? ? > : tunables : > slabdata > SCTPv6? ? ? ? ? ? ? ? 21? ? ?21? ?1536? ?21? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 1? ? ? 1? ? ? 0 > SCTP? ? ? ? ? ? ? ? ? ?0? ? ? 0? ?1408? ?23? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > kcopyd_job? ? ? ? ? ? ?0? ? ? 0? ?3312? ? 9? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > dm_uevent? ? ? ? ? ? ? 0? ? ? 0? ?2608? ?12? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > nf_conntrack_ffffffff81acbb00? 14054? 14892? ? 320 ?51? ? 4 : > tunables? ? 0? ? 0? ? 0 : slabdata? ? 292 292? ? ? 0 > lvp_cache? ? ? ? ? ? ?36? ? ?36? ? 224? ?36? ? 2 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 1? ? ? 1? ? ? 0 > lve_struct? ? ? ? ? 4140? ?4140? ? 352? ?46? ? 4 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ?90? ? ?90? ? ? 0 > fat_inode_cache? ? ? ? 0? ? ? 0? ? 744? ?44? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > fat_cache? ? ? ? ? ? ? 0? ? ? 0? ? ?40? 102? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > isofs_inode_cache? ? ? 0? ? ? 0? ? 664? ?49? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > ext4_inode_cache? ? ? 30? ? ?30? ?1088? ?30? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 1? ? ? 1? ? ? 0 > ext4_xattr? ? ? ? ? ? ?0? ? ? 0? ? ?88? ?46? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > ext4_free_data? ? ? ? ?0? ? ? 0? ? ?64? ?64? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > ext4_allocation_context? ? ?32? ? ?32? ? 128? ?32? ? 1 : tunables? ? > 0? ? 0? ? 0 : slabdata? ? ? 1? ? ? 1? ? ? 0 > ext4_io_end? ? ? ? ? ? 0? ? ? 0? ? ?72? ?56? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > ext4_extent_status? ? 102? ? 102? ? ?40? 102? ? 1 : tunables? ? 0? ? > 0? ? 0 : slabdata? ? ? 1? ? ? 1? ? ? 0 > jbd2_journal_handle? ? ? 0? ? ? 0? ? ?48? ?85? ? 1 : tunables? ? 0? ? > 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > jbd2_journal_head? ? ? 0? ? ? 0? ? 112? ?36? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > jbd2_revoke_table_s? ? 256? ? 256? ? ?16? 256? ? 1 : tunables? ? 0? ? > 0? ? 0 : slabdata? ? ? 1? ? ? 1? ? ? 0 > jbd2_revoke_record_s? ? ? 0? ? ? 0? ? ?32? 128? ? 1 : tunables? ? 0? ? > 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > kvm_async_pf? ? ? ? ? ?0? ? ? 0? ? 136? ?30? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > kvm_vcpu? ? ? ? ? ? ? ?0? ? ? 0? 18560? ? 1? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > xfs_dqtrx? ? ? ? ? ? 992? ? 992? ? 528? ?31? ? 4 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ?32? ? ?32? ? ? 0 > xfs_dquot? ? ? ? ? ?3264? ?3264? ? 472? ?34? ? 4 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ?96? ? ?96? ? ? 0 > xfs_ili? ? ? ? ? ?4342175 4774399? ? 152? ?53? ? 2 : tunables? ? 0? ? > 0? ? 0 : slabdata? 90083? 90083? ? ? 0 > xfs_inode? ? ? ? ?4915588 5486076? ?1088? ?30? ? 8 : tunables? ? 0? ? > 0? ? 0 : slabdata 182871 182871? ? ? 0 > xfs_efd_item? ? ? ? 2680? ?2760? ? 400? ?40? ? 4 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ?69? ? ?69? ? ? 0 > xfs_da_state? ? ? ? 1088? ?1088? ? 480? ?34? ? 4 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ?32? ? ?32? ? ? 0 > xfs_btree_cur? ? ? ?1248? ?1248? ? 208? ?39? ? 2 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ?32? ? ?32? ? ? 0 > xfs_log_ticket? ? ?14874? 15048? ? 184? ?44? ? 2 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 342? ? 342? ? ? 0 > xfs_ioend? ? ? ? ? 12909? 13104? ? 104? ?39? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 336? ? 336? ? ? 0 > scsi_cmd_cache? ? ? 5400? ?5652? ? 448? ?36? ? 4 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 157? ? 157? ? ? 0 > ve_struct? ? ? ? ? ? ? 0? ? ? 0? ? 848? ?38? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > ip6_dst_cache? ? ? ?1152? ?1152? ? 448? ?36? ? 4 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ?32? ? ?32? ? ? 0 > RAWv6? ? ? ? ? ? ? ? 910? ? 910? ?1216? ?26? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ?35? ? ?35? ? ? 0 > UDPLITEv6? ? ? ? ? ? ? 0? ? ? 0? ?1216? ?26? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > UDPv6? ? ? ? ? ? ? ? 832? ? 832? ?1216? ?26? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ?32? ? ?32? ? ? 0 > tw_sock_TCPv6? ? ? ?1152? ?1376? ? 256? ?32? ? 2 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ?43? ? ?43? ? ? 0 > TCPv6? ? ? ? ? ? ? ? 510? ? 510? ?2176? ?15? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ?34? ? ?34? ? ? 0 > cfq_queue? ? ? ? ? ?3698? ?5145? ? 232? ?35? ? 2 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 147? ? 147? ? ? 0 > bsg_cmd? ? ? ? ? ? ? ? 0? ? ? 0? ? 312? ?52? ? 4 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > mqueue_inode_cache? ? 136? ? 136? ? 960? ?34? ? 8 : tunables? ? 0? ? > 0? ? 0 : slabdata? ? ? 4? ? ? 4? ? ? 0 > hugetlbfs_inode_cache? ?1632? ?1632? ? 632? ?51? ? 8 : tunables? ? 0? > ? 0? ? 0 : slabdata? ? ?32? ? ?32? ? ? 0 > configfs_dir_cache? ?1472? ?1472? ? ?88? ?46? ? 1 : tunables? ? 0? ? > 0? ? 0 : slabdata? ? ?32? ? ?32? ? ? 0 > dquot? ? ? ? ? ? ? ? ? 0? ? ? 0? ? 256? ?32? ? 2 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > userfaultfd_ctx_cache? ? ?32? ? ?32? ? 128? ?32? ? 1 : tunables? ? 0? > ? 0? ? 0 : slabdata? ? ? 1? ? ? 1? ? ? 0 > fanotify_event_info? ?2336? ?2336? ? ?56? ?73? ? 1 : tunables? ? 0? ? > 0? ? 0 : slabdata? ? ?32? ? ?32? ? ? 0 > dio? ? ? ? ? ? ? ? ?6171? ?6222? ? 640? ?51? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 122? ? 122? ? ? 0 > pid_namespace? ? ? ? ?42? ? ?42? ?2192? ?14? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 3? ? ? 3? ? ? 0 > posix_timers_cache? ?1056? ?1056? ? 248? ?33? ? 2 : tunables? ? 0? ? > 0? ? 0 : slabdata? ? ?32? ? ?32? ? ? 0 > UDP-Lite? ? ? ? ? ? ? ?0? ? ? 0? ?1088? ?30? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > flow_cache? ? ? ? ? 2268? ?2296? ? 144? ?28? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ?82? ? ?82? ? ? 0 > xfrm_dst_cache? ? ? ?896? ? 896? ? 576? ?28? ? 4 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ?32? ? ?32? ? ? 0 > ip_fib_alias? ? ? ? 2720? ?2720? ? ?48? ?85? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ?32? ? ?32? ? ? 0 > RAW? ? ? ? ? ? ? ? ?3977? ?4224? ?1024? ?32? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 132? ? 132? ? ? 0 > UDP? ? ? ? ? ? ? ? ?4110? ?4110? ?1088? ?30? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 137? ? 137? ? ? 0 > tw_sock_TCP? ? ? ? ?4756? ?5216? ? 256? ?32? ? 2 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 163? ? 163? ? ? 0 > TCP? ? ? ? ? ? ? ? ?2705? ?2768? ?1984? ?16? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 173? ? 173? ? ? 0 > scsi_data_buffer? ? 5440? ?5440? ? ?24? 170? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ?32? ? ?32? ? ? 0 > blkdev_queue? ? ? ? ?154? ? 154? ?2208? ?14? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ?11? ? ?11? ? ? 0 > blkdev_requests? ?4397688 4405884? ? 384? ?42? ? 4 : tunables? ? 0? ? > 0? ? 0 : slabdata 104902 104902? ? ? 0 > blkdev_ioc? ? ? ? ?11232? 11232? ? 112? ?36? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 312? ? 312? ? ? 0 > user_namespace? ? ? ? ?0? ? ? 0? ?1304? ?25? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > sock_inode_cache? ?12282? 12282? ? 704? ?46? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 267? ? 267? ? ? 0 > file_lock_cache? ? 20056? 20960? ? 200? ?40? ? 2 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 524? ? 524? ? ? 0 > net_namespace? ? ? ? ? 6? ? ? 6? ?5056? ? 6? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 1? ? ? 1? ? ? 0 > shmem_inode_cache? 16970? 18952? ? 712? ?46? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 412? ? 412? ? ? 0 > Acpi-ParseExt? ? ? 39491? 40432? ? ?72? ?56? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 722? ? 722? ? ? 0 > Acpi-State? ? ? ? ? 1683? ?1683? ? ?80? ?51? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ?33? ? ?33? ? ? 0 > Acpi-Namespace? ? ?11424? 11424? ? ?40? 102? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 112? ? 112? ? ? 0 > task_delay_info? ? 15336? 15336? ? 112? ?36? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 426? ? 426? ? ? 0 > taskstats? ? ? ? ? ?1568? ?1568? ? 328? ?49? ? 4 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ?32? ? ?32? ? ? 0 > proc_inode_cache? 169897 190608? ? 680? ?48? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ?3971? ?3971? ? ? 0 > sigqueue? ? ? ? ? ? 2208? ?2208? ? 168? ?48? ? 2 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ?46? ? ?46? ? ? 0 > bdev_cache? ? ? ? ? ?792? ? 792? ? 896? ?36? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ?22? ? ?22? ? ? 0 > sysfs_dir_cache? ? 74698? 74698? ? 120? ?34? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ?2197? ?2197? ? ? 0 > mnt_cache? ? ? ? ?163197 163424? ? 256? ?32? ? 2 : tunables? ? 0? ? 0? > ? 0 : slabdata? ?5107? ?5107? ? ? 0 > filp? ? ? ? ? ? ? ?64607? 97257? ? 320? ?51? ? 4 : tunables? ? 0? ? 0? > ? 0 : slabdata? ?1907? ?1907? ? ? 0 > inode_cache? ? ? ?370744 370947? ? 616? ?53? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ?6999? ?6999? ? ? 0 > dentry? ? ? ? ? ? 1316262 2139228? ? 192? ?42? ? 2 : tunables? ? 0? ? > 0? ? 0 : slabdata? 50934? 50934? ? ? 0 > iint_cache? ? ? ? ? ? ?0? ? ? 0? ? ?80? ?51? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > buffer_head? ? ? ?1441470 2890290? ? 104? ?39? ? 1 : tunables? ? 0? ? > 0? ? 0 : slabdata? 74110? 74110? ? ? 0 > vm_area_struct? ? 194998 196840? ? 216? ?37? ? 2 : tunables? ? 0? ? 0? > ? 0 : slabdata? ?5320? ?5320? ? ? 0 > mm_struct? ? ? ? ? ?2679? ?2760? ?1600? ?20? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 138? ? 138? ? ? 0 > files_cache? ? ? ? ?8680? ?8925? ? 640? ?51? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 175? ? 175? ? ? 0 > signal_cache? ? ? ? 3691? ?3780? ?1152? ?28? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 135? ? 135? ? ? 0 > sighand_cache? ? ? ?1950? ?2160? ?2112? ?15? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 144? ? 144? ? ? 0 > task_xstate? ? ? ? ?8070? ?8658? ? 832? ?39? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 222? ? 222? ? ? 0 > task_struct? ? ? ? ?1913? ?2088? ?4080? ? 8? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 261? ? 261? ? ? 0 > cred_jar? ? ? ? ? ?31699? 33936? ? 192? ?42? ? 2 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 808? ? 808? ? ? 0 > anon_vma_chain? ? 164026 168704? ? ?64? ?64? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ?2636? ?2636? ? ? 0 > anon_vma? ? ? ? ? ?84104? 84594? ? ?88? ?46? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ?1839? ?1839? ? ? 0 > pid? ? ? ? ? ? ? ? 11127? 12576? ? 128? ?32? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 393? ? 393? ? ? 0 > shared_policy_node? ?9350? ?9350? ? ?48? ?85? ? 1 : tunables? ? 0? ? > 0? ? 0 : slabdata? ? 110? ? 110? ? ? 0 > numa_policy? ? ? ? ? ?62? ? ?62? ? 264? ?31? ? 2 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 2? ? ? 2? ? ? 0 > radix_tree_node? ?771778 1194312? ? 584? ?28? ? 4 : tunables? ? 0? ? > 0? ? 0 : slabdata? 42654? 42654? ? ? 0 > idr_layer_cache? ? ?2538? ?2565? ?2112? ?15? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 171? ? 171? ? ? 0 > dma-kmalloc-8192? ? ? ?0? ? ? 0? ?8192? ? 4? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > dma-kmalloc-4096? ? ? ?0? ? ? 0? ?4096? ? 8? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > dma-kmalloc-2048? ? ? ?0? ? ? 0? ?2048? ?16? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > dma-kmalloc-1024? ? ? ?0? ? ? 0? ?1024? ?32? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > dma-kmalloc-512? ? ? ? 0? ? ? 0? ? 512? ?32? ? 4 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > dma-kmalloc-256? ? ? ? 0? ? ? 0? ? 256? ?32? ? 2 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > dma-kmalloc-128? ? ? ? 0? ? ? 0? ? 128? ?32? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > dma-kmalloc-64? ? ? ? ?0? ? ? 0? ? ?64? ?64? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > dma-kmalloc-32? ? ? ? ?0? ? ? 0? ? ?32? 128? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > dma-kmalloc-16? ? ? ? ?0? ? ? 0? ? ?16? 256? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > dma-kmalloc-8? ? ? ? ? 0? ? ? 0? ? ? 8? 512? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > dma-kmalloc-192? ? ? ? 0? ? ? 0? ? 192? ?42? ? 2 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > dma-kmalloc-96? ? ? ? ?0? ? ? 0? ? ?96? ?42? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 > kmalloc-8192? ? ? ? ?385? ? 388? ?8192? ? 4? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ?97? ? ?97? ? ? 0 > kmalloc-4096? ? ? ? 9296? 10088? ?4096? ? 8? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ?1261? ?1261? ? ? 0 > kmalloc-2048? ? ? ?65061 133536? ?2048? ?16? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ?8346? ?8346? ? ? 0 > kmalloc-1024? ? ? ?11987? 21120? ?1024? ?32? ? 8 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 660? ? 660? ? ? 0 > kmalloc-512? ? ? ?107510 187072? ? 512? ?32? ? 4 : tunables? ? 0? ? 0? > ? 0 : slabdata? ?5846? ?5846? ? ? 0 > kmalloc-256? ? ? ?160498 199104? ? 256? ?32? ? 2 : tunables? ? 0? ? 0? > ? 0 : slabdata? ?6222? ?6222? ? ? 0 > kmalloc-192? ? ? ?144975 237426? ? 192? ?42? ? 2 : tunables? ? 0? ? 0? > ? 0 : slabdata? ?5653? ?5653? ? ? 0 > kmalloc-128? ? ? ? 36799 108096? ? 128? ?32? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ?3378? ?3378? ? ? 0 > kmalloc-96? ? ? ? ?99510 238896? ? ?96? ?42? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ?5688? ?5688? ? ? 0 > kmalloc-64? ? ? ? 7978152 8593280? ? ?64? ?64? ? 1 : tunables? ? 0? ? > 0? ? 0 : slabdata 134270 134270? ? ? 0 > kmalloc-32? ? ? ? 2939882 3089664? ? ?32? 128? ? 1 : tunables? ? 0? ? > 0? ? 0 : slabdata? 24138? 24138? ? ? 0 > kmalloc-16? ? ? ? 172057 172288? ? ?16? 256? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 673? ? 673? ? ? 0 > kmalloc-8? ? ? ? ?109568 109568? ? ? 8? 512? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? 214? ? 214? ? ? 0 > kmem_cache_node? ? ? 893? ? 896? ? ?64? ?64? ? 1 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ?14? ? ?14? ? ? 0 > kmem_cache? ? ? ? ? ?612? ? 612? ? 320? ?51? ? 4 : tunables? ? 0? ? 0? > ? 0 : slabdata? ? ?12? ? ?12? ? ? 0 > > ------------------------------------------------- > > > # uname -r > 3.10.0-714.10.2.lve1.5.17.1.el7.x86_64 > > -------------------------------------------------------- > > Core part of glances > http://i.imgur.com/La5JbQn.png > ----------------------------------------------------------- > > Thank you very much for looking into this > > > On Thu, Aug 2, 2018 at 12:37 PM Igor A. Ippolitov > > wrote: > > Anoop, > > I doubt this will be the solution, but may we have a look at > /proc/buddyinfo and /proc/slabinfo the moment when nginx can't > allocate memory? > > On 02.08.2018 08:15, Anoop Alias wrote: >> Hi Maxim, >> >> I enabled debug and the memalign call is happening on nginx >> reloads and the ENOMEM happen sometimes on the reload(not on all >> reloads) >> >> 2018/08/02 05:59:08 [notice] 872052#872052: signal process started >> 2018/08/02 05:59:23 [notice] 871570#871570: signal 1 (SIGHUP) >> received from 872052, reconfiguring >> 2018/08/02 05:59:23 [debug] 871570#871570: wake up, sigio 0 >> 2018/08/02 05:59:23 [notice] 871570#871570: reconfiguring >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 0000000002B0DA00:16384 @16? ? ? === > the memalign call on reload >> 2018/08/02 05:59:23 [debug] 871570#871570: malloc: >> 00000000087924D0:4560 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 000000000E442E00:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: malloc: >> 0000000005650850:4096 >> 20 >> >> >> >> >> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #71 >> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #72 >> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #73 >> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #74 >> 2018/08/02 05:48:49 [debug] 871275#871275: add cleanup: >> 000000005340D728 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >> 00000000024D3260:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >> 00000000517BAF10:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >> 0000000053854FC0:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >> 0000000053855FD0:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >> 0000000053856FE0:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >> 0000000053857FF0:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: posix_memalign: >> 0000000053859000:16384 @16 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >> 000000005385D010:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >> 000000005385E020:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >> 000000005385F030:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >> 00000000536CD160:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >> 00000000536CE170:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >> 00000000536CF180:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >> 00000000536D0190:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >> 00000000536D11A0:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >> 00000000536D21B0:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >> 00000000536D31C0:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >> 00000000536D41D0:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >> 00000000536D51E0:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >> 00000000536D61F0:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >> 00000000536D7200:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >> 00000000536D8210:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >> 00000000536D9220:4096 >> >> >> Infact there are lot of such calls during a reload >> >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA17ED00:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA1B0FF0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA1E12C0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA211590:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA243880:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA271B30:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA2A3E20:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA2D20D0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA3063E0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA334690:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA366980:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA396C50:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA3C8F40:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA3F9210:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA4294E0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA45B7D0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA489A80:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA4BBD70:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA4EA020:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA51E330:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA54C5E0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA57E8D0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA5AEBA0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA5DEE70:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA611160:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA641430:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA671700:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA6A29E0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA6D5CE0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA707FD0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA736280:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA768570:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA796820:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA7CAB30:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA7F8DE0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA82B0D0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA85B3A0:16384 @16 >> >> >> >> What is perplexing is that the system has enough free (available RAM) >> ############# >> # free -g >> ? ? ? ? ? ? ? total? ? ? ? used? ? ? ? free shared? buff/cache? >> ?available >> Mem:? ? ? ? ? ? 125? ? ? ? ? 54? ? ? ? ? 24 ? ? ?8? ? ? ? ? 46? ? >> ? ? ? 58 >> Swap:? ? ? ? ? ? ?0? ? ? ? ? ?0? ? ? ? ? ?0 >> ############# >> >> # ulimit -a >> core file size? ? ? ? ? (blocks, -c) 0 >> data seg size? ? ? ? ? ?(kbytes, -d) unlimited >> scheduling priority? ? ? ? ? ? ?(-e) 0 >> file size? ? ? ? ? ? ? ?(blocks, -f) unlimited >> pending signals? ? ? ? ? ? ? ? ?(-i) 514579 >> max locked memory? ? ? ?(kbytes, -l) 64 >> max memory size? ? ? ? ?(kbytes, -m) unlimited >> open files? ? ? ? ? ? ? ? ? ? ? (-n) 1024 >> pipe size? ? ? ? ? ? (512 bytes, -p) 8 >> POSIX message queues? ? ?(bytes, -q) 819200 >> real-time priority? ? ? ? ? ? ? (-r) 0 >> stack size? ? ? ? ? ? ? (kbytes, -s) 8192 >> cpu time? ? ? ? ? ? ? ?(seconds, -t) unlimited >> max user processes? ? ? ? ? ? ? (-u) 514579 >> virtual memory? ? ? ? ? (kbytes, -v) unlimited >> file locks? ? ? ? ? ? ? ? ? ? ? (-x) unlimited >> >> ######################################### >> >> There is no other thing limiting memory allocation >> >> Any way to prevent this or probably identify/prevent this >> >> >> On Tue, Jul 31, 2018 at 7:08 PM Maxim Dounin > > wrote: >> >> Hello! >> >> On Tue, Jul 31, 2018 at 09:52:29AM +0530, Anoop Alias wrote: >> >> > I am repeatedly seeing errors like >> > >> > ###################### >> > 2018/07/31 03:46:33 [emerg] 2854560#2854560: >> posix_memalign(16, 16384) >> > failed (12: Cannot allocate memory) >> > 2018/07/31 03:54:09 [emerg] 2890190#2890190: >> posix_memalign(16, 16384) >> > failed (12: Cannot allocate memory) >> > 2018/07/31 04:08:36 [emerg] 2939230#2939230: >> posix_memalign(16, 16384) >> > failed (12: Cannot allocate memory) >> > 2018/07/31 04:24:48 [emerg] 2992650#2992650: >> posix_memalign(16, 16384) >> > failed (12: Cannot allocate memory) >> > 2018/07/31 04:42:09 [emerg] 3053092#3053092: >> posix_memalign(16, 16384) >> > failed (12: Cannot allocate memory) >> > 2018/07/31 04:42:17 [emerg] 3053335#3053335: >> posix_memalign(16, 16384) >> > failed (12: Cannot allocate memory) >> > 2018/07/31 04:42:28 [emerg] 3053937#3053937: >> posix_memalign(16, 16384) >> > failed (12: Cannot allocate memory) >> > 2018/07/31 04:47:54 [emerg] 3070638#3070638: >> posix_memalign(16, 16384) >> > failed (12: Cannot allocate memory) >> > #################### >> > >> > on a few servers >> > >> > The servers have enough memory free and the swap usage is >> 0, yet somehow >> > the kernel denies the posix_memalign with ENOMEM ( this is >> what I think is >> > happening!) >> > >> > The numbers requested are always 16, 16k . This makes me >> suspicious >> > >> > I have no setting in nginx.conf that reference a 16k >> > >> > Is there any chance of finding out what requests this and >> why this is not >> > fulfilled >> >> There are at least some buffers which default to 16k - for >> example, ssl_buffer_size (http://nginx.org/r/ssl_buffer_size). >> >> You may try debugging log to futher find out where the >> particular >> allocation happens, see here for details: >> >> http://nginx.org/en/docs/debugging_log.html >> >> But I don't really think it worth the effort.? The error is >> pretty >> clear, and it's better to focus on why these allocations are >> denied.? Likely you are hitting some limit. >> >> -- >> Maxim Dounin >> http://mdounin.ru/ >> _______________________________________________ >> nginx mailing list >> nginx at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx >> >> >> >> -- >> *Anoop P Alias* >> >> >> >> _______________________________________________ >> nginx mailing list >> nginx at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx > > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx > > > > -- > *Anoop P Alias* > > > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.friscia at yale.edu Thu Aug 2 13:46:35 2018 From: michael.friscia at yale.edu (Friscia, Michael) Date: Thu, 2 Aug 2018 13:46:35 +0000 Subject: sub_filter not working on JSON file Message-ID: <5DA1C771-8EEE-4F55-92CD-6343420B6B10@yale.edu> I?m trying to figure out why my sub_filter is not working on a JSON file. I do have application/json and text/json listed in the sub_filter_types but the simple string replace is not happening. It causes me to think that for whatever reason, Nginx is not seeing this file as JSON. Is there a way to output what mime type the file from the upstream server is so I can make sure I have the right filter type? Or, is there something I should also be doing to make the sub_filter work on a JSON file? ___________________________________________ Michael Friscia Office of Communications Yale School of Medicine (203) 737-7932 - office (203) 931-5381 - mobile http://web.yale.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From xeioex at nginx.com Thu Aug 2 13:49:49 2018 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Thu, 2 Aug 2018 16:49:49 +0300 Subject: sub_filter not working on JSON file In-Reply-To: <5DA1C771-8EEE-4F55-92CD-6343420B6B10@yale.edu> References: <5DA1C771-8EEE-4F55-92CD-6343420B6B10@yale.edu> Message-ID: On 02.08.2018 16:46, Friscia, Michael wrote: > I?m trying to figure out why my sub_filter is not working on a JSON > file. I do have application/json and text/json listed in the > sub_filter_types but the simple string replace is not happening. It > causes me to think that for whatever reason, Nginx is not seeing this > file as JSON. Is there a way to output what mime type the file from the > upstream server is so I can make sure I have the right filter type? > > Or, is there something I should also be doing to make the sub_filter > work on a JSON file? > Hi, Do not forget to configure sub_filter_types appropriately. > ___________________________________________ > > Michael Friscia > > Office of Communications > > Yale School of Medicine > > (203) 737-7932 - office > > (203) 931-5381 - mobile > > http://web.yale.edu > > > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx > From michael.friscia at yale.edu Thu Aug 2 14:10:59 2018 From: michael.friscia at yale.edu (Friscia, Michael) Date: Thu, 2 Aug 2018 14:10:59 +0000 Subject: sub_filter not working on JSON file In-Reply-To: References: <5DA1C771-8EEE-4F55-92CD-6343420B6B10@yale.edu> Message-ID: <7A426856-B65E-46BA-92CA-8A0F5DD77496@yale.edu> Yeah, the sub_filter_types is fine, if I set it correctly it does not work. So I tested by using Sub_filter_types *; This works fine which makes me wonder what Nginx is seeing as the mime type of this json file. I'll dig through the upstream variables again but I don't recall seeing it unless it's right in front of my face and I'm being blind to it. ___________________________________________ Michael Friscia Office of Communications Yale School of Medicine (203) 737-7932 - office (203) 931-5381 - mobile http://web.yale.edu ?On 8/2/18, 9:49 AM, "Dmitry Volyntsev" wrote: On 02.08.2018 16:46, Friscia, Michael wrote: > I?m trying to figure out why my sub_filter is not working on a JSON > file. I do have application/json and text/json listed in the > sub_filter_types but the simple string replace is not happening. It > causes me to think that for whatever reason, Nginx is not seeing this > file as JSON. Is there a way to output what mime type the file from the > upstream server is so I can make sure I have the right filter type? > > Or, is there something I should also be doing to make the sub_filter > work on a JSON file? > Hi, Do not forget to configure sub_filter_types appropriately. > ___________________________________________ > > Michael Friscia > > Office of Communications > > Yale School of Medicine > > (203) 737-7932 - office > > (203) 931-5381 - mobile > > https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fweb.yale.edu&data=02%7C01%7Cmichael.friscia%40yale.edu%7Cbb4162e4f4dc4c2b17f308d5f87ed329%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C636688145950069464&sdata=Kam1ZbqWD8GXtkGJdvmR7M4sMheqYuPM5OSTEmw5RXE%3D&reserved=0 > > > > _______________________________________________ > nginx mailing list > nginx at nginx.org > https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.nginx.org%2Fmailman%2Flistinfo%2Fnginx&data=02%7C01%7Cmichael.friscia%40yale.edu%7Cbb4162e4f4dc4c2b17f308d5f87ed329%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C636688145950079469&sdata=93k%2BaN0xhqYb9BqOhE4kCwKbAjue2L2Fb%2FwAg0krldI%3D&reserved=0 > From mdounin at mdounin.ru Thu Aug 2 15:11:14 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 2 Aug 2018 18:11:14 +0300 Subject: sub_filter not working on JSON file In-Reply-To: <5DA1C771-8EEE-4F55-92CD-6343420B6B10@yale.edu> References: <5DA1C771-8EEE-4F55-92CD-6343420B6B10@yale.edu> Message-ID: <20180802151114.GP56558@mdounin.ru> Hello! On Thu, Aug 02, 2018 at 01:46:35PM +0000, Friscia, Michael wrote: > I?m trying to figure out why my sub_filter is not working on a > JSON file. I do have application/json and text/json listed in > the sub_filter_types but the simple string replace is not > happening. It causes me to think that for whatever reason, Nginx > is not seeing this file as JSON. Is there a way to output what > mime type the file from the upstream server is so I can make > sure I have the right filter type? > > Or, is there something I should also be doing to make the > sub_filter work on a JSON file? Some things to check, in no particular order: - The Content-Type header as sent by the upstream server is correct. Check the response headers in your favorite client to see the Content-Type header (you can also log $sent_http_content_type or $upstream_http_content_type, but usually checking response headers is enough). - The Content-Type in question is listed in the sub_filter_types directive. - The response is not compressed by the upstream server. If it is, consider adding `proxy_set_header Accept-Encoding "";` to the configuration. -- Maxim Dounin http://mdounin.ru/ From anoopalias01 at gmail.com Sat Aug 4 04:54:56 2018 From: anoopalias01 at gmail.com (Anoop Alias) Date: Sat, 4 Aug 2018 10:24:56 +0530 Subject: posix_memalign error In-Reply-To: References: <20180731133813.GG56558@mdounin.ru> <4fa47f31-9c32-5386-7c2d-a4e9347b3dfc@nginx.com> Message-ID: Hi Igor, Setting vm.max_map_count to 20x the normal value did not help The issue happens on a group of servers and among the group, it shows up only in servers which have ~10k server{} blocks On servers that have lower number of server{} blocks , the ENOMEM issue is not there Also, I can find that the RAM usage of the Nginx process is directly proportional to the number of server {} blocks For example on a server having the problem # ps_mem| head -1 && ps_mem |grep nginx Private + Shared = RAM used Program 1.0 GiB + 2.8 GiB = 3.8 GiB nginx (3) That is for a single worker process with 4 threads in thread_pool # pstree|grep nginx |-nginx-+-nginx---4*[{nginx}] | `-nginx Whatever config change I try the memory usage seem to mostly depend on the number of server contexts defined Now the issue mostly happen in nginx reload ,when one more worker process will be active in shutting down mode I believe the memalign error is thrown by the worker being shutdown, this is because the sites work after the error and also the pid mentioned in the error would have gone when I check ps # pmap 948965|grep 16K 00007f2923ff2000 16K r-x-- ngx_http_redis2_module.so 00007f2924fd7000 16K r---- libc-2.17.so 00007f2925431000 16K rw--- [ anon ] 00007f292584a000 16K rw--- [ anon ] Aug 4 05:50:00 b kernel: SysRq : Show Memory Aug 4 05:50:00 b kernel: Mem-Info: Aug 4 05:50:00 b kernel: active_anon:7757394 inactive_anon:1021319 isolated_anon:0#012 active_file:3733324 inactive_file:2136476 isolated_file:0#012 unevictable:0 dirty:1766 writeback:6 wbtmp:0 unstable:0#012 slab_reclaimable:2003687 slab_unreclaimable:901391#012 mapped:316734 shmem:2381810 pagetables:63163 bounce:0#012 free:4851283 free_pcp:11332 free_cma:0 Aug 4 05:50:00 bravo kernel: Node 0 DMA free:15888kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15972kB managed:15888kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 4 05:50:00 b kernel: lowmem_reserve[]: 0 1679 64139 64139 # cat /proc/buddyinfo Node 0, zone DMA 0 0 1 0 2 1 1 0 1 1 3 Node 0, zone DMA32 5284 6753 6677 1083 410 59 1 0 0 0 0 Node 0, zone Normal 500327 638958 406737 14690 872 106 11 0 0 0 0 Node 1, zone Normal 584840 291640 188 0 0 0 0 0 0 0 0 The only correlation I see in having the error is the number of server {} blocks (close to 10k) which then makes the nginx process consume ~ 4GB of mem with a single worker process and then a reload is done On Thu, Aug 2, 2018 at 6:02 PM Igor A. Ippolitov wrote: > Anoop, > > There are two guesses: either mmap allocations limit is hit or memory is > way too fragmented. > Could you please track amount of mapped regions for a worker with pmap and > amount of 16k areas in Normal zones (it is the third number)? > > You can also set vm.max_map_count to a higher number (like 20 times higher > than default) and look if the error is gone. > > Please, let me know if increasing vm.max_map_count helps you. > > On 02.08.2018 13:06, Anoop Alias wrote: > > Hi Igor, > > The error happens randomly > > 2018/08/02 06:52:42 [emerg] 874514#874514: posix_memalign(16, 16384) > failed (12: Cannot allocate memory) > 2018/08/02 09:42:53 [emerg] 872996#872996: posix_memalign(16, 16384) > failed (12: Cannot allocate memory) > 2018/08/02 10:16:14 [emerg] 877611#877611: posix_memalign(16, 16384) > failed (12: Cannot allocate memory) > 2018/08/02 10:16:48 [emerg] 879410#879410: posix_memalign(16, 16384) > failed (12: Cannot allocate memory) > 2018/08/02 10:17:55 [emerg] 876563#876563: posix_memalign(16, 16384) > failed (12: Cannot allocate memory) > 2018/08/02 10:20:21 [emerg] 879263#879263: posix_memalign(16, 16384) > failed (12: Cannot allocate memory) > 2018/08/02 10:20:51 [emerg] 878991#878991: posix_memalign(16, 16384) > failed (12: Cannot allocate memory) > > # date > Thu Aug 2 10:58:48 BST 2018 > > ------------------------------------------ > # cat /proc/buddyinfo > Node 0, zone DMA 0 0 1 0 2 1 1 > 0 1 1 3 > Node 0, zone DMA32 11722 11057 4663 1647 609 72 10 > 7 1 0 0 > Node 0, zone Normal 755026 710760 398136 21462 1114 18 1 > 0 0 0 0 > Node 1, zone Normal 341295 801810 179604 256 0 0 0 > 0 0 0 0 > ----------------------------------------- > > > slabinfo - version: 2.1 > # name > : tunables : slabdata > > SCTPv6 21 21 1536 21 8 : tunables 0 0 0 > : slabdata 1 1 0 > SCTP 0 0 1408 23 8 : tunables 0 0 0 > : slabdata 0 0 0 > kcopyd_job 0 0 3312 9 8 : tunables 0 0 0 > : slabdata 0 0 0 > dm_uevent 0 0 2608 12 8 : tunables 0 0 0 > : slabdata 0 0 0 > nf_conntrack_ffffffff81acbb00 14054 14892 320 51 4 : tunables > 0 0 0 : slabdata 292 292 0 > lvp_cache 36 36 224 36 2 : tunables 0 0 0 > : slabdata 1 1 0 > lve_struct 4140 4140 352 46 4 : tunables 0 0 0 > : slabdata 90 90 0 > fat_inode_cache 0 0 744 44 8 : tunables 0 0 0 > : slabdata 0 0 0 > fat_cache 0 0 40 102 1 : tunables 0 0 0 > : slabdata 0 0 0 > isofs_inode_cache 0 0 664 49 8 : tunables 0 0 0 > : slabdata 0 0 0 > ext4_inode_cache 30 30 1088 30 8 : tunables 0 0 0 > : slabdata 1 1 0 > ext4_xattr 0 0 88 46 1 : tunables 0 0 0 > : slabdata 0 0 0 > ext4_free_data 0 0 64 64 1 : tunables 0 0 0 > : slabdata 0 0 0 > ext4_allocation_context 32 32 128 32 1 : tunables 0 > 0 0 : slabdata 1 1 0 > ext4_io_end 0 0 72 56 1 : tunables 0 0 0 > : slabdata 0 0 0 > ext4_extent_status 102 102 40 102 1 : tunables 0 0 > 0 : slabdata 1 1 0 > jbd2_journal_handle 0 0 48 85 1 : tunables 0 0 > 0 : slabdata 0 0 0 > jbd2_journal_head 0 0 112 36 1 : tunables 0 0 0 > : slabdata 0 0 0 > jbd2_revoke_table_s 256 256 16 256 1 : tunables 0 0 > 0 : slabdata 1 1 0 > jbd2_revoke_record_s 0 0 32 128 1 : tunables 0 0 > 0 : slabdata 0 0 0 > kvm_async_pf 0 0 136 30 1 : tunables 0 0 0 > : slabdata 0 0 0 > kvm_vcpu 0 0 18560 1 8 : tunables 0 0 0 > : slabdata 0 0 0 > xfs_dqtrx 992 992 528 31 4 : tunables 0 0 0 > : slabdata 32 32 0 > xfs_dquot 3264 3264 472 34 4 : tunables 0 0 0 > : slabdata 96 96 0 > xfs_ili 4342175 4774399 152 53 2 : tunables 0 0 > 0 : slabdata 90083 90083 0 > xfs_inode 4915588 5486076 1088 30 8 : tunables 0 0 > 0 : slabdata 182871 182871 0 > xfs_efd_item 2680 2760 400 40 4 : tunables 0 0 0 > : slabdata 69 69 0 > xfs_da_state 1088 1088 480 34 4 : tunables 0 0 0 > : slabdata 32 32 0 > xfs_btree_cur 1248 1248 208 39 2 : tunables 0 0 0 > : slabdata 32 32 0 > xfs_log_ticket 14874 15048 184 44 2 : tunables 0 0 0 > : slabdata 342 342 0 > xfs_ioend 12909 13104 104 39 1 : tunables 0 0 0 > : slabdata 336 336 0 > scsi_cmd_cache 5400 5652 448 36 4 : tunables 0 0 0 > : slabdata 157 157 0 > ve_struct 0 0 848 38 8 : tunables 0 0 0 > : slabdata 0 0 0 > ip6_dst_cache 1152 1152 448 36 4 : tunables 0 0 0 > : slabdata 32 32 0 > RAWv6 910 910 1216 26 8 : tunables 0 0 0 > : slabdata 35 35 0 > UDPLITEv6 0 0 1216 26 8 : tunables 0 0 0 > : slabdata 0 0 0 > UDPv6 832 832 1216 26 8 : tunables 0 0 0 > : slabdata 32 32 0 > tw_sock_TCPv6 1152 1376 256 32 2 : tunables 0 0 0 > : slabdata 43 43 0 > TCPv6 510 510 2176 15 8 : tunables 0 0 0 > : slabdata 34 34 0 > cfq_queue 3698 5145 232 35 2 : tunables 0 0 0 > : slabdata 147 147 0 > bsg_cmd 0 0 312 52 4 : tunables 0 0 0 > : slabdata 0 0 0 > mqueue_inode_cache 136 136 960 34 8 : tunables 0 0 > 0 : slabdata 4 4 0 > hugetlbfs_inode_cache 1632 1632 632 51 8 : tunables 0 0 > 0 : slabdata 32 32 0 > configfs_dir_cache 1472 1472 88 46 1 : tunables 0 0 > 0 : slabdata 32 32 0 > dquot 0 0 256 32 2 : tunables 0 0 0 > : slabdata 0 0 0 > userfaultfd_ctx_cache 32 32 128 32 1 : tunables 0 0 > 0 : slabdata 1 1 0 > fanotify_event_info 2336 2336 56 73 1 : tunables 0 0 > 0 : slabdata 32 32 0 > dio 6171 6222 640 51 8 : tunables 0 0 0 > : slabdata 122 122 0 > pid_namespace 42 42 2192 14 8 : tunables 0 0 0 > : slabdata 3 3 0 > posix_timers_cache 1056 1056 248 33 2 : tunables 0 0 > 0 : slabdata 32 32 0 > UDP-Lite 0 0 1088 30 8 : tunables 0 0 0 > : slabdata 0 0 0 > flow_cache 2268 2296 144 28 1 : tunables 0 0 0 > : slabdata 82 82 0 > xfrm_dst_cache 896 896 576 28 4 : tunables 0 0 0 > : slabdata 32 32 0 > ip_fib_alias 2720 2720 48 85 1 : tunables 0 0 0 > : slabdata 32 32 0 > RAW 3977 4224 1024 32 8 : tunables 0 0 0 > : slabdata 132 132 0 > UDP 4110 4110 1088 30 8 : tunables 0 0 0 > : slabdata 137 137 0 > tw_sock_TCP 4756 5216 256 32 2 : tunables 0 0 0 > : slabdata 163 163 0 > TCP 2705 2768 1984 16 8 : tunables 0 0 0 > : slabdata 173 173 0 > scsi_data_buffer 5440 5440 24 170 1 : tunables 0 0 0 > : slabdata 32 32 0 > blkdev_queue 154 154 2208 14 8 : tunables 0 0 0 > : slabdata 11 11 0 > blkdev_requests 4397688 4405884 384 42 4 : tunables 0 0 > 0 : slabdata 104902 104902 0 > blkdev_ioc 11232 11232 112 36 1 : tunables 0 0 0 > : slabdata 312 312 0 > user_namespace 0 0 1304 25 8 : tunables 0 0 0 > : slabdata 0 0 0 > sock_inode_cache 12282 12282 704 46 8 : tunables 0 0 0 > : slabdata 267 267 0 > file_lock_cache 20056 20960 200 40 2 : tunables 0 0 0 > : slabdata 524 524 0 > net_namespace 6 6 5056 6 8 : tunables 0 0 0 > : slabdata 1 1 0 > shmem_inode_cache 16970 18952 712 46 8 : tunables 0 0 0 > : slabdata 412 412 0 > Acpi-ParseExt 39491 40432 72 56 1 : tunables 0 0 0 > : slabdata 722 722 0 > Acpi-State 1683 1683 80 51 1 : tunables 0 0 0 > : slabdata 33 33 0 > Acpi-Namespace 11424 11424 40 102 1 : tunables 0 0 0 > : slabdata 112 112 0 > task_delay_info 15336 15336 112 36 1 : tunables 0 0 0 > : slabdata 426 426 0 > taskstats 1568 1568 328 49 4 : tunables 0 0 0 > : slabdata 32 32 0 > proc_inode_cache 169897 190608 680 48 8 : tunables 0 0 0 > : slabdata 3971 3971 0 > sigqueue 2208 2208 168 48 2 : tunables 0 0 0 > : slabdata 46 46 0 > bdev_cache 792 792 896 36 8 : tunables 0 0 0 > : slabdata 22 22 0 > sysfs_dir_cache 74698 74698 120 34 1 : tunables 0 0 0 > : slabdata 2197 2197 0 > mnt_cache 163197 163424 256 32 2 : tunables 0 0 0 > : slabdata 5107 5107 0 > filp 64607 97257 320 51 4 : tunables 0 0 0 > : slabdata 1907 1907 0 > inode_cache 370744 370947 616 53 8 : tunables 0 0 0 > : slabdata 6999 6999 0 > dentry 1316262 2139228 192 42 2 : tunables 0 0 > 0 : slabdata 50934 50934 0 > iint_cache 0 0 80 51 1 : tunables 0 0 0 > : slabdata 0 0 0 > buffer_head 1441470 2890290 104 39 1 : tunables 0 0 > 0 : slabdata 74110 74110 0 > vm_area_struct 194998 196840 216 37 2 : tunables 0 0 0 > : slabdata 5320 5320 0 > mm_struct 2679 2760 1600 20 8 : tunables 0 0 0 > : slabdata 138 138 0 > files_cache 8680 8925 640 51 8 : tunables 0 0 0 > : slabdata 175 175 0 > signal_cache 3691 3780 1152 28 8 : tunables 0 0 0 > : slabdata 135 135 0 > sighand_cache 1950 2160 2112 15 8 : tunables 0 0 0 > : slabdata 144 144 0 > task_xstate 8070 8658 832 39 8 : tunables 0 0 0 > : slabdata 222 222 0 > task_struct 1913 2088 4080 8 8 : tunables 0 0 0 > : slabdata 261 261 0 > cred_jar 31699 33936 192 42 2 : tunables 0 0 0 > : slabdata 808 808 0 > anon_vma_chain 164026 168704 64 64 1 : tunables 0 0 0 > : slabdata 2636 2636 0 > anon_vma 84104 84594 88 46 1 : tunables 0 0 0 > : slabdata 1839 1839 0 > pid 11127 12576 128 32 1 : tunables 0 0 0 > : slabdata 393 393 0 > shared_policy_node 9350 9350 48 85 1 : tunables 0 0 > 0 : slabdata 110 110 0 > numa_policy 62 62 264 31 2 : tunables 0 0 0 > : slabdata 2 2 0 > radix_tree_node 771778 1194312 584 28 4 : tunables 0 0 > 0 : slabdata 42654 42654 0 > idr_layer_cache 2538 2565 2112 15 8 : tunables 0 0 0 > : slabdata 171 171 0 > dma-kmalloc-8192 0 0 8192 4 8 : tunables 0 0 0 > : slabdata 0 0 0 > dma-kmalloc-4096 0 0 4096 8 8 : tunables 0 0 0 > : slabdata 0 0 0 > dma-kmalloc-2048 0 0 2048 16 8 : tunables 0 0 0 > : slabdata 0 0 0 > dma-kmalloc-1024 0 0 1024 32 8 : tunables 0 0 0 > : slabdata 0 0 0 > dma-kmalloc-512 0 0 512 32 4 : tunables 0 0 0 > : slabdata 0 0 0 > dma-kmalloc-256 0 0 256 32 2 : tunables 0 0 0 > : slabdata 0 0 0 > dma-kmalloc-128 0 0 128 32 1 : tunables 0 0 0 > : slabdata 0 0 0 > dma-kmalloc-64 0 0 64 64 1 : tunables 0 0 0 > : slabdata 0 0 0 > dma-kmalloc-32 0 0 32 128 1 : tunables 0 0 0 > : slabdata 0 0 0 > dma-kmalloc-16 0 0 16 256 1 : tunables 0 0 0 > : slabdata 0 0 0 > dma-kmalloc-8 0 0 8 512 1 : tunables 0 0 0 > : slabdata 0 0 0 > dma-kmalloc-192 0 0 192 42 2 : tunables 0 0 0 > : slabdata 0 0 0 > dma-kmalloc-96 0 0 96 42 1 : tunables 0 0 0 > : slabdata 0 0 0 > kmalloc-8192 385 388 8192 4 8 : tunables 0 0 0 > : slabdata 97 97 0 > kmalloc-4096 9296 10088 4096 8 8 : tunables 0 0 0 > : slabdata 1261 1261 0 > kmalloc-2048 65061 133536 2048 16 8 : tunables 0 0 0 > : slabdata 8346 8346 0 > kmalloc-1024 11987 21120 1024 32 8 : tunables 0 0 0 > : slabdata 660 660 0 > kmalloc-512 107510 187072 512 32 4 : tunables 0 0 0 > : slabdata 5846 5846 0 > kmalloc-256 160498 199104 256 32 2 : tunables 0 0 0 > : slabdata 6222 6222 0 > kmalloc-192 144975 237426 192 42 2 : tunables 0 0 0 > : slabdata 5653 5653 0 > kmalloc-128 36799 108096 128 32 1 : tunables 0 0 0 > : slabdata 3378 3378 0 > kmalloc-96 99510 238896 96 42 1 : tunables 0 0 0 > : slabdata 5688 5688 0 > kmalloc-64 7978152 8593280 64 64 1 : tunables 0 0 > 0 : slabdata 134270 134270 0 > kmalloc-32 2939882 3089664 32 128 1 : tunables 0 0 > 0 : slabdata 24138 24138 0 > kmalloc-16 172057 172288 16 256 1 : tunables 0 0 0 > : slabdata 673 673 0 > kmalloc-8 109568 109568 8 512 1 : tunables 0 0 0 > : slabdata 214 214 0 > kmem_cache_node 893 896 64 64 1 : tunables 0 0 0 > : slabdata 14 14 0 > kmem_cache 612 612 320 51 4 : tunables 0 0 0 > : slabdata 12 12 0 > > ------------------------------------------------- > > > # uname -r > 3.10.0-714.10.2.lve1.5.17.1.el7.x86_64 > > -------------------------------------------------------- > > Core part of glances > http://i.imgur.com/La5JbQn.png > ----------------------------------------------------------- > > Thank you very much for looking into this > > > On Thu, Aug 2, 2018 at 12:37 PM Igor A. Ippolitov > wrote: > >> Anoop, >> >> I doubt this will be the solution, but may we have a look at >> /proc/buddyinfo and /proc/slabinfo the moment when nginx can't allocate >> memory? >> >> On 02.08.2018 08:15, Anoop Alias wrote: >> >> Hi Maxim, >> >> I enabled debug and the memalign call is happening on nginx reloads and >> the ENOMEM happen sometimes on the reload(not on all reloads) >> >> 2018/08/02 05:59:08 [notice] 872052#872052: signal process started >> 2018/08/02 05:59:23 [notice] 871570#871570: signal 1 (SIGHUP) received >> from 872052, reconfiguring >> 2018/08/02 05:59:23 [debug] 871570#871570: wake up, sigio 0 >> 2018/08/02 05:59:23 [notice] 871570#871570: reconfiguring >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 0000000002B0DA00:16384 @16 === > the memalign call on reload >> 2018/08/02 05:59:23 [debug] 871570#871570: malloc: 00000000087924D0:4560 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 000000000E442E00:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: malloc: 0000000005650850:4096 >> 20 >> >> >> >> >> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #71 >> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #72 >> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #73 >> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #74 >> 2018/08/02 05:48:49 [debug] 871275#871275: add cleanup: 000000005340D728 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000024D3260:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000517BAF10:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053854FC0:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053855FD0:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053856FE0:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053857FF0:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: posix_memalign: >> 0000000053859000:16384 @16 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385D010:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385E020:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385F030:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CD160:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CE170:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CF180:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D0190:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D11A0:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D21B0:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D31C0:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D41D0:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D51E0:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D61F0:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D7200:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D8210:4096 >> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D9220:4096 >> >> >> Infact there are lot of such calls during a reload >> >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA17ED00:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA1B0FF0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA1E12C0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA211590:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA243880:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA271B30:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA2A3E20:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA2D20D0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA3063E0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA334690:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA366980:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA396C50:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA3C8F40:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA3F9210:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA4294E0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA45B7D0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA489A80:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA4BBD70:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA4EA020:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA51E330:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA54C5E0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA57E8D0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA5AEBA0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA5DEE70:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA611160:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA641430:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA671700:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA6A29E0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA6D5CE0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA707FD0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA736280:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA768570:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA796820:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA7CAB30:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA7F8DE0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA82B0D0:16384 @16 >> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >> 00000000BA85B3A0:16384 @16 >> >> >> >> What is perplexing is that the system has enough free (available RAM) >> ############# >> # free -g >> total used free shared buff/cache >> available >> Mem: 125 54 24 8 46 >> 58 >> Swap: 0 0 0 >> ############# >> >> # ulimit -a >> core file size (blocks, -c) 0 >> data seg size (kbytes, -d) unlimited >> scheduling priority (-e) 0 >> file size (blocks, -f) unlimited >> pending signals (-i) 514579 >> max locked memory (kbytes, -l) 64 >> max memory size (kbytes, -m) unlimited >> open files (-n) 1024 >> pipe size (512 bytes, -p) 8 >> POSIX message queues (bytes, -q) 819200 >> real-time priority (-r) 0 >> stack size (kbytes, -s) 8192 >> cpu time (seconds, -t) unlimited >> max user processes (-u) 514579 >> virtual memory (kbytes, -v) unlimited >> file locks (-x) unlimited >> >> ######################################### >> >> There is no other thing limiting memory allocation >> >> Any way to prevent this or probably identify/prevent this >> >> >> On Tue, Jul 31, 2018 at 7:08 PM Maxim Dounin wrote: >> >>> Hello! >>> >>> On Tue, Jul 31, 2018 at 09:52:29AM +0530, Anoop Alias wrote: >>> >>> > I am repeatedly seeing errors like >>> > >>> > ###################### >>> > 2018/07/31 03:46:33 [emerg] 2854560#2854560: posix_memalign(16, 16384) >>> > failed (12: Cannot allocate memory) >>> > 2018/07/31 03:54:09 [emerg] 2890190#2890190: posix_memalign(16, 16384) >>> > failed (12: Cannot allocate memory) >>> > 2018/07/31 04:08:36 [emerg] 2939230#2939230: posix_memalign(16, 16384) >>> > failed (12: Cannot allocate memory) >>> > 2018/07/31 04:24:48 [emerg] 2992650#2992650: posix_memalign(16, 16384) >>> > failed (12: Cannot allocate memory) >>> > 2018/07/31 04:42:09 [emerg] 3053092#3053092: posix_memalign(16, 16384) >>> > failed (12: Cannot allocate memory) >>> > 2018/07/31 04:42:17 [emerg] 3053335#3053335: posix_memalign(16, 16384) >>> > failed (12: Cannot allocate memory) >>> > 2018/07/31 04:42:28 [emerg] 3053937#3053937: posix_memalign(16, 16384) >>> > failed (12: Cannot allocate memory) >>> > 2018/07/31 04:47:54 [emerg] 3070638#3070638: posix_memalign(16, 16384) >>> > failed (12: Cannot allocate memory) >>> > #################### >>> > >>> > on a few servers >>> > >>> > The servers have enough memory free and the swap usage is 0, yet >>> somehow >>> > the kernel denies the posix_memalign with ENOMEM ( this is what I >>> think is >>> > happening!) >>> > >>> > The numbers requested are always 16, 16k . This makes me suspicious >>> > >>> > I have no setting in nginx.conf that reference a 16k >>> > >>> > Is there any chance of finding out what requests this and why this is >>> not >>> > fulfilled >>> >>> There are at least some buffers which default to 16k - for >>> example, ssl_buffer_size (http://nginx.org/r/ssl_buffer_size). >>> >>> You may try debugging log to futher find out where the particular >>> allocation happens, see here for details: >>> >>> http://nginx.org/en/docs/debugging_log.html >>> >>> But I don't really think it worth the effort. The error is pretty >>> clear, and it's better to focus on why these allocations are >>> denied. Likely you are hitting some limit. >>> >>> -- >>> Maxim Dounin >>> http://mdounin.ru/ >>> _______________________________________________ >>> nginx mailing list >>> nginx at nginx.org >>> http://mailman.nginx.org/mailman/listinfo/nginx >>> >> >> >> -- >> *Anoop P Alias* >> >> >> >> _______________________________________________ >> nginx mailing listnginx at nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx >> >> >> _______________________________________________ >> nginx mailing list >> nginx at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx > > > > -- > *Anoop P Alias* > > > > _______________________________________________ > nginx mailing listnginx at nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx > > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx -- *Anoop P Alias* -------------- next part -------------- An HTML attachment was scrubbed... URL: From plato.puthur at gmail.com Sun Aug 5 20:04:59 2018 From: plato.puthur at gmail.com (Plato Puthur) Date: Mon, 6 Aug 2018 01:34:59 +0530 Subject: nginx, php7.0-fpm and laravel, not able to set it up when the url has a prefix which the server doesn't have Message-ID: Hello I am trying to set up a laravel installation in docker with php-fpm and nginx server on a separate container. The issue is the laravel is installed in a path like /home/apps/foo and the url I need is abcd.com/v11/. I thought this was fairly simple, but I am not able to set it up. Here is my location part in the nginx config location /v11/ { try_files $uri $uri/ /index.php?$query_string; location ~ \.php$ { fastcgi_index index.php; fastcgi_pass php_wbv1.0:5000; fastcgi_split_path_info ^(.+\.php)(/.*)$; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; fastcgi_param QUERY_STRING $query_string; include fastcgi_params; } } I tried rewrite tag, and tried giving alias inside the location block, both didn't work. The thing is if I remove the /v11/ from the location tag and the URL, its working without any issues. What is the right way to do this? -- Sincerely, Plato P -------------- next part -------------- An HTML attachment was scrubbed... URL: From nginx-forum at forum.nginx.org Mon Aug 6 03:21:06 2018 From: nginx-forum at forum.nginx.org (hunterqin) Date: Sun, 05 Aug 2018 23:21:06 -0400 Subject: keepalive not work with grpc Message-ID: <545ddb40a6249a93f8594293a7a13005.NginxMailingListEnglish@forum.nginx.org> Hi, everyone! I have a problem that keepalive does not work with grpc. My conf looks like this: worker_processes 1; events { worker_connections 102400; } http { include mime.types; default_type application/octet-stream; sendfile on; #tcp_nopush on; #keepalive_timeout 0; keepalive_timeout 65; keepalive_requests 1000; upstream grpc_backend { server localhost:2080; keepalive 2; } ... server { listen 1090 http2; server_name localhost; ... location / { grpc_pass grpc://grpc_backend; } } } I have a C++ gprc client and C++ grpc server that works well together without nginx. It seems like that nginx does not support the keepalive with grpc_pass at all. There are still so many connections built from nginx to grpc server that all my ports are consumed. I remember that nginx does support that, so how could it work? Where is my mistake? Posted at Nginx Forum: https://forum.nginx.org/read.php?2,280765,280765#msg-280765 From r at roze.lv Mon Aug 6 08:01:59 2018 From: r at roze.lv (Reinis Rozitis) Date: Mon, 6 Aug 2018 11:01:59 +0300 Subject: nginx, php7.0-fpm and laravel, not able to set it up when the url has a prefix which the server doesn't have In-Reply-To: References: Message-ID: <002801d42d5b$c0a9bbd0$41fd3370$@roze.lv> > I tried rewrite tag, and tried giving alias inside the location block, both didn't work. The thing is if I remove the /v11/ from the location tag and the URL, its working without any issues What is the right way to do this? The try_files needs probably be changed to: try_files $uri $uri/ /v11/index.php?$query_string; rr From iippolitov at nginx.com Mon Aug 6 13:03:12 2018 From: iippolitov at nginx.com (Igor A. Ippolitov) Date: Mon, 6 Aug 2018 16:03:12 +0300 Subject: posix_memalign error In-Reply-To: References: <20180731133813.GG56558@mdounin.ru> <4fa47f31-9c32-5386-7c2d-a4e9347b3dfc@nginx.com> Message-ID: <1fdf4937-cac6-0550-b725-36f04eff65db@nginx.com> Anoop, I suppose, most of your 10k servers are very similar, right? Please, post top level configuration and a typical server{}, please. Also, how do you reload configuration? With 'service nginx reload' or may be other commands? It looks like you have a lot of fragmented memory and only 4gb free in the second numa node. So, I'd say this is OK that you are getting errors from allocating a 16k stripes. Could you please post numastat -m output additionally. Just to make sure you have half of the memory for the second CPU. And we'll have a look if memory utilization may be optimized based on your configuration. Regards, Igor. On 04.08.2018 07:54, Anoop Alias wrote: > Hi Igor, > > Setting?vm.max_map_count to 20x the normal value did not help > > The issue happens on a group of servers and among the group, it shows > up only in servers which have ~10k? server{} blocks > > On servers that have lower number of server{} blocks , the ENOMEM > issue is not there > > Also, I can find that the RAM usage of the Nginx process is directly > proportional to the number of server {} blocks > > For example on a server having the problem > > # ps_mem| head -1 && ps_mem |grep nginx > ?Private? +? ?Shared? =? RAM used? ? ? ?Program > ? 1.0 GiB +? ?2.8 GiB =? ?3.8 GiB? ? ? ?nginx (3) > > > That is for a single worker process with 4 threads in thread_pool > # pstree|grep nginx > ? ? ? ? |-nginx-+-nginx---4*[{nginx}] > ? ? ? ? |? ? ? ?`-nginx > > Whatever config change I try the memory usage seem to mostly depend on > the number of server contexts defined > > Now the issue mostly happen in nginx reload ,when one more worker > process will be active in shutting down mode > > I believe the memalign error is thrown by the worker being shutdown, > this is because the sites work after the error and also the pid > mentioned in the error would have gone when I check ps > > > # pmap 948965|grep 16K > 00007f2923ff2000? ? ?16K r-x-- ngx_http_redis2_module.so > 00007f2924fd7000? ? ?16K r---- libc-2.17.so > 00007f2925431000? ? ?16K rw---? ?[ anon ] > 00007f292584a000? ? ?16K rw---? ?[ anon ] > > Aug? 4 05:50:00 b kernel: SysRq : Show Memory > Aug? 4 05:50:00 b kernel: Mem-Info: > Aug? 4 05:50:00 b kernel: active_anon:7757394 inactive_anon:1021319 > isolated_anon:0#012 active_file:3733324 inactive_file:2136476 > isolated_file:0#012 unevictable:0 dirty:1766 writeback:6 wbtmp:0 > unstable:0#012 slab_reclaimable:2003687 slab_unreclaimable:901391#012 > mapped:316734 shmem:2381810 pagetables:63163 bounce:0#012 free:4851283 > free_pcp:11332 free_cma:0 > Aug? 4 05:50:00 bravo kernel: Node 0 DMA free:15888kB min:8kB low:8kB > high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB > inactive_file:0kB unevictable:0kB isolated(anon):0kB > isolated(file):0kB present:15972kB managed:15888kB mlocked:0kB > dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB > slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB > bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB > pages_scanned:0 all_unreclaimable? yes > Aug? 4 05:50:00 b kernel: lowmem_reserve[]: 0 1679 64139 64139 > > # cat /proc/buddyinfo > Node 0, zone? ? ? DMA? ? ? 0? ? ? 0? ? ? 1? ? ? 0? ? ? 2 ? ? 1? ? ? 1? > ? ? 0? ? ? 1? ? ? 1? ? ? 3 > Node 0, zone? ? DMA32? ?5284? ?6753? ?6677? ?1083? ? 410 ? ?59? ? ? 1? > ? ? 0? ? ? 0? ? ? 0? ? ? 0 > Node 0, zone? ?Normal 500327 638958 406737? 14690? ? 872 ? 106? ? ?11? > ? ? 0? ? ? 0? ? ? 0? ? ? 0 > Node 1, zone? ?Normal 584840 291640? ? 188? ? ? 0? ? ? 0 ? ? 0? ? ? 0? > ? ? 0? ? ? 0? ? ? 0? ? ? 0 > > > The only correlation I see in having the error is the number of > server?{} blocks (close to 10k) which then makes the nginx?process > consume ~ 4GB of mem with a single worker process and then a reload is > done > > > > > On Thu, Aug 2, 2018 at 6:02 PM Igor A. Ippolitov > wrote: > > Anoop, > > There are two guesses: either mmap allocations limit is hit or > memory is? way too fragmented. > Could you please track amount of mapped regions for a worker with > pmap and amount of 16k areas in Normal zones (it is the third number)? > > You can also set vm.max_map_count to a higher number (like 20 > times higher than default) and look if the error is gone. > > Please, let me know if increasing vm.max_map_count helps you. > > On 02.08.2018 13:06, Anoop Alias wrote: >> Hi Igor, >> >> The error happens randomly >> >> 2018/08/02 06:52:42 [emerg] 874514#874514: posix_memalign(16, >> 16384) failed (12: Cannot allocate memory) >> 2018/08/02 09:42:53 [emerg] 872996#872996: posix_memalign(16, >> 16384) failed (12: Cannot allocate memory) >> 2018/08/02 10:16:14 [emerg] 877611#877611: posix_memalign(16, >> 16384) failed (12: Cannot allocate memory) >> 2018/08/02 10:16:48 [emerg] 879410#879410: posix_memalign(16, >> 16384) failed (12: Cannot allocate memory) >> 2018/08/02 10:17:55 [emerg] 876563#876563: posix_memalign(16, >> 16384) failed (12: Cannot allocate memory) >> 2018/08/02 10:20:21 [emerg] 879263#879263: posix_memalign(16, >> 16384) failed (12: Cannot allocate memory) >> 2018/08/02 10:20:51 [emerg] 878991#878991: posix_memalign(16, >> 16384) failed (12: Cannot allocate memory) >> >> # date >> Thu Aug? 2 10:58:48 BST 2018 >> >> ------------------------------------------ >> # cat /proc/buddyinfo >> Node 0, zone? ? ? DMA? ? ? 0? ? ? 0? ? ? 1 0? ? ? 2? ? ? 1? ? ? >> 1? ? ? 0? ? ? 1? ? ? 1 3 >> Node 0, zone? ? DMA32? 11722? 11057? ?4663 ?1647? ? 609? ? ?72? ? >> ?10? ? ? 7? ? ? 1? ? ? 0 ? 0 >> Node 0, zone? ?Normal 755026 710760 398136 21462? ?1114? ? ?18? ? >> ? 1? ? ? 0? ? ? 0? ? ? 0 ? 0 >> Node 1, zone? ?Normal 341295 801810 179604 256? ? ? 0? ? ? 0? ? ? >> 0? ? ? 0? ? ? 0? ? ? 0 0 >> ----------------------------------------- >> >> >> slabinfo - version: 2.1 >> # name? ? ? ? ? ? >> : tunables : >> slabdata >> SCTPv6? ? ? ? ? ? ? ? 21? ? ?21? ?1536? ?21 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 1 1? ? ? 0 >> SCTP? ? ? ? ? ? ? ? ? ?0? ? ? 0? ?1408? ?23 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> kcopyd_job? ? ? ? ? ? ?0? ? ? 0? ?3312? ? 9 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> dm_uevent? ? ? ? ? ? ? 0? ? ? 0? ?2608? ?12 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> nf_conntrack_ffffffff81acbb00? 14054? 14892 320? ?51? ? 4 : >> tunables? ? 0? ? 0? ? 0 : slabdata? ? 292? ? 292? ? ? 0 >> lvp_cache? ? ? ? ? ? ?36? ? ?36? ? 224? ?36 2 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 1 1? ? ? 0 >> lve_struct? ? ? ? ? 4140? ?4140? ? 352? ?46 4 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ?90 ?90? ? ? 0 >> fat_inode_cache? ? ? ? 0? ? ? 0? ? 744? ?44 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> fat_cache? ? ? ? ? ? ? 0? ? ? 0? ? ?40? 102 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> isofs_inode_cache? ? ? 0? ? ? 0? ? 664? ?49 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> ext4_inode_cache? ? ? 30? ? ?30? ?1088? ?30 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 1 1? ? ? 0 >> ext4_xattr? ? ? ? ? ? ?0? ? ? 0? ? ?88? ?46 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> ext4_free_data? ? ? ? ?0? ? ? 0? ? ?64? ?64 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> ext4_allocation_context? ? ?32? ? ?32? ? 128 ?32? ? 1 : tunables? >> ? 0? ? 0? ? 0 : slabdata 1? ? ? 1? ? ? 0 >> ext4_io_end? ? ? ? ? ? 0? ? ? 0? ? ?72? ?56 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> ext4_extent_status? ? 102? ? 102? ? ?40? 102 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 1 1? ? ? 0 >> jbd2_journal_handle? ? ? 0? ? ? 0? ? ?48? ?85 ? 1 : tunables? ? >> 0? ? 0? ? 0 : slabdata? ? ? 0 ? 0? ? ? 0 >> jbd2_journal_head? ? ? 0? ? ? 0? ? 112? ?36 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> jbd2_revoke_table_s? ? 256? ? 256? ? ?16? 256 ? 1 : tunables? ? >> 0? ? 0? ? 0 : slabdata? ? ? 1 ? 1? ? ? 0 >> jbd2_revoke_record_s? ? ? 0? ? ? 0? ? ?32? 128 ? 1 : tunables? ? >> 0? ? 0? ? 0 : slabdata? ? ? 0 ? 0? ? ? 0 >> kvm_async_pf? ? ? ? ? ?0? ? ? 0? ? 136? ?30 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> kvm_vcpu? ? ? ? ? ? ? ?0? ? ? 0? 18560? ? 1 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> xfs_dqtrx? ? ? ? ? ? 992? ? 992? ? 528? ?31 4 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ?32 ?32? ? ? 0 >> xfs_dquot? ? ? ? ? ?3264? ?3264? ? 472? ?34 4 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ?96 ?96? ? ? 0 >> xfs_ili? ? ? ? ? ?4342175 4774399? ? 152? ?53 ? 2 : tunables? ? >> 0? ? 0? ? 0 : slabdata? 90083 90083? ? ? 0 >> xfs_inode? ? ? ? ?4915588 5486076? ?1088? ?30 ? 8 : tunables? ? >> 0? ? 0? ? 0 : slabdata 182871 182871? ? ? 0 >> xfs_efd_item? ? ? ? 2680? ?2760? ? 400? ?40 4 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ?69 ?69? ? ? 0 >> xfs_da_state? ? ? ? 1088? ?1088? ? 480? ?34 4 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ?32 ?32? ? ? 0 >> xfs_btree_cur? ? ? ?1248? ?1248? ? 208? ?39 2 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ?32 ?32? ? ? 0 >> xfs_log_ticket? ? ?14874? 15048? ? 184? ?44 2 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 342 342? ? ? 0 >> xfs_ioend? ? ? ? ? 12909? 13104? ? 104? ?39 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 336 336? ? ? 0 >> scsi_cmd_cache? ? ? 5400? ?5652? ? 448? ?36 4 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 157 157? ? ? 0 >> ve_struct? ? ? ? ? ? ? 0? ? ? 0? ? 848? ?38 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> ip6_dst_cache? ? ? ?1152? ?1152? ? 448? ?36 4 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ?32 ?32? ? ? 0 >> RAWv6? ? ? ? ? ? ? ? 910? ? 910? ?1216? ?26 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ?35 ?35? ? ? 0 >> UDPLITEv6? ? ? ? ? ? ? 0? ? ? 0? ?1216? ?26 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> UDPv6? ? ? ? ? ? ? ? 832? ? 832? ?1216? ?26 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ?32 ?32? ? ? 0 >> tw_sock_TCPv6? ? ? ?1152? ?1376? ? 256? ?32 2 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ?43 ?43? ? ? 0 >> TCPv6? ? ? ? ? ? ? ? 510? ? 510? ?2176? ?15 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ?34 ?34? ? ? 0 >> cfq_queue? ? ? ? ? ?3698? ?5145? ? 232? ?35 2 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 147 147? ? ? 0 >> bsg_cmd? ? ? ? ? ? ? ? 0? ? ? 0? ? 312? ?52 4 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> mqueue_inode_cache? ? 136? ? 136? ? 960? ?34 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 4 4? ? ? 0 >> hugetlbfs_inode_cache? ?1632? ?1632? ? 632 ?51? ? 8 : tunables? ? >> 0? ? 0? ? 0 : slabdata ?32? ? ?32? ? ? 0 >> configfs_dir_cache? ?1472? ?1472? ? ?88? ?46 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ?32 ?32? ? ? 0 >> dquot? ? ? ? ? ? ? ? ? 0? ? ? 0? ? 256? ?32 2 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> userfaultfd_ctx_cache? ? ?32? ? ?32? ? 128 ?32? ? 1 : tunables? ? >> 0? ? 0? ? 0 : slabdata 1? ? ? 1? ? ? 0 >> fanotify_event_info? ?2336? ?2336? ? ?56? ?73 ? 1 : tunables? ? >> 0? ? 0? ? 0 : slabdata? ? ?32 ?32? ? ? 0 >> dio? ? ? ? ? ? ? ? ?6171? ?6222? ? 640? ?51 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 122 122? ? ? 0 >> pid_namespace? ? ? ? ?42? ? ?42? ?2192? ?14 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 3 3? ? ? 0 >> posix_timers_cache? ?1056? ?1056? ? 248? ?33 2 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ?32 ?32? ? ? 0 >> UDP-Lite? ? ? ? ? ? ? ?0? ? ? 0? ?1088? ?30 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> flow_cache? ? ? ? ? 2268? ?2296? ? 144? ?28 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ?82 ?82? ? ? 0 >> xfrm_dst_cache? ? ? ?896? ? 896? ? 576? ?28 4 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ?32 ?32? ? ? 0 >> ip_fib_alias? ? ? ? 2720? ?2720? ? ?48? ?85 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ?32 ?32? ? ? 0 >> RAW? ? ? ? ? ? ? ? ?3977? ?4224? ?1024? ?32 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 132 132? ? ? 0 >> UDP? ? ? ? ? ? ? ? ?4110? ?4110? ?1088? ?30 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 137 137? ? ? 0 >> tw_sock_TCP? ? ? ? ?4756? ?5216? ? 256? ?32 2 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 163 163? ? ? 0 >> TCP? ? ? ? ? ? ? ? ?2705? ?2768? ?1984? ?16 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 173 173? ? ? 0 >> scsi_data_buffer? ? 5440? ?5440? ? ?24? 170 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ?32 ?32? ? ? 0 >> blkdev_queue? ? ? ? ?154? ? 154? ?2208? ?14 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ?11 ?11? ? ? 0 >> blkdev_requests? ?4397688 4405884? ? 384? ?42 ? 4 : tunables? ? >> 0? ? 0? ? 0 : slabdata 104902 104902? ? ? 0 >> blkdev_ioc? ? ? ? ?11232? 11232? ? 112? ?36 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 312 312? ? ? 0 >> user_namespace? ? ? ? ?0? ? ? 0? ?1304? ?25 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> sock_inode_cache? ?12282? 12282? ? 704? ?46 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 267 267? ? ? 0 >> file_lock_cache? ? 20056? 20960? ? 200? ?40 2 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 524 524? ? ? 0 >> net_namespace? ? ? ? ? 6? ? ? 6? ?5056? ? 6 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 1 1? ? ? 0 >> shmem_inode_cache? 16970? 18952? ? 712? ?46 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 412 412? ? ? 0 >> Acpi-ParseExt? ? ? 39491? 40432? ? ?72? ?56 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 722 722? ? ? 0 >> Acpi-State? ? ? ? ? 1683? ?1683? ? ?80? ?51 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ?33 ?33? ? ? 0 >> Acpi-Namespace? ? ?11424? 11424? ? ?40? 102 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 112 112? ? ? 0 >> task_delay_info? ? 15336? 15336? ? 112? ?36 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 426 426? ? ? 0 >> taskstats? ? ? ? ? ?1568? ?1568? ? 328? ?49 4 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ?32 ?32? ? ? 0 >> proc_inode_cache? 169897 190608? ? 680? ?48 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ?3971 ?3971? ? ? 0 >> sigqueue? ? ? ? ? ? 2208? ?2208? ? 168? ?48 2 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ?46 ?46? ? ? 0 >> bdev_cache? ? ? ? ? ?792? ? 792? ? 896? ?36 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ?22 ?22? ? ? 0 >> sysfs_dir_cache? ? 74698? 74698? ? 120? ?34 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ?2197 ?2197? ? ? 0 >> mnt_cache? ? ? ? ?163197 163424? ? 256? ?32 2 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ?5107 ?5107? ? ? 0 >> filp? ? ? ? ? ? ? ?64607? 97257? ? 320? ?51 4 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ?1907 ?1907? ? ? 0 >> inode_cache? ? ? ?370744 370947? ? 616? ?53 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ?6999 ?6999? ? ? 0 >> dentry? ? ? ? ? ? 1316262 2139228? ? 192? ?42 ? 2 : tunables? ? >> 0? ? 0? ? 0 : slabdata? 50934 50934? ? ? 0 >> iint_cache? ? ? ? ? ? ?0? ? ? 0? ? ?80? ?51 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> buffer_head? ? ? ?1441470 2890290? ? 104? ?39 ? 1 : tunables? ? >> 0? ? 0? ? 0 : slabdata? 74110 74110? ? ? 0 >> vm_area_struct? ? 194998 196840? ? 216? ?37 2 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ?5320 ?5320? ? ? 0 >> mm_struct? ? ? ? ? ?2679? ?2760? ?1600? ?20 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 138 138? ? ? 0 >> files_cache? ? ? ? ?8680? ?8925? ? 640? ?51 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 175 175? ? ? 0 >> signal_cache? ? ? ? 3691? ?3780? ?1152? ?28 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 135 135? ? ? 0 >> sighand_cache? ? ? ?1950? ?2160? ?2112? ?15 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 144 144? ? ? 0 >> task_xstate? ? ? ? ?8070? ?8658? ? 832? ?39 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 222 222? ? ? 0 >> task_struct? ? ? ? ?1913? ?2088? ?4080? ? 8 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 261 261? ? ? 0 >> cred_jar? ? ? ? ? ?31699? 33936? ? 192? ?42 2 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 808 808? ? ? 0 >> anon_vma_chain? ? 164026 168704? ? ?64? ?64 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ?2636 ?2636? ? ? 0 >> anon_vma? ? ? ? ? ?84104? 84594? ? ?88? ?46 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ?1839 ?1839? ? ? 0 >> pid? ? ? ? ? ? ? ? 11127? 12576? ? 128? ?32 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 393 393? ? ? 0 >> shared_policy_node? ?9350? ?9350? ? ?48? ?85 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 110 110? ? ? 0 >> numa_policy? ? ? ? ? ?62? ? ?62? ? 264? ?31 2 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 2 2? ? ? 0 >> radix_tree_node? ?771778 1194312? ? 584? ?28 4 : tunables? ? 0? ? >> 0? ? 0 : slabdata? 42654 42654? ? ? 0 >> idr_layer_cache? ? ?2538? ?2565? ?2112? ?15 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 171 171? ? ? 0 >> dma-kmalloc-8192? ? ? ?0? ? ? 0? ?8192? ? 4 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> dma-kmalloc-4096? ? ? ?0? ? ? 0? ?4096? ? 8 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> dma-kmalloc-2048? ? ? ?0? ? ? 0? ?2048? ?16 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> dma-kmalloc-1024? ? ? ?0? ? ? 0? ?1024? ?32 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> dma-kmalloc-512? ? ? ? 0? ? ? 0? ? 512? ?32 4 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> dma-kmalloc-256? ? ? ? 0? ? ? 0? ? 256? ?32 2 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> dma-kmalloc-128? ? ? ? 0? ? ? 0? ? 128? ?32 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> dma-kmalloc-64? ? ? ? ?0? ? ? 0? ? ?64? ?64 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> dma-kmalloc-32? ? ? ? ?0? ? ? 0? ? ?32? 128 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> dma-kmalloc-16? ? ? ? ?0? ? ? 0? ? ?16? 256 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> dma-kmalloc-8? ? ? ? ? 0? ? ? 0? ? ? 8? 512 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> dma-kmalloc-192? ? ? ? 0? ? ? 0? ? 192? ?42 2 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> dma-kmalloc-96? ? ? ? ?0? ? ? 0? ? ?96? ?42 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ? 0 0? ? ? 0 >> kmalloc-8192? ? ? ? ?385? ? 388? ?8192? ? 4 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ?97 ?97? ? ? 0 >> kmalloc-4096? ? ? ? 9296? 10088? ?4096? ? 8 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ?1261 ?1261? ? ? 0 >> kmalloc-2048? ? ? ?65061 133536? ?2048? ?16 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ?8346 ?8346? ? ? 0 >> kmalloc-1024? ? ? ?11987? 21120? ?1024? ?32 8 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 660 660? ? ? 0 >> kmalloc-512? ? ? ?107510 187072? ? 512? ?32 4 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ?5846 ?5846? ? ? 0 >> kmalloc-256? ? ? ?160498 199104? ? 256? ?32 2 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ?6222 ?6222? ? ? 0 >> kmalloc-192? ? ? ?144975 237426? ? 192? ?42 2 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ?5653 ?5653? ? ? 0 >> kmalloc-128? ? ? ? 36799 108096? ? 128? ?32 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ?3378 ?3378? ? ? 0 >> kmalloc-96? ? ? ? ?99510 238896? ? ?96? ?42 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ?5688 ?5688? ? ? 0 >> kmalloc-64? ? ? ? 7978152 8593280? ? ?64? ?64 ? 1 : tunables? ? >> 0? ? 0? ? 0 : slabdata 134270 134270? ? ? 0 >> kmalloc-32? ? ? ? 2939882 3089664? ? ?32? 128 ? 1 : tunables? ? >> 0? ? 0? ? 0 : slabdata? 24138 24138? ? ? 0 >> kmalloc-16? ? ? ? 172057 172288? ? ?16? 256 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 673 673? ? ? 0 >> kmalloc-8? ? ? ? ?109568 109568? ? ? 8? 512 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? 214 214? ? ? 0 >> kmem_cache_node? ? ? 893? ? 896? ? ?64? ?64 1 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ?14 ?14? ? ? 0 >> kmem_cache? ? ? ? ? ?612? ? 612? ? 320? ?51 4 : tunables? ? 0? ? >> 0? ? 0 : slabdata? ? ?12 ?12? ? ? 0 >> >> ------------------------------------------------- >> >> >> # uname -r >> 3.10.0-714.10.2.lve1.5.17.1.el7.x86_64 >> >> -------------------------------------------------------- >> >> Core part of glances >> http://i.imgur.com/La5JbQn.png >> ----------------------------------------------------------- >> >> Thank you very much for looking into this >> >> >> On Thu, Aug 2, 2018 at 12:37 PM Igor A. Ippolitov >> > wrote: >> >> Anoop, >> >> I doubt this will be the solution, but may we have a look at >> /proc/buddyinfo and /proc/slabinfo the moment when nginx >> can't allocate memory? >> >> On 02.08.2018 08:15, Anoop Alias wrote: >>> Hi Maxim, >>> >>> I enabled debug and the memalign call is happening on nginx >>> reloads and the ENOMEM happen sometimes on the reload(not on >>> all reloads) >>> >>> 2018/08/02 05:59:08 [notice] 872052#872052: signal process >>> started >>> 2018/08/02 05:59:23 [notice] 871570#871570: signal 1 >>> (SIGHUP) received from 872052, reconfiguring >>> 2018/08/02 05:59:23 [debug] 871570#871570: wake up, sigio 0 >>> 2018/08/02 05:59:23 [notice] 871570#871570: reconfiguring >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 0000000002B0DA00:16384 @16? ? ? === > the memalign call on >>> reload >>> 2018/08/02 05:59:23 [debug] 871570#871570: malloc: >>> 00000000087924D0:4560 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 000000000E442E00:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: malloc: >>> 0000000005650850:4096 >>> 20 >>> >>> >>> >>> >>> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #71 >>> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #72 >>> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #73 >>> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #74 >>> 2018/08/02 05:48:49 [debug] 871275#871275: add cleanup: >>> 000000005340D728 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>> 00000000024D3260:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>> 00000000517BAF10:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>> 0000000053854FC0:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>> 0000000053855FD0:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>> 0000000053856FE0:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>> 0000000053857FF0:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: posix_memalign: >>> 0000000053859000:16384 @16 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>> 000000005385D010:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>> 000000005385E020:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>> 000000005385F030:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>> 00000000536CD160:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>> 00000000536CE170:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>> 00000000536CF180:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>> 00000000536D0190:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>> 00000000536D11A0:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>> 00000000536D21B0:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>> 00000000536D31C0:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>> 00000000536D41D0:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>> 00000000536D51E0:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>> 00000000536D61F0:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>> 00000000536D7200:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>> 00000000536D8210:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>> 00000000536D9220:4096 >>> >>> >>> Infact there are lot of such calls during a reload >>> >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA17ED00:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA1B0FF0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA1E12C0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA211590:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA243880:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA271B30:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA2A3E20:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA2D20D0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA3063E0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA334690:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA366980:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA396C50:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA3C8F40:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA3F9210:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA4294E0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA45B7D0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA489A80:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA4BBD70:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA4EA020:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA51E330:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA54C5E0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA57E8D0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA5AEBA0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA5DEE70:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA611160:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA641430:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA671700:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA6A29E0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA6D5CE0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA707FD0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA736280:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA768570:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA796820:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA7CAB30:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA7F8DE0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA82B0D0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA85B3A0:16384 @16 >>> >>> >>> >>> What is perplexing is that the system has enough free >>> (available RAM) >>> ############# >>> # free -g >>> ? ? ? ? ? ? ? total? ? ? ? used free? ? ? shared? >>> buff/cache? ?available >>> Mem:? ? ? ? ? ? 125? ? ? ? ? 54 24? ? ? ? ? ?8? ? ? ? ? 46? >>> ? ? ? ? 58 >>> Swap:? ? ? ? ? ? ?0? ? ? ? ? ?0 ?0 >>> ############# >>> >>> # ulimit -a >>> core file size? ? ? ? ? (blocks, -c) 0 >>> data seg size? ? ? ? ? ?(kbytes, -d) unlimited >>> scheduling priority? ? ? ? ? ? ?(-e) 0 >>> file size? ? ? ? ? ? ? ?(blocks, -f) unlimited >>> pending signals? ? ? ? ? ? ? ? ?(-i) 514579 >>> max locked memory? ? ? ?(kbytes, -l) 64 >>> max memory size? ? ? ? ?(kbytes, -m) unlimited >>> open files? ? ? ? ? ? ? ? ? ? ? (-n) 1024 >>> pipe size? ? ? ? ? ? (512 bytes, -p) 8 >>> POSIX message queues? ? ?(bytes, -q) 819200 >>> real-time priority? ? ? ? ? ? ? (-r) 0 >>> stack size? ? ? ? ? ? ? (kbytes, -s) 8192 >>> cpu time? ? ? ? ? ? ? ?(seconds, -t) unlimited >>> max user processes? ? ? ? ? ? ? (-u) 514579 >>> virtual memory? ? ? ? ? (kbytes, -v) unlimited >>> file locks? ? ? ? ? ? ? ? ? ? ? (-x) unlimited >>> >>> ######################################### >>> >>> There is no other thing limiting memory allocation >>> >>> Any way to prevent this or probably identify/prevent this >>> >>> >>> On Tue, Jul 31, 2018 at 7:08 PM Maxim Dounin >>> > wrote: >>> >>> Hello! >>> >>> On Tue, Jul 31, 2018 at 09:52:29AM +0530, Anoop Alias wrote: >>> >>> > I am repeatedly seeing errors like >>> > >>> > ###################### >>> > 2018/07/31 03:46:33 [emerg] 2854560#2854560: >>> posix_memalign(16, 16384) >>> > failed (12: Cannot allocate memory) >>> > 2018/07/31 03:54:09 [emerg] 2890190#2890190: >>> posix_memalign(16, 16384) >>> > failed (12: Cannot allocate memory) >>> > 2018/07/31 04:08:36 [emerg] 2939230#2939230: >>> posix_memalign(16, 16384) >>> > failed (12: Cannot allocate memory) >>> > 2018/07/31 04:24:48 [emerg] 2992650#2992650: >>> posix_memalign(16, 16384) >>> > failed (12: Cannot allocate memory) >>> > 2018/07/31 04:42:09 [emerg] 3053092#3053092: >>> posix_memalign(16, 16384) >>> > failed (12: Cannot allocate memory) >>> > 2018/07/31 04:42:17 [emerg] 3053335#3053335: >>> posix_memalign(16, 16384) >>> > failed (12: Cannot allocate memory) >>> > 2018/07/31 04:42:28 [emerg] 3053937#3053937: >>> posix_memalign(16, 16384) >>> > failed (12: Cannot allocate memory) >>> > 2018/07/31 04:47:54 [emerg] 3070638#3070638: >>> posix_memalign(16, 16384) >>> > failed (12: Cannot allocate memory) >>> > #################### >>> > >>> > on a few servers >>> > >>> > The servers have enough memory free and the swap usage >>> is 0, yet somehow >>> > the kernel denies the posix_memalign with ENOMEM ( >>> this is what I think is >>> > happening!) >>> > >>> > The numbers requested are always 16, 16k . This makes >>> me suspicious >>> > >>> > I have no setting in nginx.conf that reference a 16k >>> > >>> > Is there any chance of finding out what requests this >>> and why this is not >>> > fulfilled >>> >>> There are at least some buffers which default to 16k - for >>> example, ssl_buffer_size >>> (http://nginx.org/r/ssl_buffer_size). >>> >>> You may try debugging log to futher find out where the >>> particular >>> allocation happens, see here for details: >>> >>> http://nginx.org/en/docs/debugging_log.html >>> >>> But I don't really think it worth the effort. The error >>> is pretty >>> clear, and it's better to focus on why these allocations >>> are >>> denied.? Likely you are hitting some limit. >>> >>> -- >>> Maxim Dounin >>> http://mdounin.ru/ >>> _______________________________________________ >>> nginx mailing list >>> nginx at nginx.org >>> http://mailman.nginx.org/mailman/listinfo/nginx >>> >>> >>> >>> -- >>> *Anoop P Alias* >>> >>> >>> >>> _______________________________________________ >>> nginx mailing list >>> nginx at nginx.org >>> http://mailman.nginx.org/mailman/listinfo/nginx >> >> >> _______________________________________________ >> nginx mailing list >> nginx at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx >> >> >> >> -- >> *Anoop P Alias* >> >> >> >> _______________________________________________ >> nginx mailing list >> nginx at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx > > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx > > > > -- > *Anoop P Alias* > > > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx -------------- next part -------------- An HTML attachment was scrubbed... URL: From plato.puthur at gmail.com Mon Aug 6 15:29:40 2018 From: plato.puthur at gmail.com (Plato Puthur) Date: Mon, 6 Aug 2018 20:59:40 +0530 Subject: nginx, php7.0-fpm and laravel, not able to set it up when the url has a prefix which the server doesn't have In-Reply-To: <002801d42d5b$c0a9bbd0$41fd3370$@roze.lv> References: <002801d42d5b$c0a9bbd0$41fd3370$@roze.lv> Message-ID: Hello, Thanks for that input, I changed my config like you said and also set the fastcgi_param SCRIPT_FILENAME to /path/to/the/index.php; But I feel like this should be done by the rewrite directive instead of this way. Now that this is working, can someone please tell me if I can use the rewrite tag so that I can avoid hardcoding the path everywhere? On Mon, Aug 6, 2018 at 1:31 PM, Reinis Rozitis wrote: > > I tried rewrite tag, and tried giving alias inside the location block, > both didn't work. The thing is if I remove the /v11/ from the location tag > and the URL, its working without any issues What is the right way to do > this? > > > The try_files needs probably be changed to: > > try_files $uri $uri/ /v11/index.php?$query_string; > > > rr > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx > -- Sincerely, Plato P about.me/plato_p -------------- next part -------------- An HTML attachment was scrubbed... URL: From anoopalias01 at gmail.com Mon Aug 6 19:55:14 2018 From: anoopalias01 at gmail.com (Anoop Alias) Date: Tue, 7 Aug 2018 01:25:14 +0530 Subject: posix_memalign error In-Reply-To: <1fdf4937-cac6-0550-b725-36f04eff65db@nginx.com> References: <20180731133813.GG56558@mdounin.ru> <4fa47f31-9c32-5386-7c2d-a4e9347b3dfc@nginx.com> <1fdf4937-cac6-0550-b725-36f04eff65db@nginx.com> Message-ID: Hi Igor, Config is reloaded using /usr/sbin/nginx -s reload this is invoked from a python/shell script ( Nginx is installed on a web control panel ) The top-level Nginx config is in the gist below https://gist.github.com/AnoopAlias/ba5ad6749a586c7e267672ee65b32b3a It further includes ~8k server blocks or more in some servers. Out of this 2/3 are server {} blocks with TLS config and 1/3 non-TLS ones ]# pwd /etc/nginx/sites-enabled # grep "server {" *|wc -l 7886 And yes most of them are very similar and mostly proxy to upstream httpd I have tried removing all the loadable modules and even tried an older version of nginx and all produce the error # numastat -m Per-node system memory usage (in MBs): Node 0 Node 1 Total --------------- --------------- --------------- MemTotal 65430.84 65536.00 130966.84 MemFree 5491.26 40.89 5532.15 MemUsed 59939.58 65495.11 125434.69 Active 22295.61 21016.09 43311.70 Inactive 8742.76 4662.48 13405.24 Active(anon) 16717.10 16572.19 33289.29 Inactive(anon) 2931.94 1388.14 4320.08 Active(file) 5578.50 4443.91 10022.41 Inactive(file) 5810.82 3274.34 9085.16 Unevictable 0.00 0.00 0.00 Mlocked 0.00 0.00 0.00 Dirty 7.04 1.64 8.67 Writeback 0.00 0.00 0.00 FilePages 18458.93 10413.97 28872.90 Mapped 862.14 413.38 1275.52 AnonPages 12579.49 15264.37 27843.86 Shmem 7069.52 2695.71 9765.23 KernelStack 18.34 3.03 21.38 PageTables 153.14 107.77 260.90 NFS_Unstable 0.00 0.00 0.00 Bounce 0.00 0.00 0.00 WritebackTmp 0.00 0.00 0.00 Slab 4830.68 2254.55 7085.22 SReclaimable 2061.05 921.72 2982.77 SUnreclaim 2769.62 1332.83 4102.45 AnonHugePages 4.00 2.00 6.00 HugePages_Total 0.00 0.00 0.00 HugePages_Free 0.00 0.00 0.00 HugePages_Surp 0.00 0.00 0.00 Thanks, On Mon, Aug 6, 2018 at 6:33 PM Igor A. Ippolitov wrote: > Anoop, > > I suppose, most of your 10k servers are very similar, right? > Please, post top level configuration and a typical server{}, please. > > Also, how do you reload configuration? With 'service nginx reload' or may > be other commands? > > It looks like you have a lot of fragmented memory and only 4gb free in the > second numa node. > So, I'd say this is OK that you are getting errors from allocating a 16k > stripes. > > Could you please post numastat -m output additionally. Just to make sure > you have half of the memory for the second CPU. > And we'll have a look if memory utilization may be optimized based on your > configuration. > > Regards, > Igor. > > On 04.08.2018 07:54, Anoop Alias wrote: > > Hi Igor, > > Setting vm.max_map_count to 20x the normal value did not help > > The issue happens on a group of servers and among the group, it shows up > only in servers which have ~10k server{} blocks > > On servers that have lower number of server{} blocks , the ENOMEM issue is > not there > > Also, I can find that the RAM usage of the Nginx process is directly > proportional to the number of server {} blocks > > For example on a server having the problem > > # ps_mem| head -1 && ps_mem |grep nginx > Private + Shared = RAM used Program > 1.0 GiB + 2.8 GiB = 3.8 GiB nginx (3) > > > That is for a single worker process with 4 threads in thread_pool > # pstree|grep nginx > |-nginx-+-nginx---4*[{nginx}] > | `-nginx > > Whatever config change I try the memory usage seem to mostly depend on the > number of server contexts defined > > Now the issue mostly happen in nginx reload ,when one more worker process > will be active in shutting down mode > > I believe the memalign error is thrown by the worker being shutdown, this > is because the sites work after the error and also the pid mentioned in the > error would have gone when I check ps > > > # pmap 948965|grep 16K > 00007f2923ff2000 16K r-x-- ngx_http_redis2_module.so > 00007f2924fd7000 16K r---- libc-2.17.so > 00007f2925431000 16K rw--- [ anon ] > 00007f292584a000 16K rw--- [ anon ] > > Aug 4 05:50:00 b kernel: SysRq : Show Memory > Aug 4 05:50:00 b kernel: Mem-Info: > Aug 4 05:50:00 b kernel: active_anon:7757394 inactive_anon:1021319 > isolated_anon:0#012 active_file:3733324 inactive_file:2136476 isolated_ > file:0#012 unevictable:0 dirty:1766 writeback:6 wbtmp:0 unstable:0#012 > slab_reclaimable:2003687 slab_unreclaimable:901391#012 mapped:316734 > shmem:2381810 pagetables:63163 bounce:0#012 free:4851283 free_pcp:11332 > free_cma:0 > Aug 4 05:50:00 bravo kernel: Node 0 DMA free:15888kB min:8kB low:8kB > high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_ > file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB > present:15972kB managed:15888kB mlocked:0kB dirty:0kB writeback:0kB > mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB > kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB > local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 > all_unreclaimable? yes > Aug 4 05:50:00 b kernel: lowmem_reserve[]: 0 1679 64139 64139 > > # cat /proc/buddyinfo > Node 0, zone DMA 0 0 1 0 2 1 1 > 0 1 1 3 > Node 0, zone DMA32 5284 6753 6677 1083 410 59 1 > 0 0 0 0 > Node 0, zone Normal 500327 638958 406737 14690 872 106 11 > 0 0 0 0 > Node 1, zone Normal 584840 291640 188 0 0 0 0 > 0 0 0 0 > > > The only correlation I see in having the error is the number of server {} > blocks (close to 10k) which then makes the nginx process consume ~ 4GB of > mem with a single worker process and then a reload is done > > > > > On Thu, Aug 2, 2018 at 6:02 PM Igor A. Ippolitov > wrote: > >> Anoop, >> >> There are two guesses: either mmap allocations limit is hit or memory is >> way too fragmented. >> Could you please track amount of mapped regions for a worker with pmap >> and amount of 16k areas in Normal zones (it is the third number)? >> >> You can also set vm.max_map_count to a higher number (like 20 times >> higher than default) and look if the error is gone. >> >> Please, let me know if increasing vm.max_map_count helps you. >> >> On 02.08.2018 13:06, Anoop Alias wrote: >> >> Hi Igor, >> >> The error happens randomly >> >> 2018/08/02 06:52:42 [emerg] 874514#874514: posix_memalign(16, 16384) >> failed (12: Cannot allocate memory) >> 2018/08/02 09:42:53 [emerg] 872996#872996: posix_memalign(16, 16384) >> failed (12: Cannot allocate memory) >> 2018/08/02 10:16:14 [emerg] 877611#877611: posix_memalign(16, 16384) >> failed (12: Cannot allocate memory) >> 2018/08/02 10:16:48 [emerg] 879410#879410: posix_memalign(16, 16384) >> failed (12: Cannot allocate memory) >> 2018/08/02 10:17:55 [emerg] 876563#876563: posix_memalign(16, 16384) >> failed (12: Cannot allocate memory) >> 2018/08/02 10:20:21 [emerg] 879263#879263: posix_memalign(16, 16384) >> failed (12: Cannot allocate memory) >> 2018/08/02 10:20:51 [emerg] 878991#878991: posix_memalign(16, 16384) >> failed (12: Cannot allocate memory) >> >> # date >> Thu Aug 2 10:58:48 BST 2018 >> >> ------------------------------------------ >> # cat /proc/buddyinfo >> Node 0, zone DMA 0 0 1 0 2 1 1 >> 0 1 1 3 >> Node 0, zone DMA32 11722 11057 4663 1647 609 72 10 >> 7 1 0 0 >> Node 0, zone Normal 755026 710760 398136 21462 1114 18 1 >> 0 0 0 0 >> Node 1, zone Normal 341295 801810 179604 256 0 0 0 >> 0 0 0 0 >> ----------------------------------------- >> >> >> slabinfo - version: 2.1 >> # name >> : tunables : slabdata >> >> SCTPv6 21 21 1536 21 8 : tunables 0 0 >> 0 : slabdata 1 1 0 >> SCTP 0 0 1408 23 8 : tunables 0 0 >> 0 : slabdata 0 0 0 >> kcopyd_job 0 0 3312 9 8 : tunables 0 0 >> 0 : slabdata 0 0 0 >> dm_uevent 0 0 2608 12 8 : tunables 0 0 >> 0 : slabdata 0 0 0 >> nf_conntrack_ffffffff81acbb00 14054 14892 320 51 4 : tunables >> 0 0 0 : slabdata 292 292 0 >> lvp_cache 36 36 224 36 2 : tunables 0 0 >> 0 : slabdata 1 1 0 >> lve_struct 4140 4140 352 46 4 : tunables 0 0 >> 0 : slabdata 90 90 0 >> fat_inode_cache 0 0 744 44 8 : tunables 0 0 >> 0 : slabdata 0 0 0 >> fat_cache 0 0 40 102 1 : tunables 0 0 >> 0 : slabdata 0 0 0 >> isofs_inode_cache 0 0 664 49 8 : tunables 0 0 >> 0 : slabdata 0 0 0 >> ext4_inode_cache 30 30 1088 30 8 : tunables 0 0 >> 0 : slabdata 1 1 0 >> ext4_xattr 0 0 88 46 1 : tunables 0 0 >> 0 : slabdata 0 0 0 >> ext4_free_data 0 0 64 64 1 : tunables 0 0 >> 0 : slabdata 0 0 0 >> ext4_allocation_context 32 32 128 32 1 : tunables 0 >> 0 0 : slabdata 1 1 0 >> ext4_io_end 0 0 72 56 1 : tunables 0 0 >> 0 : slabdata 0 0 0 >> ext4_extent_status 102 102 40 102 1 : tunables 0 0 >> 0 : slabdata 1 1 0 >> jbd2_journal_handle 0 0 48 85 1 : tunables 0 0 >> 0 : slabdata 0 0 0 >> jbd2_journal_head 0 0 112 36 1 : tunables 0 0 >> 0 : slabdata 0 0 0 >> jbd2_revoke_table_s 256 256 16 256 1 : tunables 0 0 >> 0 : slabdata 1 1 0 >> jbd2_revoke_record_s 0 0 32 128 1 : tunables 0 0 >> 0 : slabdata 0 0 0 >> kvm_async_pf 0 0 136 30 1 : tunables 0 0 >> 0 : slabdata 0 0 0 >> kvm_vcpu 0 0 18560 1 8 : tunables 0 0 >> 0 : slabdata 0 0 0 >> xfs_dqtrx 992 992 528 31 4 : tunables 0 0 >> 0 : slabdata 32 32 0 >> xfs_dquot 3264 3264 472 34 4 : tunables 0 0 >> 0 : slabdata 96 96 0 >> xfs_ili 4342175 4774399 152 53 2 : tunables 0 0 >> 0 : slabdata 90083 90083 0 >> xfs_inode 4915588 5486076 1088 30 8 : tunables 0 0 >> 0 : slabdata 182871 182871 0 >> xfs_efd_item 2680 2760 400 40 4 : tunables 0 0 >> 0 : slabdata 69 69 0 >> xfs_da_state 1088 1088 480 34 4 : tunables 0 0 >> 0 : slabdata 32 32 0 >> xfs_btree_cur 1248 1248 208 39 2 : tunables 0 0 >> 0 : slabdata 32 32 0 >> xfs_log_ticket 14874 15048 184 44 2 : tunables 0 0 >> 0 : slabdata 342 342 0 >> xfs_ioend 12909 13104 104 39 1 : tunables 0 0 >> 0 : slabdata 336 336 0 >> scsi_cmd_cache 5400 5652 448 36 4 : tunables 0 0 >> 0 : slabdata 157 157 0 >> ve_struct 0 0 848 38 8 : tunables 0 0 >> 0 : slabdata 0 0 0 >> ip6_dst_cache 1152 1152 448 36 4 : tunables 0 0 >> 0 : slabdata 32 32 0 >> RAWv6 910 910 1216 26 8 : tunables 0 0 >> 0 : slabdata 35 35 0 >> UDPLITEv6 0 0 1216 26 8 : tunables 0 0 >> 0 : slabdata 0 0 0 >> UDPv6 832 832 1216 26 8 : tunables 0 0 >> 0 : slabdata 32 32 0 >> tw_sock_TCPv6 1152 1376 256 32 2 : tunables 0 0 >> 0 : slabdata 43 43 0 >> TCPv6 510 510 2176 15 8 : tunables 0 0 >> 0 : slabdata 34 34 0 >> cfq_queue 3698 5145 232 35 2 : tunables 0 0 >> 0 : slabdata 147 147 0 >> bsg_cmd 0 0 312 52 4 : tunables 0 0 >> 0 : slabdata 0 0 0 >> mqueue_inode_cache 136 136 960 34 8 : tunables 0 0 >> 0 : slabdata 4 4 0 >> hugetlbfs_inode_cache 1632 1632 632 51 8 : tunables 0 >> 0 0 : slabdata 32 32 0 >> configfs_dir_cache 1472 1472 88 46 1 : tunables 0 0 >> 0 : slabdata 32 32 0 >> dquot 0 0 256 32 2 : tunables 0 0 >> 0 : slabdata 0 0 0 >> userfaultfd_ctx_cache 32 32 128 32 1 : tunables 0 >> 0 0 : slabdata 1 1 0 >> fanotify_event_info 2336 2336 56 73 1 : tunables 0 0 >> 0 : slabdata 32 32 0 >> dio 6171 6222 640 51 8 : tunables 0 0 >> 0 : slabdata 122 122 0 >> pid_namespace 42 42 2192 14 8 : tunables 0 0 >> 0 : slabdata 3 3 0 >> posix_timers_cache 1056 1056 248 33 2 : tunables 0 0 >> 0 : slabdata 32 32 0 >> UDP-Lite 0 0 1088 30 8 : tunables 0 0 >> 0 : slabdata 0 0 0 >> flow_cache 2268 2296 144 28 1 : tunables 0 0 >> 0 : slabdata 82 82 0 >> xfrm_dst_cache 896 896 576 28 4 : tunables 0 0 >> 0 : slabdata 32 32 0 >> ip_fib_alias 2720 2720 48 85 1 : tunables 0 0 >> 0 : slabdata 32 32 0 >> RAW 3977 4224 1024 32 8 : tunables 0 0 >> 0 : slabdata 132 132 0 >> UDP 4110 4110 1088 30 8 : tunables 0 0 >> 0 : slabdata 137 137 0 >> tw_sock_TCP 4756 5216 256 32 2 : tunables 0 0 >> 0 : slabdata 163 163 0 >> TCP 2705 2768 1984 16 8 : tunables 0 0 >> 0 : slabdata 173 173 0 >> scsi_data_buffer 5440 5440 24 170 1 : tunables 0 0 >> 0 : slabdata 32 32 0 >> blkdev_queue 154 154 2208 14 8 : tunables 0 0 >> 0 : slabdata 11 11 0 >> blkdev_requests 4397688 4405884 384 42 4 : tunables 0 0 >> 0 : slabdata 104902 104902 0 >> blkdev_ioc 11232 11232 112 36 1 : tunables 0 0 >> 0 : slabdata 312 312 0 >> user_namespace 0 0 1304 25 8 : tunables 0 0 >> 0 : slabdata 0 0 0 >> sock_inode_cache 12282 12282 704 46 8 : tunables 0 0 >> 0 : slabdata 267 267 0 >> file_lock_cache 20056 20960 200 40 2 : tunables 0 0 >> 0 : slabdata 524 524 0 >> net_namespace 6 6 5056 6 8 : tunables 0 0 >> 0 : slabdata 1 1 0 >> shmem_inode_cache 16970 18952 712 46 8 : tunables 0 0 >> 0 : slabdata 412 412 0 >> Acpi-ParseExt 39491 40432 72 56 1 : tunables 0 0 >> 0 : slabdata 722 722 0 >> Acpi-State 1683 1683 80 51 1 : tunables 0 0 >> 0 : slabdata 33 33 0 >> Acpi-Namespace 11424 11424 40 102 1 : tunables 0 0 >> 0 : slabdata 112 112 0 >> task_delay_info 15336 15336 112 36 1 : tunables 0 0 >> 0 : slabdata 426 426 0 >> taskstats 1568 1568 328 49 4 : tunables 0 0 >> 0 : slabdata 32 32 0 >> proc_inode_cache 169897 190608 680 48 8 : tunables 0 0 >> 0 : slabdata 3971 3971 0 >> sigqueue 2208 2208 168 48 2 : tunables 0 0 >> 0 : slabdata 46 46 0 >> bdev_cache 792 792 896 36 8 : tunables 0 0 >> 0 : slabdata 22 22 0 >> sysfs_dir_cache 74698 74698 120 34 1 : tunables 0 0 >> 0 : slabdata 2197 2197 0 >> mnt_cache 163197 163424 256 32 2 : tunables 0 0 >> 0 : slabdata 5107 5107 0 >> filp 64607 97257 320 51 4 : tunables 0 0 >> 0 : slabdata 1907 1907 0 >> inode_cache 370744 370947 616 53 8 : tunables 0 0 >> 0 : slabdata 6999 6999 0 >> dentry 1316262 2139228 192 42 2 : tunables 0 0 >> 0 : slabdata 50934 50934 0 >> iint_cache 0 0 80 51 1 : tunables 0 0 >> 0 : slabdata 0 0 0 >> buffer_head 1441470 2890290 104 39 1 : tunables 0 0 >> 0 : slabdata 74110 74110 0 >> vm_area_struct 194998 196840 216 37 2 : tunables 0 0 >> 0 : slabdata 5320 5320 0 >> mm_struct 2679 2760 1600 20 8 : tunables 0 0 >> 0 : slabdata 138 138 0 >> files_cache 8680 8925 640 51 8 : tunables 0 0 >> 0 : slabdata 175 175 0 >> signal_cache 3691 3780 1152 28 8 : tunables 0 0 >> 0 : slabdata 135 135 0 >> sighand_cache 1950 2160 2112 15 8 : tunables 0 0 >> 0 : slabdata 144 144 0 >> task_xstate 8070 8658 832 39 8 : tunables 0 0 >> 0 : slabdata 222 222 0 >> task_struct 1913 2088 4080 8 8 : tunables 0 0 >> 0 : slabdata 261 261 0 >> cred_jar 31699 33936 192 42 2 : tunables 0 0 >> 0 : slabdata 808 808 0 >> anon_vma_chain 164026 168704 64 64 1 : tunables 0 0 >> 0 : slabdata 2636 2636 0 >> anon_vma 84104 84594 88 46 1 : tunables 0 0 >> 0 : slabdata 1839 1839 0 >> pid 11127 12576 128 32 1 : tunables 0 0 >> 0 : slabdata 393 393 0 >> shared_policy_node 9350 9350 48 85 1 : tunables 0 0 >> 0 : slabdata 110 110 0 >> numa_policy 62 62 264 31 2 : tunables 0 0 >> 0 : slabdata 2 2 0 >> radix_tree_node 771778 1194312 584 28 4 : tunables 0 0 >> 0 : slabdata 42654 42654 0 >> idr_layer_cache 2538 2565 2112 15 8 : tunables 0 0 >> 0 : slabdata 171 171 0 >> dma-kmalloc-8192 0 0 8192 4 8 : tunables 0 0 >> 0 : slabdata 0 0 0 >> dma-kmalloc-4096 0 0 4096 8 8 : tunables 0 0 >> 0 : slabdata 0 0 0 >> dma-kmalloc-2048 0 0 2048 16 8 : tunables 0 0 >> 0 : slabdata 0 0 0 >> dma-kmalloc-1024 0 0 1024 32 8 : tunables 0 0 >> 0 : slabdata 0 0 0 >> dma-kmalloc-512 0 0 512 32 4 : tunables 0 0 >> 0 : slabdata 0 0 0 >> dma-kmalloc-256 0 0 256 32 2 : tunables 0 0 >> 0 : slabdata 0 0 0 >> dma-kmalloc-128 0 0 128 32 1 : tunables 0 0 >> 0 : slabdata 0 0 0 >> dma-kmalloc-64 0 0 64 64 1 : tunables 0 0 >> 0 : slabdata 0 0 0 >> dma-kmalloc-32 0 0 32 128 1 : tunables 0 0 >> 0 : slabdata 0 0 0 >> dma-kmalloc-16 0 0 16 256 1 : tunables 0 0 >> 0 : slabdata 0 0 0 >> dma-kmalloc-8 0 0 8 512 1 : tunables 0 0 >> 0 : slabdata 0 0 0 >> dma-kmalloc-192 0 0 192 42 2 : tunables 0 0 >> 0 : slabdata 0 0 0 >> dma-kmalloc-96 0 0 96 42 1 : tunables 0 0 >> 0 : slabdata 0 0 0 >> kmalloc-8192 385 388 8192 4 8 : tunables 0 0 >> 0 : slabdata 97 97 0 >> kmalloc-4096 9296 10088 4096 8 8 : tunables 0 0 >> 0 : slabdata 1261 1261 0 >> kmalloc-2048 65061 133536 2048 16 8 : tunables 0 0 >> 0 : slabdata 8346 8346 0 >> kmalloc-1024 11987 21120 1024 32 8 : tunables 0 0 >> 0 : slabdata 660 660 0 >> kmalloc-512 107510 187072 512 32 4 : tunables 0 0 >> 0 : slabdata 5846 5846 0 >> kmalloc-256 160498 199104 256 32 2 : tunables 0 0 >> 0 : slabdata 6222 6222 0 >> kmalloc-192 144975 237426 192 42 2 : tunables 0 0 >> 0 : slabdata 5653 5653 0 >> kmalloc-128 36799 108096 128 32 1 : tunables 0 0 >> 0 : slabdata 3378 3378 0 >> kmalloc-96 99510 238896 96 42 1 : tunables 0 0 >> 0 : slabdata 5688 5688 0 >> kmalloc-64 7978152 8593280 64 64 1 : tunables 0 0 >> 0 : slabdata 134270 134270 0 >> kmalloc-32 2939882 3089664 32 128 1 : tunables 0 0 >> 0 : slabdata 24138 24138 0 >> kmalloc-16 172057 172288 16 256 1 : tunables 0 0 >> 0 : slabdata 673 673 0 >> kmalloc-8 109568 109568 8 512 1 : tunables 0 0 >> 0 : slabdata 214 214 0 >> kmem_cache_node 893 896 64 64 1 : tunables 0 0 >> 0 : slabdata 14 14 0 >> kmem_cache 612 612 320 51 4 : tunables 0 0 >> 0 : slabdata 12 12 0 >> >> ------------------------------------------------- >> >> >> # uname -r >> 3.10.0-714.10.2.lve1.5.17.1.el7.x86_64 >> >> -------------------------------------------------------- >> >> Core part of glances >> http://i.imgur.com/La5JbQn.png >> ----------------------------------------------------------- >> >> Thank you very much for looking into this >> >> >> On Thu, Aug 2, 2018 at 12:37 PM Igor A. Ippolitov >> wrote: >> >>> Anoop, >>> >>> I doubt this will be the solution, but may we have a look at >>> /proc/buddyinfo and /proc/slabinfo the moment when nginx can't allocate >>> memory? >>> >>> On 02.08.2018 08:15, Anoop Alias wrote: >>> >>> Hi Maxim, >>> >>> I enabled debug and the memalign call is happening on nginx reloads and >>> the ENOMEM happen sometimes on the reload(not on all reloads) >>> >>> 2018/08/02 05:59:08 [notice] 872052#872052: signal process started >>> 2018/08/02 05:59:23 [notice] 871570#871570: signal 1 (SIGHUP) received >>> from 872052, reconfiguring >>> 2018/08/02 05:59:23 [debug] 871570#871570: wake up, sigio 0 >>> 2018/08/02 05:59:23 [notice] 871570#871570: reconfiguring >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 0000000002B0DA00:16384 @16 === > the memalign call on reload >>> 2018/08/02 05:59:23 [debug] 871570#871570: malloc: 00000000087924D0:4560 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 000000000E442E00:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: malloc: 0000000005650850:4096 >>> 20 >>> >>> >>> >>> >>> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #71 >>> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #72 >>> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #73 >>> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #74 >>> 2018/08/02 05:48:49 [debug] 871275#871275: add cleanup: 000000005340D728 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000024D3260:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000517BAF10:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053854FC0:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053855FD0:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053856FE0:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053857FF0:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: posix_memalign: >>> 0000000053859000:16384 @16 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385D010:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385E020:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385F030:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CD160:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CE170:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CF180:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D0190:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D11A0:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D21B0:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D31C0:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D41D0:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D51E0:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D61F0:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D7200:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D8210:4096 >>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D9220:4096 >>> >>> >>> Infact there are lot of such calls during a reload >>> >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA17ED00:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA1B0FF0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA1E12C0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA211590:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA243880:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA271B30:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA2A3E20:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA2D20D0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA3063E0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA334690:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA366980:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA396C50:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA3C8F40:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA3F9210:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA4294E0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA45B7D0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA489A80:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA4BBD70:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA4EA020:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA51E330:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA54C5E0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA57E8D0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA5AEBA0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA5DEE70:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA611160:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA641430:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA671700:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA6A29E0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA6D5CE0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA707FD0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA736280:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA768570:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA796820:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA7CAB30:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA7F8DE0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA82B0D0:16384 @16 >>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>> 00000000BA85B3A0:16384 @16 >>> >>> >>> >>> What is perplexing is that the system has enough free (available RAM) >>> ############# >>> # free -g >>> total used free shared buff/cache >>> available >>> Mem: 125 54 24 8 46 >>> 58 >>> Swap: 0 0 0 >>> ############# >>> >>> # ulimit -a >>> core file size (blocks, -c) 0 >>> data seg size (kbytes, -d) unlimited >>> scheduling priority (-e) 0 >>> file size (blocks, -f) unlimited >>> pending signals (-i) 514579 >>> max locked memory (kbytes, -l) 64 >>> max memory size (kbytes, -m) unlimited >>> open files (-n) 1024 >>> pipe size (512 bytes, -p) 8 >>> POSIX message queues (bytes, -q) 819200 >>> real-time priority (-r) 0 >>> stack size (kbytes, -s) 8192 >>> cpu time (seconds, -t) unlimited >>> max user processes (-u) 514579 >>> virtual memory (kbytes, -v) unlimited >>> file locks (-x) unlimited >>> >>> ######################################### >>> >>> There is no other thing limiting memory allocation >>> >>> Any way to prevent this or probably identify/prevent this >>> >>> >>> On Tue, Jul 31, 2018 at 7:08 PM Maxim Dounin wrote: >>> >>>> Hello! >>>> >>>> On Tue, Jul 31, 2018 at 09:52:29AM +0530, Anoop Alias wrote: >>>> >>>> > I am repeatedly seeing errors like >>>> > >>>> > ###################### >>>> > 2018/07/31 03:46:33 [emerg] 2854560#2854560: posix_memalign(16, 16384) >>>> > failed (12: Cannot allocate memory) >>>> > 2018/07/31 03:54:09 [emerg] 2890190#2890190: posix_memalign(16, 16384) >>>> > failed (12: Cannot allocate memory) >>>> > 2018/07/31 04:08:36 [emerg] 2939230#2939230: posix_memalign(16, 16384) >>>> > failed (12: Cannot allocate memory) >>>> > 2018/07/31 04:24:48 [emerg] 2992650#2992650: posix_memalign(16, 16384) >>>> > failed (12: Cannot allocate memory) >>>> > 2018/07/31 04:42:09 [emerg] 3053092#3053092: posix_memalign(16, 16384) >>>> > failed (12: Cannot allocate memory) >>>> > 2018/07/31 04:42:17 [emerg] 3053335#3053335: posix_memalign(16, 16384) >>>> > failed (12: Cannot allocate memory) >>>> > 2018/07/31 04:42:28 [emerg] 3053937#3053937: posix_memalign(16, 16384) >>>> > failed (12: Cannot allocate memory) >>>> > 2018/07/31 04:47:54 [emerg] 3070638#3070638: posix_memalign(16, 16384) >>>> > failed (12: Cannot allocate memory) >>>> > #################### >>>> > >>>> > on a few servers >>>> > >>>> > The servers have enough memory free and the swap usage is 0, yet >>>> somehow >>>> > the kernel denies the posix_memalign with ENOMEM ( this is what I >>>> think is >>>> > happening!) >>>> > >>>> > The numbers requested are always 16, 16k . This makes me suspicious >>>> > >>>> > I have no setting in nginx.conf that reference a 16k >>>> > >>>> > Is there any chance of finding out what requests this and why this is >>>> not >>>> > fulfilled >>>> >>>> There are at least some buffers which default to 16k - for >>>> example, ssl_buffer_size (http://nginx.org/r/ssl_buffer_size). >>>> >>>> You may try debugging log to futher find out where the particular >>>> allocation happens, see here for details: >>>> >>>> http://nginx.org/en/docs/debugging_log.html >>>> >>>> But I don't really think it worth the effort. The error is pretty >>>> clear, and it's better to focus on why these allocations are >>>> denied. Likely you are hitting some limit. >>>> >>>> -- >>>> Maxim Dounin >>>> http://mdounin.ru/ >>>> _______________________________________________ >>>> nginx mailing list >>>> nginx at nginx.org >>>> http://mailman.nginx.org/mailman/listinfo/nginx >>>> >>> >>> >>> -- >>> *Anoop P Alias* >>> >>> >>> >>> _______________________________________________ >>> nginx mailing listnginx at nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx >>> >>> >>> _______________________________________________ >>> nginx mailing list >>> nginx at nginx.org >>> http://mailman.nginx.org/mailman/listinfo/nginx >> >> >> >> -- >> *Anoop P Alias* >> >> >> >> _______________________________________________ >> nginx mailing listnginx at nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx >> >> >> _______________________________________________ >> nginx mailing list >> nginx at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx > > > > -- > *Anoop P Alias* > > > > _______________________________________________ > nginx mailing listnginx at nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx > > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx -- *Anoop P Alias* -------------- next part -------------- An HTML attachment was scrubbed... URL: From cameron.kerr at otago.ac.nz Tue Aug 7 02:45:02 2018 From: cameron.kerr at otago.ac.nz (Cameron Kerr) Date: Tue, 7 Aug 2018 02:45:02 +0000 Subject: limit_rate based on User-Agent; how to exempt /robots.txt ? Message-ID: <1fda9bac56e6491ba870cf8445e8e6a6@its-mail-p05.registry.otago.ac.nz> Hi all, I?ve recently deployed a rate-limiting configuration aimed at protecting myself from spiders. nginx version: nginx/1.15.1 (RPM from nginx.org) I did this based on the excellent Nginx blog post at https://www.nginx.com/blog/rate-limiting-nginx/ and have consulted the documentation for limit_req and limit_req_zone. I understand that you can have multiple zones in play, and that the most-restrictive of all matches will apply for any matching request. I want to go the other way though. I want to exempt /robots.txt from being rate limited by spiders. To put this in context, here is the gist of the relevant config, which aims to implement a caching (and rate-limiting) layer in front of a much more complex request routing layer (httpd). http { map $http_user_agent $user_agent_rate_key { default ""; "~our-crawler" "wanted-robot"; "~*(bot/|crawler|robot|spider)" "robot"; "~ScienceBrowser/Nutch" "robot"; "~Arachni/" "robot"; } limit_req_zone $user_agent_rate_key zone=per_spider_class:1m rate=100r/m; limit_req_status 429; server { limit_req zone=per_spider_class; location / { proxy_pass http://routing_layer_http/; } } } Option 1: (working, but has issues) Should I instead put the limit_req inside the "location / {}" stanza, and have a separate "location /robots.txt {}" (or some generalised form using a map) and not have limit_req inside that stanza That would mean that any other configuration inside the location stanzas would get duplicated, which would be a manageability concern. I just want to override the limit_req. server { location /robots.txt { proxy_pass http://routing_layer_http/; } location / { limit_req zone=per_spider_class; proxy_pass http://routing_layer_http/; } } I've tested this, and it works. Option 2: (working, but has issues) Should I create a "location /robots.txt {}" stanza that has a limit_req with a high burst, say burst=500? It's not a whitelist, but perhaps something still useful? But I still end up with replicated location stanzas... I don't think I like this approach. server { limit_req zone=per_spider_class; location /robots.txt { limit_req zone=per_spider_class burst=500; proxy_pass https://routing_layer_https/; } location / { proxy_pass https://routing_layer_https/; } } Option 3: (does not work) Some other way... perhaps I need to create some map that takes the path and produces a $path_exempt variable, and then somehow use that with the $user_agent_rate_key, returning "" when $path_exempt, or $user_agent_rate_key otherwise. map $http_user_agent $user_agent_rate_key { default ""; "~otago-crawler" "wanted-robot"; "~*(bot/|crawler|robot|spider)" "robot"; "~ScienceBrowser/Nutch" "robot"; "~Arachni/" "robot"; } map $uri $rate_for_spider_exempting { default $user_agent_rate_key; "/robots.txt" ""; } #limit_req_zone $user_agent_rate_key zone=per_spider_class:1m rate=100r/m; limit_req_zone $rate_for_spider_exempting zone=per_spider_class:1m rate=100r/m; However, this does not work because the second map is not returning $user_agent_rate_key; the effect is that non-robots are affected (and the load-balancer health-probes start getting rate-limited). I'm guessing my reasoning of how this works is incorrect, or there is a limitation or some sort of implicit ordering issue. Option 4: (does not work) http://nginx.org/en/docs/http/ngx_http_core_module.html#limit_rate I see that there is a variable $limit_rate that can be used, and this would seem to be the cleanest, except in testing it doesn't seem to work (still gets 429 responses as a User-Agent that is a bot) server { limit_req zone=per_spider_class; location /robots.txt { set $limit_rate 0; } location / { proxy_pass http://routing_layer_http/; } } I'm still fairly new with Nginx, so wanting something that decomposes cleanly into an Nginx configuration. I would quite like to be able just have one place where I specify the map of URLs I wish to exempt (I imagine there could be others, such as ~/.well-known/something that could pop up). Thank you very much for your time. -- Cameron Kerr Systems Engineer, Information Technology Services University of Otago From peter_booth at me.com Tue Aug 7 05:56:03 2018 From: peter_booth at me.com (Peter Booth) Date: Tue, 07 Aug 2018 01:56:03 -0400 Subject: limit_rate based on User-Agent; how to exempt /robots.txt ? In-Reply-To: <1fda9bac56e6491ba870cf8445e8e6a6@its-mail-p05.registry.otago.ac.nz> References: <1fda9bac56e6491ba870cf8445e8e6a6@its-mail-p05.registry.otago.ac.nz> Message-ID: <885FAE40-7AC6-498E-AB8D-DE4ADCE6F2E0@me.com> So it?s very easy to get caught up in he trap if having unrealistic mental models of how we servers work when dealing with web servers. If your host is a recent (< 5 years) single Dickey host then you can probably support 300,000 requests per second fir your robots.txt file. That?s because the file will be served from your Linux file ca he (memory) Sent from my iPhone > On Aug 6, 2018, at 10:45 PM, Cameron Kerr wrote: > > Hi all, I?ve recently deployed a rate-limiting configuration aimed at protecting myself from spiders. > > nginx version: nginx/1.15.1 (RPM from nginx.org) > > I did this based on the excellent Nginx blog post at https://www.nginx.com/blog/rate-limiting-nginx/ and have consulted the documentation for limit_req and limit_req_zone. > > I understand that you can have multiple zones in play, and that the most-restrictive of all matches will apply for any matching request. I want to go the other way though. I want to exempt /robots.txt from being rate limited by spiders. > > To put this in context, here is the gist of the relevant config, which aims to implement a caching (and rate-limiting) layer in front of a much more complex request routing layer (httpd). > > http { > map $http_user_agent $user_agent_rate_key { > default ""; > "~our-crawler" "wanted-robot"; > "~*(bot/|crawler|robot|spider)" "robot"; > "~ScienceBrowser/Nutch" "robot"; > "~Arachni/" "robot"; > } > > limit_req_zone $user_agent_rate_key zone=per_spider_class:1m rate=100r/m; > limit_req_status 429; > > server { > limit_req zone=per_spider_class; > > location / { > proxy_pass http://routing_layer_http/; > } > } > } > > > > Option 1: (working, but has issues) > > Should I instead put the limit_req inside the "location / {}" stanza, and have a separate "location /robots.txt {}" (or some generalised form using a map) and not have limit_req inside that stanza > > That would mean that any other configuration inside the location stanzas would get duplicated, which would be a manageability concern. I just want to override the limit_req. > > server { > location /robots.txt { > proxy_pass http://routing_layer_http/; > } > > location / { > limit_req zone=per_spider_class; > proxy_pass http://routing_layer_http/; > } > } > > I've tested this, and it works. > > > Option 2: (working, but has issues) > > Should I create a "location /robots.txt {}" stanza that has a limit_req with a high burst, say burst=500? It's not a whitelist, but perhaps something still useful? > > But I still end up with replicated location stanzas... I don't think I like this approach. > > server { > limit_req zone=per_spider_class; > > location /robots.txt { > limit_req zone=per_spider_class burst=500; > proxy_pass https://routing_layer_https/; > } > > location / { > proxy_pass https://routing_layer_https/; > } > } > > > Option 3: (does not work) > > Some other way... perhaps I need to create some map that takes the path and produces a $path_exempt variable, and then somehow use that with the $user_agent_rate_key, returning "" when $path_exempt, or $user_agent_rate_key otherwise. > > map $http_user_agent $user_agent_rate_key { > default ""; > "~otago-crawler" "wanted-robot"; > "~*(bot/|crawler|robot|spider)" "robot"; > "~ScienceBrowser/Nutch" "robot"; > "~Arachni/" "robot"; > } > > map $uri $rate_for_spider_exempting { > default $user_agent_rate_key; > "/robots.txt" ""; > } > > #limit_req_zone $user_agent_rate_key zone=per_spider_class:1m rate=100r/m; > limit_req_zone $rate_for_spider_exempting zone=per_spider_class:1m rate=100r/m; > > > However, this does not work because the second map is not returning $user_agent_rate_key; the effect is that non-robots are affected (and the load-balancer health-probes start getting rate-limited). > > I'm guessing my reasoning of how this works is incorrect, or there is a limitation or some sort of implicit ordering issue. > > > Option 4: (does not work) > > http://nginx.org/en/docs/http/ngx_http_core_module.html#limit_rate > > I see that there is a variable $limit_rate that can be used, and this would seem to be the cleanest, except in testing it doesn't seem to work (still gets 429 responses as a User-Agent that is a bot) > > server { > limit_req zone=per_spider_class; > > location /robots.txt { > set $limit_rate 0; > } > > location / { > proxy_pass http://routing_layer_http/; > } > } > > > I'm still fairly new with Nginx, so wanting something that decomposes cleanly into an Nginx configuration. I would quite like to be able just have one place where I specify the map of URLs I wish to exempt (I imagine there could be others, such as ~/.well-known/something that could pop up). > > Thank you very much for your time. > > -- > Cameron Kerr > Systems Engineer, Information Technology Services > University of Otago > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx From r at roze.lv Tue Aug 7 11:28:49 2018 From: r at roze.lv (Reinis Rozitis) Date: Tue, 7 Aug 2018 14:28:49 +0300 Subject: nginx, php7.0-fpm and laravel, not able to set it up when the url has a prefix which the server doesn't have In-Reply-To: References: <002801d42d5b$c0a9bbd0$41fd3370$@roze.lv> Message-ID: <00a301d42e41$d0408bf0$70c1a3d0$@roze.lv> > Hello, > Thanks for that input, I changed my config like you said and also set the fastcgi_param SCRIPT_FILENAME to /path/to/the/index.php; > But I feel like this should be done by the rewrite directive instead of this way. Now that this is working, can someone please tell me if I can use the rewrite tag so that I can avoid > hardcoding the path everywhere? Well you didn't specify the actual rewrite. Also from the initial mail the actual setup is a bit unclear - if the docker is only running php-fpm (and the nginx is on host) or both nginx and php are inside docker, but I assume it's the first. As an idea - while you can mangle/rewrite the root/path in the nginx config, when configuring wordpress (+php-fpm) to run inside a docker container I found more easy just to set the container volume to have the same path (mount point) as the host server. So while isolated, both (nginx on host and php inside container) have identical directory tree. rr From iippolitov at nginx.com Tue Aug 7 11:54:13 2018 From: iippolitov at nginx.com (Igor A. Ippolitov) Date: Tue, 7 Aug 2018 14:54:13 +0300 Subject: posix_memalign error In-Reply-To: References: <20180731133813.GG56558@mdounin.ru> <4fa47f31-9c32-5386-7c2d-a4e9347b3dfc@nginx.com> <1fdf4937-cac6-0550-b725-36f04eff65db@nginx.com> Message-ID: Anoop, I don't see any troubles with your configuration. Also, if you have 120G of RAM and a single worker - the problem is not in nginx. Do you have other software running on the host? Basically, you just run out of memory. You can optimize your reload though: use "service nginx reload" (or "kill -SIGHUP") to reload nginx configuration. When you do nginx -s reload - you make nginx parse configuration (and it requires memory) and then send a signal to the running master. You can avoid this overhead with 'service' command as it uses 'kill' documented in the manual page. On 06.08.2018 22:55, Anoop Alias wrote: > Hi Igor, > > Config is reloaded using > > /usr/sbin/nginx -s reload > > this is invoked from a python/shell script ( Nginx is installed on a > web control panel ) > > The top-level?Nginx config is in the gist below > > https://gist.github.com/AnoopAlias/ba5ad6749a586c7e267672ee65b32b3a > > It further includes ~8k server blocks or more in some servers. Out of > this 2/3 are server?{} blocks with TLS config and 1/3 non-TLS ones > > ]# pwd > /etc/nginx/sites-enabled > # grep "server {" *|wc -l > 7886 > > And yes most of them are very similar and mostly proxy to upstream httpd > > I have tried removing all the loadable modules and even tried an older > version of nginx?and all produce the error > > > # numastat -m > > Per-node system memory usage (in MBs): > ? ? ? ? ? ? ? ? ? ? ? ? ? Node 0? ? ? ? ? Node 1 ?Total > ? ? ? ? ? ? ? ? ?--------------- --------------- --------------- > MemTotal? ? ? ? ? ? ? ? 65430.84? ? ? ? 65536.00 ?130966.84 > MemFree? ? ? ? ? ? ? ? ? 5491.26? ? ? ? ? ?40.89 ?5532.15 > MemUsed? ? ? ? ? ? ? ? ?59939.58? ? ? ? 65495.11 ?125434.69 > Active? ? ? ? ? ? ? ? ? 22295.61? ? ? ? 21016.09 43311.70 > Inactive? ? ? ? ? ? ? ? ?8742.76? ? ? ? ?4662.48 13405.24 > Active(anon)? ? ? ? ? ? 16717.10? ? ? ? 16572.19 33289.29 > Inactive(anon)? ? ? ? ? ?2931.94? ? ? ? ?1388.14 ?4320.08 > Active(file)? ? ? ? ? ? ?5578.50? ? ? ? ?4443.91 10022.41 > Inactive(file)? ? ? ? ? ?5810.82? ? ? ? ?3274.34 ?9085.16 > Unevictable? ? ? ? ? ? ? ? ?0.00? ? ? ? ? ? 0.00 ? 0.00 > Mlocked? ? ? ? ? ? ? ? ? ? ?0.00? ? ? ? ? ? 0.00 ? 0.00 > Dirty? ? ? ? ? ? ? ? ? ? ? ?7.04? ? ? ? ? ? 1.64 ? 8.67 > Writeback? ? ? ? ? ? ? ? ? ?0.00? ? ? ? ? ? 0.00 ? 0.00 > FilePages? ? ? ? ? ? ? ?18458.93? ? ? ? 10413.97 28872.90 > Mapped? ? ? ? ? ? ? ? ? ? 862.14? ? ? ? ? 413.38 ?1275.52 > AnonPages? ? ? ? ? ? ? ?12579.49? ? ? ? 15264.37 27843.86 > Shmem? ? ? ? ? ? ? ? ? ? 7069.52? ? ? ? ?2695.71 ?9765.23 > KernelStack? ? ? ? ? ? ? ? 18.34? ? ? ? ? ? 3.03 ?21.38 > PageTables? ? ? ? ? ? ? ? 153.14? ? ? ? ? 107.77 260.90 > NFS_Unstable? ? ? ? ? ? ? ? 0.00? ? ? ? ? ? 0.00 ? 0.00 > Bounce? ? ? ? ? ? ? ? ? ? ? 0.00? ? ? ? ? ? 0.00 ? 0.00 > WritebackTmp? ? ? ? ? ? ? ? 0.00? ? ? ? ? ? 0.00 ? 0.00 > Slab? ? ? ? ? ? ? ? ? ? ?4830.68? ? ? ? ?2254.55 ?7085.22 > SReclaimable? ? ? ? ? ? ?2061.05? ? ? ? ? 921.72 ?2982.77 > SUnreclaim? ? ? ? ? ? ? ?2769.62? ? ? ? ?1332.83 ?4102.45 > AnonHugePages? ? ? ? ? ? ? ?4.00? ? ? ? ? ? 2.00 ? 6.00 > HugePages_Total? ? ? ? ? ? ?0.00? ? ? ? ? ? 0.00 ? 0.00 > HugePages_Free? ? ? ? ? ? ? 0.00? ? ? ? ? ? 0.00 ? 0.00 > HugePages_Surp? ? ? ? ? ? ? 0.00? ? ? ? ? ? 0.00 ? 0.00 > > > Thanks, > > > > > > On Mon, Aug 6, 2018 at 6:33 PM Igor A. Ippolitov > wrote: > > Anoop, > > I suppose, most of your 10k servers are very similar, right? > Please, post top level configuration and a typical server{}, please. > > Also, how do you reload configuration? With 'service nginx reload' > or may be other commands? > > It looks like you have a lot of fragmented memory and only 4gb > free in the second numa node. > So, I'd say this is OK that you are getting errors from allocating > a 16k stripes. > > Could you please post numastat -m output additionally. Just to > make sure you have half of the memory for the second CPU. > And we'll have a look if memory utilization may be optimized based > on your configuration. > > Regards, > Igor. > > On 04.08.2018 07:54, Anoop Alias wrote: >> Hi Igor, >> >> Setting?vm.max_map_count to 20x the normal value did not help >> >> The issue happens on a group of servers and among the group, it >> shows up only in servers which have ~10k? server{} blocks >> >> On servers that have lower number of server{} blocks , the ENOMEM >> issue is not there >> >> Also, I can find that the RAM usage of the Nginx process is >> directly proportional to the number of server {} blocks >> >> For example on a server having the problem >> >> # ps_mem| head -1 && ps_mem |grep nginx >> ?Private? +? ?Shared? =? RAM used? ? ? ?Program >> ? 1.0 GiB +? ?2.8 GiB =? ?3.8 GiB? ? ? ?nginx (3) >> >> >> That is for a single worker process with 4 threads in thread_pool >> # pstree|grep nginx >> ? ? ? ? |-nginx-+-nginx---4*[{nginx}] >> ? ? ? ? |? ? ? ?`-nginx >> >> Whatever config change I try the memory usage seem to mostly >> depend on the number of server contexts defined >> >> Now the issue mostly happen in nginx reload ,when one more worker >> process will be active in shutting down mode >> >> I believe the memalign error is thrown by the worker being >> shutdown, this is because the sites work after the error and also >> the pid mentioned in the error would have gone when I check ps >> >> >> # pmap 948965|grep 16K >> 00007f2923ff2000? ? ?16K r-x-- ngx_http_redis2_module.so >> 00007f2924fd7000? ? ?16K r---- libc-2.17.so >> 00007f2925431000? ? ?16K rw---? ?[ anon ] >> 00007f292584a000? ? ?16K rw---? ?[ anon ] >> >> Aug? 4 05:50:00 b kernel: SysRq : Show Memory >> Aug? 4 05:50:00 b kernel: Mem-Info: >> Aug? 4 05:50:00 b kernel: active_anon:7757394 >> inactive_anon:1021319 isolated_anon:0#012 active_file:3733324 >> inactive_file:2136476 isolated_file:0#012 unevictable:0 >> dirty:1766 writeback:6 wbtmp:0 unstable:0#012 >> slab_reclaimable:2003687 slab_unreclaimable:901391#012 >> mapped:316734 shmem:2381810 pagetables:63163 bounce:0#012 >> free:4851283 free_pcp:11332 free_cma:0 >> Aug? 4 05:50:00 bravo kernel: Node 0 DMA free:15888kB min:8kB >> low:8kB high:12kB active_anon:0kB inactive_anon:0kB >> active_file:0kB inactive_file:0kB unevictable:0kB >> isolated(anon):0kB isolated(file):0kB present:15972kB >> managed:15888kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB >> shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB >> kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB >> free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB >> pages_scanned:0 all_unreclaimable? yes >> Aug? 4 05:50:00 b kernel: lowmem_reserve[]: 0 1679 64139 64139 >> >> # cat /proc/buddyinfo >> Node 0, zone? ? ? DMA? ? ? 0? ? ? 0? ? ? 1 0? ? ? 2? ? ? 1? ? ? >> 1? ? ? 0? ? ? 1? ? ? 1? ? ? 3 >> Node 0, zone? ? DMA32? ?5284? ?6753? ?6677 ?1083? ? 410? ? ?59? ? >> ? 1? ? ? 0? ? ? 0? ? ? 0 0 >> Node 0, zone? ?Normal 500327 638958 406737 14690? ? 872? ? 106? ? >> ?11? ? ? 0? ? ? 0? ? ? 0 0 >> Node 1, zone? ?Normal 584840 291640? ? 188 0? ? ? 0? ? ? 0? ? ? >> 0? ? ? 0? ? ? 0? ? ? 0? ? ? 0 >> >> >> The only correlation I see in having the error is the number of >> server?{} blocks (close to 10k) which then makes the >> nginx?process consume ~ 4GB of mem with a single worker process >> and then a reload is done >> >> >> >> >> On Thu, Aug 2, 2018 at 6:02 PM Igor A. Ippolitov >> > wrote: >> >> Anoop, >> >> There are two guesses: either mmap allocations limit is hit >> or memory is? way too fragmented. >> Could you please track amount of mapped regions for a worker >> with pmap and amount of 16k areas in Normal zones (it is the >> third number)? >> >> You can also set vm.max_map_count to a higher number (like 20 >> times higher than default) and look if the error is gone. >> >> Please, let me know if increasing vm.max_map_count helps you. >> >> On 02.08.2018 13:06, Anoop Alias wrote: >>> Hi Igor, >>> >>> The error happens randomly >>> >>> 2018/08/02 06:52:42 [emerg] 874514#874514: >>> posix_memalign(16, 16384) failed (12: Cannot allocate memory) >>> 2018/08/02 09:42:53 [emerg] 872996#872996: >>> posix_memalign(16, 16384) failed (12: Cannot allocate memory) >>> 2018/08/02 10:16:14 [emerg] 877611#877611: >>> posix_memalign(16, 16384) failed (12: Cannot allocate memory) >>> 2018/08/02 10:16:48 [emerg] 879410#879410: >>> posix_memalign(16, 16384) failed (12: Cannot allocate memory) >>> 2018/08/02 10:17:55 [emerg] 876563#876563: >>> posix_memalign(16, 16384) failed (12: Cannot allocate memory) >>> 2018/08/02 10:20:21 [emerg] 879263#879263: >>> posix_memalign(16, 16384) failed (12: Cannot allocate memory) >>> 2018/08/02 10:20:51 [emerg] 878991#878991: >>> posix_memalign(16, 16384) failed (12: Cannot allocate memory) >>> >>> # date >>> Thu Aug? 2 10:58:48 BST 2018 >>> >>> ------------------------------------------ >>> # cat /proc/buddyinfo >>> Node 0, zone? ? ? DMA? ? ? 0? ? ? 0 ? 1? ? ? 0? ? ? 2? ? ? >>> 1? ? ? 1? ? ? 0 ? 1? ? ? 1? ? ? 3 >>> Node 0, zone? ? DMA32? 11722? 11057 ?4663? ?1647? ? 609? ? >>> ?72? ? ?10? ? ? 7 ? ? 1? ? ? 0? ? ? 0 >>> Node 0, zone? ?Normal 755026 710760 398136? 21462? ?1114? ? >>> ?18? ? ? 1? ? ? 0 ? ? 0? ? ? 0? ? ? 0 >>> Node 1, zone? ?Normal 341295 801810 179604? ? 256? ? ? 0? ? >>> ? 0? ? ? 0? ? ? 0 ? ? 0? ? ? 0? ? ? 0 >>> ----------------------------------------- >>> >>> >>> slabinfo - version: 2.1 >>> # name? ? ? ? ? ? >>> : tunables >>> : slabdata >>> >>> SCTPv6? ? ? ? ? ? ? ? 21? ? ?21? ?1536 ?21? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 1? ? ? 1? ? ? 0 >>> SCTP? ? ? ? ? ? ? ? ? ?0? ? ? 0? ?1408 ?23? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> kcopyd_job? ? ? ? ? ? ?0? ? ? 0? ?3312 ? 9? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> dm_uevent? ? ? ? ? ? ? 0? ? ? 0? ?2608 ?12? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> nf_conntrack_ffffffff81acbb00? 14054 14892? ? 320? ?51? ? 4 >>> : tunables? ? 0 0? ? 0 : slabdata? ? 292? ? 292? ? ? 0 >>> lvp_cache? ? ? ? ? ? ?36? ? ?36? ? 224 ?36? ? 2 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 1? ? ? 1? ? ? 0 >>> lve_struct? ? ? ? ? 4140? ?4140? ? 352 ?46? ? 4 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ?90? ? ?90? ? ? 0 >>> fat_inode_cache? ? ? ? 0? ? ? 0? ? 744 ?44? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> fat_cache? ? ? ? ? ? ? 0? ? ? 0? ? ?40 102? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> isofs_inode_cache? ? ? 0? ? ? 0? ? 664 ?49? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> ext4_inode_cache? ? ? 30? ? ?30? ?1088 ?30? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 1? ? ? 1? ? ? 0 >>> ext4_xattr? ? ? ? ? ? ?0? ? ? 0? ? ?88 ?46? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> ext4_free_data? ? ? ? ?0? ? ? 0? ? ?64 ?64? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> ext4_allocation_context? ? ?32? ? ?32 ? 128? ?32? ? 1 : >>> tunables? ? 0? ? 0? ? 0 : slabdata? ? ? 1? ? ? 1? ? ? 0 >>> ext4_io_end? ? ? ? ? ? 0? ? ? 0? ? ?72 ?56? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> ext4_extent_status? ? 102? ? 102 ?40? 102? ? 1 : tunables? ? >>> 0? ? 0? ? 0 : slabdata? ? ? 1? ? ? 1? ? ? 0 >>> jbd2_journal_handle? ? ? 0? ? ? 0 ?48? ?85? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> jbd2_journal_head? ? ? 0? ? ? 0? ? 112 ?36? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> jbd2_revoke_table_s? ? 256? ? 256 ?16? 256? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 1? ? ? 1? ? ? 0 >>> jbd2_revoke_record_s? ? ? 0? ? ? 0 ?32? 128? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> kvm_async_pf? ? ? ? ? ?0? ? ? 0? ? 136 ?30? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> kvm_vcpu? ? ? ? ? ? ? ?0? ? ? 0? 18560 ? 1? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> xfs_dqtrx? ? ? ? ? ? 992? ? 992? ? 528 ?31? ? 4 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ?32? ? ?32? ? ? 0 >>> xfs_dquot? ? ? ? ? ?3264? ?3264? ? 472 ?34? ? 4 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ?96? ? ?96? ? ? 0 >>> xfs_ili? ? ? ? ? ?4342175 4774399 152? ?53? ? 2 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? 90083? 90083? ? ? 0 >>> xfs_inode? ? ? ? ?4915588 5486076 ?1088? ?30? ? 8 : >>> tunables? ? 0? ? 0? ? 0 : slabdata 182871 182871? ? ? 0 >>> xfs_efd_item? ? ? ? 2680? ?2760? ? 400 ?40? ? 4 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ?69? ? ?69? ? ? 0 >>> xfs_da_state? ? ? ? 1088? ?1088? ? 480 ?34? ? 4 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ?32? ? ?32? ? ? 0 >>> xfs_btree_cur? ? ? ?1248? ?1248? ? 208 ?39? ? 2 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ?32? ? ?32? ? ? 0 >>> xfs_log_ticket? ? ?14874? 15048? ? 184 ?44? ? 2 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 342? ? 342? ? ? 0 >>> xfs_ioend? ? ? ? ? 12909? 13104? ? 104 ?39? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 336? ? 336? ? ? 0 >>> scsi_cmd_cache? ? ? 5400? ?5652? ? 448 ?36? ? 4 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 157? ? 157? ? ? 0 >>> ve_struct? ? ? ? ? ? ? 0? ? ? 0? ? 848 ?38? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> ip6_dst_cache? ? ? ?1152? ?1152? ? 448 ?36? ? 4 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ?32? ? ?32? ? ? 0 >>> RAWv6? ? ? ? ? ? ? ? 910? ? 910? ?1216 ?26? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ?35? ? ?35? ? ? 0 >>> UDPLITEv6? ? ? ? ? ? ? 0? ? ? 0? ?1216 ?26? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> UDPv6? ? ? ? ? ? ? ? 832? ? 832? ?1216 ?26? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ?32? ? ?32? ? ? 0 >>> tw_sock_TCPv6? ? ? ?1152? ?1376? ? 256 ?32? ? 2 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ?43? ? ?43? ? ? 0 >>> TCPv6? ? ? ? ? ? ? ? 510? ? 510? ?2176 ?15? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ?34? ? ?34? ? ? 0 >>> cfq_queue? ? ? ? ? ?3698? ?5145? ? 232 ?35? ? 2 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 147? ? 147? ? ? 0 >>> bsg_cmd? ? ? ? ? ? ? ? 0? ? ? 0? ? 312 ?52? ? 4 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> mqueue_inode_cache? ? 136? ? 136 960? ?34? ? 8 : tunables? ? >>> 0? ? 0? ? 0 : slabdata? ? ? 4? ? ? 4? ? ? 0 >>> hugetlbfs_inode_cache? ?1632? ?1632 632? ?51? ? 8 : >>> tunables? ? 0? ? 0? ? 0 : slabdata? ? ?32? ? ?32? ? ? 0 >>> configfs_dir_cache? ?1472? ?1472 ?88? ?46? ? 1 : tunables? ? >>> 0? ? 0? ? 0 : slabdata? ? ?32? ? ?32? ? ? 0 >>> dquot? ? ? ? ? ? ? ? ? 0? ? ? 0? ? 256 ?32? ? 2 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> userfaultfd_ctx_cache? ? ?32? ? ?32 128? ?32? ? 1 : >>> tunables? ? 0? ? 0? ? 0 : slabdata? ? ? 1? ? ? 1? ? ? 0 >>> fanotify_event_info? ?2336? ?2336 ?56? ?73? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ?32? ? ?32? ? ? 0 >>> dio? ? ? ? ? ? ? ? ?6171? ?6222? ? 640 ?51? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 122? ? 122? ? ? 0 >>> pid_namespace? ? ? ? ?42? ? ?42? ?2192 ?14? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 3? ? ? 3? ? ? 0 >>> posix_timers_cache? ?1056? ?1056 248? ?33? ? 2 : tunables? ? >>> 0? ? 0? ? 0 : slabdata? ? ?32? ? ?32? ? ? 0 >>> UDP-Lite? ? ? ? ? ? ? ?0? ? ? 0? ?1088 ?30? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> flow_cache? ? ? ? ? 2268? ?2296? ? 144 ?28? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ?82? ? ?82? ? ? 0 >>> xfrm_dst_cache? ? ? ?896? ? 896? ? 576 ?28? ? 4 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ?32? ? ?32? ? ? 0 >>> ip_fib_alias? ? ? ? 2720? ?2720? ? ?48 ?85? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ?32? ? ?32? ? ? 0 >>> RAW? ? ? ? ? ? ? ? ?3977? ?4224? ?1024 ?32? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 132? ? 132? ? ? 0 >>> UDP? ? ? ? ? ? ? ? ?4110? ?4110? ?1088 ?30? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 137? ? 137? ? ? 0 >>> tw_sock_TCP? ? ? ? ?4756? ?5216? ? 256 ?32? ? 2 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 163? ? 163? ? ? 0 >>> TCP? ? ? ? ? ? ? ? ?2705? ?2768? ?1984 ?16? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 173? ? 173? ? ? 0 >>> scsi_data_buffer? ? 5440? ?5440? ? ?24 170? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ?32? ? ?32? ? ? 0 >>> blkdev_queue? ? ? ? ?154? ? 154? ?2208 ?14? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ?11? ? ?11? ? ? 0 >>> blkdev_requests? ?4397688 4405884 384? ?42? ? 4 : tunables? >>> ? 0? ? 0? ? 0 : slabdata 104902 104902? ? ? 0 >>> blkdev_ioc? ? ? ? ?11232? 11232? ? 112 ?36? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 312? ? 312? ? ? 0 >>> user_namespace? ? ? ? ?0? ? ? 0? ?1304 ?25? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> sock_inode_cache? ?12282? 12282? ? 704 ?46? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 267? ? 267? ? ? 0 >>> file_lock_cache? ? 20056? 20960? ? 200 ?40? ? 2 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 524? ? 524? ? ? 0 >>> net_namespace? ? ? ? ? 6? ? ? 6? ?5056 ? 6? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 1? ? ? 1? ? ? 0 >>> shmem_inode_cache? 16970? 18952? ? 712 ?46? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 412? ? 412? ? ? 0 >>> Acpi-ParseExt? ? ? 39491? 40432? ? ?72 ?56? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 722? ? 722? ? ? 0 >>> Acpi-State? ? ? ? ? 1683? ?1683? ? ?80 ?51? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ?33? ? ?33? ? ? 0 >>> Acpi-Namespace? ? ?11424? 11424? ? ?40 102? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 112? ? 112? ? ? 0 >>> task_delay_info? ? 15336? 15336? ? 112 ?36? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 426? ? 426? ? ? 0 >>> taskstats? ? ? ? ? ?1568? ?1568? ? 328 ?49? ? 4 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ?32? ? ?32? ? ? 0 >>> proc_inode_cache? 169897 190608? ? 680 ?48? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ?3971? ?3971? ? ? 0 >>> sigqueue? ? ? ? ? ? 2208? ?2208? ? 168 ?48? ? 2 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ?46? ? ?46? ? ? 0 >>> bdev_cache? ? ? ? ? ?792? ? 792? ? 896 ?36? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ?22? ? ?22? ? ? 0 >>> sysfs_dir_cache? ? 74698? 74698? ? 120 ?34? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ?2197? ?2197? ? ? 0 >>> mnt_cache? ? ? ? ?163197 163424? ? 256 ?32? ? 2 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ?5107? ?5107? ? ? 0 >>> filp? ? ? ? ? ? ? ?64607? 97257? ? 320 ?51? ? 4 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ?1907? ?1907? ? ? 0 >>> inode_cache? ? ? ?370744 370947? ? 616 ?53? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ?6999? ?6999? ? ? 0 >>> dentry? ? ? ? ? ? 1316262 2139228 192? ?42? ? 2 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? 50934? 50934? ? ? 0 >>> iint_cache? ? ? ? ? ? ?0? ? ? 0? ? ?80 ?51? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> buffer_head? ? ? ?1441470 2890290 104? ?39? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? 74110? 74110? ? ? 0 >>> vm_area_struct? ? 194998 196840? ? 216 ?37? ? 2 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ?5320? ?5320? ? ? 0 >>> mm_struct? ? ? ? ? ?2679? ?2760? ?1600 ?20? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 138? ? 138? ? ? 0 >>> files_cache? ? ? ? ?8680? ?8925? ? 640 ?51? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 175? ? 175? ? ? 0 >>> signal_cache? ? ? ? 3691? ?3780? ?1152 ?28? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 135? ? 135? ? ? 0 >>> sighand_cache? ? ? ?1950? ?2160? ?2112 ?15? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 144? ? 144? ? ? 0 >>> task_xstate? ? ? ? ?8070? ?8658? ? 832 ?39? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 222? ? 222? ? ? 0 >>> task_struct? ? ? ? ?1913? ?2088? ?4080 ? 8? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 261? ? 261? ? ? 0 >>> cred_jar? ? ? ? ? ?31699? 33936? ? 192 ?42? ? 2 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 808? ? 808? ? ? 0 >>> anon_vma_chain? ? 164026 168704? ? ?64 ?64? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ?2636? ?2636? ? ? 0 >>> anon_vma? ? ? ? ? ?84104? 84594? ? ?88 ?46? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ?1839? ?1839? ? ? 0 >>> pid? ? ? ? ? ? ? ? 11127? 12576? ? 128 ?32? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 393? ? 393? ? ? 0 >>> shared_policy_node? ?9350? ?9350 ?48? ?85? ? 1 : tunables? ? >>> 0? ? 0? ? 0 : slabdata? ? 110? ? 110? ? ? 0 >>> numa_policy? ? ? ? ? ?62? ? ?62? ? 264 ?31? ? 2 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 2? ? ? 2? ? ? 0 >>> radix_tree_node? ?771778 1194312 584? ?28? ? 4 : tunables? ? >>> 0? ? 0? ? 0 : slabdata? 42654? 42654? ? ? 0 >>> idr_layer_cache? ? ?2538? ?2565? ?2112 ?15? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 171? ? 171? ? ? 0 >>> dma-kmalloc-8192? ? ? ?0? ? ? 0? ?8192 ? 4? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> dma-kmalloc-4096? ? ? ?0? ? ? 0? ?4096 ? 8? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> dma-kmalloc-2048? ? ? ?0? ? ? 0? ?2048 ?16? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> dma-kmalloc-1024? ? ? ?0? ? ? 0? ?1024 ?32? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> dma-kmalloc-512? ? ? ? 0? ? ? 0? ? 512 ?32? ? 4 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> dma-kmalloc-256? ? ? ? 0? ? ? 0? ? 256 ?32? ? 2 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> dma-kmalloc-128? ? ? ? 0? ? ? 0? ? 128 ?32? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> dma-kmalloc-64? ? ? ? ?0? ? ? 0? ? ?64 ?64? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> dma-kmalloc-32? ? ? ? ?0? ? ? 0? ? ?32 128? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> dma-kmalloc-16? ? ? ? ?0? ? ? 0? ? ?16 256? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> dma-kmalloc-8? ? ? ? ? 0? ? ? 0? ? ? 8 512? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> dma-kmalloc-192? ? ? ? 0? ? ? 0? ? 192 ?42? ? 2 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> dma-kmalloc-96? ? ? ? ?0? ? ? 0? ? ?96 ?42? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ? 0? ? ? 0? ? ? 0 >>> kmalloc-8192? ? ? ? ?385? ? 388? ?8192 ? 4? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ?97? ? ?97? ? ? 0 >>> kmalloc-4096? ? ? ? 9296? 10088? ?4096 ? 8? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ?1261? ?1261? ? ? 0 >>> kmalloc-2048? ? ? ?65061 133536? ?2048 ?16? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ?8346? ?8346? ? ? 0 >>> kmalloc-1024? ? ? ?11987? 21120? ?1024 ?32? ? 8 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 660? ? 660? ? ? 0 >>> kmalloc-512? ? ? ?107510 187072? ? 512 ?32? ? 4 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ?5846? ?5846? ? ? 0 >>> kmalloc-256? ? ? ?160498 199104? ? 256 ?32? ? 2 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ?6222? ?6222? ? ? 0 >>> kmalloc-192? ? ? ?144975 237426? ? 192 ?42? ? 2 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ?5653? ?5653? ? ? 0 >>> kmalloc-128? ? ? ? 36799 108096? ? 128 ?32? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ?3378? ?3378? ? ? 0 >>> kmalloc-96? ? ? ? ?99510 238896? ? ?96 ?42? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ?5688? ?5688? ? ? 0 >>> kmalloc-64? ? ? ? 7978152 8593280 ?64? ?64? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata 134270 134270? ? ? 0 >>> kmalloc-32? ? ? ? 2939882 3089664 ?32? 128? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? 24138? 24138? ? ? 0 >>> kmalloc-16? ? ? ? 172057 172288? ? ?16 256? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 673? ? 673? ? ? 0 >>> kmalloc-8? ? ? ? ?109568 109568? ? ? 8 512? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? 214? ? 214? ? ? 0 >>> kmem_cache_node? ? ? 893? ? 896? ? ?64 ?64? ? 1 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ?14? ? ?14? ? ? 0 >>> kmem_cache? ? ? ? ? ?612? ? 612? ? 320 ?51? ? 4 : tunables? >>> ? 0? ? 0? ? 0 : slabdata? ? ?12? ? ?12? ? ? 0 >>> >>> ------------------------------------------------- >>> >>> >>> # uname -r >>> 3.10.0-714.10.2.lve1.5.17.1.el7.x86_64 >>> >>> -------------------------------------------------------- >>> >>> Core part of glances >>> http://i.imgur.com/La5JbQn.png >>> ----------------------------------------------------------- >>> >>> Thank you very much for looking into this >>> >>> >>> On Thu, Aug 2, 2018 at 12:37 PM Igor A. Ippolitov >>> > wrote: >>> >>> Anoop, >>> >>> I doubt this will be the solution, but may we have a >>> look at /proc/buddyinfo and /proc/slabinfo the moment >>> when nginx can't allocate memory? >>> >>> On 02.08.2018 08:15, Anoop Alias wrote: >>>> Hi Maxim, >>>> >>>> I enabled debug and the memalign call is happening on >>>> nginx reloads and the ENOMEM happen sometimes on the >>>> reload(not on all reloads) >>>> >>>> 2018/08/02 05:59:08 [notice] 872052#872052: signal >>>> process started >>>> 2018/08/02 05:59:23 [notice] 871570#871570: signal 1 >>>> (SIGHUP) received from 872052, reconfiguring >>>> 2018/08/02 05:59:23 [debug] 871570#871570: wake up, sigio 0 >>>> 2018/08/02 05:59:23 [notice] 871570#871570: reconfiguring >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 0000000002B0DA00:16384 @16? ? ? === > >>>> the memalign call on reload >>>> 2018/08/02 05:59:23 [debug] 871570#871570: malloc: >>>> 00000000087924D0:4560 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 000000000E442E00:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: malloc: >>>> 0000000005650850:4096 >>>> 20 >>>> >>>> >>>> >>>> >>>> 2018/08/02 05:48:49 [debug] 871275#871275: bind() >>>> xxxx:443 #71 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: bind() >>>> xxxx:443 #72 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: bind() >>>> xxxx:443 #73 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: bind() >>>> xxxx:443 #74 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: add cleanup: >>>> 000000005340D728 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>> 00000000024D3260:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>> 00000000517BAF10:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>> 0000000053854FC0:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>> 0000000053855FD0:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>> 0000000053856FE0:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>> 0000000053857FF0:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: >>>> posix_memalign: 0000000053859000:16384 @16 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>> 000000005385D010:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>> 000000005385E020:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>> 000000005385F030:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>> 00000000536CD160:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>> 00000000536CE170:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>> 00000000536CF180:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>> 00000000536D0190:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>> 00000000536D11A0:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>> 00000000536D21B0:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>> 00000000536D31C0:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>> 00000000536D41D0:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>> 00000000536D51E0:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>> 00000000536D61F0:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>> 00000000536D7200:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>> 00000000536D8210:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>> 00000000536D9220:4096 >>>> >>>> >>>> Infact there are lot of such calls during a reload >>>> >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA17ED00:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA1B0FF0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA1E12C0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA211590:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA243880:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA271B30:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA2A3E20:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA2D20D0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA3063E0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA334690:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA366980:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA396C50:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA3C8F40:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA3F9210:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA4294E0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA45B7D0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA489A80:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA4BBD70:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA4EA020:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA51E330:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA54C5E0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA57E8D0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA5AEBA0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA5DEE70:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA611160:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA641430:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA671700:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA6A29E0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA6D5CE0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA707FD0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA736280:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA768570:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA796820:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA7CAB30:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA7F8DE0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA82B0D0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: >>>> posix_memalign: 00000000BA85B3A0:16384 @16 >>>> >>>> >>>> >>>> What is perplexing is that the system has enough free >>>> (available RAM) >>>> ############# >>>> # free -g >>>> ? ? ? ? ? ? ? total? ? ? ? used ? ? ? free? ? ? shared? >>>> buff/cache ?available >>>> Mem:? ? ? ? ? ? 125? ? ? ? ? 54 ? ? ? ? 24? ? ? ? ? ?8? >>>> ? ? ? ? 46 ? ? ? ? 58 >>>> Swap:? ? ? ? ? ? ?0? ? ? ? ? ?0 ? ? ? ? ?0 >>>> ############# >>>> >>>> # ulimit -a >>>> core file size? ? ? ? ? (blocks, -c) 0 >>>> data seg size? ? ? ? ? ?(kbytes, -d) unlimited >>>> scheduling priority ?(-e) 0 >>>> file size? ? ? ? ? ? ? ?(blocks, -f) unlimited >>>> pending signals ?(-i) 514579 >>>> max locked memory? ? ? ?(kbytes, -l) 64 >>>> max memory size? ? ? ? ?(kbytes, -m) unlimited >>>> open files (-n) 1024 >>>> pipe size? ? ? ? ? ? (512 bytes, -p) 8 >>>> POSIX message queues? ? ?(bytes, -q) 819200 >>>> real-time priority (-r) 0 >>>> stack size? ? ? ? ? ? ? (kbytes, -s) 8192 >>>> cpu time? ? ? ? ? ? ? ?(seconds, -t) unlimited >>>> max user processes (-u) 514579 >>>> virtual memory? ? ? ? ? (kbytes, -v) unlimited >>>> file locks (-x) unlimited >>>> >>>> ######################################### >>>> >>>> There is no other thing limiting memory allocation >>>> >>>> Any way to prevent this or probably identify/prevent this >>>> >>>> >>>> On Tue, Jul 31, 2018 at 7:08 PM Maxim Dounin >>>> > wrote: >>>> >>>> Hello! >>>> >>>> On Tue, Jul 31, 2018 at 09:52:29AM +0530, Anoop >>>> Alias wrote: >>>> >>>> > I am repeatedly seeing errors like >>>> > >>>> > ###################### >>>> > 2018/07/31 03:46:33 [emerg] 2854560#2854560: >>>> posix_memalign(16, 16384) >>>> > failed (12: Cannot allocate memory) >>>> > 2018/07/31 03:54:09 [emerg] 2890190#2890190: >>>> posix_memalign(16, 16384) >>>> > failed (12: Cannot allocate memory) >>>> > 2018/07/31 04:08:36 [emerg] 2939230#2939230: >>>> posix_memalign(16, 16384) >>>> > failed (12: Cannot allocate memory) >>>> > 2018/07/31 04:24:48 [emerg] 2992650#2992650: >>>> posix_memalign(16, 16384) >>>> > failed (12: Cannot allocate memory) >>>> > 2018/07/31 04:42:09 [emerg] 3053092#3053092: >>>> posix_memalign(16, 16384) >>>> > failed (12: Cannot allocate memory) >>>> > 2018/07/31 04:42:17 [emerg] 3053335#3053335: >>>> posix_memalign(16, 16384) >>>> > failed (12: Cannot allocate memory) >>>> > 2018/07/31 04:42:28 [emerg] 3053937#3053937: >>>> posix_memalign(16, 16384) >>>> > failed (12: Cannot allocate memory) >>>> > 2018/07/31 04:47:54 [emerg] 3070638#3070638: >>>> posix_memalign(16, 16384) >>>> > failed (12: Cannot allocate memory) >>>> > #################### >>>> > >>>> > on a few servers >>>> > >>>> > The servers have enough memory free and the swap >>>> usage is 0, yet somehow >>>> > the kernel denies the posix_memalign with ENOMEM >>>> ( this is what I think is >>>> > happening!) >>>> > >>>> > The numbers requested are always 16, 16k . This >>>> makes me suspicious >>>> > >>>> > I have no setting in nginx.conf that reference a 16k >>>> > >>>> > Is there any chance of finding out what requests >>>> this and why this is not >>>> > fulfilled >>>> >>>> There are at least some buffers which default to >>>> 16k - for >>>> example, ssl_buffer_size >>>> (http://nginx.org/r/ssl_buffer_size). >>>> >>>> You may try debugging log to futher find out where >>>> the particular >>>> allocation happens, see here for details: >>>> >>>> http://nginx.org/en/docs/debugging_log.html >>>> >>>> But I don't really think it worth the effort.? The >>>> error is pretty >>>> clear, and it's better to focus on why these >>>> allocations are >>>> denied.? Likely you are hitting some limit. >>>> >>>> -- >>>> Maxim Dounin >>>> http://mdounin.ru/ >>>> _______________________________________________ >>>> nginx mailing list >>>> nginx at nginx.org >>>> http://mailman.nginx.org/mailman/listinfo/nginx >>>> >>>> >>>> >>>> -- >>>> *Anoop P Alias* >>>> >>>> >>>> >>>> _______________________________________________ >>>> nginx mailing list >>>> nginx at nginx.org >>>> http://mailman.nginx.org/mailman/listinfo/nginx >>> >>> >>> _______________________________________________ >>> nginx mailing list >>> nginx at nginx.org >>> http://mailman.nginx.org/mailman/listinfo/nginx >>> >>> >>> >>> -- >>> *Anoop P Alias* >>> >>> >>> >>> _______________________________________________ >>> nginx mailing list >>> nginx at nginx.org >>> http://mailman.nginx.org/mailman/listinfo/nginx >> >> >> _______________________________________________ >> nginx mailing list >> nginx at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx >> >> >> >> -- >> *Anoop P Alias* >> >> >> >> _______________________________________________ >> nginx mailing list >> nginx at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx > > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx > > > > -- > *Anoop P Alias* > > > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Tue Aug 7 11:58:22 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 7 Aug 2018 14:58:22 +0300 Subject: limit_rate based on User-Agent; how to exempt /robots.txt ? In-Reply-To: <1fda9bac56e6491ba870cf8445e8e6a6@its-mail-p05.registry.otago.ac.nz> References: <1fda9bac56e6491ba870cf8445e8e6a6@its-mail-p05.registry.otago.ac.nz> Message-ID: <20180807115822.GU56558@mdounin.ru> Hello! On Tue, Aug 07, 2018 at 02:45:02AM +0000, Cameron Kerr wrote: > Hi all, I?ve recently deployed a rate-limiting configuration > aimed at protecting myself from spiders. > > nginx version: nginx/1.15.1 (RPM from nginx.org) > > I did this based on the excellent Nginx blog post at > https://www.nginx.com/blog/rate-limiting-nginx/ and have > consulted the documentation for limit_req and limit_req_zone. > > I understand that you can have multiple zones in play, and that > the most-restrictive of all matches will apply for any matching > request. I want to go the other way though. I want to exempt > /robots.txt from being rate limited by spiders. > > To put this in context, here is the gist of the relevant config, > which aims to implement a caching (and rate-limiting) layer in > front of a much more complex request routing layer (httpd). > > http { > map $http_user_agent $user_agent_rate_key { > default ""; > "~our-crawler" "wanted-robot"; > "~*(bot/|crawler|robot|spider)" "robot"; > "~ScienceBrowser/Nutch" "robot"; > "~Arachni/" "robot"; > } > > limit_req_zone $user_agent_rate_key zone=per_spider_class:1m rate=100r/m; > limit_req_status 429; > > server { > limit_req zone=per_spider_class; > > location / { > proxy_pass http://routing_layer_http/; > } > } > } > > > > Option 1: (working, but has issues) > > Should I instead put the limit_req inside the "location / {}" > stanza, and have a separate "location /robots.txt {}" (or some > generalised form using a map) and not have limit_req inside that > stanza > > That would mean that any other configuration inside the location > stanzas would get duplicated, which would be a manageability > concern. I just want to override the limit_req. > > server { > location /robots.txt { > proxy_pass http://routing_layer_http/; > } > > location / { > limit_req zone=per_spider_class; > proxy_pass http://routing_layer_http/; > } > } > > I've tested this, and it works. This is most simple and nginx-way: provide exact configurations in particular locations. And this is what I would recommend to use. [...] > Option 3: (does not work) > > Some other way... perhaps I need to create some map that takes > the path and produces a $path_exempt variable, and then somehow > use that with the $user_agent_rate_key, returning "" when > $path_exempt, or $user_agent_rate_key otherwise. > > map $http_user_agent $user_agent_rate_key { > default ""; > "~otago-crawler" "wanted-robot"; > "~*(bot/|crawler|robot|spider)" "robot"; > "~ScienceBrowser/Nutch" "robot"; > "~Arachni/" "robot"; > } > > map $uri $rate_for_spider_exempting { > default $user_agent_rate_key; > "/robots.txt" ""; > } > > #limit_req_zone $user_agent_rate_key zone=per_spider_class:1m rate=100r/m; > limit_req_zone $rate_for_spider_exempting zone=per_spider_class:1m rate=100r/m; > > > However, this does not work because the second map is not > returning $user_agent_rate_key; the effect is that non-robots > are affected (and the load-balancer health-probes start getting > rate-limited). > > I'm guessing my reasoning of how this works is incorrect, or > there is a limitation or some sort of implicit ordering issue. This approach is expected to work fine (assuming you've used limit_req somewhere), and I've just tested the exact configuration snipped provided to be sure. If it doesn't work for you, the problem is likely elsewhere. > Option 4: (does not work) > > http://nginx.org/en/docs/http/ngx_http_core_module.html#limit_rate > > I see that there is a variable $limit_rate that can be used, and > this would seem to be the cleanest, except in testing it doesn't > seem to work (still gets 429 responses as a User-Agent that is a > bot) The limit_rate directive (and the $limit_rate variable) controls bandwidth, and it is completely unrelated to the limit_req module. -- Maxim Dounin http://mdounin.ru/ From fugee279 at gmail.com Tue Aug 7 19:22:45 2018 From: fugee279 at gmail.com (fugee ohu) Date: Tue, 7 Aug 2018 15:22:45 -0400 Subject: HTTPS over port 443 Message-ID: I'm trying to enable site wide ssl over port 443 on a site that runs on http port 80 In nginx.conf i have `listen 443 ssl;` for the server but requests for the server get routed to the first available host on port 80, another of my sites also in the nginx.conf How can I diagnose this to see what's going on? From mailinglist at unix-solution.de Tue Aug 7 19:28:08 2018 From: mailinglist at unix-solution.de (basti) Date: Tue, 7 Aug 2018 21:28:08 +0200 Subject: HTTPS over port 443 In-Reply-To: References: Message-ID: please show us your config On 07.08.2018 21:22, fugee ohu wrote: > I'm trying to enable site wide ssl over port 443 on a site that runs > on http port 80 > In nginx.conf i have `listen 443 ssl;` for the server but requests for > the server get routed to the first available host on port 80, another > of my sites also in the nginx.conf How can I diagnose this to see > what's going on? > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx > From fugee279 at gmail.com Tue Aug 7 19:37:30 2018 From: fugee279 at gmail.com (fugee ohu) Date: Tue, 7 Aug 2018 15:37:30 -0400 Subject: HTTPS over port 443 In-Reply-To: References: Message-ID: server { listen 443 ssl; ssl_certificate /etc/ssl/certs/ignatzmouse/certificate.crt; ssl_certificate_key /etc/ssl/certs/ignatzmouse/private.key; server_name *.ignatzmouse.com www.ignatzmouse.com ignatzmouse.com; charset utf-8; location / { root /usr/home/fugee/websites/ignatzmouse/public; rails_env production; passenger_enabled on; } } On Tue, Aug 7, 2018 at 3:28 PM, basti wrote: > please show us your config > > On 07.08.2018 21:22, fugee ohu wrote: >> I'm trying to enable site wide ssl over port 443 on a site that runs >> on http port 80 >> In nginx.conf i have `listen 443 ssl;` for the server but requests for >> the server get routed to the first available host on port 80, another >> of my sites also in the nginx.conf How can I diagnose this to see >> what's going on? >> _______________________________________________ >> nginx mailing list >> nginx at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx >> > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx From fugee279 at gmail.com Tue Aug 7 19:38:24 2018 From: fugee279 at gmail.com (fugee ohu) Date: Tue, 7 Aug 2018 15:38:24 -0400 Subject: HTTPS over port 443 In-Reply-To: References: Message-ID: server { listen 80; # listen 443 ssl; # ssl_certificate /etc/ssl/certs/ignatzmouse/certificate.crt; # ssl_certificate_key /etc/ssl/certs/ignatzmouse/private.key; server_name *.sitename.com www.sitename.com sitename.com; charset utf-8; location / { root /usr/home/fugee/websites/ignatzmouse/public; rails_env production; passenger_enabled on; } } On Tue, Aug 7, 2018 at 3:28 PM, basti wrote: > please show us your config > > On 07.08.2018 21:22, fugee ohu wrote: >> I'm trying to enable site wide ssl over port 443 on a site that runs >> on http port 80 >> In nginx.conf i have `listen 443 ssl;` for the server but requests for >> the server get routed to the first available host on port 80, another >> of my sites also in the nginx.conf How can I diagnose this to see >> what's going on? >> _______________________________________________ >> nginx mailing list >> nginx at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx >> > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx From fugee279 at gmail.com Tue Aug 7 19:39:47 2018 From: fugee279 at gmail.com (fugee ohu) Date: Tue, 7 Aug 2018 15:39:47 -0400 Subject: HTTPS over port 443 In-Reply-To: References: Message-ID: server { listen 443 ssl; ssl_certificate /etc/ssl/certs/sitename/certificate.crt; ssl_certificate_key /etc/ssl/certs/sitename/private.key; server_name *.sitename.com www.sitename.com sitename; charset utf-8; location / { root /usr/home/fugee/websites/sitename/public; rails_env production; passenger_enabled on; } } On Tue, Aug 7, 2018 at 3:28 PM, basti wrote: > please show us your config > > On 07.08.2018 21:22, fugee ohu wrote: >> I'm trying to enable site wide ssl over port 443 on a site that runs >> on http port 80 >> In nginx.conf i have `listen 443 ssl;` for the server but requests for >> the server get routed to the first available host on port 80, another >> of my sites also in the nginx.conf How can I diagnose this to see >> what's going on? >> _______________________________________________ >> nginx mailing list >> nginx at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx >> > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx From mailinglist at unix-solution.de Tue Aug 7 19:45:51 2018 From: mailinglist at unix-solution.de (basti) Date: Tue, 7 Aug 2018 21:45:51 +0200 Subject: HTTPS over port 443 In-Reply-To: References: Message-ID: <92f37798-b8ed-6d5c-7620-038fd17bf43b@unix-solution.de> first of all use wget or curl for testing to bypass browser cache. when you have also a port 80 config and call http://example.com you *must* have a redirect to https. when you call https://example.com it should work, i think (not tested) On 07.08.2018 21:39, fugee ohu wrote: > server { > listen 443 ssl; > ssl_certificate /etc/ssl/certs/sitename/certificate.crt; > ssl_certificate_key /etc/ssl/certs/sitename/private.key; > server_name *.sitename.com www.sitename.com sitename; > charset utf-8; > location / { > root /usr/home/fugee/websites/sitename/public; > rails_env production; > passenger_enabled on; > } > } > > On Tue, Aug 7, 2018 at 3:28 PM, basti wrote: >> please show us your config >> >> On 07.08.2018 21:22, fugee ohu wrote: >>> I'm trying to enable site wide ssl over port 443 on a site that runs >>> on http port 80 >>> In nginx.conf i have `listen 443 ssl;` for the server but requests for >>> the server get routed to the first available host on port 80, another >>> of my sites also in the nginx.conf How can I diagnose this to see >>> what's going on? >>> _______________________________________________ >>> nginx mailing list >>> nginx at nginx.org >>> http://mailman.nginx.org/mailman/listinfo/nginx >>> >> _______________________________________________ >> nginx mailing list >> nginx at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx > From fugee279 at gmail.com Tue Aug 7 19:58:26 2018 From: fugee279 at gmail.com (fugee ohu) Date: Tue, 7 Aug 2018 15:58:26 -0400 Subject: HTTPS over port 443 In-Reply-To: <92f37798-b8ed-6d5c-7620-038fd17bf43b@unix-solution.de> References: <92f37798-b8ed-6d5c-7620-038fd17bf43b@unix-solution.de> Message-ID: What do you mean by "also have a port 80 config" ? The port 80 configs are my other sites that I haven't created certificates for yet The port 80 config for the site in question is commented out in favor of `listen 443 ssl;` On Tue, Aug 7, 2018 at 3:45 PM, basti wrote: > first of all use wget or curl for testing to bypass browser cache. > when you have also a port 80 config and call http://example.com you > *must* have a redirect to https. when you call https://example.com it > should work, i think (not tested) > > On 07.08.2018 21:39, fugee ohu wrote: >> server { >> listen 443 ssl; >> ssl_certificate /etc/ssl/certs/sitename/certificate.crt; >> ssl_certificate_key /etc/ssl/certs/sitename/private.key; >> server_name *.sitename.com www.sitename.com sitename; >> charset utf-8; >> location / { >> root /usr/home/fugee/websites/sitename/public; >> rails_env production; >> passenger_enabled on; >> } >> } >> >> On Tue, Aug 7, 2018 at 3:28 PM, basti wrote: >>> please show us your config >>> >>> On 07.08.2018 21:22, fugee ohu wrote: >>>> I'm trying to enable site wide ssl over port 443 on a site that runs >>>> on http port 80 >>>> In nginx.conf i have `listen 443 ssl;` for the server but requests for >>>> the server get routed to the first available host on port 80, another >>>> of my sites also in the nginx.conf How can I diagnose this to see >>>> what's going on? >>>> _______________________________________________ >>>> nginx mailing list >>>> nginx at nginx.org >>>> http://mailman.nginx.org/mailman/listinfo/nginx >>>> >>> _______________________________________________ >>> nginx mailing list >>> nginx at nginx.org >>> http://mailman.nginx.org/mailman/listinfo/nginx >> _______________________________________________ >> nginx mailing list >> nginx at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx >> > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx From jeff.dyke at gmail.com Tue Aug 7 22:23:16 2018 From: jeff.dyke at gmail.com (Jeff Dyke) Date: Tue, 7 Aug 2018 18:23:16 -0400 Subject: HTTPS over port 443 In-Reply-To: References: <92f37798-b8ed-6d5c-7620-038fd17bf43b@unix-solution.de> Message-ID: how about adding server { listen 80; redirect https://$host$request_uri 301; //YYMV as to what destination you need them to end up at. } On Tue, Aug 7, 2018 at 3:58 PM, fugee ohu wrote: > What do you mean by "also have a port 80 config" ? The port 80 configs > are my other sites that I haven't created certificates for yet The > port 80 config for the site in question is commented out in favor of > `listen 443 ssl;` > > On Tue, Aug 7, 2018 at 3:45 PM, basti > wrote: > > first of all use wget or curl for testing to bypass browser cache. > > when you have also a port 80 config and call http://example.com you > > *must* have a redirect to https. when you call https://example.com it > > should work, i think (not tested) > > > > On 07.08.2018 21:39, fugee ohu wrote: > >> server { > >> listen 443 ssl; > >> ssl_certificate /etc/ssl/certs/sitename/certificate.crt; > >> ssl_certificate_key /etc/ssl/certs/sitename/private.key; > >> server_name *.sitename.com www.sitename.com sitename; > >> charset utf-8; > >> location / { > >> root /usr/home/fugee/websites/sitename/public; > >> rails_env production; > >> passenger_enabled on; > >> } > >> } > >> > >> On Tue, Aug 7, 2018 at 3:28 PM, basti > wrote: > >>> please show us your config > >>> > >>> On 07.08.2018 21:22, fugee ohu wrote: > >>>> I'm trying to enable site wide ssl over port 443 on a site that runs > >>>> on http port 80 > >>>> In nginx.conf i have `listen 443 ssl;` for the server but requests for > >>>> the server get routed to the first available host on port 80, another > >>>> of my sites also in the nginx.conf How can I diagnose this to see > >>>> what's going on? > >>>> _______________________________________________ > >>>> nginx mailing list > >>>> nginx at nginx.org > >>>> http://mailman.nginx.org/mailman/listinfo/nginx > >>>> > >>> _______________________________________________ > >>> nginx mailing list > >>> nginx at nginx.org > >>> http://mailman.nginx.org/mailman/listinfo/nginx > >> _______________________________________________ > >> nginx mailing list > >> nginx at nginx.org > >> http://mailman.nginx.org/mailman/listinfo/nginx > >> > > _______________________________________________ > > nginx mailing list > > nginx at nginx.org > > http://mailman.nginx.org/mailman/listinfo/nginx > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx > -------------- next part -------------- An HTML attachment was scrubbed... URL: From francis at daoine.org Tue Aug 7 22:26:25 2018 From: francis at daoine.org (Francis Daly) Date: Tue, 7 Aug 2018 23:26:25 +0100 Subject: HTTPS over port 443 In-Reply-To: References: Message-ID: <20180807222625.GC18403@daoine.org> On Tue, Aug 07, 2018 at 03:22:45PM -0400, fugee ohu wrote: Hi there, > I'm trying to enable site wide ssl over port 443 on a site that runs > on http port 80 > In nginx.conf i have `listen 443 ssl;` for the server but requests for > the server get routed to the first available host on port 80, another > of my sites also in the nginx.conf How can I diagnose this to see > what's going on? curl -v https://www.sitename.com The response will be interesting. My guess is that your rails/passenger settings (outside of nginx) are causing a http redirect to something like http://www.sitename.com; and because you have disabled the http server for that name, it is handled by the default server on that ip:port. If that is the case -- check your rails/passenger setting to see if they refer to http anywhere, and make them not. f -- Francis Daly francis at daoine.org From cameron.kerr at otago.ac.nz Tue Aug 7 22:27:23 2018 From: cameron.kerr at otago.ac.nz (Cameron Kerr) Date: Tue, 7 Aug 2018 22:27:23 +0000 Subject: limit_rate based on User-Agent; how to exempt /robots.txt ? In-Reply-To: <20180807115822.GU56558@mdounin.ru> References: <1fda9bac56e6491ba870cf8445e8e6a6@its-mail-p05.registry.otago.ac.nz> <20180807115822.GU56558@mdounin.ru> Message-ID: Hi Maxim, that's very helpful... > -----Original Message----- > From: nginx [mailto:nginx-bounces at nginx.org] On Behalf Of Maxim Dounin > On Tue, Aug 07, 2018 at 02:45:02AM +0000, Cameron Kerr wrote: > > Option 3: (does not work) > This approach is expected to work fine (assuming you've used limit_req > somewhere), and I've just tested the exact configuration snipped provided > to be sure. If it doesn't work for you, the problem is likely elsewhere. Thank you for the confirmation; I've retried it, and testing with ab, it seems to work, so I'm not sure what I was doing wrong previously. I like the pattern of chaining maps; its nicely functional in my way of thinking. For the sake of others, my configuration looks like the following: http { map $http_user_agent $user_agent_rate_key { default ""; "~*(bot[/-]|crawler|robot|spider)" "robot"; "~ScienceBrowser/Nutch" "robot"; "~Arachni/" "robot"; } map $uri $rate_for_spider_exempting { default $user_agent_rate_key; "/robots.txt" ''; } limit_req_zone $rate_for_spider_exempting zone=per_spider_class:1m rate=100r/m; limit_req_status 429; server_tokens off; server { limit_req zone=per_spider_class; location / { proxy_pass http://routing_layer_http/; } } } And my testing: // spider with non-exempted (ie. rate-limited for spiders) URI $ ab -H 'User-Agent: spider' -n100 https://.../hostname | grep -e '^Complete requests:' -e '^Failed requests:' Complete requests: 100 Failed requests: 98 // spider with exempted (ie. no-rate-limiting for spiders) URI $ ab -H 'User-Agent: spider' -n100 https://.../robots.txt | grep -e '^Complete requests:' -e '^Failed requests:' Complete requests: 100 Failed requests: 0 // non-spider with exempted (ie. rate-limited for spiders) URI $ ab -n100 https://.../robots.txt | grep -e '^Complete requests:' -e '^Failed requests:' Complete requests: 100 Failed requests: 0 // non-spider with non-exempted (ie. no-rate-limiting for spiders) URI $ ab -n100 https://.../hostname | grep -e '^Complete requests:' -e '^Failed requests:' Complete requests: 100 Failed requests: 0 Thanks again for your feedback Cheers, Cameron From fugee279 at gmail.com Tue Aug 7 23:18:20 2018 From: fugee279 at gmail.com (fugee ohu) Date: Tue, 7 Aug 2018 19:18:20 -0400 Subject: HTTPS over port 443 In-Reply-To: References: <92f37798-b8ed-6d5c-7620-038fd17bf43b@unix-solution.de> Message-ID: If I have `listen 80 ssl;` would that do the redirect automatically for me without the explicit redirect you gave me? On Tue, Aug 7, 2018 at 6:23 PM, Jeff Dyke wrote: > how about adding > server { > listen 80; > redirect https://$host$request_uri 301; //YYMV as to what destination > you need them to end up at. > } > > On Tue, Aug 7, 2018 at 3:58 PM, fugee ohu wrote: >> >> What do you mean by "also have a port 80 config" ? The port 80 configs >> are my other sites that I haven't created certificates for yet The >> port 80 config for the site in question is commented out in favor of >> `listen 443 ssl;` >> >> >> On Tue, Aug 7, 2018 at 3:45 PM, basti >> wrote: >> > first of all use wget or curl for testing to bypass browser cache. >> > when you have also a port 80 config and call http://example.com you >> > *must* have a redirect to https. when you call https://example.com it >> > should work, i think (not tested) >> > >> > On 07.08.2018 21:39, fugee ohu wrote: >> >> server { >> >> listen 443 ssl; >> >> ssl_certificate /etc/ssl/certs/sitename/certificate.crt; >> >> ssl_certificate_key /etc/ssl/certs/sitename/private.key; >> >> server_name *.sitename.com www.sitename.com sitename; >> >> charset utf-8; >> >> location / { >> >> root /usr/home/fugee/websites/sitename/public; >> >> rails_env production; >> >> passenger_enabled on; >> >> } >> >> } >> >> >> >> On Tue, Aug 7, 2018 at 3:28 PM, basti >> >> wrote: >> >>> please show us your config >> >>> >> >>> On 07.08.2018 21:22, fugee ohu wrote: >> >>>> I'm trying to enable site wide ssl over port 443 on a site that runs >> >>>> on http port 80 >> >>>> In nginx.conf i have `listen 443 ssl;` for the server but requests >> >>>> for >> >>>> the server get routed to the first available host on port 80, another >> >>>> of my sites also in the nginx.conf How can I diagnose this to see >> >>>> what's going on? >> >>>> _______________________________________________ >> >>>> nginx mailing list >> >>>> nginx at nginx.org >> >>>> http://mailman.nginx.org/mailman/listinfo/nginx >> >>>> >> >>> _______________________________________________ >> >>> nginx mailing list >> >>> nginx at nginx.org >> >>> http://mailman.nginx.org/mailman/listinfo/nginx >> >> _______________________________________________ >> >> nginx mailing list >> >> nginx at nginx.org >> >> http://mailman.nginx.org/mailman/listinfo/nginx >> >> >> > _______________________________________________ >> > nginx mailing list >> > nginx at nginx.org >> > http://mailman.nginx.org/mailman/listinfo/nginx >> _______________________________________________ >> nginx mailing list >> nginx at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx > > > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx From fugee279 at gmail.com Tue Aug 7 23:20:58 2018 From: fugee279 at gmail.com (fugee ohu) Date: Tue, 7 Aug 2018 19:20:58 -0400 Subject: HTTPS over port 443 In-Reply-To: <20180807222625.GC18403@daoine.org> References: <20180807222625.GC18403@daoine.org> Message-ID: I don't think there's rails passenger settings Adding passenger to the Gemfile only causes the app to use passenger in development mode instead of the default (presently puma) On Tue, Aug 7, 2018 at 6:26 PM, Francis Daly wrote: > On Tue, Aug 07, 2018 at 03:22:45PM -0400, fugee ohu wrote: > > Hi there, > >> I'm trying to enable site wide ssl over port 443 on a site that runs >> on http port 80 >> In nginx.conf i have `listen 443 ssl;` for the server but requests for >> the server get routed to the first available host on port 80, another >> of my sites also in the nginx.conf How can I diagnose this to see >> what's going on? > > curl -v https://www.sitename.com > > The response will be interesting. > > My guess is that your rails/passenger settings (outside of nginx) are > causing a http redirect to something like http://www.sitename.com; and > because you have disabled the http server for that name, it is handled > by the default server on that ip:port. > > If that is the case -- check your rails/passenger setting to see if they > refer to http anywhere, and make them not. > > f > -- > Francis Daly francis at daoine.org > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx From anoopalias01 at gmail.com Wed Aug 8 03:33:39 2018 From: anoopalias01 at gmail.com (Anoop Alias) Date: Wed, 8 Aug 2018 09:03:39 +0530 Subject: posix_memalign error In-Reply-To: References: <20180731133813.GG56558@mdounin.ru> <4fa47f31-9c32-5386-7c2d-a4e9347b3dfc@nginx.com> <1fdf4937-cac6-0550-b725-36f04eff65db@nginx.com> Message-ID: Hi Igor, Yes the server runs other software including httpd with a similar number of vhost # grep " wrote: > Anoop, > > I don't see any troubles with your configuration. > Also, if you have 120G of RAM and a single worker - the problem is not in > nginx. > Do you have other software running on the host? > > Basically, you just run out of memory. > > You can optimize your reload though: use "service nginx reload" (or "kill > -SIGHUP") to reload nginx configuration. > When you do nginx -s reload - you make nginx parse configuration (and it > requires memory) and then send a signal to the running master. You can > avoid this overhead with 'service' command as it uses 'kill' documented in > the manual page. > > On 06.08.2018 22:55, Anoop Alias wrote: > > Hi Igor, > > Config is reloaded using > > /usr/sbin/nginx -s reload > > this is invoked from a python/shell script ( Nginx is installed on a web > control panel ) > > The top-level Nginx config is in the gist below > > https://gist.github.com/AnoopAlias/ba5ad6749a586c7e267672ee65b32b3a > > It further includes ~8k server blocks or more in some servers. Out of this > 2/3 are server {} blocks with TLS config and 1/3 non-TLS ones > > ]# pwd > /etc/nginx/sites-enabled > # grep "server {" *|wc -l > 7886 > > And yes most of them are very similar and mostly proxy to upstream httpd > > I have tried removing all the loadable modules and even tried an older > version of nginx and all produce the error > > > # numastat -m > > Per-node system memory usage (in MBs): > Node 0 Node 1 Total > --------------- --------------- --------------- > MemTotal 65430.84 65536.00 130966.84 > MemFree 5491.26 40.89 5532.15 > MemUsed 59939.58 65495.11 125434.69 > Active 22295.61 21016.09 43311.70 > Inactive 8742.76 4662.48 13405.24 > Active(anon) 16717.10 16572.19 33289.29 > Inactive(anon) 2931.94 1388.14 4320.08 > Active(file) 5578.50 4443.91 10022.41 > Inactive(file) 5810.82 3274.34 9085.16 > Unevictable 0.00 0.00 0.00 > Mlocked 0.00 0.00 0.00 > Dirty 7.04 1.64 8.67 > Writeback 0.00 0.00 0.00 > FilePages 18458.93 10413.97 28872.90 > Mapped 862.14 413.38 1275.52 > AnonPages 12579.49 15264.37 27843.86 > Shmem 7069.52 2695.71 9765.23 > KernelStack 18.34 3.03 21.38 > PageTables 153.14 107.77 260.90 > NFS_Unstable 0.00 0.00 0.00 > Bounce 0.00 0.00 0.00 > WritebackTmp 0.00 0.00 0.00 > Slab 4830.68 2254.55 7085.22 > SReclaimable 2061.05 921.72 2982.77 > SUnreclaim 2769.62 1332.83 4102.45 > AnonHugePages 4.00 2.00 6.00 > HugePages_Total 0.00 0.00 0.00 > HugePages_Free 0.00 0.00 0.00 > HugePages_Surp 0.00 0.00 0.00 > > > Thanks, > > > > > > On Mon, Aug 6, 2018 at 6:33 PM Igor A. Ippolitov > wrote: > >> Anoop, >> >> I suppose, most of your 10k servers are very similar, right? >> Please, post top level configuration and a typical server{}, please. >> >> Also, how do you reload configuration? With 'service nginx reload' or may >> be other commands? >> >> It looks like you have a lot of fragmented memory and only 4gb free in >> the second numa node. >> So, I'd say this is OK that you are getting errors from allocating a 16k >> stripes. >> >> Could you please post numastat -m output additionally. Just to make sure >> you have half of the memory for the second CPU. >> And we'll have a look if memory utilization may be optimized based on >> your configuration. >> >> Regards, >> Igor. >> >> On 04.08.2018 07:54, Anoop Alias wrote: >> >> Hi Igor, >> >> Setting vm.max_map_count to 20x the normal value did not help >> >> The issue happens on a group of servers and among the group, it shows up >> only in servers which have ~10k server{} blocks >> >> On servers that have lower number of server{} blocks , the ENOMEM issue >> is not there >> >> Also, I can find that the RAM usage of the Nginx process is directly >> proportional to the number of server {} blocks >> >> For example on a server having the problem >> >> # ps_mem| head -1 && ps_mem |grep nginx >> Private + Shared = RAM used Program >> 1.0 GiB + 2.8 GiB = 3.8 GiB nginx (3) >> >> >> That is for a single worker process with 4 threads in thread_pool >> # pstree|grep nginx >> |-nginx-+-nginx---4*[{nginx}] >> | `-nginx >> >> Whatever config change I try the memory usage seem to mostly depend on >> the number of server contexts defined >> >> Now the issue mostly happen in nginx reload ,when one more worker process >> will be active in shutting down mode >> >> I believe the memalign error is thrown by the worker being shutdown, this >> is because the sites work after the error and also the pid mentioned in the >> error would have gone when I check ps >> >> >> # pmap 948965|grep 16K >> 00007f2923ff2000 16K r-x-- ngx_http_redis2_module.so >> 00007f2924fd7000 16K r---- libc-2.17.so >> 00007f2925431000 16K rw--- [ anon ] >> 00007f292584a000 16K rw--- [ anon ] >> >> Aug 4 05:50:00 b kernel: SysRq : Show Memory >> Aug 4 05:50:00 b kernel: Mem-Info: >> Aug 4 05:50:00 b kernel: active_anon:7757394 inactive_anon:1021319 >> isolated_anon:0#012 active_file:3733324 inactive_file:2136476 isolated_ >> file:0#012 unevictable:0 dirty:1766 writeback:6 wbtmp:0 unstable:0#012 >> slab_reclaimable:2003687 slab_unreclaimable:901391#012 mapped:316734 >> shmem:2381810 pagetables:63163 bounce:0#012 free:4851283 free_pcp:11332 >> free_cma:0 >> Aug 4 05:50:00 bravo kernel: Node 0 DMA free:15888kB min:8kB low:8kB >> high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_ >> file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB >> present:15972kB managed:15888kB mlocked:0kB dirty:0kB writeback:0kB >> mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB >> kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB >> local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 >> all_unreclaimable? yes >> Aug 4 05:50:00 b kernel: lowmem_reserve[]: 0 1679 64139 64139 >> >> # cat /proc/buddyinfo >> Node 0, zone DMA 0 0 1 0 2 1 1 >> 0 1 1 3 >> Node 0, zone DMA32 5284 6753 6677 1083 410 59 1 >> 0 0 0 0 >> Node 0, zone Normal 500327 638958 406737 14690 872 106 11 >> 0 0 0 0 >> Node 1, zone Normal 584840 291640 188 0 0 0 0 >> 0 0 0 0 >> >> >> The only correlation I see in having the error is the number of server {} >> blocks (close to 10k) which then makes the nginx process consume ~ 4GB of >> mem with a single worker process and then a reload is done >> >> >> >> >> On Thu, Aug 2, 2018 at 6:02 PM Igor A. Ippolitov >> wrote: >> >>> Anoop, >>> >>> There are two guesses: either mmap allocations limit is hit or memory >>> is way too fragmented. >>> Could you please track amount of mapped regions for a worker with pmap >>> and amount of 16k areas in Normal zones (it is the third number)? >>> >>> You can also set vm.max_map_count to a higher number (like 20 times >>> higher than default) and look if the error is gone. >>> >>> Please, let me know if increasing vm.max_map_count helps you. >>> >>> On 02.08.2018 13:06, Anoop Alias wrote: >>> >>> Hi Igor, >>> >>> The error happens randomly >>> >>> 2018/08/02 06:52:42 [emerg] 874514#874514: posix_memalign(16, 16384) >>> failed (12: Cannot allocate memory) >>> 2018/08/02 09:42:53 [emerg] 872996#872996: posix_memalign(16, 16384) >>> failed (12: Cannot allocate memory) >>> 2018/08/02 10:16:14 [emerg] 877611#877611: posix_memalign(16, 16384) >>> failed (12: Cannot allocate memory) >>> 2018/08/02 10:16:48 [emerg] 879410#879410: posix_memalign(16, 16384) >>> failed (12: Cannot allocate memory) >>> 2018/08/02 10:17:55 [emerg] 876563#876563: posix_memalign(16, 16384) >>> failed (12: Cannot allocate memory) >>> 2018/08/02 10:20:21 [emerg] 879263#879263: posix_memalign(16, 16384) >>> failed (12: Cannot allocate memory) >>> 2018/08/02 10:20:51 [emerg] 878991#878991: posix_memalign(16, 16384) >>> failed (12: Cannot allocate memory) >>> >>> # date >>> Thu Aug 2 10:58:48 BST 2018 >>> >>> ------------------------------------------ >>> # cat /proc/buddyinfo >>> Node 0, zone DMA 0 0 1 0 2 1 1 >>> 0 1 1 3 >>> Node 0, zone DMA32 11722 11057 4663 1647 609 72 10 >>> 7 1 0 0 >>> Node 0, zone Normal 755026 710760 398136 21462 1114 18 1 >>> 0 0 0 0 >>> Node 1, zone Normal 341295 801810 179604 256 0 0 0 >>> 0 0 0 0 >>> ----------------------------------------- >>> >>> >>> slabinfo - version: 2.1 >>> # name >>> : tunables : slabdata >>> >>> SCTPv6 21 21 1536 21 8 : tunables 0 0 >>> 0 : slabdata 1 1 0 >>> SCTP 0 0 1408 23 8 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> kcopyd_job 0 0 3312 9 8 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> dm_uevent 0 0 2608 12 8 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> nf_conntrack_ffffffff81acbb00 14054 14892 320 51 4 : tunables >>> 0 0 0 : slabdata 292 292 0 >>> lvp_cache 36 36 224 36 2 : tunables 0 0 >>> 0 : slabdata 1 1 0 >>> lve_struct 4140 4140 352 46 4 : tunables 0 0 >>> 0 : slabdata 90 90 0 >>> fat_inode_cache 0 0 744 44 8 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> fat_cache 0 0 40 102 1 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> isofs_inode_cache 0 0 664 49 8 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> ext4_inode_cache 30 30 1088 30 8 : tunables 0 0 >>> 0 : slabdata 1 1 0 >>> ext4_xattr 0 0 88 46 1 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> ext4_free_data 0 0 64 64 1 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> ext4_allocation_context 32 32 128 32 1 : tunables 0 >>> 0 0 : slabdata 1 1 0 >>> ext4_io_end 0 0 72 56 1 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> ext4_extent_status 102 102 40 102 1 : tunables 0 0 >>> 0 : slabdata 1 1 0 >>> jbd2_journal_handle 0 0 48 85 1 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> jbd2_journal_head 0 0 112 36 1 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> jbd2_revoke_table_s 256 256 16 256 1 : tunables 0 0 >>> 0 : slabdata 1 1 0 >>> jbd2_revoke_record_s 0 0 32 128 1 : tunables 0 >>> 0 0 : slabdata 0 0 0 >>> kvm_async_pf 0 0 136 30 1 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> kvm_vcpu 0 0 18560 1 8 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> xfs_dqtrx 992 992 528 31 4 : tunables 0 0 >>> 0 : slabdata 32 32 0 >>> xfs_dquot 3264 3264 472 34 4 : tunables 0 0 >>> 0 : slabdata 96 96 0 >>> xfs_ili 4342175 4774399 152 53 2 : tunables 0 0 >>> 0 : slabdata 90083 90083 0 >>> xfs_inode 4915588 5486076 1088 30 8 : tunables 0 0 >>> 0 : slabdata 182871 182871 0 >>> xfs_efd_item 2680 2760 400 40 4 : tunables 0 0 >>> 0 : slabdata 69 69 0 >>> xfs_da_state 1088 1088 480 34 4 : tunables 0 0 >>> 0 : slabdata 32 32 0 >>> xfs_btree_cur 1248 1248 208 39 2 : tunables 0 0 >>> 0 : slabdata 32 32 0 >>> xfs_log_ticket 14874 15048 184 44 2 : tunables 0 0 >>> 0 : slabdata 342 342 0 >>> xfs_ioend 12909 13104 104 39 1 : tunables 0 0 >>> 0 : slabdata 336 336 0 >>> scsi_cmd_cache 5400 5652 448 36 4 : tunables 0 0 >>> 0 : slabdata 157 157 0 >>> ve_struct 0 0 848 38 8 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> ip6_dst_cache 1152 1152 448 36 4 : tunables 0 0 >>> 0 : slabdata 32 32 0 >>> RAWv6 910 910 1216 26 8 : tunables 0 0 >>> 0 : slabdata 35 35 0 >>> UDPLITEv6 0 0 1216 26 8 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> UDPv6 832 832 1216 26 8 : tunables 0 0 >>> 0 : slabdata 32 32 0 >>> tw_sock_TCPv6 1152 1376 256 32 2 : tunables 0 0 >>> 0 : slabdata 43 43 0 >>> TCPv6 510 510 2176 15 8 : tunables 0 0 >>> 0 : slabdata 34 34 0 >>> cfq_queue 3698 5145 232 35 2 : tunables 0 0 >>> 0 : slabdata 147 147 0 >>> bsg_cmd 0 0 312 52 4 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> mqueue_inode_cache 136 136 960 34 8 : tunables 0 0 >>> 0 : slabdata 4 4 0 >>> hugetlbfs_inode_cache 1632 1632 632 51 8 : tunables 0 >>> 0 0 : slabdata 32 32 0 >>> configfs_dir_cache 1472 1472 88 46 1 : tunables 0 0 >>> 0 : slabdata 32 32 0 >>> dquot 0 0 256 32 2 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> userfaultfd_ctx_cache 32 32 128 32 1 : tunables 0 >>> 0 0 : slabdata 1 1 0 >>> fanotify_event_info 2336 2336 56 73 1 : tunables 0 0 >>> 0 : slabdata 32 32 0 >>> dio 6171 6222 640 51 8 : tunables 0 0 >>> 0 : slabdata 122 122 0 >>> pid_namespace 42 42 2192 14 8 : tunables 0 0 >>> 0 : slabdata 3 3 0 >>> posix_timers_cache 1056 1056 248 33 2 : tunables 0 0 >>> 0 : slabdata 32 32 0 >>> UDP-Lite 0 0 1088 30 8 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> flow_cache 2268 2296 144 28 1 : tunables 0 0 >>> 0 : slabdata 82 82 0 >>> xfrm_dst_cache 896 896 576 28 4 : tunables 0 0 >>> 0 : slabdata 32 32 0 >>> ip_fib_alias 2720 2720 48 85 1 : tunables 0 0 >>> 0 : slabdata 32 32 0 >>> RAW 3977 4224 1024 32 8 : tunables 0 0 >>> 0 : slabdata 132 132 0 >>> UDP 4110 4110 1088 30 8 : tunables 0 0 >>> 0 : slabdata 137 137 0 >>> tw_sock_TCP 4756 5216 256 32 2 : tunables 0 0 >>> 0 : slabdata 163 163 0 >>> TCP 2705 2768 1984 16 8 : tunables 0 0 >>> 0 : slabdata 173 173 0 >>> scsi_data_buffer 5440 5440 24 170 1 : tunables 0 0 >>> 0 : slabdata 32 32 0 >>> blkdev_queue 154 154 2208 14 8 : tunables 0 0 >>> 0 : slabdata 11 11 0 >>> blkdev_requests 4397688 4405884 384 42 4 : tunables 0 0 >>> 0 : slabdata 104902 104902 0 >>> blkdev_ioc 11232 11232 112 36 1 : tunables 0 0 >>> 0 : slabdata 312 312 0 >>> user_namespace 0 0 1304 25 8 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> sock_inode_cache 12282 12282 704 46 8 : tunables 0 0 >>> 0 : slabdata 267 267 0 >>> file_lock_cache 20056 20960 200 40 2 : tunables 0 0 >>> 0 : slabdata 524 524 0 >>> net_namespace 6 6 5056 6 8 : tunables 0 0 >>> 0 : slabdata 1 1 0 >>> shmem_inode_cache 16970 18952 712 46 8 : tunables 0 0 >>> 0 : slabdata 412 412 0 >>> Acpi-ParseExt 39491 40432 72 56 1 : tunables 0 0 >>> 0 : slabdata 722 722 0 >>> Acpi-State 1683 1683 80 51 1 : tunables 0 0 >>> 0 : slabdata 33 33 0 >>> Acpi-Namespace 11424 11424 40 102 1 : tunables 0 0 >>> 0 : slabdata 112 112 0 >>> task_delay_info 15336 15336 112 36 1 : tunables 0 0 >>> 0 : slabdata 426 426 0 >>> taskstats 1568 1568 328 49 4 : tunables 0 0 >>> 0 : slabdata 32 32 0 >>> proc_inode_cache 169897 190608 680 48 8 : tunables 0 0 >>> 0 : slabdata 3971 3971 0 >>> sigqueue 2208 2208 168 48 2 : tunables 0 0 >>> 0 : slabdata 46 46 0 >>> bdev_cache 792 792 896 36 8 : tunables 0 0 >>> 0 : slabdata 22 22 0 >>> sysfs_dir_cache 74698 74698 120 34 1 : tunables 0 0 >>> 0 : slabdata 2197 2197 0 >>> mnt_cache 163197 163424 256 32 2 : tunables 0 0 >>> 0 : slabdata 5107 5107 0 >>> filp 64607 97257 320 51 4 : tunables 0 0 >>> 0 : slabdata 1907 1907 0 >>> inode_cache 370744 370947 616 53 8 : tunables 0 0 >>> 0 : slabdata 6999 6999 0 >>> dentry 1316262 2139228 192 42 2 : tunables 0 0 >>> 0 : slabdata 50934 50934 0 >>> iint_cache 0 0 80 51 1 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> buffer_head 1441470 2890290 104 39 1 : tunables 0 0 >>> 0 : slabdata 74110 74110 0 >>> vm_area_struct 194998 196840 216 37 2 : tunables 0 0 >>> 0 : slabdata 5320 5320 0 >>> mm_struct 2679 2760 1600 20 8 : tunables 0 0 >>> 0 : slabdata 138 138 0 >>> files_cache 8680 8925 640 51 8 : tunables 0 0 >>> 0 : slabdata 175 175 0 >>> signal_cache 3691 3780 1152 28 8 : tunables 0 0 >>> 0 : slabdata 135 135 0 >>> sighand_cache 1950 2160 2112 15 8 : tunables 0 0 >>> 0 : slabdata 144 144 0 >>> task_xstate 8070 8658 832 39 8 : tunables 0 0 >>> 0 : slabdata 222 222 0 >>> task_struct 1913 2088 4080 8 8 : tunables 0 0 >>> 0 : slabdata 261 261 0 >>> cred_jar 31699 33936 192 42 2 : tunables 0 0 >>> 0 : slabdata 808 808 0 >>> anon_vma_chain 164026 168704 64 64 1 : tunables 0 0 >>> 0 : slabdata 2636 2636 0 >>> anon_vma 84104 84594 88 46 1 : tunables 0 0 >>> 0 : slabdata 1839 1839 0 >>> pid 11127 12576 128 32 1 : tunables 0 0 >>> 0 : slabdata 393 393 0 >>> shared_policy_node 9350 9350 48 85 1 : tunables 0 0 >>> 0 : slabdata 110 110 0 >>> numa_policy 62 62 264 31 2 : tunables 0 0 >>> 0 : slabdata 2 2 0 >>> radix_tree_node 771778 1194312 584 28 4 : tunables 0 0 >>> 0 : slabdata 42654 42654 0 >>> idr_layer_cache 2538 2565 2112 15 8 : tunables 0 0 >>> 0 : slabdata 171 171 0 >>> dma-kmalloc-8192 0 0 8192 4 8 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> dma-kmalloc-4096 0 0 4096 8 8 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> dma-kmalloc-2048 0 0 2048 16 8 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> dma-kmalloc-1024 0 0 1024 32 8 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> dma-kmalloc-512 0 0 512 32 4 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> dma-kmalloc-256 0 0 256 32 2 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> dma-kmalloc-128 0 0 128 32 1 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> dma-kmalloc-64 0 0 64 64 1 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> dma-kmalloc-32 0 0 32 128 1 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> dma-kmalloc-16 0 0 16 256 1 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> dma-kmalloc-8 0 0 8 512 1 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> dma-kmalloc-192 0 0 192 42 2 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> dma-kmalloc-96 0 0 96 42 1 : tunables 0 0 >>> 0 : slabdata 0 0 0 >>> kmalloc-8192 385 388 8192 4 8 : tunables 0 0 >>> 0 : slabdata 97 97 0 >>> kmalloc-4096 9296 10088 4096 8 8 : tunables 0 0 >>> 0 : slabdata 1261 1261 0 >>> kmalloc-2048 65061 133536 2048 16 8 : tunables 0 0 >>> 0 : slabdata 8346 8346 0 >>> kmalloc-1024 11987 21120 1024 32 8 : tunables 0 0 >>> 0 : slabdata 660 660 0 >>> kmalloc-512 107510 187072 512 32 4 : tunables 0 0 >>> 0 : slabdata 5846 5846 0 >>> kmalloc-256 160498 199104 256 32 2 : tunables 0 0 >>> 0 : slabdata 6222 6222 0 >>> kmalloc-192 144975 237426 192 42 2 : tunables 0 0 >>> 0 : slabdata 5653 5653 0 >>> kmalloc-128 36799 108096 128 32 1 : tunables 0 0 >>> 0 : slabdata 3378 3378 0 >>> kmalloc-96 99510 238896 96 42 1 : tunables 0 0 >>> 0 : slabdata 5688 5688 0 >>> kmalloc-64 7978152 8593280 64 64 1 : tunables 0 0 >>> 0 : slabdata 134270 134270 0 >>> kmalloc-32 2939882 3089664 32 128 1 : tunables 0 0 >>> 0 : slabdata 24138 24138 0 >>> kmalloc-16 172057 172288 16 256 1 : tunables 0 0 >>> 0 : slabdata 673 673 0 >>> kmalloc-8 109568 109568 8 512 1 : tunables 0 0 >>> 0 : slabdata 214 214 0 >>> kmem_cache_node 893 896 64 64 1 : tunables 0 0 >>> 0 : slabdata 14 14 0 >>> kmem_cache 612 612 320 51 4 : tunables 0 0 >>> 0 : slabdata 12 12 0 >>> >>> ------------------------------------------------- >>> >>> >>> # uname -r >>> 3.10.0-714.10.2.lve1.5.17.1.el7.x86_64 >>> >>> -------------------------------------------------------- >>> >>> Core part of glances >>> http://i.imgur.com/La5JbQn.png >>> ----------------------------------------------------------- >>> >>> Thank you very much for looking into this >>> >>> >>> On Thu, Aug 2, 2018 at 12:37 PM Igor A. Ippolitov >>> wrote: >>> >>>> Anoop, >>>> >>>> I doubt this will be the solution, but may we have a look at >>>> /proc/buddyinfo and /proc/slabinfo the moment when nginx can't allocate >>>> memory? >>>> >>>> On 02.08.2018 08:15, Anoop Alias wrote: >>>> >>>> Hi Maxim, >>>> >>>> I enabled debug and the memalign call is happening on nginx reloads and >>>> the ENOMEM happen sometimes on the reload(not on all reloads) >>>> >>>> 2018/08/02 05:59:08 [notice] 872052#872052: signal process started >>>> 2018/08/02 05:59:23 [notice] 871570#871570: signal 1 (SIGHUP) received >>>> from 872052, reconfiguring >>>> 2018/08/02 05:59:23 [debug] 871570#871570: wake up, sigio 0 >>>> 2018/08/02 05:59:23 [notice] 871570#871570: reconfiguring >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 0000000002B0DA00:16384 @16 === > the memalign call on reload >>>> 2018/08/02 05:59:23 [debug] 871570#871570: malloc: 00000000087924D0:4560 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 000000000E442E00:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: malloc: 0000000005650850:4096 >>>> 20 >>>> >>>> >>>> >>>> >>>> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #71 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #72 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #73 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #74 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: add cleanup: 000000005340D728 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000024D3260:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000517BAF10:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053854FC0:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053855FD0:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053856FE0:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 0000000053857FF0:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: posix_memalign: >>>> 0000000053859000:16384 @16 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385D010:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385E020:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 000000005385F030:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CD160:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CE170:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536CF180:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D0190:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D11A0:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D21B0:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D31C0:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D41D0:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D51E0:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D61F0:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D7200:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D8210:4096 >>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: 00000000536D9220:4096 >>>> >>>> >>>> Infact there are lot of such calls during a reload >>>> >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA17ED00:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA1B0FF0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA1E12C0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA211590:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA243880:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA271B30:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA2A3E20:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA2D20D0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA3063E0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA334690:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA366980:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA396C50:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA3C8F40:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA3F9210:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA4294E0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA45B7D0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA489A80:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA4BBD70:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA4EA020:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA51E330:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA54C5E0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA57E8D0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA5AEBA0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA5DEE70:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA611160:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA641430:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA671700:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA6A29E0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA6D5CE0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA707FD0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA736280:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA768570:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA796820:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA7CAB30:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA7F8DE0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA82B0D0:16384 @16 >>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>> 00000000BA85B3A0:16384 @16 >>>> >>>> >>>> >>>> What is perplexing is that the system has enough free (available RAM) >>>> ############# >>>> # free -g >>>> total used free shared buff/cache >>>> available >>>> Mem: 125 54 24 8 46 >>>> 58 >>>> Swap: 0 0 0 >>>> ############# >>>> >>>> # ulimit -a >>>> core file size (blocks, -c) 0 >>>> data seg size (kbytes, -d) unlimited >>>> scheduling priority (-e) 0 >>>> file size (blocks, -f) unlimited >>>> pending signals (-i) 514579 >>>> max locked memory (kbytes, -l) 64 >>>> max memory size (kbytes, -m) unlimited >>>> open files (-n) 1024 >>>> pipe size (512 bytes, -p) 8 >>>> POSIX message queues (bytes, -q) 819200 >>>> real-time priority (-r) 0 >>>> stack size (kbytes, -s) 8192 >>>> cpu time (seconds, -t) unlimited >>>> max user processes (-u) 514579 >>>> virtual memory (kbytes, -v) unlimited >>>> file locks (-x) unlimited >>>> >>>> ######################################### >>>> >>>> There is no other thing limiting memory allocation >>>> >>>> Any way to prevent this or probably identify/prevent this >>>> >>>> >>>> On Tue, Jul 31, 2018 at 7:08 PM Maxim Dounin >>>> wrote: >>>> >>>>> Hello! >>>>> >>>>> On Tue, Jul 31, 2018 at 09:52:29AM +0530, Anoop Alias wrote: >>>>> >>>>> > I am repeatedly seeing errors like >>>>> > >>>>> > ###################### >>>>> > 2018/07/31 03:46:33 [emerg] 2854560#2854560: posix_memalign(16, >>>>> 16384) >>>>> > failed (12: Cannot allocate memory) >>>>> > 2018/07/31 03:54:09 [emerg] 2890190#2890190: posix_memalign(16, >>>>> 16384) >>>>> > failed (12: Cannot allocate memory) >>>>> > 2018/07/31 04:08:36 [emerg] 2939230#2939230: posix_memalign(16, >>>>> 16384) >>>>> > failed (12: Cannot allocate memory) >>>>> > 2018/07/31 04:24:48 [emerg] 2992650#2992650: posix_memalign(16, >>>>> 16384) >>>>> > failed (12: Cannot allocate memory) >>>>> > 2018/07/31 04:42:09 [emerg] 3053092#3053092: posix_memalign(16, >>>>> 16384) >>>>> > failed (12: Cannot allocate memory) >>>>> > 2018/07/31 04:42:17 [emerg] 3053335#3053335: posix_memalign(16, >>>>> 16384) >>>>> > failed (12: Cannot allocate memory) >>>>> > 2018/07/31 04:42:28 [emerg] 3053937#3053937: posix_memalign(16, >>>>> 16384) >>>>> > failed (12: Cannot allocate memory) >>>>> > 2018/07/31 04:47:54 [emerg] 3070638#3070638: posix_memalign(16, >>>>> 16384) >>>>> > failed (12: Cannot allocate memory) >>>>> > #################### >>>>> > >>>>> > on a few servers >>>>> > >>>>> > The servers have enough memory free and the swap usage is 0, yet >>>>> somehow >>>>> > the kernel denies the posix_memalign with ENOMEM ( this is what I >>>>> think is >>>>> > happening!) >>>>> > >>>>> > The numbers requested are always 16, 16k . This makes me suspicious >>>>> > >>>>> > I have no setting in nginx.conf that reference a 16k >>>>> > >>>>> > Is there any chance of finding out what requests this and why this >>>>> is not >>>>> > fulfilled >>>>> >>>>> There are at least some buffers which default to 16k - for >>>>> example, ssl_buffer_size (http://nginx.org/r/ssl_buffer_size). >>>>> >>>>> You may try debugging log to futher find out where the particular >>>>> allocation happens, see here for details: >>>>> >>>>> http://nginx.org/en/docs/debugging_log.html >>>>> >>>>> But I don't really think it worth the effort. The error is pretty >>>>> clear, and it's better to focus on why these allocations are >>>>> denied. Likely you are hitting some limit. >>>>> >>>>> -- >>>>> Maxim Dounin >>>>> http://mdounin.ru/ >>>>> _______________________________________________ >>>>> nginx mailing list >>>>> nginx at nginx.org >>>>> http://mailman.nginx.org/mailman/listinfo/nginx >>>>> >>>> >>>> >>>> -- >>>> *Anoop P Alias* >>>> >>>> >>>> >>>> _______________________________________________ >>>> nginx mailing listnginx at nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx >>>> >>>> >>>> _______________________________________________ >>>> nginx mailing list >>>> nginx at nginx.org >>>> http://mailman.nginx.org/mailman/listinfo/nginx >>> >>> >>> >>> -- >>> *Anoop P Alias* >>> >>> >>> >>> _______________________________________________ >>> nginx mailing listnginx at nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx >>> >>> >>> _______________________________________________ >>> nginx mailing list >>> nginx at nginx.org >>> http://mailman.nginx.org/mailman/listinfo/nginx >> >> >> >> -- >> *Anoop P Alias* >> >> >> >> _______________________________________________ >> nginx mailing listnginx at nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx >> >> >> _______________________________________________ >> nginx mailing list >> nginx at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx > > > > -- > *Anoop P Alias* > > > > _______________________________________________ > nginx mailing listnginx at nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx > > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx -- *Anoop P Alias* -------------- next part -------------- An HTML attachment was scrubbed... URL: From francis at daoine.org Wed Aug 8 07:17:04 2018 From: francis at daoine.org (Francis Daly) Date: Wed, 8 Aug 2018 08:17:04 +0100 Subject: HTTPS over port 443 In-Reply-To: References: <20180807222625.GC18403@daoine.org> Message-ID: <20180808071704.GD18403@daoine.org> On Tue, Aug 07, 2018 at 07:20:58PM -0400, fugee ohu wrote: > I don't think there's rails passenger settings Adding passenger to the > Gemfile only causes the app to use passenger in development mode > instead of the default (presently puma) Ok. What is the output of curl -v https://www.sitename.com f -- Francis Daly francis at daoine.org From francis at daoine.org Wed Aug 8 08:24:37 2018 From: francis at daoine.org (Francis Daly) Date: Wed, 8 Aug 2018 09:24:37 +0100 Subject: Getting NGINX to view an alias In-Reply-To: References: Message-ID: <20180808082437.GE18403@daoine.org> On Tue, Jul 31, 2018 at 12:49:31AM -0700, Gregory Schultz wrote: Hi there, > I'm new at NGINX and I'm having difficulty in setting up to read an alias. alias should work per its documentation (http://nginx.org/r/alias), or else there is a significant bug. What I suspect is happening here is that your "include" file has other location{} blocks, and that one of those locations is the one chosen by nginx to process this request. > I'm setting up adminer on NGINX to use an alias to see a file outside of > it's main directory. The file is called latest.php in /usr/share/adminer. I > created a synlink to link adminer.php to latest.php. I'm trying to access > adminer through /admin/adminer.php but returns a 404 when I try to access > the file. > > my config file: > > server { > listen 80; > listen [::]:80; > include /etc/nginx-rc/conf.d/[site]/main.conf; > location /admin/ { > alias /usr/share/adminer/; > } > } You can show "alias" working correctly by temporarily removing that "include" directive. All of the rest of your web site will (probably) fail, but the request for /admin/adminer.php should return the content of the file /usr/share/adminer/adminer.php That output is (almost certainly) not what you want; you probably want to proxy_pass or fastcgi_pass to a separate service to do some processing of the php file contents. For fastcgi, the two things you need to know are: what is the incoming url that you want to process; and what is the name of the file that you want to tell the fastcgi server to use. When you have those, you can see about writing the nginx config to match it. f -- Francis Daly francis at daoine.org From plato.puthur at gmail.com Wed Aug 8 08:29:57 2018 From: plato.puthur at gmail.com (Plato Puthur) Date: Wed, 8 Aug 2018 13:59:57 +0530 Subject: nginx, php7.0-fpm and laravel, not able to set it up when the url has a prefix which the server doesn't have In-Reply-To: <00a301d42e41$d0408bf0$70c1a3d0$@roze.lv> References: <002801d42d5b$c0a9bbd0$41fd3370$@roze.lv> <00a301d42e41$d0408bf0$70c1a3d0$@roze.lv> Message-ID: Hi, Thanks for replying. I have 3 containers, one with nginx, another 2 containers with php-fpm on them. Both php-fpm are sharing a volume mounted from local system, but on separate paths. The php-fpm container files are serving as api webservices. Both are having different set of files, but the path is the same. ie, the php files will be on this path: /home/apps/apis. Now lets the the name of the containers are nginx_wb, php_wbv1.0, php_wbv1.1 If the nginx receives the request for http://abc.com/request_phonenumber It forwards the requests to the php_wbv1.0 container with the url http://abc.com/request_phonenumber if the nginx receives the request for http://abc.com/v11/request_phonenumber It forwards the request to the php_wbv1.1 container with the url http://abc.com/request_phonenumber //Notice there is no v11 in the path. This is what I wanted, although now its working fine, I wanted to use the rewrite mechanism to do this, instead of hardcoding the SCRIPT_FILENAME. Thanks. On Tue, Aug 7, 2018 at 4:59 PM Reinis Rozitis wrote: > > Hello, > > Thanks for that input, I changed my config like you said and also set > the fastcgi_param SCRIPT_FILENAME to /path/to/the/index.php; > > > But I feel like this should be done by the rewrite directive instead of > this way. Now that this is working, can someone please tell me if I can use > the rewrite tag so that I can avoid > > hardcoding the path everywhere? > > > Well you didn't specify the actual rewrite. > > Also from the initial mail the actual setup is a bit unclear - if the > docker is only running php-fpm (and the nginx is on host) or both nginx and > php are inside docker, but I assume it's the first. > > As an idea - while you can mangle/rewrite the root/path in the nginx > config, when configuring wordpress (+php-fpm) to run inside a docker > container I found more easy just to set the container volume to have the > same path (mount point) as the host server. So while isolated, both (nginx > on host and php inside container) have identical directory tree. > > rr > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx > -- Sincerely, Plato P about.me/plato_p -------------- next part -------------- An HTML attachment was scrubbed... URL: From mustafa at aldemir.net Wed Aug 8 14:06:41 2018 From: mustafa at aldemir.net (Mustafa Aldemir) Date: Wed, 8 Aug 2018 16:06:41 +0200 Subject: How to install njs after building from sources Message-ID: Hello, I built NGINX and njs from sources. Now I have njs executable and some static object files under ~/njs/build. How should I install njs or configure NGINX to run with njs? regards, mustafa -------------- next part -------------- An HTML attachment was scrubbed... URL: From xeioex at nginx.com Wed Aug 8 14:11:15 2018 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Wed, 8 Aug 2018 17:11:15 +0300 Subject: How to install njs after building from sources In-Reply-To: References: Message-ID: <9da1bb14-a05d-98a8-df47-503d50616973@nginx.com> On 08.08.2018 17:06, Mustafa Aldemir wrote: > Hello, > > I built NGINX and njs?from sources. Now I have njs?executable and some > static object files under ~/njs/build. > > How should I install njs?or configure NGINX to run with njs? Hi Mustafa, Please, use the instruction from here http://nginx.org/en/docs/njs_about.html#install_sources > > regards, > mustafa > > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx > From fugee279 at gmail.com Wed Aug 8 22:42:42 2018 From: fugee279 at gmail.com (fugee ohu) Date: Wed, 8 Aug 2018 18:42:42 -0400 Subject: HTTPS over port 443 In-Reply-To: <20180808071704.GD18403@daoine.org> References: <20180807222625.GC18403@daoine.org> <20180808071704.GD18403@daoine.org> Message-ID: I made a little progress What happens in a browser now is it says "ERR_TOO_MANY_REDIRECTS" $ curl -v sitename.com * Rebuilt URL to: sitename.com/ * Trying 108.41.240.225... * TCP_NODELAY set * Connected to sitename.com (108.41.240.225) port 80 (#0) > GET / HTTP/1.1 > Host: sitename.com > User-Agent: curl/7.58.0 > Accept: */* > < HTTP/1.1 301 Moved Permanently < Server: nginx/1.14.0 < Date: Wed, 08 Aug 2018 22:39:54 GMT < Content-Type: text/html < Content-Length: 185 < Connection: keep-alive < Location: https://sitename.com/ < 301 Moved Permanently

301 Moved Permanently


nginx/1.14.0
* Connection #0 to host sitename.com left intact On Wed, Aug 8, 2018 at 3:17 AM, Francis Daly wrote: > On Tue, Aug 07, 2018 at 07:20:58PM -0400, fugee ohu wrote: >> I don't think there's rails passenger settings Adding passenger to the >> Gemfile only causes the app to use passenger in development mode >> instead of the default (presently puma) > > Ok. > > What is the output of > > curl -v https://www.sitename.com > > f > -- > Francis Daly francis at daoine.org > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx From fugee279 at gmail.com Wed Aug 8 22:58:29 2018 From: fugee279 at gmail.com (fugee ohu) Date: Wed, 8 Aug 2018 18:58:29 -0400 Subject: HTTPS over port 443 In-Reply-To: <20180808071704.GD18403@daoine.org> References: <20180807222625.GC18403@daoine.org> <20180808071704.GD18403@daoine.org> Message-ID: server { listen 80; listen [::]:80; return 301 https://$host$request_uri; ssl_certificate /etc/ssl/certs/sitename/certificate.crt; ssl_certificate_key /etc/ssl/certs/sitename/private.key; server_name *.sitename.com www.sitename.com sitename.co$ charset utf-8; location / { root /usr/home/fugee/websites/sitename/public; rails_env production; # passenger_enabled on; } } On Wed, Aug 8, 2018 at 3:17 AM, Francis Daly wrote: > On Tue, Aug 07, 2018 at 07:20:58PM -0400, fugee ohu wrote: >> I don't think there's rails passenger settings Adding passenger to the >> Gemfile only causes the app to use passenger in development mode >> instead of the default (presently puma) > > Ok. > > What is the output of > > curl -v https://www.sitename.com > > f > -- > Francis Daly francis at daoine.org > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx From fugee279 at gmail.com Thu Aug 9 00:57:00 2018 From: fugee279 at gmail.com (fugee ohu) Date: Wed, 8 Aug 2018 20:57:00 -0400 Subject: Trying to setup SSL - Error too many redirects Message-ID: server { listen 80; listen [::]:80; return 301 https://$host$request_uri; ssl_certificate /etc/ssl/certs/bluegrasscounty/certificate.crt; ssl_certificate_key /etc/ssl/certs/bluegrasscounty/private.key; server_name *.bluegrasscounty.com www.bluegrasscounty.com bluegrasscounty.com; charset utf-8; location / { root /usr/home/fugee/websites/bluegrasscounty/public; rails_env production; # passenger_enabled on; } } From francis at daoine.org Thu Aug 9 07:30:11 2018 From: francis at daoine.org (Francis Daly) Date: Thu, 9 Aug 2018 08:30:11 +0100 Subject: HTTPS over port 443 In-Reply-To: References: <20180807222625.GC18403@daoine.org> <20180808071704.GD18403@daoine.org> Message-ID: <20180809073011.GF18403@daoine.org> On Wed, Aug 08, 2018 at 06:42:42PM -0400, fugee ohu wrote: Hi there, > I made a little progress What happens in a browser now is it says > "ERR_TOO_MANY_REDIRECTS" It's good that you have a definite error message. > $ curl -v sitename.com ... > < HTTP/1.1 301 Moved Permanently > < Location: https://sitename.com/ That is the output of curl -v http://sitename.com/ What is the output of curl -v https://sitename.com/ ? (And if that is a http 301 redirect to another Location, what is the output of "curl -v that-location"?) f -- Francis Daly francis at daoine.org From mustafa at aldemir.net Thu Aug 9 09:34:36 2018 From: mustafa at aldemir.net (Mustafa Aldemir) Date: Thu, 9 Aug 2018 11:34:36 +0200 Subject: How to install njs after building from sources In-Reply-To: <9da1bb14-a05d-98a8-df47-503d50616973@nginx.com> References: <9da1bb14-a05d-98a8-df47-503d50616973@nginx.com> Message-ID: Hello Dmitry, I followed the guides. To build njs: *./configure --with-compat* *make* To build and install NGINX: *./auto/configure --sbin-path=/usr/local/nginx/nginx --conf-path=/usr/local/nginx/nginx.conf --pid-path=/usr/local/nginx/nginx.pid --with-http_ssl_module --with-pcre=../pcre-8.41 --with-zlib=../zlib-1.2.11 --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-compat --add-dynamic-module=/home/ubuntu/patched/njs/nginx --with-debug* *make modules* *make* *sudo make install* Then, I put a configuration and Javascript file which work well with a pre-built installation. When I start NGINX, I get an error: nginx: [emerg] failed to create JS VM in /usr/local/nginx/nginx.conf:14 The 14th line of nginx.conf is: js_include mqtt.js; bye, mustafa nginx.conf: *worker_processes 1;* *error_log /var/log/nginx/error.log warn;* *pid /var/run/nginx.pid;* *load_module modules/ngx_stream_js_module.so;* *events {* * worker_connections 1024;* *}* *stream {* * js_include mqtt.js;* * js_set $mqtt_client_id setClientId;* * upstream awsiot {* * server iot.us-east-1.amazonaws.com:8883 ;* * hash $mqtt_client_id consistent;* * }* * log_format mqtt '$remote_addr [$time_local] $protocol $status $bytes_received'* * '$bytes_sent $upstream_addr $mqtt_client_id'; # Include MQTT ClientId* * server {* * listen 1883;* * preread_buffer_size 1k; # Big enough to read CONNECT packet header* * tcp_nodelay on;* * js_preread getClientId; # Parse CONNECT packet for ClientId* * js_filter filter;* * proxy_pass awsiot;* * proxy_ssl on;* * proxy_ssl_certificate nginx-as-client.pem;* * proxy_ssl_certificate_key nginx-as-client.key;* * proxy_ssl_trusted_certificate root-ca-for-awsiot.pem;* * proxy_ssl_verify off;* * proxy_ssl_session_reuse off;* * access_log /var/log/nginx/mqtt_access.log mqtt;* * error_log /var/log/nginx/mqtt_error.log debug;* * }* *}* On Wed, Aug 8, 2018 at 4:11 PM, Dmitry Volyntsev wrote: > > > On 08.08.2018 17:06, Mustafa Aldemir wrote: > >> Hello, >> >> I built NGINX and njs from sources. Now I have njs executable and some >> static object files under ~/njs/build. >> >> How should I install njs or configure NGINX to run with njs? >> > > > Hi Mustafa, > > Please, use the instruction from here > http://nginx.org/en/docs/njs_about.html#install_sources > > > > > >> regards, >> mustafa >> >> >> _______________________________________________ >> nginx mailing list >> nginx at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx >> >> -- Mustafa Aldemir, http://mustafa.aldemir.net -------------- next part -------------- An HTML attachment was scrubbed... URL: From nginx-forum at forum.nginx.org Thu Aug 9 19:14:33 2018 From: nginx-forum at forum.nginx.org (jlangr) Date: Thu, 09 Aug 2018 15:14:33 -0400 Subject: Using the mirror module In-Reply-To: <20180313213456.GB832@Romans-MacBook-Air.local> References: <20180313213456.GB832@Romans-MacBook-Air.local> Message-ID: Thanks for the posts Kenny & pbooth. I have this problem as well. I am running nginx 1.15 under MacOS and also experience this under Windows. Requests successfully go to the server specified under the location but are not received at the proxy_pass defined in the mirror location's proxy_pass. Switching between the two servers (swapping the mirror for the proxy_pass in the primary location) exhibits the same problem. My config is: ``` http { server { listen 3333; server_name localhost; location / { mirror /other; proxy_pass http://localhost:3001; } location = /other { internal; proxy_pass http://localhost:3002; } } } events { } ``` Based on your other post, it does not look like I need a resolver. I've also tried with two servers--one localhost and one established server on another machine. Any help greatly appreciated! Jeff Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279036,280828#msg-280828 From nginx-forum at forum.nginx.org Thu Aug 9 21:08:25 2018 From: nginx-forum at forum.nginx.org (jlangr) Date: Thu, 09 Aug 2018 17:08:25 -0400 Subject: Using the mirror module In-Reply-To: References: <20180313213456.GB832@Romans-MacBook-Air.local> Message-ID: <719ab521b2bf2a53b842ab7ddef0d918.NginxMailingListEnglish@forum.nginx.org> hmm looks like the mirror kills the url. I need to understand how to do URL rewrites, probably, but the original request's path isn't coming across to the mirror location. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279036,280829#msg-280829 From francis at daoine.org Thu Aug 9 22:27:53 2018 From: francis at daoine.org (Francis Daly) Date: Thu, 9 Aug 2018 23:27:53 +0100 Subject: Using the mirror module In-Reply-To: <719ab521b2bf2a53b842ab7ddef0d918.NginxMailingListEnglish@forum.nginx.org> References: <20180313213456.GB832@Romans-MacBook-Air.local> <719ab521b2bf2a53b842ab7ddef0d918.NginxMailingListEnglish@forum.nginx.org> Message-ID: <20180809222753.GH18403@daoine.org> On Thu, Aug 09, 2018 at 05:08:25PM -0400, jlangr wrote: Hi there, > hmm looks like the mirror kills the url. I need to understand how to do URL > rewrites, probably, but the original request's path isn't coming across to > the mirror location. Does it work for you if you use the Example Configuration in the documentation? http://nginx.org/en/docs/http/ngx_http_mirror_module.html f -- Francis Daly francis at daoine.org From nginx-forum at forum.nginx.org Thu Aug 9 23:04:50 2018 From: nginx-forum at forum.nginx.org (jlangr) Date: Thu, 09 Aug 2018 19:04:50 -0400 Subject: Using the mirror module In-Reply-To: <20180809222753.GH18403@daoine.org> References: <20180809222753.GH18403@daoine.org> Message-ID: Hi Francis-- We got this working to wrap up the day. My ignorance of nginx meant that I was interpreting the example literally--I viewed the "/mirror" as the name for the internal route, but that actually becomes part of the URL the mirror location proxy forwards to. http { server { listen 3333; server_name localhost; location / { mirror /; resolver 8.8.8.8; # I shouldn't need this, will test tomorrow proxy_pass http://1.2.3.4:3001; } location = / { internal; resolver 8.8.8.8; set $upstream_endpoint http://1.2.3.4:3002; proxy_pass $upstream_endpoint$request_uri; } } } That works like a champ. Initially I was trying to use localhost as the proxy_pass destination, and had turned on port forwarding at my router, but for whatever reason that was having troubles. Many thanks in any case! Jeff Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279036,280832#msg-280832 From nginx-forum at forum.nginx.org Thu Aug 9 23:05:11 2018 From: nginx-forum at forum.nginx.org (jlangr) Date: Thu, 09 Aug 2018 19:05:11 -0400 Subject: Using the mirror module In-Reply-To: References: <20180809222753.GH18403@daoine.org> Message-ID: <8f188d81c3a9c5006134b24fb79e694a.NginxMailingListEnglish@forum.nginx.org> ps how do you format code in these posts? (It's not markdown) Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279036,280833#msg-280833 From pablo at pablo.com.mx Fri Aug 10 03:00:33 2018 From: pablo at pablo.com.mx (Pablo Fischer) Date: Thu, 9 Aug 2018 20:00:33 -0700 Subject: Error_log format (json) or suggestions? Message-ID: Howdy, I've a project that I need to send the error log but it only accepts json format (I know I know), in the past I used logstash but right now that is not an option so looking to see if anyone has done any patch or ways to go around it. Ideally this should either be any of: a) Log to current error_log as json (through a patch) b) Or log to the current error_log in the current format and have a separate process that processes it and creates a new log file with the json format (we are fine with ending with two files) Any ideas? Thanks! -- Pablo From anoopalias01 at gmail.com Fri Aug 10 06:42:20 2018 From: anoopalias01 at gmail.com (Anoop Alias) Date: Fri, 10 Aug 2018 12:12:20 +0530 Subject: posix_memalign error In-Reply-To: References: <20180731133813.GG56558@mdounin.ru> <4fa47f31-9c32-5386-7c2d-a4e9347b3dfc@nginx.com> <1fdf4937-cac6-0550-b725-36f04eff65db@nginx.com> Message-ID: I may have found the root cause of this issue. Many thanks to Igor for the valuable inputs The issue had something to do with the fact that I was calling nginx -s reload from python subprocess module and I believe the error was coming from the fork in python https://stackoverflow.com/questions/1367373/python-subprocess-popen-oserror-errno-12-cannot-allocate-memory https://stackoverflow.com/questions/5306075/python-memory-allocation-error-using-subprocess-popen As Igor suggested I changed the subprocess call in python to signal (SIGHUP to master binary) and the logs don't have the ENOMEM error anymore, at least in the past 12+ hours The memory usage for ~10k virtual hosts are a bit high though, but things do work as expected and no more errors Thank you all again On Wed, Aug 8, 2018 at 9:03 AM Anoop Alias wrote: > Hi Igor, > > Yes the server runs other software including httpd with a similar number > of vhost > > # grep " 5168 > > I haven't found issue with the other softwares in the logs relating to > memory > > Infact httpd (event mpm) use lesser memory to load similar config > > # ps_mem| head -1 && ps_mem |grep httpd > Private + Shared = RAM used Program > 585.6 MiB + 392.0 MiB = 977.6 MiB httpd (63) > > # ps_mem| head -1 && ps_mem |grep nginx > Private + Shared = RAM used Program > 999.8 MiB + 2.5 GiB = 3.5 GiB nginx (3) > > The server is a shared hosting one and runs CloudLinux , but as far as I > know ,CloudLinux applies limits to only user level process and not nginx > > The nginx HUP is needed as this is triggered by changes in apache > configuration and nginx need to reload the new config . For log file reload > SIGUSR1 is used > > > > > > On Tue, Aug 7, 2018 at 5:50 PM Igor A. Ippolitov > wrote: > >> Anoop, >> >> I don't see any troubles with your configuration. >> Also, if you have 120G of RAM and a single worker - the problem is not in >> nginx. >> Do you have other software running on the host? >> >> Basically, you just run out of memory. >> >> You can optimize your reload though: use "service nginx reload" (or "kill >> -SIGHUP") to reload nginx configuration. >> When you do nginx -s reload - you make nginx parse configuration (and it >> requires memory) and then send a signal to the running master. You can >> avoid this overhead with 'service' command as it uses 'kill' documented in >> the manual page. >> >> On 06.08.2018 22:55, Anoop Alias wrote: >> >> Hi Igor, >> >> Config is reloaded using >> >> /usr/sbin/nginx -s reload >> >> this is invoked from a python/shell script ( Nginx is installed on a web >> control panel ) >> >> The top-level Nginx config is in the gist below >> >> https://gist.github.com/AnoopAlias/ba5ad6749a586c7e267672ee65b32b3a >> >> It further includes ~8k server blocks or more in some servers. Out of >> this 2/3 are server {} blocks with TLS config and 1/3 non-TLS ones >> >> ]# pwd >> /etc/nginx/sites-enabled >> # grep "server {" *|wc -l >> 7886 >> >> And yes most of them are very similar and mostly proxy to upstream httpd >> >> I have tried removing all the loadable modules and even tried an older >> version of nginx and all produce the error >> >> >> # numastat -m >> >> Per-node system memory usage (in MBs): >> Node 0 Node 1 Total >> --------------- --------------- --------------- >> MemTotal 65430.84 65536.00 130966.84 >> MemFree 5491.26 40.89 5532.15 >> MemUsed 59939.58 65495.11 125434.69 >> Active 22295.61 21016.09 43311.70 >> Inactive 8742.76 4662.48 13405.24 >> Active(anon) 16717.10 16572.19 33289.29 >> Inactive(anon) 2931.94 1388.14 4320.08 >> Active(file) 5578.50 4443.91 10022.41 >> Inactive(file) 5810.82 3274.34 9085.16 >> Unevictable 0.00 0.00 0.00 >> Mlocked 0.00 0.00 0.00 >> Dirty 7.04 1.64 8.67 >> Writeback 0.00 0.00 0.00 >> FilePages 18458.93 10413.97 28872.90 >> Mapped 862.14 413.38 1275.52 >> AnonPages 12579.49 15264.37 27843.86 >> Shmem 7069.52 2695.71 9765.23 >> KernelStack 18.34 3.03 21.38 >> PageTables 153.14 107.77 260.90 >> NFS_Unstable 0.00 0.00 0.00 >> Bounce 0.00 0.00 0.00 >> WritebackTmp 0.00 0.00 0.00 >> Slab 4830.68 2254.55 7085.22 >> SReclaimable 2061.05 921.72 2982.77 >> SUnreclaim 2769.62 1332.83 4102.45 >> AnonHugePages 4.00 2.00 6.00 >> HugePages_Total 0.00 0.00 0.00 >> HugePages_Free 0.00 0.00 0.00 >> HugePages_Surp 0.00 0.00 0.00 >> >> >> Thanks, >> >> >> >> >> >> On Mon, Aug 6, 2018 at 6:33 PM Igor A. Ippolitov >> wrote: >> >>> Anoop, >>> >>> I suppose, most of your 10k servers are very similar, right? >>> Please, post top level configuration and a typical server{}, please. >>> >>> Also, how do you reload configuration? With 'service nginx reload' or >>> may be other commands? >>> >>> It looks like you have a lot of fragmented memory and only 4gb free in >>> the second numa node. >>> So, I'd say this is OK that you are getting errors from allocating a 16k >>> stripes. >>> >>> Could you please post numastat -m output additionally. Just to make sure >>> you have half of the memory for the second CPU. >>> And we'll have a look if memory utilization may be optimized based on >>> your configuration. >>> >>> Regards, >>> Igor. >>> >>> On 04.08.2018 07:54, Anoop Alias wrote: >>> >>> Hi Igor, >>> >>> Setting vm.max_map_count to 20x the normal value did not help >>> >>> The issue happens on a group of servers and among the group, it shows up >>> only in servers which have ~10k server{} blocks >>> >>> On servers that have lower number of server{} blocks , the ENOMEM issue >>> is not there >>> >>> Also, I can find that the RAM usage of the Nginx process is directly >>> proportional to the number of server {} blocks >>> >>> For example on a server having the problem >>> >>> # ps_mem| head -1 && ps_mem |grep nginx >>> Private + Shared = RAM used Program >>> 1.0 GiB + 2.8 GiB = 3.8 GiB nginx (3) >>> >>> >>> That is for a single worker process with 4 threads in thread_pool >>> # pstree|grep nginx >>> |-nginx-+-nginx---4*[{nginx}] >>> | `-nginx >>> >>> Whatever config change I try the memory usage seem to mostly depend on >>> the number of server contexts defined >>> >>> Now the issue mostly happen in nginx reload ,when one more worker >>> process will be active in shutting down mode >>> >>> I believe the memalign error is thrown by the worker being shutdown, >>> this is because the sites work after the error and also the pid mentioned >>> in the error would have gone when I check ps >>> >>> >>> # pmap 948965|grep 16K >>> 00007f2923ff2000 16K r-x-- ngx_http_redis2_module.so >>> 00007f2924fd7000 16K r---- libc-2.17.so >>> 00007f2925431000 16K rw--- [ anon ] >>> 00007f292584a000 16K rw--- [ anon ] >>> >>> Aug 4 05:50:00 b kernel: SysRq : Show Memory >>> Aug 4 05:50:00 b kernel: Mem-Info: >>> Aug 4 05:50:00 b kernel: active_anon:7757394 inactive_anon:1021319 >>> isolated_anon:0#012 active_file:3733324 inactive_file:2136476 isolated_ >>> file:0#012 unevictable:0 dirty:1766 writeback:6 wbtmp:0 unstable:0#012 >>> slab_reclaimable:2003687 slab_unreclaimable:901391#012 mapped:316734 >>> shmem:2381810 pagetables:63163 bounce:0#012 free:4851283 free_pcp:11332 >>> free_cma:0 >>> Aug 4 05:50:00 bravo kernel: Node 0 DMA free:15888kB min:8kB low:8kB >>> high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_ >>> file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB >>> present:15972kB managed:15888kB mlocked:0kB dirty:0kB writeback:0kB >>> mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB >>> kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB >>> local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 >>> all_unreclaimable? yes >>> Aug 4 05:50:00 b kernel: lowmem_reserve[]: 0 1679 64139 64139 >>> >>> # cat /proc/buddyinfo >>> Node 0, zone DMA 0 0 1 0 2 1 1 >>> 0 1 1 3 >>> Node 0, zone DMA32 5284 6753 6677 1083 410 59 1 >>> 0 0 0 0 >>> Node 0, zone Normal 500327 638958 406737 14690 872 106 11 >>> 0 0 0 0 >>> Node 1, zone Normal 584840 291640 188 0 0 0 0 >>> 0 0 0 0 >>> >>> >>> The only correlation I see in having the error is the number of >>> server {} blocks (close to 10k) which then makes the nginx process consume >>> ~ 4GB of mem with a single worker process and then a reload is done >>> >>> >>> >>> >>> On Thu, Aug 2, 2018 at 6:02 PM Igor A. Ippolitov >>> wrote: >>> >>>> Anoop, >>>> >>>> There are two guesses: either mmap allocations limit is hit or memory >>>> is way too fragmented. >>>> Could you please track amount of mapped regions for a worker with pmap >>>> and amount of 16k areas in Normal zones (it is the third number)? >>>> >>>> You can also set vm.max_map_count to a higher number (like 20 times >>>> higher than default) and look if the error is gone. >>>> >>>> Please, let me know if increasing vm.max_map_count helps you. >>>> >>>> On 02.08.2018 13:06, Anoop Alias wrote: >>>> >>>> Hi Igor, >>>> >>>> The error happens randomly >>>> >>>> 2018/08/02 06:52:42 [emerg] 874514#874514: posix_memalign(16, 16384) >>>> failed (12: Cannot allocate memory) >>>> 2018/08/02 09:42:53 [emerg] 872996#872996: posix_memalign(16, 16384) >>>> failed (12: Cannot allocate memory) >>>> 2018/08/02 10:16:14 [emerg] 877611#877611: posix_memalign(16, 16384) >>>> failed (12: Cannot allocate memory) >>>> 2018/08/02 10:16:48 [emerg] 879410#879410: posix_memalign(16, 16384) >>>> failed (12: Cannot allocate memory) >>>> 2018/08/02 10:17:55 [emerg] 876563#876563: posix_memalign(16, 16384) >>>> failed (12: Cannot allocate memory) >>>> 2018/08/02 10:20:21 [emerg] 879263#879263: posix_memalign(16, 16384) >>>> failed (12: Cannot allocate memory) >>>> 2018/08/02 10:20:51 [emerg] 878991#878991: posix_memalign(16, 16384) >>>> failed (12: Cannot allocate memory) >>>> >>>> # date >>>> Thu Aug 2 10:58:48 BST 2018 >>>> >>>> ------------------------------------------ >>>> # cat /proc/buddyinfo >>>> Node 0, zone DMA 0 0 1 0 2 1 1 >>>> 0 1 1 3 >>>> Node 0, zone DMA32 11722 11057 4663 1647 609 72 10 >>>> 7 1 0 0 >>>> Node 0, zone Normal 755026 710760 398136 21462 1114 18 1 >>>> 0 0 0 0 >>>> Node 1, zone Normal 341295 801810 179604 256 0 0 0 >>>> 0 0 0 0 >>>> ----------------------------------------- >>>> >>>> >>>> slabinfo - version: 2.1 >>>> # name >>>> : tunables : slabdata >>>> >>>> SCTPv6 21 21 1536 21 8 : tunables 0 0 >>>> 0 : slabdata 1 1 0 >>>> SCTP 0 0 1408 23 8 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> kcopyd_job 0 0 3312 9 8 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> dm_uevent 0 0 2608 12 8 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> nf_conntrack_ffffffff81acbb00 14054 14892 320 51 4 : >>>> tunables 0 0 0 : slabdata 292 292 0 >>>> lvp_cache 36 36 224 36 2 : tunables 0 0 >>>> 0 : slabdata 1 1 0 >>>> lve_struct 4140 4140 352 46 4 : tunables 0 0 >>>> 0 : slabdata 90 90 0 >>>> fat_inode_cache 0 0 744 44 8 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> fat_cache 0 0 40 102 1 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> isofs_inode_cache 0 0 664 49 8 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> ext4_inode_cache 30 30 1088 30 8 : tunables 0 0 >>>> 0 : slabdata 1 1 0 >>>> ext4_xattr 0 0 88 46 1 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> ext4_free_data 0 0 64 64 1 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> ext4_allocation_context 32 32 128 32 1 : tunables 0 >>>> 0 0 : slabdata 1 1 0 >>>> ext4_io_end 0 0 72 56 1 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> ext4_extent_status 102 102 40 102 1 : tunables 0 0 >>>> 0 : slabdata 1 1 0 >>>> jbd2_journal_handle 0 0 48 85 1 : tunables 0 >>>> 0 0 : slabdata 0 0 0 >>>> jbd2_journal_head 0 0 112 36 1 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> jbd2_revoke_table_s 256 256 16 256 1 : tunables 0 >>>> 0 0 : slabdata 1 1 0 >>>> jbd2_revoke_record_s 0 0 32 128 1 : tunables 0 >>>> 0 0 : slabdata 0 0 0 >>>> kvm_async_pf 0 0 136 30 1 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> kvm_vcpu 0 0 18560 1 8 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> xfs_dqtrx 992 992 528 31 4 : tunables 0 0 >>>> 0 : slabdata 32 32 0 >>>> xfs_dquot 3264 3264 472 34 4 : tunables 0 0 >>>> 0 : slabdata 96 96 0 >>>> xfs_ili 4342175 4774399 152 53 2 : tunables 0 >>>> 0 0 : slabdata 90083 90083 0 >>>> xfs_inode 4915588 5486076 1088 30 8 : tunables 0 >>>> 0 0 : slabdata 182871 182871 0 >>>> xfs_efd_item 2680 2760 400 40 4 : tunables 0 0 >>>> 0 : slabdata 69 69 0 >>>> xfs_da_state 1088 1088 480 34 4 : tunables 0 0 >>>> 0 : slabdata 32 32 0 >>>> xfs_btree_cur 1248 1248 208 39 2 : tunables 0 0 >>>> 0 : slabdata 32 32 0 >>>> xfs_log_ticket 14874 15048 184 44 2 : tunables 0 0 >>>> 0 : slabdata 342 342 0 >>>> xfs_ioend 12909 13104 104 39 1 : tunables 0 0 >>>> 0 : slabdata 336 336 0 >>>> scsi_cmd_cache 5400 5652 448 36 4 : tunables 0 0 >>>> 0 : slabdata 157 157 0 >>>> ve_struct 0 0 848 38 8 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> ip6_dst_cache 1152 1152 448 36 4 : tunables 0 0 >>>> 0 : slabdata 32 32 0 >>>> RAWv6 910 910 1216 26 8 : tunables 0 0 >>>> 0 : slabdata 35 35 0 >>>> UDPLITEv6 0 0 1216 26 8 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> UDPv6 832 832 1216 26 8 : tunables 0 0 >>>> 0 : slabdata 32 32 0 >>>> tw_sock_TCPv6 1152 1376 256 32 2 : tunables 0 0 >>>> 0 : slabdata 43 43 0 >>>> TCPv6 510 510 2176 15 8 : tunables 0 0 >>>> 0 : slabdata 34 34 0 >>>> cfq_queue 3698 5145 232 35 2 : tunables 0 0 >>>> 0 : slabdata 147 147 0 >>>> bsg_cmd 0 0 312 52 4 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> mqueue_inode_cache 136 136 960 34 8 : tunables 0 0 >>>> 0 : slabdata 4 4 0 >>>> hugetlbfs_inode_cache 1632 1632 632 51 8 : tunables 0 >>>> 0 0 : slabdata 32 32 0 >>>> configfs_dir_cache 1472 1472 88 46 1 : tunables 0 0 >>>> 0 : slabdata 32 32 0 >>>> dquot 0 0 256 32 2 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> userfaultfd_ctx_cache 32 32 128 32 1 : tunables 0 >>>> 0 0 : slabdata 1 1 0 >>>> fanotify_event_info 2336 2336 56 73 1 : tunables 0 >>>> 0 0 : slabdata 32 32 0 >>>> dio 6171 6222 640 51 8 : tunables 0 0 >>>> 0 : slabdata 122 122 0 >>>> pid_namespace 42 42 2192 14 8 : tunables 0 0 >>>> 0 : slabdata 3 3 0 >>>> posix_timers_cache 1056 1056 248 33 2 : tunables 0 0 >>>> 0 : slabdata 32 32 0 >>>> UDP-Lite 0 0 1088 30 8 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> flow_cache 2268 2296 144 28 1 : tunables 0 0 >>>> 0 : slabdata 82 82 0 >>>> xfrm_dst_cache 896 896 576 28 4 : tunables 0 0 >>>> 0 : slabdata 32 32 0 >>>> ip_fib_alias 2720 2720 48 85 1 : tunables 0 0 >>>> 0 : slabdata 32 32 0 >>>> RAW 3977 4224 1024 32 8 : tunables 0 0 >>>> 0 : slabdata 132 132 0 >>>> UDP 4110 4110 1088 30 8 : tunables 0 0 >>>> 0 : slabdata 137 137 0 >>>> tw_sock_TCP 4756 5216 256 32 2 : tunables 0 0 >>>> 0 : slabdata 163 163 0 >>>> TCP 2705 2768 1984 16 8 : tunables 0 0 >>>> 0 : slabdata 173 173 0 >>>> scsi_data_buffer 5440 5440 24 170 1 : tunables 0 0 >>>> 0 : slabdata 32 32 0 >>>> blkdev_queue 154 154 2208 14 8 : tunables 0 0 >>>> 0 : slabdata 11 11 0 >>>> blkdev_requests 4397688 4405884 384 42 4 : tunables 0 >>>> 0 0 : slabdata 104902 104902 0 >>>> blkdev_ioc 11232 11232 112 36 1 : tunables 0 0 >>>> 0 : slabdata 312 312 0 >>>> user_namespace 0 0 1304 25 8 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> sock_inode_cache 12282 12282 704 46 8 : tunables 0 0 >>>> 0 : slabdata 267 267 0 >>>> file_lock_cache 20056 20960 200 40 2 : tunables 0 0 >>>> 0 : slabdata 524 524 0 >>>> net_namespace 6 6 5056 6 8 : tunables 0 0 >>>> 0 : slabdata 1 1 0 >>>> shmem_inode_cache 16970 18952 712 46 8 : tunables 0 0 >>>> 0 : slabdata 412 412 0 >>>> Acpi-ParseExt 39491 40432 72 56 1 : tunables 0 0 >>>> 0 : slabdata 722 722 0 >>>> Acpi-State 1683 1683 80 51 1 : tunables 0 0 >>>> 0 : slabdata 33 33 0 >>>> Acpi-Namespace 11424 11424 40 102 1 : tunables 0 0 >>>> 0 : slabdata 112 112 0 >>>> task_delay_info 15336 15336 112 36 1 : tunables 0 0 >>>> 0 : slabdata 426 426 0 >>>> taskstats 1568 1568 328 49 4 : tunables 0 0 >>>> 0 : slabdata 32 32 0 >>>> proc_inode_cache 169897 190608 680 48 8 : tunables 0 0 >>>> 0 : slabdata 3971 3971 0 >>>> sigqueue 2208 2208 168 48 2 : tunables 0 0 >>>> 0 : slabdata 46 46 0 >>>> bdev_cache 792 792 896 36 8 : tunables 0 0 >>>> 0 : slabdata 22 22 0 >>>> sysfs_dir_cache 74698 74698 120 34 1 : tunables 0 0 >>>> 0 : slabdata 2197 2197 0 >>>> mnt_cache 163197 163424 256 32 2 : tunables 0 0 >>>> 0 : slabdata 5107 5107 0 >>>> filp 64607 97257 320 51 4 : tunables 0 0 >>>> 0 : slabdata 1907 1907 0 >>>> inode_cache 370744 370947 616 53 8 : tunables 0 0 >>>> 0 : slabdata 6999 6999 0 >>>> dentry 1316262 2139228 192 42 2 : tunables 0 >>>> 0 0 : slabdata 50934 50934 0 >>>> iint_cache 0 0 80 51 1 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> buffer_head 1441470 2890290 104 39 1 : tunables 0 >>>> 0 0 : slabdata 74110 74110 0 >>>> vm_area_struct 194998 196840 216 37 2 : tunables 0 0 >>>> 0 : slabdata 5320 5320 0 >>>> mm_struct 2679 2760 1600 20 8 : tunables 0 0 >>>> 0 : slabdata 138 138 0 >>>> files_cache 8680 8925 640 51 8 : tunables 0 0 >>>> 0 : slabdata 175 175 0 >>>> signal_cache 3691 3780 1152 28 8 : tunables 0 0 >>>> 0 : slabdata 135 135 0 >>>> sighand_cache 1950 2160 2112 15 8 : tunables 0 0 >>>> 0 : slabdata 144 144 0 >>>> task_xstate 8070 8658 832 39 8 : tunables 0 0 >>>> 0 : slabdata 222 222 0 >>>> task_struct 1913 2088 4080 8 8 : tunables 0 0 >>>> 0 : slabdata 261 261 0 >>>> cred_jar 31699 33936 192 42 2 : tunables 0 0 >>>> 0 : slabdata 808 808 0 >>>> anon_vma_chain 164026 168704 64 64 1 : tunables 0 0 >>>> 0 : slabdata 2636 2636 0 >>>> anon_vma 84104 84594 88 46 1 : tunables 0 0 >>>> 0 : slabdata 1839 1839 0 >>>> pid 11127 12576 128 32 1 : tunables 0 0 >>>> 0 : slabdata 393 393 0 >>>> shared_policy_node 9350 9350 48 85 1 : tunables 0 0 >>>> 0 : slabdata 110 110 0 >>>> numa_policy 62 62 264 31 2 : tunables 0 0 >>>> 0 : slabdata 2 2 0 >>>> radix_tree_node 771778 1194312 584 28 4 : tunables 0 0 >>>> 0 : slabdata 42654 42654 0 >>>> idr_layer_cache 2538 2565 2112 15 8 : tunables 0 0 >>>> 0 : slabdata 171 171 0 >>>> dma-kmalloc-8192 0 0 8192 4 8 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> dma-kmalloc-4096 0 0 4096 8 8 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> dma-kmalloc-2048 0 0 2048 16 8 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> dma-kmalloc-1024 0 0 1024 32 8 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> dma-kmalloc-512 0 0 512 32 4 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> dma-kmalloc-256 0 0 256 32 2 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> dma-kmalloc-128 0 0 128 32 1 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> dma-kmalloc-64 0 0 64 64 1 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> dma-kmalloc-32 0 0 32 128 1 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> dma-kmalloc-16 0 0 16 256 1 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> dma-kmalloc-8 0 0 8 512 1 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> dma-kmalloc-192 0 0 192 42 2 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> dma-kmalloc-96 0 0 96 42 1 : tunables 0 0 >>>> 0 : slabdata 0 0 0 >>>> kmalloc-8192 385 388 8192 4 8 : tunables 0 0 >>>> 0 : slabdata 97 97 0 >>>> kmalloc-4096 9296 10088 4096 8 8 : tunables 0 0 >>>> 0 : slabdata 1261 1261 0 >>>> kmalloc-2048 65061 133536 2048 16 8 : tunables 0 0 >>>> 0 : slabdata 8346 8346 0 >>>> kmalloc-1024 11987 21120 1024 32 8 : tunables 0 0 >>>> 0 : slabdata 660 660 0 >>>> kmalloc-512 107510 187072 512 32 4 : tunables 0 0 >>>> 0 : slabdata 5846 5846 0 >>>> kmalloc-256 160498 199104 256 32 2 : tunables 0 0 >>>> 0 : slabdata 6222 6222 0 >>>> kmalloc-192 144975 237426 192 42 2 : tunables 0 0 >>>> 0 : slabdata 5653 5653 0 >>>> kmalloc-128 36799 108096 128 32 1 : tunables 0 0 >>>> 0 : slabdata 3378 3378 0 >>>> kmalloc-96 99510 238896 96 42 1 : tunables 0 0 >>>> 0 : slabdata 5688 5688 0 >>>> kmalloc-64 7978152 8593280 64 64 1 : tunables 0 >>>> 0 0 : slabdata 134270 134270 0 >>>> kmalloc-32 2939882 3089664 32 128 1 : tunables 0 >>>> 0 0 : slabdata 24138 24138 0 >>>> kmalloc-16 172057 172288 16 256 1 : tunables 0 0 >>>> 0 : slabdata 673 673 0 >>>> kmalloc-8 109568 109568 8 512 1 : tunables 0 0 >>>> 0 : slabdata 214 214 0 >>>> kmem_cache_node 893 896 64 64 1 : tunables 0 0 >>>> 0 : slabdata 14 14 0 >>>> kmem_cache 612 612 320 51 4 : tunables 0 0 >>>> 0 : slabdata 12 12 0 >>>> >>>> ------------------------------------------------- >>>> >>>> >>>> # uname -r >>>> 3.10.0-714.10.2.lve1.5.17.1.el7.x86_64 >>>> >>>> -------------------------------------------------------- >>>> >>>> Core part of glances >>>> http://i.imgur.com/La5JbQn.png >>>> ----------------------------------------------------------- >>>> >>>> Thank you very much for looking into this >>>> >>>> >>>> On Thu, Aug 2, 2018 at 12:37 PM Igor A. Ippolitov >>>> wrote: >>>> >>>>> Anoop, >>>>> >>>>> I doubt this will be the solution, but may we have a look at >>>>> /proc/buddyinfo and /proc/slabinfo the moment when nginx can't allocate >>>>> memory? >>>>> >>>>> On 02.08.2018 08:15, Anoop Alias wrote: >>>>> >>>>> Hi Maxim, >>>>> >>>>> I enabled debug and the memalign call is happening on nginx reloads >>>>> and the ENOMEM happen sometimes on the reload(not on all reloads) >>>>> >>>>> 2018/08/02 05:59:08 [notice] 872052#872052: signal process started >>>>> 2018/08/02 05:59:23 [notice] 871570#871570: signal 1 (SIGHUP) received >>>>> from 872052, reconfiguring >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: wake up, sigio 0 >>>>> 2018/08/02 05:59:23 [notice] 871570#871570: reconfiguring >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 0000000002B0DA00:16384 @16 === > the memalign call on reload >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: malloc: >>>>> 00000000087924D0:4560 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 000000000E442E00:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: malloc: >>>>> 0000000005650850:4096 >>>>> 20 >>>>> >>>>> >>>>> >>>>> >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #71 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #72 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #73 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: bind() xxxx:443 #74 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: add cleanup: >>>>> 000000005340D728 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>>> 00000000024D3260:4096 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>>> 00000000517BAF10:4096 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>>> 0000000053854FC0:4096 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>>> 0000000053855FD0:4096 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>>> 0000000053856FE0:4096 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>>> 0000000053857FF0:4096 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: posix_memalign: >>>>> 0000000053859000:16384 @16 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>>> 000000005385D010:4096 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>>> 000000005385E020:4096 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>>> 000000005385F030:4096 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>>> 00000000536CD160:4096 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>>> 00000000536CE170:4096 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>>> 00000000536CF180:4096 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>>> 00000000536D0190:4096 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>>> 00000000536D11A0:4096 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>>> 00000000536D21B0:4096 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>>> 00000000536D31C0:4096 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>>> 00000000536D41D0:4096 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>>> 00000000536D51E0:4096 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>>> 00000000536D61F0:4096 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>>> 00000000536D7200:4096 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>>> 00000000536D8210:4096 >>>>> 2018/08/02 05:48:49 [debug] 871275#871275: malloc: >>>>> 00000000536D9220:4096 >>>>> >>>>> >>>>> Infact there are lot of such calls during a reload >>>>> >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA17ED00:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA1B0FF0:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA1E12C0:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA211590:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA243880:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA271B30:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA2A3E20:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA2D20D0:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA3063E0:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA334690:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA366980:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA396C50:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA3C8F40:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA3F9210:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA4294E0:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA45B7D0:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA489A80:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA4BBD70:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA4EA020:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA51E330:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA54C5E0:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA57E8D0:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA5AEBA0:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA5DEE70:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA611160:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA641430:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA671700:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA6A29E0:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA6D5CE0:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA707FD0:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA736280:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA768570:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA796820:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA7CAB30:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA7F8DE0:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA82B0D0:16384 @16 >>>>> 2018/08/02 05:59:23 [debug] 871570#871570: posix_memalign: >>>>> 00000000BA85B3A0:16384 @16 >>>>> >>>>> >>>>> >>>>> What is perplexing is that the system has enough free (available RAM) >>>>> ############# >>>>> # free -g >>>>> total used free shared buff/cache >>>>> available >>>>> Mem: 125 54 24 8 46 >>>>> 58 >>>>> Swap: 0 0 0 >>>>> ############# >>>>> >>>>> # ulimit -a >>>>> core file size (blocks, -c) 0 >>>>> data seg size (kbytes, -d) unlimited >>>>> scheduling priority (-e) 0 >>>>> file size (blocks, -f) unlimited >>>>> pending signals (-i) 514579 >>>>> max locked memory (kbytes, -l) 64 >>>>> max memory size (kbytes, -m) unlimited >>>>> open files (-n) 1024 >>>>> pipe size (512 bytes, -p) 8 >>>>> POSIX message queues (bytes, -q) 819200 >>>>> real-time priority (-r) 0 >>>>> stack size (kbytes, -s) 8192 >>>>> cpu time (seconds, -t) unlimited >>>>> max user processes (-u) 514579 >>>>> virtual memory (kbytes, -v) unlimited >>>>> file locks (-x) unlimited >>>>> >>>>> ######################################### >>>>> >>>>> There is no other thing limiting memory allocation >>>>> >>>>> Any way to prevent this or probably identify/prevent this >>>>> >>>>> >>>>> On Tue, Jul 31, 2018 at 7:08 PM Maxim Dounin >>>>> wrote: >>>>> >>>>>> Hello! >>>>>> >>>>>> On Tue, Jul 31, 2018 at 09:52:29AM +0530, Anoop Alias wrote: >>>>>> >>>>>> > I am repeatedly seeing errors like >>>>>> > >>>>>> > ###################### >>>>>> > 2018/07/31 03:46:33 [emerg] 2854560#2854560: posix_memalign(16, >>>>>> 16384) >>>>>> > failed (12: Cannot allocate memory) >>>>>> > 2018/07/31 03:54:09 [emerg] 2890190#2890190: posix_memalign(16, >>>>>> 16384) >>>>>> > failed (12: Cannot allocate memory) >>>>>> > 2018/07/31 04:08:36 [emerg] 2939230#2939230: posix_memalign(16, >>>>>> 16384) >>>>>> > failed (12: Cannot allocate memory) >>>>>> > 2018/07/31 04:24:48 [emerg] 2992650#2992650: posix_memalign(16, >>>>>> 16384) >>>>>> > failed (12: Cannot allocate memory) >>>>>> > 2018/07/31 04:42:09 [emerg] 3053092#3053092: posix_memalign(16, >>>>>> 16384) >>>>>> > failed (12: Cannot allocate memory) >>>>>> > 2018/07/31 04:42:17 [emerg] 3053335#3053335: posix_memalign(16, >>>>>> 16384) >>>>>> > failed (12: Cannot allocate memory) >>>>>> > 2018/07/31 04:42:28 [emerg] 3053937#3053937: posix_memalign(16, >>>>>> 16384) >>>>>> > failed (12: Cannot allocate memory) >>>>>> > 2018/07/31 04:47:54 [emerg] 3070638#3070638: posix_memalign(16, >>>>>> 16384) >>>>>> > failed (12: Cannot allocate memory) >>>>>> > #################### >>>>>> > >>>>>> > on a few servers >>>>>> > >>>>>> > The servers have enough memory free and the swap usage is 0, yet >>>>>> somehow >>>>>> > the kernel denies the posix_memalign with ENOMEM ( this is what I >>>>>> think is >>>>>> > happening!) >>>>>> > >>>>>> > The numbers requested are always 16, 16k . This makes me suspicious >>>>>> > >>>>>> > I have no setting in nginx.conf that reference a 16k >>>>>> > >>>>>> > Is there any chance of finding out what requests this and why this >>>>>> is not >>>>>> > fulfilled >>>>>> >>>>>> There are at least some buffers which default to 16k - for >>>>>> example, ssl_buffer_size (http://nginx.org/r/ssl_buffer_size). >>>>>> >>>>>> You may try debugging log to futher find out where the particular >>>>>> allocation happens, see here for details: >>>>>> >>>>>> http://nginx.org/en/docs/debugging_log.html >>>>>> >>>>>> But I don't really think it worth the effort. The error is pretty >>>>>> clear, and it's better to focus on why these allocations are >>>>>> denied. Likely you are hitting some limit. >>>>>> >>>>>> -- >>>>>> Maxim Dounin >>>>>> http://mdounin.ru/ >>>>>> _______________________________________________ >>>>>> nginx mailing list >>>>>> nginx at nginx.org >>>>>> http://mailman.nginx.org/mailman/listinfo/nginx >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Anoop P Alias* >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> nginx mailing listnginx at nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx >>>>> >>>>> >>>>> _______________________________________________ >>>>> nginx mailing list >>>>> nginx at nginx.org >>>>> http://mailman.nginx.org/mailman/listinfo/nginx >>>> >>>> >>>> >>>> -- >>>> *Anoop P Alias* >>>> >>>> >>>> >>>> _______________________________________________ >>>> nginx mailing listnginx at nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx >>>> >>>> >>>> _______________________________________________ >>>> nginx mailing list >>>> nginx at nginx.org >>>> http://mailman.nginx.org/mailman/listinfo/nginx >>> >>> >>> >>> -- >>> *Anoop P Alias* >>> >>> >>> >>> _______________________________________________ >>> nginx mailing listnginx at nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx >>> >>> >>> _______________________________________________ >>> nginx mailing list >>> nginx at nginx.org >>> http://mailman.nginx.org/mailman/listinfo/nginx >> >> >> >> -- >> *Anoop P Alias* >> >> >> >> _______________________________________________ >> nginx mailing listnginx at nginx.orghttp://mailman.nginx.org/mailman/listinfo/nginx >> >> >> _______________________________________________ >> nginx mailing list >> nginx at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx > > > > -- > *Anoop P Alias* > > -- *Anoop P Alias* -------------- next part -------------- An HTML attachment was scrubbed... URL: From nginx-forum at forum.nginx.org Fri Aug 10 07:10:11 2018 From: nginx-forum at forum.nginx.org (hunterqin) Date: Fri, 10 Aug 2018 03:10:11 -0400 Subject: keepalive not work with grpc In-Reply-To: <545ddb40a6249a93f8594293a7a13005.NginxMailingListEnglish@forum.nginx.org> References: <545ddb40a6249a93f8594293a7a13005.NginxMailingListEnglish@forum.nginx.org> Message-ID: I found the problem. I used the wireshark to analyse the packets between nginx server and grpc server and I found that in the last packet from grpc server to nginx client, there are five frames,SETTINGS, HEADERS, DATA, HEADERS, WINDOW_UPDATE. The second HEADERS set the 'end_stream' and ngx_http_grpc_filter checked it (about the 2100th line in the ngx_http_grpc_module.c) if (ctx->end_stream) { u->length = 0; if (ctx->in == NULL && ctx->out == NULL && ctx->output_closed && b->last == b->pos) { u->keepalive = 1; } return NGX_OK; } everthing is true but b->last == b->pos because there is still a WINDOW_UPDATE frame in buffer and the difference between b->last and b->pos just is the length of WINDOW_UPDATE frame. That's why keepalive in the upstream not work, it is not set by the grpc module. Please check the bug. The keepalive problem also happens between grpc client and nginx but it is not so serious like this one but it needs your attention, too. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,280765,280836#msg-280836 From mdounin at mdounin.ru Fri Aug 10 10:17:44 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 10 Aug 2018 13:17:44 +0300 Subject: keepalive not work with grpc In-Reply-To: References: <545ddb40a6249a93f8594293a7a13005.NginxMailingListEnglish@forum.nginx.org> Message-ID: <20180810101744.GD56558@mdounin.ru> Hello! On Fri, Aug 10, 2018 at 03:10:11AM -0400, hunterqin wrote: > I found the problem. > I used the wireshark to analyse the packets between nginx server and grpc > server and I found that in the last packet from grpc server to nginx client, > there are five frames,SETTINGS, HEADERS, DATA, HEADERS, WINDOW_UPDATE. The > second HEADERS set the 'end_stream' and ngx_http_grpc_filter checked it > (about the 2100th line in the ngx_http_grpc_module.c) > if (ctx->end_stream) { > u->length = 0; > > if (ctx->in == NULL > && ctx->out == NULL > && ctx->output_closed > && b->last == b->pos) > { > u->keepalive = 1; > } > > return NGX_OK; > } > everthing is true but b->last == b->pos > because there is still a WINDOW_UPDATE frame in buffer and the difference > between b->last and b->pos just is the length of WINDOW_UPDATE frame. > That's why keepalive in the upstream not work, it is not set by the grpc > module. > Please check the bug. The keepalive problem also happens between grpc client > and nginx but it is not so serious like this one but it needs your > attention, too. With gRPC, much like with other protocols, connections are only preserved in keepalive cache if there are no pending frames to be sent or processed when the request is complete. It is expected that some connections might be closed if there are occasional pending frames. If you see keepalive not working for you at all, this might be because of unfortunate behaviour of your gRPC server. If this is indeed the case, please provide more details about the gRPC library you are using and the gRPC server code. An example code gRPC backend server would be ideal. -- Maxim Dounin http://mdounin.ru/ From satcse88 at gmail.com Fri Aug 10 11:38:49 2018 From: satcse88 at gmail.com (Sathish Kumar) Date: Fri, 10 Aug 2018 19:38:49 +0800 Subject: Multiple Domain CORS Message-ID: Hi All, I would like to use cloudfront.net content inside my webapp and its throwing Access-Control-Allow-Origin error and have added the header for single host on Nginx to make it work now. The problem is we have multiple environments which is trying to do the same and I have to whitelisted all domains. I don't want to add '*'. Chrome is throwing error and not accepting multiple values. Is there anyway to allow CORS domain like based on Host Origin.For Options, Get and other methods. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ano at bestmx.net Fri Aug 10 12:17:10 2018 From: ano at bestmx.net (Andrey Oktyabrskiy) Date: Fri, 10 Aug 2018 15:17:10 +0300 Subject: Multiple Domain CORS In-Reply-To: References: Message-ID: On 10.08.2018 14:38, Sathish Kumar wrote: > Is there anyway to allow CORS domain like based on Host Origin.For > Options, Get and other methods. Something like this should do what you want: location / { include inc/cors_options.inc; ... include inc/cors_headers.inc; } ### /etc/nginx/inc/cors_options.inc if ($request_method = 'OPTIONS') { add_header Access-Control-Allow-Credentials true; add_header Access-Control-Allow-Origin $cors_origin; add_header Access-Control-Allow-Methods OPTIONS; add_header Access-Control-Allow-Headers $http_access_control_request_headers; add_header Access-Control-Max-Age 86400; add_header Content-Type 'text/plain; charset=utf-8'; add_header Content-Length 0; return 204; } ### /etc/nginx/inc/cors_headers.inc add_header Access-Control-Allow-Credentials true always; add_header Access-Control-Allow-Origin $cors_origin always; add_header Access-Control-Allow-Methods $cors_method always; add_header Access-Control-Allow-Headers $http_access_control_request_headers always; add_header Access-Control-Max-Age 86400 always; ### /etc/nginx/sites-enabled/_map_cors_method map $http_access_control_request_method $cors_method { GET GET; HEAD HEAD; default OPTIONS; } $ cat /etc/nginx/sites-enabled/_map_cors_origin map $http_origin $cors_origin { https://domain.ru https://domain.ru; ... default ""; } From ano at bestmx.net Fri Aug 10 12:46:32 2018 From: ano at bestmx.net (Andrey Oktyabrskiy) Date: Fri, 10 Aug 2018 15:46:32 +0300 Subject: Multiple Domain CORS In-Reply-To: References: Message-ID: On 10.08.2018 15:17, Andrey Oktyabrskiy wrote: > ### /etc/nginx/inc/cors_options.inc > if ($request_method = 'OPTIONS') { > ? add_header Access-Control-Allow-Credentials??????? true; > ? add_header Access-Control-Allow-Origin???? $cors_origin; > ? add_header Access-Control-Allow-Methods??? OPTIONS; - add_header Access-Control-Allow-Methods OPTIONS; + add_header Access-Control-Allow-Methods $cors_method; From roger at netskrt.io Fri Aug 10 16:05:30 2018 From: roger at netskrt.io (Roger Fischer) Date: Fri, 10 Aug 2018 09:05:30 -0700 Subject: Actions after cache miss is detected Message-ID: Hello, Is there a way to perform an action after a cache miss is detected but before the request is forwarded to the upstream server? Specifically, on a cache miss I want to: Return a response instead of forwarding the request to the upstream server. Trigger a handler (module or script) that executes asynchronously to the request. The latter I could do by looking at the logs. But I was wondering if there is a more elegant way. Thanks? Roger -------------- next part -------------- An HTML attachment was scrubbed... URL: From arunachalamamsam at gmail.com Sun Aug 12 05:33:23 2018 From: arunachalamamsam at gmail.com (Arunachalam Parthasarathy) Date: Sun, 12 Aug 2018 11:03:23 +0530 Subject: Regarding nginx load balancing based on cpu, memory Message-ID: hi experts, We had tried hash type load balancing (explorer other types too), but we could not find a rule / config where it can load balance based on cpu or memory limit of application server Do we need write plugin ourselves for the same ? Thanks, -------------- next part -------------- An HTML attachment was scrubbed... URL: From ciapnz at gmail.com Sun Aug 12 20:33:25 2018 From: ciapnz at gmail.com (Danila Vershinin) Date: Sun, 12 Aug 2018 23:33:25 +0300 Subject: PROXY protocol to upstream server Message-ID: Hi, It seems that nginx can accept PROXY protocol fine, but when it comes to forwarding, it can only do so only within a stream { server { ? proxy_protocol on; } } . Are there any plans to add proxy_protocol on; for regular HTTP server blocks so it can be used alongside proxy_pass? This would come in very handy in a situation where NGINX is used as SSL terminator, e.g.: NGINX (SSL) ? (Proxy protocol) ? Varnish. Varnish supports accepting PROXY protocol. Best Regards, Danila -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From mdounin at mdounin.ru Sun Aug 12 21:12:24 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 13 Aug 2018 00:12:24 +0300 Subject: PROXY protocol to upstream server In-Reply-To: References: Message-ID: <20180812211224.GJ56558@mdounin.ru> Hello! On Sun, Aug 12, 2018 at 11:33:25PM +0300, Danila Vershinin wrote: > It seems that nginx can accept PROXY protocol fine, but when it > comes to forwarding, it can only do so only within a stream { > server { ? proxy_protocol on; } } . > > Are there any plans to add proxy_protocol on; for regular HTTP > server blocks so it can be used alongside proxy_pass? This would > come in very handy in a situation where NGINX is used as SSL > terminator, e.g.: > > NGINX (SSL) ? (Proxy protocol) ? Varnish. > > Varnish supports accepting PROXY protocol. There are no such plans, because in HTTP the same connection can be used for requests from different clients. Consider using X-Forwarded-For instead. -- Maxim Dounin http://mdounin.ru/ From mdounin at mdounin.ru Sun Aug 12 21:26:05 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 13 Aug 2018 00:26:05 +0300 Subject: Actions after cache miss is detected In-Reply-To: References: Message-ID: <20180812212605.GK56558@mdounin.ru> Hello! On Fri, Aug 10, 2018 at 09:05:30AM -0700, Roger Fischer wrote: > Is there a way to perform an action after a cache miss is > detected but before the request is forwarded to the upstream > server? > > Specifically, on a cache miss I want to: > Return a response instead of forwarding the request to the > upstream server. > Trigger a handler (module or script) that executes > asynchronously to the request. > > The latter I could do by looking at the logs. But I was > wondering if there is a more elegant way. There is no direct way to do this. There are multiple of ways to do this indirectly though, including: - configure a backend which returns a non-cacheable response and triggers appropriate actions; - configure a non-responding backend and error_page 502 to return a response and trigger appropriate processing. Alternatively, in some cases it might be more convenient to use proxy_store instead of proxy_cache, and then work with responses on the file system - with try_files and so on. -- Maxim Dounin http://mdounin.ru/ From ciapnz at gmail.com Sun Aug 12 22:38:38 2018 From: ciapnz at gmail.com (Danila Vershinin) Date: Mon, 13 Aug 2018 01:38:38 +0300 Subject: PROXY protocol to upstream server In-Reply-To: <20180812211224.GJ56558@mdounin.ru> References: <20180812211224.GJ56558@mdounin.ru> Message-ID: <412A45DC-2702-4C9E-92E8-F6D8E6893964@gmail.com> Hi Maxim, I understand. Followup question is: Is NGINX capable of presenting clients with different SSL certificate based on SNI? As in: stream { server { ssl_certificate foo.example.com .crt; ssl_certificate bar.example.com .crt; ... } } Best Regards, Danila > On 13 Aug 2018, at 00:12, Maxim Dounin wrote: > > Hello! > > On Sun, Aug 12, 2018 at 11:33:25PM +0300, Danila Vershinin wrote: > >> It seems that nginx can accept PROXY protocol fine, but when it >> comes to forwarding, it can only do so only within a stream { >> server { ? proxy_protocol on; } } . >> >> Are there any plans to add proxy_protocol on; for regular HTTP >> server blocks so it can be used alongside proxy_pass? This would >> come in very handy in a situation where NGINX is used as SSL >> terminator, e.g.: >> >> NGINX (SSL) ? (Proxy protocol) ? Varnish. >> >> Varnish supports accepting PROXY protocol. > > There are no such plans, because in HTTP the same connection can > be used for requests from different clients. Consider using > X-Forwarded-For instead. > > -- > Maxim Dounin > http://mdounin.ru/ > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From mdounin at mdounin.ru Mon Aug 13 14:20:05 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 13 Aug 2018 17:20:05 +0300 Subject: PROXY protocol to upstream server In-Reply-To: <412A45DC-2702-4C9E-92E8-F6D8E6893964@gmail.com> References: <20180812211224.GJ56558@mdounin.ru> <412A45DC-2702-4C9E-92E8-F6D8E6893964@gmail.com> Message-ID: <20180813142005.GL56558@mdounin.ru> Hello! On Mon, Aug 13, 2018 at 01:38:38AM +0300, Danila Vershinin wrote: > I understand. Followup question is: > > Is NGINX capable of presenting clients with different SSL certificate based on SNI? > As in: > > stream { > server { > ssl_certificate foo.example.com .crt; > ssl_certificate bar.example.com .crt; > ... > } > } No, currently there is no SNI support in the stream module (except $ssl_preread_server_name in the stream ssl preread module, see http://nginx.org/en/docs/stream/ngx_stream_ssl_preread_module.html). -- Maxim Dounin http://mdounin.ru/ From kelsey.dannels at nginx.com Mon Aug 13 16:40:16 2018 From: kelsey.dannels at nginx.com (Kelsey Dannels) Date: Mon, 13 Aug 2018 09:40:16 -0700 Subject: Reminder: Help Shape the Future of NGINX in the 2018 User Survey Message-ID: Hello- Sending out a friendly reminder: our annual NGINX User Survey is now open. Your feedback is incredibly important in helping us shape our products and services to better serve you. If you haven?t yet, please take ten minutes to share your thoughts: *https://nkadmin.typeform.com/to/e1A4mJ?source=email * Best, Kelsey -- *Join us at NGINX Conf 2018 , Oct 8-11, Atlanta, GA* Kelsey Dannels Marketing Communication Specialist Mobile: 650 773 1046 San Francisco -------------- next part -------------- An HTML attachment was scrubbed... URL: From nginx-forum at forum.nginx.org Tue Aug 14 01:12:42 2018 From: nginx-forum at forum.nginx.org (5lava) Date: Mon, 13 Aug 2018 21:12:42 -0400 Subject: Custom HTTP code in limit_except In-Reply-To: <796d9175587cfa2ca0acc407b42f19e7.NginxMailingListEnglish@forum.nginx.org> References: <796d9175587cfa2ca0acc407b42f19e7.NginxMailingListEnglish@forum.nginx.org> Message-ID: Bump Posted at Nginx Forum: https://forum.nginx.org/read.php?2,280084,280867#msg-280867 From francis at daoine.org Tue Aug 14 07:44:04 2018 From: francis at daoine.org (Francis Daly) Date: Tue, 14 Aug 2018 08:44:04 +0100 Subject: Using the mirror module In-Reply-To: References: <20180809222753.GH18403@daoine.org> Message-ID: <20180814074404.GI18403@daoine.org> On Thu, Aug 09, 2018 at 07:04:50PM -0400, jlangr wrote: Hi there, > We got this working to wrap up the day. Good that you have something working for you. > My ignorance of nginx meant that I > was interpreting the example literally--I viewed the "/mirror" as the name > for the internal route, but that actually becomes part of the URL the mirror > location proxy forwards to. It's actually a bit more subtle than that. The argument to the "mirror" directive is the local url of the subrequest; it should start with "/" or "@". Then, in the matching location{}, if you choose to proxy_pass, then the proxy_pass rules come in to play. And it is proxy_pass that decides what request to make to the upstream server. http://nginx.org/r/proxy_pass > mirror /; That means that the subrequest will be exactly "/"... > location = / { ...which will be handled in this location... > set $upstream_endpoint http://1.2.3.4:3002; > proxy_pass $upstream_endpoint$request_uri; ...and the proxy_pass argument includes something after the host:port, so that is what the request to upstream will be. An alternative could have been to use a named location -- "mirror @mirror;", with "location @mirror { proxy_pass http://1.2.3.4:3002; }" -- which would avoid the need for some of the extra variables. But, once you have something that works, there is no need to change it :-) Cheers, f -- Francis Daly francis at daoine.org From vervaeck.sam at skynet.be Tue Aug 14 12:41:12 2018 From: vervaeck.sam at skynet.be (Sam Vervaeck) Date: Tue, 14 Aug 2018 14:41:12 +0200 Subject: Nginx C API: Traversing the configuration tree Message-ID: Hi all, I?m writing a small monitoring module and I need to perform some set-up after the configuration object has ben initialised. I use the postconfiguration-hook in ngx_http_module_t and would like this hook to traverse over all locations that have been enabled in my module in order to initialise some state and set up some (database) connections. Does anybody know how to traverse the locations when given the root ngx_conf_t? I?ve found ngx_http_location_tree_node_t but I?ve failed to decipher how exactly it works. Besides that, is this the right way to do so, or is there some clever location-specific postconfiguration-hook that I?m missing? Many thanks, Sam -------------- next part -------------- An HTML attachment was scrubbed... URL: From vervaeck.sam at skynet.be Tue Aug 14 13:27:19 2018 From: vervaeck.sam at skynet.be (Sam Vervaeck) Date: Tue, 14 Aug 2018 15:27:19 +0200 Subject: Nginx C API: Traversing the configuration tree In-Reply-To: References: Message-ID: <0743AE05-04F7-47EB-9682-D05D0970ED00@skynet.be> Sorry, looks like I overlooked something in the API! It is possible to have a location-specific postconfiguration-hook: the last field of nginx_command_t. My bad! Regards, Sam > On 14 Aug 2018, at 14:41, Sam Vervaeck wrote: > > Hi all, > > I?m writing a small monitoring module and I need to perform some set-up after the configuration object has ben initialised. I use the postconfiguration-hook in ngx_http_module_t and would like this hook to traverse over all locations that have been enabled in my module in order to initialise some state and set up some (database) connections. Does anybody know how to traverse the locations when given the root ngx_conf_t? I?ve found ngx_http_location_tree_node_t but I?ve failed to decipher how exactly it works. Besides that, is this the right way to do so, or is there some clever location-specific postconfiguration-hook that I?m missing? > > Many thanks, > Sam > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx -------------- next part -------------- An HTML attachment was scrubbed... URL: From nginx-forum at forum.nginx.org Tue Aug 14 14:02:24 2018 From: nginx-forum at forum.nginx.org (jlangr) Date: Tue, 14 Aug 2018 10:02:24 -0400 Subject: Using the mirror module In-Reply-To: <20180814074404.GI18403@daoine.org> References: <20180814074404.GI18403@daoine.org> Message-ID: <48170afa7544b8778480198e7eaffbd8.NginxMailingListEnglish@forum.nginx.org> thanks Francis for the additional info, very helpful! Posted at Nginx Forum: https://forum.nginx.org/read.php?2,279036,280872#msg-280872 From jim at k4vqc.com Tue Aug 14 20:53:23 2018 From: jim at k4vqc.com (Jim Popovitch) Date: Tue, 14 Aug 2018 16:53:23 -0400 Subject: hostname based listen directive with A/AAAA RRs Message-ID: <1534280003.3785.1.camel@k4vqc.com> Hello, Will the following listen statement bind to all addrs (2xIPv4 and 2xIPv6)? server { listen www.nginx.org:443 ssh http2 ipv6only=no bind; .... } -Jim P. From mdounin at mdounin.ru Wed Aug 15 00:06:10 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 15 Aug 2018 03:06:10 +0300 Subject: hostname based listen directive with A/AAAA RRs In-Reply-To: <1534280003.3785.1.camel@k4vqc.com> References: <1534280003.3785.1.camel@k4vqc.com> Message-ID: <20180815000610.GT56558@mdounin.ru> Hello! On Tue, Aug 14, 2018 at 04:53:23PM -0400, Jim Popovitch wrote: > Hello, > > Will the following listen statement bind to all addrs (2xIPv4 and > 2xIPv6)? > > server { > listen www.nginx.org:443 ssh http2 ipv6only=no bind; No. If a hostname is used in the listen directive, it will use the first address the name resolves to. -- Maxim Dounin http://mdounin.ru/ From jim at k4vqc.com Wed Aug 15 03:30:39 2018 From: jim at k4vqc.com (Jim Popovitch) Date: Tue, 14 Aug 2018 23:30:39 -0400 Subject: [Nginx] Re: hostname based listen directive with A/AAAA RRs In-Reply-To: <20180815000610.GT56558@mdounin.ru> References: <1534280003.3785.1.camel@k4vqc.com> <20180815000610.GT56558@mdounin.ru> Message-ID: <1534303839.7109.2.camel@k4vqc.com> On Wed, 2018-08-15 at 03:06 +0300, Maxim Dounin wrote: > Hello! > > On Tue, Aug 14, 2018 at 04:53:23PM -0400, Jim Popovitch wrote: > > > Hello, > > > > Will the following listen statement bind to all addrs (2xIPv4 and > > 2xIPv6)? > > > > server { > > ???listen www.nginx.org:443 ssh http2 ipv6only=no bind; > > No.??If a hostname is used in the listen directive, it will use? > the first address the name resolves to. Thank you, -Jim P. From gerben.wierda at rna.nl Wed Aug 15 10:02:27 2018 From: gerben.wierda at rna.nl (Gerben Wierda) Date: Wed, 15 Aug 2018 12:02:27 +0200 Subject: nginx error (connect() failed, 61 "Connection refused", but everything works) Message-ID: I have a set of minio (S3-compatoble block storage) servers running behind nginx on macOS High Sierra. While everything seems to work OK, I have noticed an unexplained error in the nginx logs. For instance, when I use minio?s mc to ls a file: mc ls nginx/gerbentest/duplicati-ifdb6b7ac174b4e5094b04e7321d10c6b.dindex.zip.aes mc reports (as expected): [2017-12-21 20:36:41 CET] 36KiB duplicati-ifdb6b7ac174b4e5094b04e7321d10c6b.dindex.zip.aes So, everything works. So does duplicati (the backup solution that uses those mini backends). I can also connect direct to the servers and it works fine. But nginx reports (apparently once per session): 2018/08/15 11:34:48 [error] 242#0: *881 kevent() reported that connect() failed (61: Connection refused) while connecting to upstream, client: 192.168.2.67, server: MYHOST, request: ?GET /gerbentest/?delimiter=%2F&max-keys=1000&prefix=duplicati-ifdb6b7ac174b4e5094b04e7321d10c6b.dindex.zip.aes HTTP/1.1?, upstream: ?http://[::1]:9003/gerbentest/?delimiter=%2F&max-keys=1000&prefix=duplicati-ifdb6b7ac174b4e5094b04e7321d10c6b.dindex.zip.aes?, host: ?MYHOST:9000? I?d like to find out why this happens. Can someone help me find the cause of these errors? The config for the minio servers is: server { listen 9000 ssl; server_name MYHOST; ssl_certificate minio_certificate_chained.crt; ssl_certificate_key minio_certificate.key; ssl_protocols TLSv1.2; proxy_buffering off; client_max_body_size 1000m; location / { proxy_set_header Host $http_host; if ($http_authorization ~* "^AWS4-HMAC-SHA256 Credential=REMOVED") { proxy_pass http://localhost:9001; } if ($http_authorization ~* "^AWS4-HMAC-SHA256 Credential=REMOVED") { proxy_pass http://localhost:9002; } if ($http_authorization ~* "^AWS4-HMAC-SHA256 Credential=REMOVED") { proxy_pass http://localhost:9003; } if ($http_authorization ~* "^AWS4-HMAC-SHA256 Credential=REMOVED") { proxy_pass http://localhost:9004; } } } Gerben Wierda Chess and the Art of Enterprise Architecture Mastering ArchiMate Architecture for Real Enterprises at InfoWorld On Slippery Ice at EAPJ -------------- next part -------------- An HTML attachment was scrubbed... URL: From gerben.wierda at rna.nl Wed Aug 15 11:06:05 2018 From: gerben.wierda at rna.nl (Gerben Wierda) Date: Wed, 15 Aug 2018 13:06:05 +0200 Subject: nginx error (connect() failed, 61 "Connection refused", but everything works) In-Reply-To: References: Message-ID: <0574401F-6B24-4785-B46C-BF18FC75B035@rna.nl> Answering myself: if I change localhost to 127.0.0.1 the error message goes away. Why, though? Gerben Wierda Chess and the Art of Enterprise Architecture Mastering ArchiMate Architecture for Real Enterprises at InfoWorld On Slippery Ice at EAPJ > On 15 Aug 2018, at 12:02, Gerben Wierda wrote: > > I have a set of minio (S3-compatoble block storage) servers running behind nginx on macOS High Sierra. While everything seems to work OK, I have noticed an unexplained error in the nginx logs. > For instance, when I use minio?s mc to ls a file: > mc ls nginx/gerbentest/duplicati-ifdb6b7ac174b4e5094b04e7321d10c6b.dindex.zip.aes > mc reports (as expected): > [2017-12-21 20:36:41 CET] 36KiB duplicati-ifdb6b7ac174b4e5094b04e7321d10c6b.dindex.zip.aes > > So, everything works. So does duplicati (the backup solution that uses those mini backends). I can also connect direct to the servers and it works fine. > > But nginx reports (apparently once per session): > 2018/08/15 11:34:48 [error] 242#0: *881 kevent() reported that connect() failed (61: Connection refused) while connecting to upstream, client: 192.168.2.67, server: MYHOST, request: ?GET /gerbentest/?delimiter=%2F&max-keys=1000&prefix=duplicati-ifdb6b7ac174b4e5094b04e7321d10c6b.dindex.zip.aes HTTP/1.1?, upstream: ?http://[::1]:9003/gerbentest/?delimiter=%2F&max-keys=1000&prefix=duplicati-ifdb6b7ac174b4e5094b04e7321d10c6b.dindex.zip.aes ?, host: ?MYHOST:9000? > > I?d like to find out why this happens. Can someone help me find the cause of these errors? The config for the minio servers is: > > server { > listen 9000 ssl; > server_name MYHOST; > ssl_certificate minio_certificate_chained.crt; > ssl_certificate_key minio_certificate.key; > ssl_protocols TLSv1.2; > proxy_buffering off; > client_max_body_size 1000m; > location / { > proxy_set_header Host $http_host; > if ($http_authorization ~* "^AWS4-HMAC-SHA256 Credential=REMOVED") { > proxy_pass http://localhost:9001 ; > } > if ($http_authorization ~* "^AWS4-HMAC-SHA256 Credential=REMOVED") { > proxy_pass http://localhost:9002 ; > } > if ($http_authorization ~* "^AWS4-HMAC-SHA256 Credential=REMOVED") { > proxy_pass http://localhost:9003 ; > } > if ($http_authorization ~* "^AWS4-HMAC-SHA256 Credential=REMOVED") { > proxy_pass http://localhost:9004 ; > } > } > } > > > Gerben Wierda > Chess and the Art of Enterprise Architecture > Mastering ArchiMate > Architecture for Real Enterprises at InfoWorld > On Slippery Ice at EAPJ > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx -------------- next part -------------- An HTML attachment was scrubbed... URL: From pluknet at nginx.com Wed Aug 15 12:05:51 2018 From: pluknet at nginx.com (Sergey Kandaurov) Date: Wed, 15 Aug 2018 15:05:51 +0300 Subject: nginx error (connect() failed, 61 "Connection refused", but everything works) In-Reply-To: References: Message-ID: > On 15 Aug 2018, at 13:02, Gerben Wierda wrote: [..] > But nginx reports (apparently once per session): > 2018/08/15 11:34:48 [error] 242#0: *881 kevent() reported that connect() failed (61: Connection refused) while connecting to upstream, client: 192.168.2.67, server: MYHOST, request: ?GET /gerbentest/?delimiter=%2F&max-keys=1000&prefix=duplicati-ifdb6b7ac174b4e5094b04e7321d10c6b.dindex.zip.aes HTTP/1.1?, upstream: ?http://[::1]:9003/gerbentest/?delimiter=%2F&max-keys=1000&prefix=duplicati-ifdb6b7ac174b4e5094b04e7321d10c6b.dindex.zip.aes?, host: ?MYHOST:9000? Note [::1]:9003 in upstream, which is likely the address nobody listens. > I?d like to find out why this happens. Can someone help me find the cause of these errors? The config for the minio servers is: > > server { > listen 9000 ssl; > server_name MYHOST; > ssl_certificate minio_certificate_chained.crt; > ssl_certificate_key minio_certificate.key; > ssl_protocols TLSv1.2; > proxy_buffering off; > client_max_body_size 1000m; > location / { > proxy_set_header Host $http_host; > if ($http_authorization ~* "^AWS4-HMAC-SHA256 Credential=REMOVED") { > proxy_pass http://localhost:9001; > } If a domain name resolves to several addresses, which is apparently the case for "localhost", all of them will be used in a round-robin fashion. See for details: http://nginx.org/r/proxy_pass -- Sergey Kandaurov From gerben.wierda at rna.nl Wed Aug 15 12:49:56 2018 From: gerben.wierda at rna.nl (Gerben Wierda) Date: Wed, 15 Aug 2018 14:49:56 +0200 Subject: nginx error (connect() failed, 61 "Connection refused", but everything works) In-Reply-To: References: Message-ID: <7833994C-7FC8-4503-B243-DA74575F7FE3@rna.nl> > On 15 Aug 2018, at 14:05, Sergey Kandaurov wrote: > > >> On 15 Aug 2018, at 13:02, Gerben Wierda wrote: > > [..] > >> But nginx reports (apparently once per session): >> 2018/08/15 11:34:48 [error] 242#0: *881 kevent() reported that connect() failed (61: Connection refused) while connecting to upstream, client: 192.168.2.67, server: MYHOST, request: ?GET /gerbentest/?delimiter=%2F&max-keys=1000&prefix=duplicati-ifdb6b7ac174b4e5094b04e7321d10c6b.dindex.zip.aes HTTP/1.1?, upstream: ?http://[::1]:9003/gerbentest/?delimiter=%2F&max-keys=1000&prefix=duplicati-ifdb6b7ac174b4e5094b04e7321d10c6b.dindex.zip.aes?, host: ?MYHOST:9000? > > Note [::1]:9003 in upstream, which is likely the address nobody listens. Ha, yes, now I see, the address is the IPv6 address for localhost. Silly me, I though in the network stack 127.0.0.1 and [::1] worked more or less as aliases for the same port, but apparently not. So, while minio is listening on 127.0.0.1:9003 it is not listening on [::1]:9003 > I?d like to find out why this happens. Can someone help me find the cause of these errors? The config for the minio servers is: >> >> server { >> listen 9000 ssl; >> server_name MYHOST; >> ssl_certificate minio_certificate_chained.crt; >> ssl_certificate_key minio_certificate.key; >> ssl_protocols TLSv1.2; >> proxy_buffering off; >> client_max_body_size 1000m; >> location / { >> proxy_set_header Host $http_host; >> if ($http_authorization ~* "^AWS4-HMAC-SHA256 Credential=REMOVED") { >> proxy_pass http://localhost:9001; >> } > > If a domain name resolves to several addresses, > which is apparently the case for "localhost?, > all of them will be used in a round-robin fashion. > > See for details: > http://nginx.org/r/proxy_pass If I use dig or nslookup to resolve localhost, I get just 127.0.0.1. But my /etc/hosts contains both IPv4 and IPv6 for localhost and nginx round-robins over both apparently. Using 127.0.0.1 instead of localhost is the correct solution, then. G > > -- > Sergey Kandaurov > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx -------------- next part -------------- An HTML attachment was scrubbed... URL: From nginx-forum at forum.nginx.org Thu Aug 16 12:56:22 2018 From: nginx-forum at forum.nginx.org (ankitschopra) Date: Thu, 16 Aug 2018 08:56:22 -0400 Subject: Using Nginx for TCP Mirroring Message-ID: <01e874454425b8cc665a6608c0cf1597.NginxMailingListEnglish@forum.nginx.org> Hi, Can any one help in using Nginx for TCP mirroring. i.e We want to send 1 TCP request to 2 servers. There is a module ngx_http_mirror_module, http://nginx.org/en/docs/http/ngx_http_mirror_module.html which can do http mirroring, but we want to do for TCP mirroring. Thanks, Ankit Chopra Posted at Nginx Forum: https://forum.nginx.org/read.php?2,280895,280895#msg-280895 From arut at nginx.com Thu Aug 16 13:32:31 2018 From: arut at nginx.com (Roman Arutyunyan) Date: Thu, 16 Aug 2018 16:32:31 +0300 Subject: Using Nginx for TCP Mirroring In-Reply-To: <01e874454425b8cc665a6608c0cf1597.NginxMailingListEnglish@forum.nginx.org> References: <01e874454425b8cc665a6608c0cf1597.NginxMailingListEnglish@forum.nginx.org> Message-ID: <20180816133231.GP43355@Romans-MacBook-Air.local> Hi, On Thu, Aug 16, 2018 at 08:56:22AM -0400, ankitschopra wrote: > Hi, > > Can any one help in using Nginx for TCP mirroring. i.e We want to send 1 > TCP request to 2 servers. There's no such module in nginx. The stream (TCP) module has no information about the underlying protocol. This means that it's not aware what is a request and where it ends. However, it is really interesting to know your use case. You want to mirror the request (let's assume we have information about the protocol), but what about the response? > There is a module ngx_http_mirror_module, > http://nginx.org/en/docs/http/ngx_http_mirror_module.html which can do http > mirroring, but we want to do for TCP mirroring. The HTTP mirror module is in fact way more generic than that. It just creates a background subrequest, which may proxy the request to a remote server and do a bunch of other things. -- Roman Arutyunyan From nginx-forum at forum.nginx.org Fri Aug 17 07:36:15 2018 From: nginx-forum at forum.nginx.org (hunterqin) Date: Fri, 17 Aug 2018 03:36:15 -0400 Subject: keepalive not work with grpc In-Reply-To: <20180810101744.GD56558@mdounin.ru> References: <20180810101744.GD56558@mdounin.ru> Message-ID: <78d922c0744b79e2abec23db26d39bda.NginxMailingListEnglish@forum.nginx.org> I used protobuf v310, grpc v181, nginx v1152 and v1140. The server is just same as the example async_server on grpc website. https://grpc.io/docs/tutorials/async/helloasync-cpp.html You can use it directly. https://github.com/grpc/grpc/blob/v1.14.1/examples/cpp/helloworld/greeter_async_server.cc By the way, there is another problem found in nginx. Maybe you are intersted in it. After fixing the problem above, I tried more. The rpc service is simple, it just transfers a short string(less than 10 bytes). At first, the grpc client , nginx and grpc server works well together. But after some time, the service stopped. I used the wireshark to check the problem. The last DATA frame from nginx to grpc server is wrong. For example, in correct packet, the DATA length is 13, message in protobuf is "Hunter", so the length of string in protobuf is 6. protobuf part: 0a 06 48 75 6e 74 65 72. But the last one is wrong, its DATA frame length is 10, it didn't send the message completely 0a 06 48 75 6e. Then the grpc server just send back a TCP ACK and ignore the wrong message, finally nginx stop the TCP connection. Then I test for longer string. When the string length is 1000bytes, it can finish for about 4000 times. When the string length is 500bytes, it turns to 8000+times. I guess some problems may happen in nginx buffer. You could check it , too. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,280765,280902#msg-280902 From nginx-forum at forum.nginx.org Fri Aug 17 13:49:41 2018 From: nginx-forum at forum.nginx.org (conquistadorjd) Date: Fri, 17 Aug 2018 09:49:41 -0400 Subject: WordPress multisite with root url as Laravel application Message-ID: I am running a wordpress multisite website with subdomain option. example.me is my main website https://one.example.me, https://two.example.me/two etc are the network websites. Now I am planning to have a small laravel application accessible at example.me. I tried using different combination but not able to figure this out. Could not find any examples in search results as well.Can someone let me know how can I use laravel webapp at main/root address https://example.me and mutilsite with subdomain ? I am new to NGINX. Here is my nginx code. map $http_host $blogid { default 0; include /var/www/goingplacesme/wp-content/uploads/nginx-helper/map.conf; } server { listen 80; listen [::]:80; root /var/www/exampleme; index index.html index.htm index.nginx-debian.html; server_name example.me www.example.me; location / { #try_files $uri $uri/ =404; try_files $uri /index.php$is_args$args; } # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000 location ~ \.php$ { include snippets/fastcgi-php.conf; # With php7.0-cgi alone: #fastcgi_pass 127.0.0.1:9000; # With php7.0-fpm: fastcgi_pass unix:/run/php/php7.0-fpm.sock; } # deny access to .htaccess files, if Apache's document root # concurs with nginx's one # #location ~ /\.ht { # deny all; #} location ~ ^/files/(.*)$ { try_files /wp-content/blogs.dir/$blogid/$uri /wp-includes/ms-files.php?file=$1 ; access_log off; log_not_found off; expires max; } location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ { expires 24h; log_not_found off; } location ^~ /blogs.dir { internal; alias /var/www/pathtoyoursite/web/wp-content/blogs.dir ; access_log off; log_not_found off; expires max; } if (!-e $request_filename) { rewrite /wp-admin$ $scheme://$host$uri/ permanent; rewrite ^/[_0-9a-zA-Z-]+(/wp-.*) $1 last; rewrite ^/[_0-9a-zA-Z-]+(/.*\.php)$ $1 last; } } Posted at Nginx Forum: https://forum.nginx.org/read.php?2,280904,280904#msg-280904 From ciapnz at gmail.com Fri Aug 17 18:46:54 2018 From: ciapnz at gmail.com (Danila Vershinin) Date: Fri, 17 Aug 2018 21:46:54 +0300 Subject: WordPress multisite with root url as Laravel application In-Reply-To: References: Message-ID: <1B43387F-5F9D-41E3-955F-A164DE9566C9@gmail.com> Hi, Why not just create 2 server blocks, one for root domain and the other for WP multisite subdomains? As in: server { server_name example.com www.example.com ; ? Laravel directives go here } server { server_name *.example.com ; .. WP multisite directives go here } Best Regards, Danila > On 17 Aug 2018, at 16:49, conquistadorjd wrote: > > I am running a wordpress multisite website with subdomain option. example.me > is my main website https://one.example.me, https://two.example.me/two etc > are the network websites. > > Now I am planning to have a small laravel application accessible at > example.me. I tried using different combination but not able to figure this > out. Could not find any examples in search results as well.Can someone let > me know how can I use laravel webapp at main/root address https://example.me > and mutilsite with subdomain ? > > I am new to NGINX. Here is my nginx code. > > map $http_host $blogid { > default 0; > include > /var/www/goingplacesme/wp-content/uploads/nginx-helper/map.conf; > } > > server { > listen 80; > listen [::]:80; > > root /var/www/exampleme; > > index index.html index.htm index.nginx-debian.html; > > server_name example.me www.example.me; > > location / { > #try_files $uri $uri/ =404; > try_files $uri /index.php$is_args$args; > } > > # pass the PHP scripts to FastCGI server listening on > 127.0.0.1:9000 > location ~ \.php$ { > include snippets/fastcgi-php.conf; > > # With php7.0-cgi alone: > #fastcgi_pass 127.0.0.1:9000; > # With php7.0-fpm: > fastcgi_pass unix:/run/php/php7.0-fpm.sock; > } > > # deny access to .htaccess files, if Apache's document root > # concurs with nginx's one > # > #location ~ /\.ht { > # deny all; > #} > location ~ ^/files/(.*)$ { > try_files /wp-content/blogs.dir/$blogid/$uri > /wp-includes/ms-files.php?file=$1 ; > access_log off; log_not_found off; expires max; > } > > location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ { > expires 24h; > log_not_found off; > } > > location ^~ /blogs.dir { > internal; > alias /var/www/pathtoyoursite/web/wp-content/blogs.dir ; > access_log off; log_not_found off; expires max; > } > > > > if (!-e $request_filename) { > rewrite /wp-admin$ $scheme://$host$uri/ permanent; > rewrite ^/[_0-9a-zA-Z-]+(/wp-.*) $1 last; > rewrite ^/[_0-9a-zA-Z-]+(/.*\.php)$ $1 last; > } > > } > > Posted at Nginx Forum: https://forum.nginx.org/read.php?2,280904,280904#msg-280904 > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From sudarshan12s at gmail.com Fri Aug 17 19:00:27 2018 From: sudarshan12s at gmail.com (Sudarshan Soma) Date: Sat, 18 Aug 2018 00:30:27 +0530 Subject: making telnet secure using Nginx redirect options Message-ID: Hi All, We have a traditional custom server which is based on telnet at port 9932. Can we make it secure by just using Nginx.conf setting. something like: location /mytelnet { grpc_pass telnet localhost 9932; } Assume telnet client is my own application, it can iniitate https connection with mytelnet prefix. 9932 is example port where custom server is running. Regards, -------------- next part -------------- An HTML attachment was scrubbed... URL: From nginx-forum at forum.nginx.org Mon Aug 20 10:09:10 2018 From: nginx-forum at forum.nginx.org (petecooper) Date: Mon, 20 Aug 2018 06:09:10 -0400 Subject: rewrite rule: MediaWiki to static site Message-ID: <042f1776e5146b0ab2e1c69c9aaca94f.NginxMailingListEnglish@forum.nginx.org> Hello. I'm diverting traffic from an retired MediaWiki site to an active static site. I'm somewhat confused with the process of `rewrite`-ing old URLs with a query string to standard URLs *without* a query string on the static site. For clarity, I do not need or want query strings on the new site. Retired URLs examples: https://example.com/wiki/index.php?title=en/Main_Page https://example.com/wiki/index.php?title=Glossary https://example.com/wiki/index.php?title=Category:Troubleshooting New URLs examples: https://example.net/MainPage https://example.net/Glossary https://example.net/Troubleshooting All the retired URLs are attached to `/wiki/index.php` with a single query string. There is no direct correlation between the old URLs and the new ones, and I have a before-and-after list of URLs to put into a 200+ entry `rewrite` list. My question is two-fold: 1) Is it better (objectively) to have all the rewrite rules inside a `location` referring to `/wiki/index.php` since the only non-Mediawiki URL on the retired site is `/`? 2) How can I best build the `rewrite` list without using `if` or nested `map`? I am grateful for your time and attention, and I would very much appreciate an example rewrite rule and/or some reading recommendations for `rewrite` with query strings. Thank you. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,280913,280913#msg-280913 From francis at daoine.org Mon Aug 20 11:29:02 2018 From: francis at daoine.org (Francis Daly) Date: Mon, 20 Aug 2018 12:29:02 +0100 Subject: rewrite rule: MediaWiki to static site In-Reply-To: <042f1776e5146b0ab2e1c69c9aaca94f.NginxMailingListEnglish@forum.nginx.org> References: <042f1776e5146b0ab2e1c69c9aaca94f.NginxMailingListEnglish@forum.nginx.org> Message-ID: <20180820112902.GB3101@daoine.org> On Mon, Aug 20, 2018 at 06:09:10AM -0400, petecooper wrote: Hi there, > I'm diverting traffic from an retired MediaWiki site to an active static > site. I'm somewhat confused with the process of `rewrite`-ing old URLs with > a query string to standard URLs *without* a query string on the static site. > For clarity, I do not need or want query strings on the new site. In this case, it looks to me that you can probably "return" rather than "rewrite", since everything should probably be a http redirect. > Retired URLs examples: > > https://example.com/wiki/index.php?title=en/Main_Page > https://example.com/wiki/index.php?title=Glossary > https://example.com/wiki/index.php?title=Category:Troubleshooting > > New URLs examples: > > https://example.net/MainPage > https://example.net/Glossary > https://example.net/Troubleshooting > > All the retired URLs are attached to `/wiki/index.php` with a single query > string. > > There is no direct correlation between the old URLs and the new ones, and I > have a before-and-after list of URLs to put into a 200+ entry `rewrite` > list. > > My question is two-fold: > > 1) Is it better (objectively) to have all the rewrite rules inside a > `location` referring to `/wiki/index.php` since the only non-Mediawiki URL > on the retired site is `/`? Yes. Except I would have the config be not a lot more than location = /wiki/index.php { return 301 https://example.net/$my_static_wiki; } > 2) How can I best build the `rewrite` list without using `if` or nested > `map`? map $arg_title $my_static_wiki { default ""; # or whatever en/Main_Page MainPage; Glossary Glossary; Category:Troubleshooting Troubleshooting; } Make that map (http://nginx.org/r/map) be as big as you like. And you can "include" an external file if you find that easier. Note: I use "$arg_title" are the string. You could use "$query_string", but then the matching values would be of the form title=en/Main_Page. And because I put "http://example.net/" in the "return" directive, I don't put that part of the destination url in the map. Adjust to taste. > I am grateful for your time and attention, and I would very much appreciate > an example rewrite rule and/or some reading recommendations for `rewrite` > with query strings. If you still want to know about rewrite with query strings, you can look at http://nginx.org/r/rewrite Basically -- the incoming query string is preserved unless you ask for it not to be. Good luck with it, f -- Francis Daly francis at daoine.org From kelsey.dannels at nginx.com Mon Aug 20 15:07:47 2018 From: kelsey.dannels at nginx.com (Kelsey Dannels) Date: Mon, 20 Aug 2018 08:07:47 -0700 Subject: Help Shape the Future of NGINX in the 2018 User Survey Message-ID: *Hello- Our annual NGINX User Survey closes Wednesday (8/22). Your feedback is incredibly valuable to us, so if you haven?t yet, please take 10 minutes to complete the survey.Link to survey: https://nkadmin.typeform.com/to/e1A4mJ?source=email Best,Kelsey * -- *Join us at NGINX Conf 2018 , Oct 8-11, Atlanta, GA* Kelsey Dannels Marketing Communication Specialist Mobile: 650 773 1046 San Francisco -------------- next part -------------- An HTML attachment was scrubbed... URL: From nginx-forum at forum.nginx.org Mon Aug 20 20:10:19 2018 From: nginx-forum at forum.nginx.org (petecooper) Date: Mon, 20 Aug 2018 16:10:19 -0400 Subject: rewrite rule: MediaWiki to static site In-Reply-To: <20180820112902.GB3101@daoine.org> References: <20180820112902.GB3101@daoine.org> Message-ID: Francis Daly Wrote: > In this case, it looks to me that you can probably "return" rather > than > "rewrite", since everything should probably be a http redirect. > non-Mediawiki URL > [...] Except I would have the config be not a lot more than > > location = /wiki/index.php { return 301 > https://example.net/$my_static_wiki; } > map $arg_title $my_static_wiki { > default ""; # or whatever > en/Main_Page MainPage; > Glossary Glossary; > Category:Troubleshooting Troubleshooting; > } > > Make that map (http://nginx.org/r/map) be as big as you like. And you > can > "include" an external file if you find that easier. > Good luck with it, Hello Francis -- thank you very much, that worked perfectly. The length of the `map` list meant I had to increase `map_hash_bucket_size` to `128`, but everything works as you described. with very best regards, and my heartfelt thanks; Pete Posted at Nginx Forum: https://forum.nginx.org/read.php?2,280913,280922#msg-280922 From jeff.dyke at gmail.com Mon Aug 20 22:35:18 2018 From: jeff.dyke at gmail.com (Jeff Dyke) Date: Mon, 20 Aug 2018 18:35:18 -0400 Subject: rewrite rule: MediaWiki to static site In-Reply-To: References: <20180820112902.GB3101@daoine.org> Message-ID: Seems like you have a solution, but i've done this recently in the reverse and found this configuration to be incredibly helpful, with little reason to ever update. I check the primary (static in my case) site first and if it 404's it is sent to a different proxy_pass. As you can guess i use the Play Framework. location @corporate { proxy_pass http://corporate-server$uri; proxy_intercept_errors on; recursive_error_pages on; error_page 404 = @play; } location @play { proxy_pass http://play-server$uri; } then in the main location block i simply have: location / { include /etc/nginx/shared_foo; #this is simply shared headers, nothing to do with the logic try_files $uri @corporate; } As @corporate will send to @play only if required. I mainly put it out here b/c if someone is searching, this is a way to go about it, without creating a map. We will add more to static and more to Play overtime, as we already have, and this keeps on chugging. Best, Jeff On Mon, Aug 20, 2018 at 4:10 PM petecooper wrote: > Francis Daly Wrote: > > > In this case, it looks to me that you can probably "return" rather > > than > > "rewrite", since everything should probably be a http redirect. > > non-Mediawiki URL > > > [...] Except I would have the config be not a lot more than > > > > location = /wiki/index.php { return 301 > > https://example.net/$my_static_wiki; } > > > map $arg_title $my_static_wiki { > > default ""; # or whatever > > en/Main_Page MainPage; > > Glossary Glossary; > > Category:Troubleshooting Troubleshooting; > > } > > > > Make that map (http://nginx.org/r/map) be as big as you like. And you > > can > > "include" an external file if you find that easier. > > > Good luck with it, > > Hello Francis -- thank you very much, that worked perfectly. > > The length of the `map` list meant I had to increase `map_hash_bucket_size` > to `128`, but everything works as you described. > > with very best regards, and my heartfelt thanks; > > Pete > > Posted at Nginx Forum: > https://forum.nginx.org/read.php?2,280913,280922#msg-280922 > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nginx-forum at forum.nginx.org Tue Aug 21 05:30:04 2018 From: nginx-forum at forum.nginx.org (chenstrace) Date: Tue, 21 Aug 2018 01:30:04 -0400 Subject: How to produce stale event for nginx Message-ID: <063b8f0e96e51b6fce90379660e4ec2b.NginxMailingListEnglish@forum.nginx.org> I am studying nginx by reading and debugging source code. I want to produce the stale event, what is the simplest way to make it happen? Thanks very much Posted at Nginx Forum: https://forum.nginx.org/read.php?2,280927,280927#msg-280927 From mdounin at mdounin.ru Wed Aug 22 01:57:50 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 22 Aug 2018 04:57:50 +0300 Subject: keepalive not work with grpc In-Reply-To: <78d922c0744b79e2abec23db26d39bda.NginxMailingListEnglish@forum.nginx.org> References: <20180810101744.GD56558@mdounin.ru> <78d922c0744b79e2abec23db26d39bda.NginxMailingListEnglish@forum.nginx.org> Message-ID: <20180822015750.GB56558@mdounin.ru> Hello! On Fri, Aug 17, 2018 at 03:36:15AM -0400, hunterqin wrote: > I used protobuf v310, grpc v181, nginx v1152 and v1140. > The server is just same as the example async_server on grpc website. > https://grpc.io/docs/tutorials/async/helloasync-cpp.html > You can use it directly. > https://github.com/grpc/grpc/blob/v1.14.1/examples/cpp/helloworld/greeter_async_server.cc Thank you for details, I was able to reproduce the problem. Please try the patch below. > By the way, there is another problem found in nginx. Maybe you are intersted > in it. > After fixing the problem above, I tried more. The rpc service is simple, it > just transfers a short string(less than 10 bytes). At first, the grpc client > , nginx and grpc server works well together. But after some time, the > service stopped. I used the wireshark to check the problem. The last DATA > frame from nginx to grpc server is wrong. For example, in correct packet, > the DATA length is 13, message in protobuf is "Hunter", so the length of > string in protobuf is 6. > protobuf part: > 0a 06 48 75 6e 74 65 72. > But the last one is wrong, its DATA frame length is 10, it didn't send the > message completely > 0a 06 48 75 6e. > Then the grpc server just send back a TCP ACK and ignore the wrong message, > finally nginx stop the TCP connection. > Then I test for longer string. When the string length is 1000bytes, it can > finish for about 4000 times. When the string length is 500bytes, it turns to > 8000+times. I guess some problems may happen in nginx buffer. You could > check it , too. The "After fixing the problem above" part is most likely the culprit. If you've simply modified nginx to keep connections alive despite the fact that there were unparsed control frames, this will quickly result in problems due to incorrect flow control because of lost window update frames. And this matches symptoms you've observed. Please try the following patch to see if it helps: # HG changeset patch # User Maxim Dounin # Date 1534897290 -10800 # Wed Aug 22 03:21:30 2018 +0300 # Node ID 2cc5ef571e7a9a4c299bb3ffd6234529cc2183f1 # Parent 70c6b08973a02551612da4a4273757dc77c70ae2 gRPC: improved keepalive handling. The code is now able to parse additional control frames after the response is received, and can send control frames as well. This fixes keepalive problems as observed with grpc-c, which can send window update and ping frames after the response. diff --git a/src/http/modules/ngx_http_grpc_module.c b/src/http/modules/ngx_http_grpc_module.c --- a/src/http/modules/ngx_http_grpc_module.c +++ b/src/http/modules/ngx_http_grpc_module.c @@ -111,6 +111,7 @@ typedef struct { unsigned output_closed:1; unsigned parsing_headers:1; unsigned end_stream:1; + unsigned done:1; unsigned status:1; ngx_http_request_t *request; @@ -1074,6 +1075,7 @@ ngx_http_grpc_reinit_request(ngx_http_re ctx->output_closed = 0; ctx->parsing_headers = 0; ctx->end_stream = 0; + ctx->done = 0; ctx->status = 0; ctx->connection = NULL; @@ -1407,6 +1409,27 @@ ngx_http_grpc_body_output_filter(void *d rc = NGX_AGAIN; } + if (ctx->done) { + + /* + * We have already got the response and were sending some additional + * control frames. Even if there is still something unsent, stop + * here anyway. + */ + + r->upstream->length = 0; + + if (ctx->in == NULL + && ctx->out == NULL + && ctx->output_closed + && ctx->state == ngx_http_grpc_st_start) + { + r->upstream->keepalive = 1; + } + + ngx_post_event(r->upstream->peer.connection->read, &ngx_posted_events); + } + return rc; } @@ -1832,6 +1855,33 @@ ngx_http_grpc_filter(void *data, ssize_t rc = ngx_http_grpc_parse_frame(r, ctx, b); if (rc == NGX_AGAIN) { + + if (ctx->done) { + + /* + * We have finished parsing the response and the + * remaining control frames. If there are unsent + * control frames, post a write event to sent them. + */ + + if (ctx->out) { + ngx_post_event(u->peer.connection->write, + &ngx_posted_events); + return NGX_AGAIN; + } + + u->length = 0; + + if (ctx->in == NULL + && ctx->output_closed + && ctx->state == ngx_http_grpc_st_start) + { + u->keepalive = 1; + } + + return NGX_OK; + } + return NGX_AGAIN; } @@ -1898,6 +1948,13 @@ ngx_http_grpc_filter(void *data, ssize_t return NGX_ERROR; } + if (ctx->stream_id && ctx->done) { + ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, + "upstream sent frame for closed stream %ui", + ctx->stream_id); + return NGX_ERROR; + } + ctx->padding = 0; } @@ -1914,17 +1971,7 @@ ngx_http_grpc_filter(void *data, ssize_t ctx->state = ngx_http_grpc_st_start; if (ctx->flags & NGX_HTTP_V2_END_STREAM_FLAG) { - u->length = 0; - - if (ctx->in == NULL - && ctx->out == NULL - && ctx->output_closed - && b->last == b->pos) - { - u->keepalive = 1; - } - - break; + ctx->done = 1; } continue; @@ -2094,17 +2141,8 @@ ngx_http_grpc_filter(void *data, ssize_t "grpc trailer done"); if (ctx->end_stream) { - u->length = 0; - - if (ctx->in == NULL - && ctx->out == NULL - && ctx->output_closed - && b->last == b->pos) - { - u->keepalive = 1; - } - - return NGX_OK; + ctx->done = 1; + break; } ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, @@ -2121,6 +2159,10 @@ ngx_http_grpc_filter(void *data, ssize_t return NGX_ERROR; } + if (rc == NGX_HTTP_PARSE_HEADER_DONE) { + continue; + } + /* rc == NGX_AGAIN */ if (ctx->rest == 0) { @@ -2237,17 +2279,7 @@ ngx_http_grpc_filter(void *data, ssize_t ctx->state = ngx_http_grpc_st_start; if (ctx->flags & NGX_HTTP_V2_END_STREAM_FLAG) { - u->length = 0; - - if (ctx->in == NULL - && ctx->out == NULL - && ctx->output_closed - && b->last == b->pos) - { - u->keepalive = 1; - } - - break; + ctx->done = 1; } } -- Maxim Dounin http://mdounin.ru/ From nginx-forum at forum.nginx.org Wed Aug 22 02:58:15 2018 From: nginx-forum at forum.nginx.org (jinsam.kim) Date: Tue, 21 Aug 2018 22:58:15 -0400 Subject: =?UTF-8?Q?=E2=80=9Cmax_conns=E2=80=9D_in_upstream_is_not_working_what_I=27?= =?UTF-8?Q?ve_expected=3B?= Message-ID: <02289aa0fa5e0c22e9562245273b68bd.NginxMailingListEnglish@forum.nginx.org> Hello Super Heroes. I want to limit connections in a service. So I used max_conns directive in upstream. But it allows twice connections as many as I've set. So, I suspected myself. Maybe? Am I used the direction in wrong way. And I found some descriptions about ?max_conns? exception rules. http://nginx.org/en/docs/http/ngx_http_upstream_module.html#server ---- If idle keepalive connections, multiple workers, and the shared memory are enabled, the total number of active and idle connections to the proxied server may exceed the max_conns value. ---- And I?ve tried change configuration of ?keepalive?, ?worker numbers? and ?shared memory?. But it still the same. It allowed twice connections as many as I expected. My config file is blow. # # max_conns Test Server # proxy_cache_path /etc/nginx/cache levels=1:2 keys_zone=my_cache:10m max_size=10g inactive=60s use_temp_path=off; proxy_cache_key "$scheme$request_method$host$request_uri"; upstream t10 { zone t1_z 1M; server localhost:8080 max_conns=5 max_fails=0 fail_timeout=1s; } server { listen 8000 ; location /test/1 { proxy_pass http://t10; # it takes 1 seconds } } And I run a shell script blow. for v in {1..40} do curl 'localhost:8000/test/1' -i -l >> 30_2.log & sleep 0.01 done ---- The url ?localhost:8080/test/1? takes 1 seconds. I expected 5 succeed and 35 fail. But it always 10 succeed, 30 failed. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,280930,280930#msg-280930 From nginx-forum at forum.nginx.org Wed Aug 22 04:07:24 2018 From: nginx-forum at forum.nginx.org (hunterqin) Date: Wed, 22 Aug 2018 00:07:24 -0400 Subject: keepalive not work with grpc In-Reply-To: <20180822015750.GB56558@mdounin.ru> References: <20180822015750.GB56558@mdounin.ru> Message-ID: <3ff8a98d6be105ef1f0c75d33c72537c.NginxMailingListEnglish@forum.nginx.org> It works. Nice job! Thank you for help. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,280765,280932#msg-280932 From nginx-forum at forum.nginx.org Wed Aug 22 09:20:54 2018 From: nginx-forum at forum.nginx.org (jeffin.joy) Date: Wed, 22 Aug 2018 05:20:54 -0400 Subject: Openssl Dynamic engine support for Nginx Message-ID: HI Nginx Team, I am working on a Dynamic engine for OpenSSL for RSA and AES. I need to add this Dynamic Engine in Nginx config file. How can I do that? I am able to do the OpenSSL speed test with my engine. What configuration I need to add for Nginx to use this Dynamic engine. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,280934,280934#msg-280934 From zchao1995 at gmail.com Wed Aug 22 09:25:40 2018 From: zchao1995 at gmail.com (tokers) Date: Wed, 22 Aug 2018 02:25:40 -0700 Subject: Openssl Dynamic engine support for Nginx In-Reply-To: References: Message-ID: Hello! How about using the openssl.cnf to load your engine? https://www.openssl.org/docs/manmaster/man5/config.html Alternatively, you may want to reference the Intel/async_mode_nginx. https://github.com/intel/asynch_mode_nginx Best Regards Alex Zhang https://github.com/tokers On August 22, 2018 at 17:21:01, jeffin.joy (nginx-forum at forum.nginx.org) wrote: HI Nginx Team, I am working on a Dynamic engine for OpenSSL for RSA and AES. I need to add this Dynamic Engine in Nginx config file. How can I do that? I am able to do the OpenSSL speed test with my engine. What configuration I need to add for Nginx to use this Dynamic engine. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,280934,280934#msg-280934 _______________________________________________ nginx mailing list nginx at nginx.org http://mailman.nginx.org/mailman/listinfo/nginx -------------- next part -------------- An HTML attachment was scrubbed... URL: From dineshbabuk14 at gmail.com Wed Aug 22 09:36:27 2018 From: dineshbabuk14 at gmail.com (K Dinesh Babu) Date: Wed, 22 Aug 2018 15:06:27 +0530 Subject: nginx for windows - reg Message-ID: Hi, I have questions regarding nginx for windows, 1. Is nginx for windows stable?(I read here that it is in beta) 2. Are there any thirdparty nginx builds for windows platform that are stable? 3. I have seen nginx built with cywin here and here .Are these good for production? Kindly help me out.Thanks in advance. Regard, Dinesh Babu K -------------- next part -------------- An HTML attachment was scrubbed... URL: From vl at nginx.com Wed Aug 22 09:57:21 2018 From: vl at nginx.com (Vladimir Homutov) Date: Wed, 22 Aug 2018 12:57:21 +0300 Subject: =?UTF-8?Q?Re=3A_=E2=80=9Cmax_conns=E2=80=9D_in_upstream_is_not_working_wha?= =?UTF-8?Q?t_I=27ve_expected=3B?= In-Reply-To: <02289aa0fa5e0c22e9562245273b68bd.NginxMailingListEnglish@forum.nginx.org> References: <02289aa0fa5e0c22e9562245273b68bd.NginxMailingListEnglish@forum.nginx.org> Message-ID: <20180822095720.GA18695@vlpc> On Tue, Aug 21, 2018 at 10:58:15PM -0400, jinsam.kim wrote: > Hello Super Heroes. > > I want to limit connections in a service. So I used max_conns directive in > upstream. > > But it allows twice connections as many as I've set. > > So, I suspected myself. Maybe? Am I used the direction in wrong way. > > And I found some descriptions about ?max_conns? exception rules. > > http://nginx.org/en/docs/http/ngx_http_upstream_module.html#server > > ---- > If idle keepalive connections, multiple workers, and the shared memory are > enabled, the total number of active and idle connections to the proxied > server may exceed the max_conns value. > ---- > > And I?ve tried change configuration of ?keepalive?, ?worker numbers? and > ?shared memory?. But it still the same. It allowed twice connections as many > as I expected. > > > My config file is blow. > > # > # max_conns Test Server > # > > proxy_cache_path /etc/nginx/cache levels=1:2 keys_zone=my_cache:10m > max_size=10g inactive=60s use_temp_path=off; > > proxy_cache_key "$scheme$request_method$host$request_uri"; > > upstream t10 { > zone t1_z 1M; > server localhost:8080 max_conns=5 max_fails=0 fail_timeout=1s; > } > > server { > listen 8000 ; > location /test/1 { > proxy_pass http://t10; # it takes 1 seconds > } > } > > And I run a shell script blow. > > for v in {1..40} > do > curl 'localhost:8000/test/1' -i -l >> 30_2.log & > sleep 0.01 > done > > ---- > > The url ?localhost:8080/test/1? takes 1 seconds. > > I expected 5 succeed and 35 fail. But it always 10 succeed, 30 failed. > Note that you are using 'localhost' in your upstream definition, not IP address. Most probably it resolves into 2 addresses (127.0.0.1 and [::1] I assume), so what you see is expected - each of peers processes 5 connections, giving 10 summary, and 30 are rejected, and 'no live upstreams' are shown in error.log; the configuration with hostname in your case is equal to: upstream t10 { zone t1_z 1M; server 127.0.0.1:8080 max_conns=5 max_fails=0 fail_timeout=1s; server [::1]:8080 max_conns=5 max_fails=0 fail_timeout=1s; } From joseph.wonesh at sticknfind.com Wed Aug 22 14:24:23 2018 From: joseph.wonesh at sticknfind.com (Joseph Wonesh) Date: Wed, 22 Aug 2018 10:24:23 -0400 Subject: Proxy_Protocol with local Proxy_Pass to local secondary nginx process Message-ID: Hello, I am trying to route requests such that those requiring websockets will route to a long-lived nginx process, and all others will go to the general reverse-proxy which handles all other traffic. These nginx processes exist in our AWS cloud behind an ELB that has been configured to use Proxy Protocol. Note that all of this works correctly with our current setup which uses only one nginx process that is configured to use proxy_protocol. The change to this setup is as follows: The first nginx server handling all ingress uses proxy_protocol and forwards requests to either the websocket or non-websocket nginx servers locally: server { listen 8080 proxy_protocol; real_ip_header proxy_protocol; charset utf-8; client_max_body_size 20M; #send to websocket process location /client { proxy_pass http://localhost:8084; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Real-IP $proxy_protocol_addr; proxy_set_header X-NginX-Proxy true; proxy_set_header X-Proxy-Scheme $scheme; proxy_set_header X-Proxy-Port $proxy_port; proxy_set_header X-ELB-Proxy-Scheme "https"; proxy_set_header X-ELB-Proxy-Port "443"; # Always support web socket connection upgrades proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; } #send to non-websocket process location / { proxy_pass http://localhost:8082; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Real-IP $proxy_protocol_addr; proxy_set_header X-NginX-Proxy true; proxy_set_header X-Proxy-Scheme $scheme; proxy_set_header X-Proxy-Port $proxy_port; proxy_set_header X-ELB-Proxy-Scheme "https"; proxy_set_header X-ELB-Proxy-Port "443"; # Always support web socket connection upgrades proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; } When any non-websocket request is sent to localhost:8082, I get an empty reply. If I remove proxy_protocol from the first server, I get a response as expected. Obviously, I need proxy_protocl to support the ingress from our ELB, so removing it is not an option. However, I would like to know what pieces I am missing to route traffic correctly -- and I would also like to know why proxying a request locally from a proxy_protocol enabled server to another nginx process (regardless of this second process using proxy_protocol or not) fails. For reference, the basic configuration of this secondary nginx process is below: upstream console { server localhost:3000 max_fails=3 fail_timeout=60 weight=1; } server { listen 8082; client_max_body_size 20M; location /console { proxy_pass http://console } . . . } Thank you all for your time, Joseph Wonesh -- This message is private and confidential. If you have received message in error, please notify us and remove from your system.? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bee.lists at gmail.com Thu Aug 23 00:32:01 2018 From: bee.lists at gmail.com (Bee.Lists) Date: Wed, 22 Aug 2018 20:32:01 -0400 Subject: Proper User Directive Message-ID: <648BFFE0-721E-44A9-89A6-67E9AC2DA899@gmail.com> Hi there. My question is about using a proper user. This directive was initially blank, then Passenger (nginx on FreeBSD) threw an error: env: bash: No such file or directory uid=65534(nobody) gid=65534(nobody) groups=65534(nobody) PWD=/usr/local/www/pneb HOME=/nonexistent SHELL=/usr/sbin/nologin LOGNAME=nobody USER=nobody PASSENGER_APP_ENV=production NODE_ENV=production WSGI_ENV=production RACK_ENV=production RAILS_ENV=production NODE_PATH=/usr/local/lib/ruby/gems/2.4/gems/passenger-5.3.4/src/nodejs_supportlib PYTHONUNBUFFERED=1 PASSENGER_SPAWN_WORK_DIR=/tmp/passenger.spawn.8FRO7ztegy IN_PASSENGER=1 SERVER_SOFTWARE=nginx/1.14.0 Phusion_Passenger/5.3.4 PASSENGER_USE_FEEDBACK_FD=true PATH=/sbin:/bin:/usr/sbin:/usr/bin RC_PID=35522 So the lack of a user closed up the $PATH to bash for some reason. I tried www and my own username, resulting in the same errors. The example on nginx.com states "www www? as the user, but that gives me the same error. I checked the documentation and it is not even two lines long: http://nginx.org/en/docs/ngx_core_module.html#user So I?m without a user and/or group to run nginx. Is there something I?m missing? Everywhere I look, people are confused about the user/group to run nginx as. ?root? doesn?t work, and I?ve been told not to run it as root. How can I get this to work? Cheers, Bee From nginx-forum at forum.nginx.org Thu Aug 23 01:34:50 2018 From: nginx-forum at forum.nginx.org (jinsam.kim) Date: Wed, 22 Aug 2018 21:34:50 -0400 Subject: =?UTF-8?Q?Re=3A_=E2=80=9Cmax_conns=E2=80=9D_in_upstream_is_not_working_wha?= =?UTF-8?Q?t_I=27ve_expected=3B?= In-Reply-To: <02289aa0fa5e0c22e9562245273b68bd.NginxMailingListEnglish@forum.nginx.org> References: <02289aa0fa5e0c22e9562245273b68bd.NginxMailingListEnglish@forum.nginx.org> Message-ID: I changed "localhost:8000" ==> "127.0.0.1:8000", and it work as I expected! Posted at Nginx Forum: https://forum.nginx.org/read.php?2,280930,280944#msg-280944 From arut at nginx.com Thu Aug 23 10:46:11 2018 From: arut at nginx.com (Roman Arutyunyan) Date: Thu, 23 Aug 2018 13:46:11 +0300 Subject: Proxy_Protocol with local Proxy_Pass to local secondary nginx process In-Reply-To: References: Message-ID: <20180823104611.GU43355@Romans-MacBook-Air.local> Hello Joseph, On Wed, Aug 22, 2018 at 10:24:23AM -0400, Joseph Wonesh wrote: > Hello, > > I am trying to route requests such that those requiring websockets will > route to a long-lived nginx process, and all others will go to the general > reverse-proxy which handles all other traffic. These nginx processes exist > in our AWS cloud behind an ELB that has been configured to use Proxy > Protocol. Note that all of this works correctly with our current setup > which uses only one nginx process that is configured to use proxy_protocol. > > The change to this setup is as follows: > > The first nginx server handling all ingress uses proxy_protocol and > forwards requests to either the websocket or non-websocket nginx servers > locally: > > server { > listen 8080 proxy_protocol; > real_ip_header proxy_protocol; > charset utf-8; > client_max_body_size 20M; > > #send to websocket process > location /client { > proxy_pass http://localhost:8084; > proxy_set_header Host $host; > proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; > proxy_set_header X-Real-IP $proxy_protocol_addr; > proxy_set_header X-NginX-Proxy true; > proxy_set_header X-Proxy-Scheme $scheme; > proxy_set_header X-Proxy-Port $proxy_port; > proxy_set_header X-ELB-Proxy-Scheme "https"; > proxy_set_header X-ELB-Proxy-Port "443"; > > # Always support web socket connection upgrades > proxy_http_version 1.1; > proxy_set_header Upgrade $http_upgrade; > proxy_set_header Connection "upgrade"; > } > > #send to non-websocket process > location / { > proxy_pass http://localhost:8082; > proxy_set_header Host $host; > proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; > proxy_set_header X-Real-IP $proxy_protocol_addr; > proxy_set_header X-NginX-Proxy true; > proxy_set_header X-Proxy-Scheme $scheme; > proxy_set_header X-Proxy-Port $proxy_port; > proxy_set_header X-ELB-Proxy-Scheme "https"; > proxy_set_header X-ELB-Proxy-Port "443"; > > # Always support web socket connection upgrades > proxy_http_version 1.1; > proxy_set_header Upgrade $http_upgrade; > proxy_set_header Connection "upgrade"; > } > > > When any non-websocket request is sent to localhost:8082, I get an empty > reply. If I remove proxy_protocol from the first server, I get a response > as expected. If removing the proxy_protocol parameter from a server's listen directive fixes the issue with a specific connection, this means that this connection does not support the PROXY protocol, whether it is curl or a proxied connection from AWS. So please make sure the PROXY protocol header is actually sent over the connection. It's easy to check by tcpdumping at most 107 first bytes, especially if PROXY protocol v1 is used. > Obviously, I need proxy_protocl to support the ingress from > our ELB, so removing it is not an option. However, I would like to know > what pieces I am missing to route traffic correctly -- and I would also > like to know why proxying a request locally from a proxy_protocol enabled > server to another nginx process (regardless of this second process using > proxy_protocol or not) fails. If you configure nginx to expect PROXY protocol (listen ... proxy_protocol), then the client should actually send it. AWS balancers can send it, as well as nginx stream module can (proxy_protocol on). The nginx http module cannot send PROXY protocol header (use X-Forwarded-For instead), so you cannot proxy an HTTP connection to a PROXY-protocol-enabled server. > For reference, the basic configuration of this secondary nginx process is > below: > > upstream console { > server localhost:3000 max_fails=3 fail_timeout=60 weight=1; > } > > server { > listen 8082; > client_max_body_size 20M; > > location /console { > proxy_pass http://console > } > > . > . > . > > > } > > > > Thank you all for your time, > Joseph Wonesh > > -- > This message is private and confidential. If you have received message in > error, please notify us and remove from your system.? > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx -- Roman Arutyunyan From michael.friscia at yale.edu Thu Aug 23 11:21:02 2018 From: michael.friscia at yale.edu (Friscia, Michael) Date: Thu, 23 Aug 2018 11:21:02 +0000 Subject: gzip_types Message-ID: <8FFECB47-DA9F-4EF0-B8B6-EBF84A19CCB4@yale.edu> What would be the reason that setting Gzip_types *; Is bad? I?m running into a compression problem and if I set it to * everything gzips just fine but if I list them out explicitly the type image/jpeg is not being gzip?d via proxy request but all others are gzip?d by proxy. Gzip_proxied is set to any. ___________________________________________ Michael Friscia Office of Communications Yale School of Medicine (203) 737-7932 - office (203) 931-5381 - mobile http://web.yale.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From joseph.wonesh at sticknfind.com Thu Aug 23 14:32:04 2018 From: joseph.wonesh at sticknfind.com (Joseph Wonesh) Date: Thu, 23 Aug 2018 10:32:04 -0400 Subject: Proxy_Protocol with local Proxy_Pass to local secondary nginx process In-Reply-To: <20180823104611.GU43355@Romans-MacBook-Air.local> References: <20180823104611.GU43355@Romans-MacBook-Air.local> Message-ID: Thank you for your reply, Roman. It turns out the issue was adding the various proxy and forwarded headers in that entry. Changing it to the below block fixed my issue: #send to non-websocket process location / { proxy_pass http://localhost:8082; proxy_set_header Host $host; } Now everything is working as expected. Cheers, Joseph Wonesh On Thu, Aug 23, 2018 at 6:46 AM, Roman Arutyunyan wrote: > Hello Joseph, > > On Wed, Aug 22, 2018 at 10:24:23AM -0400, Joseph Wonesh wrote: > > Hello, > > > > I am trying to route requests such that those requiring websockets will > > route to a long-lived nginx process, and all others will go to the > general > > reverse-proxy which handles all other traffic. These nginx processes > exist > > in our AWS cloud behind an ELB that has been configured to use Proxy > > Protocol. Note that all of this works correctly with our current setup > > which uses only one nginx process that is configured to use > proxy_protocol. > > > > The change to this setup is as follows: > > > > The first nginx server handling all ingress uses proxy_protocol and > > forwards requests to either the websocket or non-websocket nginx servers > > locally: > > > > server { > > listen 8080 proxy_protocol; > > real_ip_header proxy_protocol; > > charset utf-8; > > client_max_body_size 20M; > > > > #send to websocket process > > location /client { > > proxy_pass http://localhost:8084; > > proxy_set_header Host $host; > > proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; > > proxy_set_header X-Real-IP $proxy_protocol_addr; > > proxy_set_header X-NginX-Proxy true; > > proxy_set_header X-Proxy-Scheme $scheme; > > proxy_set_header X-Proxy-Port $proxy_port; > > proxy_set_header X-ELB-Proxy-Scheme "https"; > > proxy_set_header X-ELB-Proxy-Port "443"; > > > > # Always support web socket connection upgrades > > proxy_http_version 1.1; > > proxy_set_header Upgrade $http_upgrade; > > proxy_set_header Connection "upgrade"; > > } > > > > #send to non-websocket process > > location / { > > proxy_pass http://localhost:8082; > > proxy_set_header Host $host; > > proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; > > proxy_set_header X-Real-IP $proxy_protocol_addr; > > proxy_set_header X-NginX-Proxy true; > > proxy_set_header X-Proxy-Scheme $scheme; > > proxy_set_header X-Proxy-Port $proxy_port; > > proxy_set_header X-ELB-Proxy-Scheme "https"; > > proxy_set_header X-ELB-Proxy-Port "443"; > > > > # Always support web socket connection upgrades > > proxy_http_version 1.1; > > proxy_set_header Upgrade $http_upgrade; > > proxy_set_header Connection "upgrade"; > > } > > > > > > When any non-websocket request is sent to localhost:8082, I get an empty > > reply. If I remove proxy_protocol from the first server, I get a response > > as expected. > > If removing the proxy_protocol parameter from a server's listen directive > fixes > the issue with a specific connection, this means that this connection does > not > support the PROXY protocol, whether it is curl or a proxied connection > from AWS. > So please make sure the PROXY protocol header is actually sent over the > connection. It's easy to check by tcpdumping at most 107 first bytes, > especially if PROXY protocol v1 is used. > > > Obviously, I need proxy_protocl to support the ingress from > > our ELB, so removing it is not an option. However, I would like to know > > what pieces I am missing to route traffic correctly -- and I would also > > like to know why proxying a request locally from a proxy_protocol enabled > > server to another nginx process (regardless of this second process using > > proxy_protocol or not) fails. > > If you configure nginx to expect PROXY protocol (listen ... > proxy_protocol), > then the client should actually send it. AWS balancers can send it, as > well > as nginx stream module can (proxy_protocol on). The nginx http module > cannot send PROXY protocol header (use X-Forwarded-For instead), so you > cannot proxy an HTTP connection to a PROXY-protocol-enabled server. > > > For reference, the basic configuration of this secondary nginx process is > > below: > > > > upstream console { > > server localhost:3000 max_fails=3 fail_timeout=60 weight=1; > > } > > > > server { > > listen 8082; > > client_max_body_size 20M; > > > > location /console { > > proxy_pass http://console > > } > > > > . > > . > > . > > > > > > } > > > > > > > > Thank you all for your time, > > Joseph Wonesh > > > > -- > > This message is private and confidential. If you have received message > in > > error, please notify us and remove from your system. > > > _______________________________________________ > > nginx mailing list > > nginx at nginx.org > > http://mailman.nginx.org/mailman/listinfo/nginx > > > -- > Roman Arutyunyan > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx > -- This message is private and confidential. If you have received message in error, please notify us and remove from your system.? -------------- next part -------------- An HTML attachment was scrubbed... URL: From nginx-forum at forum.nginx.org Thu Aug 23 16:55:16 2018 From: nginx-forum at forum.nginx.org (shepardeg) Date: Thu, 23 Aug 2018 12:55:16 -0400 Subject: Build nginx for ARMV7l Message-ID: Good afternoon, community! There was a need to compile nginx under the ARMV7L platform, but the guides on the links do not help. https://superuser.com/questions/1098001/nginx-on-armv7l-installation or this http://tiebing.blogspot.com/2014/09/cross-compile-nginx-for-arm.html Maybe there is another way? Posted at Nginx Forum: https://forum.nginx.org/read.php?2,280952,280952#msg-280952 From mdounin at mdounin.ru Thu Aug 23 17:08:18 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 23 Aug 2018 20:08:18 +0300 Subject: Build nginx for ARMV7l In-Reply-To: References: Message-ID: <20180823170818.GE56558@mdounin.ru> Hello! On Thu, Aug 23, 2018 at 12:55:16PM -0400, shepardeg wrote: > Good afternoon, community! There was a need to compile nginx under the > ARMV7L platform, but the guides on the links do not help. > https://superuser.com/questions/1098001/nginx-on-armv7l-installation > or this > http://tiebing.blogspot.com/2014/09/cross-compile-nginx-for-arm.html > > Maybe there is another way? Don't try to cross-compile nginx unless you understand what are you doing and ready handle miscompiled results; nginx does not support cross-compilation. Use native compilation instead. That is, compile nginx on an ARM7L host - either on the host you are planning to use it on, or in a virtual machine. There should no problems with native compilation, just a usual "./configure && make" will do the trick (see http://nginx.org/en/docs/configure.html for details). -- Maxim Dounin http://mdounin.ru/ From mdounin at mdounin.ru Thu Aug 23 17:30:48 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 23 Aug 2018 20:30:48 +0300 Subject: nginx for windows - reg In-Reply-To: References: Message-ID: <20180823173047.GF56558@mdounin.ru> Hello! On Wed, Aug 22, 2018 at 03:06:27PM +0530, K Dinesh Babu wrote: > 1. Is nginx for windows stable?(I read here > that it is in beta) The information on https://nginx.org/en/docs/windows.html is actual. That is, nginx/Windows is in beta and have various known issues. It is good enough for various development and testing tasks, but it may not be a good idea to use it in production. -- Maxim Dounin http://mdounin.ru/ From mdounin at mdounin.ru Thu Aug 23 18:00:21 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 23 Aug 2018 21:00:21 +0300 Subject: gzip_types In-Reply-To: <8FFECB47-DA9F-4EF0-B8B6-EBF84A19CCB4@yale.edu> References: <8FFECB47-DA9F-4EF0-B8B6-EBF84A19CCB4@yale.edu> Message-ID: <20180823180021.GG56558@mdounin.ru> Hello! On Thu, Aug 23, 2018 at 11:21:02AM +0000, Friscia, Michael wrote: > What would be the reason that setting > Gzip_types *; > Is bad? Usually there are responses which are not compressible, so trying to gzip them is just a waste of resources. Also there can be cases when the client announces gzip support, but in fact have problems when some types of responses are compressed - for example, there were problems with compressed javascripts in some browsers in the past. If you are sure that in a particular configuration all possible responses must be gzipped, "gzip_types *;" should be fine. You may want to limit it to a particular location or server though. > I?m running into a compression problem and if I set it to * > everything gzips just fine but if I list them out explicitly the > type image/jpeg is not being gzip?d via proxy request but all > others are gzip?d by proxy. Gzip_proxied is set to any. As long as Content-Type of the response is "image/jpeg", there should be not difference between "gzip_types *" and "gzip_types image/jpeg ...". If you see a difference, please provide more details - full configuration and a debug log would be ideal. -- Maxim Dounin http://mdounin.ru/ From james_ at catbus.co.uk Fri Aug 24 07:59:09 2018 From: james_ at catbus.co.uk (James Beal) Date: Fri, 24 Aug 2018 08:59:09 +0100 Subject: cache key > 1049 characters results in 502 Message-ID: We currently use caching for guests, our search pages use long urls to pass the parameters to our application. Currently searches that worked for logged in users don't work for guests. I can show the issue with these two curl examples ( which are not obviously valid searches ) As a guest james_ at Sophie:/mnt/c/Users/james$ curl -I ? ?"https://archiveofourown.org/works?a=111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111" HTTP/1.1 502 Bad Gateway Server: nginx/1.13.7 Content-Type: text/html Connection: keep-alive X-Proxy-Cache: MISS X-Hostname: ao3-front01 Date: Fri, 24 Aug 2018 07:19:10 GMT X-Page-Speed: 1.13.35.1-0 Cache-Control: max-age=0, no-cache, s-maxage=10 Faked logged in james_ at Sophie:/mnt/c/Users/james$ ?curl -I ? --cookie "user_credentials=Yes" ? ?"https://archiveofourown.org/works?a=111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111" HTTP/1.1 200 OK Server: nginx/1.13.7 Content-Type: text/html; charset=utf-8 Connection: keep-alive X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block X-Content-Type-Options: nosniff ETag: W/"1790278d744d2e531bec864667a1668c" Set-Cookie: _otwarchive_session=*******c1ab; path=/; expires=Fri, 07 Sep 2018 07:55:19 -0000; HttpOnly X-Request-Id: 236abd18-2463-4163-ab7e-24b3b10e572e X-Runtime: 0.182444 X-Aooo-Debug1: Archive Unicorn X-Clacks-Overhead: GNU Terry Pratchett Potential_upstream: unicorn_story X-Hostname: ao3-front01 Date: Fri, 24 Aug 2018 07:55:19 GMT X-Page-Speed: 1.13.35.1-0 Cache-Control: max-age=0, no-cache, must-revalidate One less character james_ at Sophie:/mnt/c/Users/james$ curl -I ? ?"https://archiveofourown.org/works?a=11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111" HTTP/1.1 200 OK Server: nginx/1.13.7 Content-Type: text/html; charset=utf-8 Connection: keep-alive X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block X-Content-Type-Options: nosniff X-Request-Id: d4cb7c78-78bd-4de5-a974-1695db5a6d23 X-Runtime: 0.188035 X-Proxy-Cache: HIT X-Hostname: ao3-front01 X-Clacks-Overhead: GNU Terry Pratchett X-ao3-caching-backend: unicorn_story X-Page-Speed: 1.13.35.1-0 X-Aooo-Debug1: Archive Unicorn X-Clacks-Overhead: GNU Terry Pratchett Potential_upstream: unicorn_cache X-Hostname: ao3-front01 Date: Fri, 24 Aug 2018 07:19:17 GMT X-Page-Speed: 1.13.35.1-0 Cache-Control: max-age=0, no-cache I think the relevant bits of the config are: ? ? client_body_buffer_size 2048k; ? ? proxy_buffers ? ? 32 64k; ? ? proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; ? ? proxy_set_header Host $host; ? ? proxy_set_header X-Queue-Start "t=${msec}000"; ? ? proxy_redirect off; ? ? proxy_read_timeout 300; ? ? proxy_cache_path /var/cache/nginx/downloads keys_zone=downloads_cache:4096m loader_threshold=300 loader_files=200 max_size=300g inactive=30d ?levels=2:2; ? ? proxy_cache_path /var/cache/nginx/ao3 ? ? ? keys_zone=ao3_cache:4096m ? ? ? loader_threshold=300 loader_files=200 max_size=70g inactive=1h ? levels=2:2; ? ? proxy_cache_path /var/cache/nginx/tmpfs ? ? keys_zone=ao3_tmpfs:4096m ? ? ?loader_threshold=300 loader_files=200 max_size=40g ?inactive=1h ? levels=2:2; ? ? large_client_header_buffers 4 32k; And ? ? ? ? location / { ? ? ? ? ? proxy_cache ao3_tmpfs; ? ? ? ? ? proxy_cache_key "$request_uri"; ? ? ? ? ? proxy_cache_valid 200 302 40m; ? ? ? ? ? proxy_cache_valid 404 ? ? 10m; ? ? ? ? ? proxy_cache_use_stale ? error timeout invalid_header updating; ? ? ? ? ? proxy_cache_lock on ; ? ? ? ? ? proxy_cache_min_uses 1 ; ? ? ? ? ? proxy_buffering ? ? ? ?on ; ? ? ? ? ? proxy_hide_header ? ? ?Cache-Control ; ? ? ? ? ? proxy_hide_header ? ? ?Set-Cookie ; ? ? ? ? ? proxy_ignore_headers ? Cache-Control ; ? ? ? ? ? proxy_ignore_headers ? Set-Cookie ; ? ? ? ? proxy_http_version 1.1; ? ? ? ? proxy_set_header X-Forwarded-For "$http_x_forwarded_for,$realip_remote_addr" ; ? ? ? ? proxy_set_header X-Forwarded-Proto $scheme ; ? ? ? ? proxy_headers_hash_bucket_size 4096 ; ? ? ? ? proxy_redirect off; ? ? ? ? proxy_set_header Connection ""; ? ? ? ? real_ip_recursive on ; ? ? ? ? set_real_ip_from 10.0.0.0/8 ; ? ? ? ? real_ip_header X-Forwarded-For; ? ? ? ? client_max_body_size 4G; ? ? ? ? keepalive_timeout 5; ? ? ? ? ?proxy_set_header Host archiveofourown.org ; ? ? ? ? ? add_header ? ? ? ? ? ? X-Proxy-Cache $upstream_cache_status always; ? ? ? ? ? add_header ? ? ? ? ? ? X-Hostname $hostname always; ? ? ? ? ? add_header ? ? ? ? ? ? Cache-Control "public" always; ? ? ? ? ? add_header ? ? ? ? ? ? X-Clacks-Overhead "GNU Terry Pratchett" ; ? ? ? ? ? add_header ? ? ? ? ? ? X-ao3-caching-backend "$unicorn_cache_backend" ; ? ? ? ? ? proxy_pass http://$unicorn_cache_backend; ? ? ? ? ? expires 40m ; ? ? ? ? ?} -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdi_knght at hotmail.com Fri Aug 24 10:55:02 2018 From: jdi_knght at hotmail.com (Matt Gadient) Date: Fri, 24 Aug 2018 10:55:02 +0000 Subject: gzip_static module does not obey mime types Message-ID: I've noticed for a while that when gzip_static is enabled it checks *every* file it serves for a .gz extension, not just the ones listed in the gzip_types. For example in a page with 1 html and 30 images you'll have 61 requests to the file system: 1 for the correct .html.gz file, 30 for the non-existent .png.gz files, and 30 for the .png files. You can verify this on a running server that has gzip_static on, with "strace -p PID 2>&1 | grep gz" (where PID is an nginx worker's PID). Visit a page of the site that has images and you'll get a whole bunch of "-1 ENOENT (No such file or directory)" in the output attached to /var/www/website/image.png.gz files. I am wondering if this is intentional or if it is a bug. Thanks for your time. From michael.friscia at yale.edu Fri Aug 24 10:56:00 2018 From: michael.friscia at yale.edu (Friscia, Michael) Date: Fri, 24 Aug 2018 10:56:00 +0000 Subject: gzip_types In-Reply-To: <20180823180021.GG56558@mdounin.ru> References: <8FFECB47-DA9F-4EF0-B8B6-EBF84A19CCB4@yale.edu> <20180823180021.GG56558@mdounin.ru> Message-ID: <5AE71A8E-342E-4A77-A5A7-159BB16B0D35@yale.edu> Thank you for that explanation. I'm still debugging this with our CDN vendor. They have identified a number of bugs on their end which explain the odd behavior I've been seeing. ___________________________________________ Michael Friscia Office of Communications Yale School of Medicine (203) 737-7932 - office (203) 931-5381 - mobile http://web.yale.edu ?On 8/23/18, 2:00 PM, "nginx on behalf of Maxim Dounin" wrote: Hello! On Thu, Aug 23, 2018 at 11:21:02AM +0000, Friscia, Michael wrote: > What would be the reason that setting > Gzip_types *; > Is bad? Usually there are responses which are not compressible, so trying to gzip them is just a waste of resources. Also there can be cases when the client announces gzip support, but in fact have problems when some types of responses are compressed - for example, there were problems with compressed javascripts in some browsers in the past. If you are sure that in a particular configuration all possible responses must be gzipped, "gzip_types *;" should be fine. You may want to limit it to a particular location or server though. > I?m running into a compression problem and if I set it to * > everything gzips just fine but if I list them out explicitly the > type image/jpeg is not being gzip?d via proxy request but all > others are gzip?d by proxy. Gzip_proxied is set to any. As long as Content-Type of the response is "image/jpeg", there should be not difference between "gzip_types *" and "gzip_types image/jpeg ...". If you see a difference, please provide more details - full configuration and a debug log would be ideal. -- Maxim Dounin https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmdounin.ru%2F&data=02%7C01%7Cmichael.friscia%40yale.edu%7C6c5e1fd929a949e6066108d609224e47%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C636706440293411790&sdata=cjPwik9wTdi0Wpw0InRg9zlE2lvOY3ejsvkvlhBO4f8%3D&reserved=0 _______________________________________________ nginx mailing list nginx at nginx.org https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.nginx.org%2Fmailman%2Flistinfo%2Fnginx&data=02%7C01%7Cmichael.friscia%40yale.edu%7C6c5e1fd929a949e6066108d609224e47%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C636706440293411790&sdata=Vjpk7hlLViXxOIXo2Tav57J%2FcgRyA6fI5ppkDzs6E18%3D&reserved=0 From r1ch+nginx at teamliquid.net Fri Aug 24 15:11:27 2018 From: r1ch+nginx at teamliquid.net (Richard Stanway) Date: Fri, 24 Aug 2018 17:11:27 +0200 Subject: gzip_static module does not obey mime types In-Reply-To: References: Message-ID: This seems intentional, per the documentation at http://nginx.org/en/docs/http/ngx_http_gzip_static_module.html: "The following directives are also taken into account: gzip_http_version, gzip_proxied, gzip_disable, and gzip_vary." No mention of gzip_types. I recommend only enabling gzip_static in a location block where you know you have pre-compressed assets present. On Fri, Aug 24, 2018 at 12:55 PM Matt Gadient wrote: > I've noticed for a while that when gzip_static is enabled it checks > *every* file it serves for a .gz extension, not just the ones listed in the > gzip_types. For example in a page with 1 html and 30 images you'll have 61 > requests to the file system: 1 for the correct .html.gz file, 30 for the > non-existent .png.gz files, and 30 for the .png files. > > You can verify this on a running server that has gzip_static on, with > "strace -p PID 2>&1 | grep gz" (where PID is an nginx worker's PID). Visit > a page of the site that has images and you'll get a whole bunch of "-1 > ENOENT (No such file or directory)" in the output attached to > /var/www/website/image.png.gz files. > > I am wondering if this is intentional or if it is a bug. Thanks for your > time. > > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx > -------------- next part -------------- An HTML attachment was scrubbed... URL: From james_ at catbus.co.uk Fri Aug 24 15:37:54 2018 From: james_ at catbus.co.uk (James Beal) Date: Fri, 24 Aug 2018 16:37:54 +0100 Subject: cache key > 1049 characters results in 502 In-Reply-To: References: Message-ID: On 24/08/2018 08:59:09, James Beal wrote: We currently use caching for guests, our search pages use long urls to pass the parameters to our application. Currently searches that worked for logged in users don't work for guests. I can show the issue with these two curl examples ( which are not obviously valid searches ) James Beal:? After applying strace the solution appears to be: ?proxy_buffer_size ? 128k; -------------- next part -------------- An HTML attachment was scrubbed... URL: From aldernetwork at gmail.com Fri Aug 24 21:54:31 2018 From: aldernetwork at gmail.com (Alder Netw) Date: Fri, 24 Aug 2018 14:54:31 -0700 Subject: Questions regarding worker_connections Message-ID: Got a question regarding the worker_connections configuration. Is a worker_connection essentially a tcp socket connection or http session? Via the http_stub_status_module, I can see there are connections in waiting state, but any new request will result in a "worker connections are not enough" alert in error.log. Why it cannot accept new connections if there are existing connections in waiting state? Is that a bug? I am using nginx 1.6.2. Thanks, Alder -------------- next part -------------- An HTML attachment was scrubbed... URL: From danjel at jungersen.dk Sun Aug 26 08:12:23 2018 From: danjel at jungersen.dk (Jungersen, Danjel - Jungersen Grafisk ApS) Date: Sun, 26 Aug 2018 10:12:23 +0200 Subject: reverse proxy https not working Message-ID: <5B8260E7.23731.3D11D923@danjel.jungersen.dk> An HTML attachment was scrubbed... URL: From lucas at lucasrolff.com Sun Aug 26 08:19:28 2018 From: lucas at lucasrolff.com (Lucas Rolff) Date: Sun, 26 Aug 2018 08:19:28 +0000 Subject: reverse proxy https not working In-Reply-To: <5B8260E7.23731.3D11D923@danjel.jungersen.dk> References: <5B8260E7.23731.3D11D923@danjel.jungersen.dk> Message-ID: Which functions do not work? Be aware some software (WordPress being a good example) doesn?t always work with reverse proxies that easy. Could you possibly include your nginx configuration? Especially your proxy parts. From: nginx on behalf of "Jungersen, Danjel - Jungersen Grafisk ApS" Organization: Jungersen Grafisk ApS Reply-To: "nginx at nginx.org" Date: Sunday, 26 August 2018 at 10.13 To: "nginx at nginx.org" Subject: reverse proxy https not working Hi there. I have a setup that almost works. :-) I have a handful of domains that works as they should. Traffic as accepted and forwarded to my apache on another server (also in dmz). I have setup certificates with certbot. I have green (encrypted) icon on my browser when I visit my sites. 1 site is running on my green network. When I connect to that site all seems to work. However, certain functions fail, but only when connected via https. If I change the setup so that port 80 is not redirected to 443, everything works as long as I stay with http. As soon as I chenge the url to https:// some functions fail. I have tried but cannot understand the debug log. I don't see any hits on my firewall. Any clues? I will be happy to send config and logfiles, but I'm not sure exactly what to send. Best regards Danjel -------------- next part -------------- An HTML attachment was scrubbed... URL: From danjel at jungersen.dk Sun Aug 26 08:32:13 2018 From: danjel at jungersen.dk (Jungersen, Danjel - Jungersen Grafisk ApS) Date: Sun, 26 Aug 2018 10:32:13 +0200 Subject: reverse proxy https not working In-Reply-To: References: <5B8260E7.23731.3D11D923@danjel.jungersen.dk>, Message-ID: <5B82658D.12118.3D23FFDD@danjel.jungersen.dk> An HTML attachment was scrubbed... URL: From lucas at lucasrolff.com Sun Aug 26 08:47:03 2018 From: lucas at lucasrolff.com (Lucas Rolff) Date: Sun, 26 Aug 2018 08:47:03 +0000 Subject: reverse proxy https not working In-Reply-To: <5B82658D.12118.3D23FFDD@danjel.jungersen.dk> References: <5B8260E7.23731.3D11D923@danjel.jungersen.dk> <5B82658D.12118.3D23FFDD@danjel.jungersen.dk> Message-ID: > The vendor recommended me to use a reverse proxy.... Ideally the vendor should have a working config in that case, but, I do see a few things that can be an issue. You?re serving https but proxying to an http backend ? depending on how the software works, a lot of the reverse URLs that is sent back, might be linking to http:// instead of https:// This in itself can break a lot of functionality, you might want to try to proxy to an https backend ? this might require that you create a self-signed certificate on the backend (can be valid for 10 years) ? the backend software itself, if it has a way to enable ?https?, you?d have to set this as well. I also recommend removing the / (slash) in the end of the proxy_pass, this will pass through the request URI from the client, as per documentation: > If proxy_pass is specified without a URI, the request URI is passed to the server in the same form as sent by a client when the original request is processed, or the full normalized request URI is passed when processing the changed URI Alternatively do proxy_pass http://192.168.1.3$request_uri; or proxy_pass https://192.168.1.3$request_uri; Additionally, if your software uses Location or Refresh headers, then you might want to look into proxy_redirect ( http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_redirect ) to rewrite this on the ?return? to the user. Best Regards, Lucas Rolff From: nginx on behalf of "Jungersen, Danjel - Jungersen Grafisk ApS" Organization: Jungersen Grafisk ApS Reply-To: "nginx at nginx.org" Date: Sunday, 26 August 2018 at 10.33 To: "nginx at nginx.org" Subject: Re: reverse proxy https not working From: Lucas Rolff To: "nginx at nginx.org" Subject: Re: reverse proxy https not working Date sent: Sun, 26 Aug 2018 08:19:28 +0000 Send reply to: nginx at nginx.org > Which functions do not work? Thats a bit hard to say, but I'll try.. It's a print production system. 1 part is approval of pages in a job. When I try to open a page for approval the system should open up the page in large size. That does not happen. The thumbnails on the side works. And as stated, when I do the same thing when connected via http, there are no issues. > > Be aware some software (WordPress being a good example) doesn?t always work with reverse > proxies that easy. The vendor recommended me to use a reverse proxy.... > > Could you possibly include your nginx configuration? Especially your proxy parts. server { server_name portal.printlight.dk; client_max_body_size 1000m; # (I tried with and without this line) error_log /etc/nginx/log warn; location / { proxy_pass http://192.168.1.3:80/; proxy_set_header Host $host; } listen 80; listen 443 ssl; # managed by Certbot ssl_certificate /etc/letsencrypt/live/portal.printlight.dk/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/portal.printlight.dk/privkey.pem; # managed by Certbot include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot } > > From: nginx on behalf of "Jungersen, Danjel - > Jungersen Grafisk ApS" > Organization: Jungersen Grafisk ApS > Reply-To: "nginx at nginx.org" > Date: Sunday, 26 August 2018 at 10.13 > To: "nginx at nginx.org" > Subject: reverse proxy https not working > > Hi there. > > I have a setup that almost works. > :-) > > I have a handful of domains that works as they should. > Traffic as accepted and forwarded to my apache on another server (also in dmz). > I have setup certificates with certbot. > I have green (encrypted) icon on my browser when I visit my sites. > > 1 site is running on my green network. > When I connect to that site all seems to work. > However, certain functions fail, but only when connected via https. > If I change the setup so that port 80 is not redirected to 443, everything works as long as I > stay with http. > As soon as I chenge the url to https:// some functions fail. > I have tried but cannot understand the debug log. > > I don't see any hits on my firewall. > > Any clues? > I will be happy to send config and logfiles, but I'm not sure exactly what to send. > > Best regards > Danjel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From danjel at jungersen.dk Sun Aug 26 09:28:26 2018 From: danjel at jungersen.dk (Jungersen, Danjel - Jungersen Grafisk ApS) Date: Sun, 26 Aug 2018 11:28:26 +0200 Subject: reverse proxy https not working In-Reply-To: References: <5B8260E7.23731.3D11D923@danjel.jungersen.dk>, <5B82658D.12118.3D23FFDD@danjel.jungersen.dk>, Message-ID: <5B8272BA.13309.3D577989@danjel.jungersen.dk> An HTML attachment was scrubbed... URL: From lucas at lucasrolff.com Sun Aug 26 11:42:48 2018 From: lucas at lucasrolff.com (Lucas Rolff) Date: Sun, 26 Aug 2018 11:42:48 +0000 Subject: reverse proxy https not working In-Reply-To: <5B8272BA.13309.3D577989@danjel.jungersen.dk> References: <5B8260E7.23731.3D11D923@danjel.jungersen.dk> <5B82658D.12118.3D23FFDD@danjel.jungersen.dk> <5B8272BA.13309.3D577989@danjel.jungersen.dk> Message-ID: <8FA39F92-C180-45CA-B3B3-E1B90E71FDF3@lucasrolff.com> > Both did the trick, but which one is better? I personally prefer the $request_uri one because it?s very clear exactly what it does. > I think I read somewhere that nginx would connect unencrypted to the backend, and do the encryption / decryption, is this wrong then? Nginx will connect the way you?ve told it to connect, if you?re connecting to a http backend, it will do plain communication over http ? if you?re connecting to a https backend, it will establish a secure connection with the backend, and decrypt the response before encrypting it again when going to the client. > It works on some of my other domains, so is this just an exeption? > What I really ask is this: Should I change my other domains also, or should I kepp them as they are as long as they work? I would change it for consistency across your configs, but that?s my opinion ? if it works then it?s all OK anyway, I don?t know the specific case when it will and will not work ? so I by default set $request_uri because it works in 99% of the cases, and I?ll only modify it if another behaviour is required. Best Regards, Lucas Rolff From: nginx on behalf of "Jungersen, Danjel - Jungersen Grafisk ApS" Organization: Jungersen Grafisk ApS Reply-To: "nginx at nginx.org" Date: Sunday, 26 August 2018 at 11.29 To: "nginx at nginx.org" Subject: Re: reverse proxy https not working Thanks !!! proxy_pass https://192.168.1.3; proxy_pass https://192.168.1.3$request_uri; Both did the trick, but which one is better? I will now try to re-enable all the "force encryption" settings. And closing firewall ports to see what I can avoid having open. I'm a bit of novice at proxies, so please be patient :-) I will read the documentation sections you mentioned. I think I read somewhere that nginx would connect unencrypted to the backend, and do the encryption / decryption, is this wrong then? It works on some of my other domains, so is this just an exeption? What I really ask is this: Should I change my other domains also, or should I kepp them as they are as long as they work? It sounds like you recommend removing the "/" on all sites(?) A current typical setup: server { server_name www.printlight.dk; server_name printlight.dk; location / { proxy_pass http://192.168.20.3/; proxy_set_header Host $host; } listen 443 ssl; # managed by Certbot ssl_certificate /etc/letsencrypt/live/printlight.dk/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/printlight.dk/privkey.pem; # managed by Certbot include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot } server { if ($host = www.printlight.dk) { return 301 https://$host$request_uri; } # managed by Certbot if ($host = printlight.dk) { return 301 https://$host$request_uri; } # managed by Certbot listen 80; server_name www.printlight.dk; server_name printlight.dk; return 404; # managed by Certbot } Best regards Danjel From: Lucas Rolff To: "nginx at nginx.org" Subject: Re: reverse proxy https not working Date sent: Sun, 26 Aug 2018 08:47:03 +0000 Send reply to: nginx at nginx.org > > The vendor recommended me to use a reverse proxy.... > > Ideally the vendor should have a working config in that case, but, I do see a few things that can > be an issue. > > You?re serving https but proxying to an http backend ? depending on how the software works, a > lot of the reverse URLs that is sent back, might be linking to http:// instead of https:// > > This in itself can break a lot of functionality, you might want to try to proxy to an https backend > ? this might require that you create a self-signed certificate on the backend (can be valid for 10 > years) ? the backend software itself, if it has a way to enable ?https?, you?d have to set this as > well. > > I also recommend removing the / (slash) in the end of the proxy_pass, this will pass through the > request URI from the client, as per documentation: > > > If proxy_pass is specified without a URI, the request URI is passed to the server in the same > form as sent by a client when the original request is processed, or the full normalized request > URI is passed when processing the changed URI > > Alternatively do proxy_pass http://192.168.1.3$request_uri; or proxy_pass > https://192.168.1.3$request_uri; > > Additionally, if your software uses Location or Refresh headers, then you might want to look > into proxy_redirect ( > http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_redirect ) to rewrite this on > the ?return? to the user. > > Best Regards, > Lucas Rolff > > From: nginx on behalf of "Jungersen, Danjel - > Jungersen Grafisk ApS" > Organization: Jungersen Grafisk ApS > Reply-To: "nginx at nginx.org" > Date: Sunday, 26 August 2018 at 10.33 > To: "nginx at nginx.org" > Subject: Re: reverse proxy https not working > > > > From: Lucas Rolff > To: "nginx at nginx.org" > Subject: Re: reverse proxy https not working > Date sent: Sun, 26 Aug 2018 08:19:28 +0000 > Send reply to: nginx at nginx.org > > > Which functions do not work? > Thats a bit hard to say, but I'll try.. > > It's a print production system. > 1 part is approval of pages in a job. > > When I try to open a page for approval the system should open up the page in large size. > That does not happen. > The thumbnails on the side works. > And as stated, when I do the same thing when connected via http, there are no issues. > > > > > Be aware some software (WordPress being a good example) doesn?t always work with > reverse > > proxies that easy. > The vendor recommended me to use a reverse proxy.... > > > > > Could you possibly include your nginx configuration? Especially your proxy parts. > > server { > > server_name portal.printlight.dk; > > client_max_body_size 1000m; # (I tried with and without this line) > > error_log /etc/nginx/log warn; > > location / { > > proxy_pass http://192.168.1.3:80/; > > proxy_set_header Host $host; > > } > > listen 80; > listen 443 ssl; # managed by Certbot > ssl_certificate /etc/letsencrypt/live/portal.printlight.dk/fullchain.pem; # managed by Certbot > ssl_certificate_key /etc/letsencrypt/live/portal.printlight.dk/privkey.pem; # managed by > Certbot > include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot > ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot > > } > > > > > > From: nginx on behalf of "Jungersen, Danjel - > > Jungersen Grafisk ApS" > > Organization: Jungersen Grafisk ApS > > Reply-To: "nginx at nginx.org" > > Date: Sunday, 26 August 2018 at 10.13 > > To: "nginx at nginx.org" > > Subject: reverse proxy https not working > > > > Hi there. > > > > I have a setup that almost works. > > :-) > > > > I have a handful of domains that works as they should. > > Traffic as accepted and forwarded to my apache on another server (also in dmz). > > I have setup certificates with certbot. > > I have green (encrypted) icon on my browser when I visit my sites. > > > > 1 site is running on my green network. > > When I connect to that site all seems to work. > > However, certain functions fail, but only when connected via https. > > If I change the setup so that port 80 is not redirected to 443, everything works as long > as I > > stay with http. > > As soon as I chenge the url to https:// some functions fail. > > I have tried but cannot understand the debug log. > > > > I don't see any hits on my firewall. > > > > Any clues? > > I will be happy to send config and logfiles, but I'm not sure exactly what to send. > > > > Best regards > > Danjel > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From danjel at jungersen.dk Sun Aug 26 12:00:15 2018 From: danjel at jungersen.dk (Jungersen, Danjel - Jungersen Grafisk ApS) Date: Sun, 26 Aug 2018 14:00:15 +0200 Subject: reverse proxy https not working In-Reply-To: <8FA39F92-C180-45CA-B3B3-E1B90E71FDF3@lucasrolff.com> References: <5B8260E7.23731.3D11D923@danjel.jungersen.dk>, <5B8272BA.13309.3D577989@danjel.jungersen.dk>, <8FA39F92-C180-45CA-B3B3-E1B90E71FDF3@lucasrolff.com> Message-ID: <5B82964F.20198.3DE277D0@danjel.jungersen.dk> An HTML attachment was scrubbed... URL: From danjel at jungersen.dk Mon Aug 27 06:21:04 2018 From: danjel at jungersen.dk (Jungersen, Danjel - Jungersen Grafisk ApS) Date: Mon, 27 Aug 2018 08:21:04 +0200 Subject: reverse proxy https not working In-Reply-To: References: <5B8260E7.23731.3D11D923@danjel.jungersen.dk>, Message-ID: <5B839850.5699.41D24957@danjel.jungersen.dk> An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Mon Aug 27 11:56:15 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 27 Aug 2018 14:56:15 +0300 Subject: Questions regarding worker_connections In-Reply-To: References: Message-ID: <20180827115615.GJ56558@mdounin.ru> Hello! On Fri, Aug 24, 2018 at 02:54:31PM -0700, Alder Netw wrote: > Got a question regarding the worker_connections configuration. Is a > worker_connection > essentially > a tcp socket connection or http session? The worker_connections directive configures maximum number of connections a worker process can open. That is, basically this means "TCP sockets". See http://nginx.org/r/worker_connections for additional details. > Via the http_stub_status_module, I can see there are > connections in waiting state, but any new request > will result in a "worker connections are not enough" > alert in error.log. Why it cannot accept new connections > if there are existing connections in waiting state? Is that > a bug? I am using nginx 1.6.2. Connections in the waiting state are real established connections. While in some cases nginx can close and re-use these connections quickly enough to don't trigger the "worker connections are not enough" alert, this is not something guaranteed. In general, you should tune worker_connections so it's higher than expected number of active connections - that is, all connections including waiting ones. -- Maxim Dounin http://mdounin.ru/ From jsharan15 at gmail.com Mon Aug 27 13:26:01 2018 From: jsharan15 at gmail.com (Sharan J) Date: Mon, 27 Aug 2018 18:56:01 +0530 Subject: Resolver not working as expected Message-ID: Hi, Sample conf: http{ resolver x.x.x.x; server { server_name _; location / { proxy_pass http://somedomain.com; } } I have nameservers configured in my resolv.conf. But, somedomain.com will be configured in x.x.x.x DNS server only. So, I have specified resolver in my nginx.conf. However, during startup/reload I get "host not found in upstream error". Why nginx is not considering resolver conf and searches only in the nameservers configured in resolv.conf? Thanks, Shanthu -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Mon Aug 27 13:30:03 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 27 Aug 2018 16:30:03 +0300 Subject: Resolver not working as expected In-Reply-To: References: Message-ID: <20180827133003.GK56558@mdounin.ru> Hello! On Mon, Aug 27, 2018 at 06:56:01PM +0530, Sharan J wrote: > Hi, > > Sample conf: > > http{ > resolver x.x.x.x; > server { > server_name _; > location / { > proxy_pass http://somedomain.com; > } > } > > I have nameservers configured in my resolv.conf. But, somedomain.com will > be configured in x.x.x.x DNS server only. So, I have specified resolver in > my nginx.conf. However, during startup/reload I get "host not found in > upstream error". Why nginx is not considering resolver conf and searches > only in the nameservers configured in resolv.conf? Names as written in the configuration are resolved while parsing the configuration using the system resolver. DNS servers defined by the "resolver" directive are only used for dynamic / run-time name resolution - for example, when variables are used in the "proxy_pass" directive. -- Maxim Dounin http://mdounin.ru/ From aldernetwork at gmail.com Mon Aug 27 17:31:13 2018 From: aldernetwork at gmail.com (Alder Netw) Date: Mon, 27 Aug 2018 10:31:13 -0700 Subject: Questions regarding worker_connections In-Reply-To: <20180827115615.GJ56558@mdounin.ru> References: <20180827115615.GJ56558@mdounin.ru> Message-ID: Thanks Maxim for the info. So worker connections in waiting state may not be ready to be reused. Is there any maximum waiting time? We seem to run into a condition that those connections are permanently stuck in waiting state. Also, what would be the ball-park number of worker_connections should be set if we want to support expected number of active connections, say 150? Thanks, On Mon, Aug 27, 2018 at 4:56 AM, Maxim Dounin wrote: > Hello! > > On Fri, Aug 24, 2018 at 02:54:31PM -0700, Alder Netw wrote: > > > Got a question regarding the worker_connections configuration. Is a > > worker_connection > > essentially > > a tcp socket connection or http session? > > The worker_connections directive configures maximum number of > connections a worker process can open. That is, basically this > means "TCP sockets". > > See http://nginx.org/r/worker_connections for additional details. > > > Via the http_stub_status_module, I can see there are > > connections in waiting state, but any new request > > will result in a "worker connections are not enough" > > alert in error.log. Why it cannot accept new connections > > if there are existing connections in waiting state? Is that > > a bug? I am using nginx 1.6.2. > > Connections in the waiting state are real established connections. > While in some cases nginx can close and re-use these connections > quickly enough to don't trigger the "worker connections are not > enough" alert, this is not something guaranteed. > > In general, you should tune worker_connections so it's higher than > expected number of active connections - that is, all connections > including waiting ones. > > -- > Maxim Dounin > http://mdounin.ru/ > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Mon Aug 27 18:18:13 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 27 Aug 2018 21:18:13 +0300 Subject: Questions regarding worker_connections In-Reply-To: References: <20180827115615.GJ56558@mdounin.ru> Message-ID: <20180827181813.GM56558@mdounin.ru> Hello! On Mon, Aug 27, 2018 at 10:31:13AM -0700, Alder Netw wrote: > Thanks Maxim for the info. So worker connections in waiting state may not > be > ready to be reused. Is there any maximum waiting time? We seem to run into > a condition that those connections are permanently stuck in waiting state. Waiting connections are connections in the keepalive state and/or idle HTTP/2 connections. Relevant timeouts can be tuned using the keepalive_timeout and http2_idle_timeout directives, see docs here: http://nginx.org/r/keepalive_timeout http://nginx.org/r/http2_idle_timeout > Also, what would be the ball-park number of worker_connections should be > set > if we want to support expected number of active connections, say 150? If you are sure the number of active connections (that is, reading + writing + waiting) will never exceed 150, you can set worker_connections to a similar number, adding some for various other uses - including listening sockets and upstream connections. Usually 2x of the expected maximum number is a good estimate. In the particular case of 150 active connections, it should be a good idea to preserve "worker_connections 1024" as suggested by the default configuration. Simply because it hardly make sense to use anything smaller. -- Maxim Dounin http://mdounin.ru/ From kiriyama.kentaro at tis.co.jp Tue Aug 28 00:49:07 2018 From: kiriyama.kentaro at tis.co.jp (=?iso-2022-jp?B?GyRCNk07MyEhN3JCQE86GyhC?=) Date: Tue, 28 Aug 2018 00:49:07 +0000 Subject: [nginx]access log and error log Message-ID: Hello, I?m suffering a problem with access.log and error.log for nginx.service on RHEL 7.4. The problem is either access.log and error.log are not recording the log. To solve this issue, I have tried multiple time changing logrotation setting for nginx. Below is the setting of logrotation for nginx.service.(/etc/logrotate.d/nginx) /var/log/nginx/*.log { daily missingok rotate 7 compress create 644 nginx adm sharedscripts postrotate if [ -f /var/run/nginx.pid ]; then kill -USR1 'cat /var/run/nginx.pid' fi endscript } By the way, the log will start recording on both log files after restarting the nginx.service. I would like to know the issue on here and how to solve it. Regards, Kentaro -------------- next part -------------- An HTML attachment was scrubbed... URL: From away.ww at gmail.com Tue Aug 28 02:57:30 2018 From: away.ww at gmail.com (=?UTF-8?B?546L57u0?=) Date: Tue, 28 Aug 2018 10:57:30 +0800 Subject: No subject Message-ID: -------------- next part -------------- An HTML attachment was scrubbed... URL: From cooley.josh at gmail.com Tue Aug 28 05:01:39 2018 From: cooley.josh at gmail.com (Josh Cooley) Date: Tue, 28 Aug 2018 00:01:39 -0500 Subject: [nginx]access log and error log In-Reply-To: References: Message-ID: Your postrotate script contains single ticks (') instead of backticks (`), but you should really use the $( command ) construct if your /bin/sh is bash. INCORRECT (what you have now): if [ -f /var/run/nginx.pid ]; then kill -USR1 'cat /var/run/nginx.pid' fi CORRECT, BUT ARCHAIC: if [ -f /var/run/nginx.pid ]; then kill -USR1 `cat /var/run/nginx.pid` fi CORRECT AND MODERN: if [ -f /var/run/nginx.pid ]; then kill -USR1 $(cat /var/run/nginx.pid) fi On Mon, Aug 27, 2018 at 7:49 PM, ?? ??? wrote: > Hello, > > > > I?m suffering a problem with access.log and error.log for nginx.service on > RHEL 7.4. > > > > The problem is either access.log and error.log are not recording the log. > > > To solve this issue, I have tried multiple time changing logrotation > setting for nginx. > > > > Below is the setting of logrotation for nginx.service.(/etc/logrotate. > d/nginx) > > > > /var/log/nginx/*.log { > > daily > > missingok > > rotate 7 > > compress > > create 644 nginx adm > > sharedscripts > > postrotate > > if [ -f /var/run/nginx.pid ]; then > > kill -USR1 'cat /var/run/nginx.pid' > > fi > > endscript > > } > > > > By the way, the log will start recording on both log files after > restarting the nginx.service. > > > > I would like to know the issue on here and how to solve it. > > > > Regards, > > Kentaro > > > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kiriyama.kentaro at tis.co.jp Tue Aug 28 06:12:28 2018 From: kiriyama.kentaro at tis.co.jp (=?utf-8?B?5qGQ5bGx44CA5YGl5aSq6YOO?=) Date: Tue, 28 Aug 2018 06:12:28 +0000 Subject: [nginx]access log and error log In-Reply-To: References: Message-ID: Hi, Josh Appreciate for your advice. I will try either both of those and will try to fix this issue. Regards, Kentaro From: nginx [mailto:nginx-bounces at nginx.org] On Behalf Of Josh Cooley Sent: Tuesday, August 28, 2018 2:02 PM To: nginx at nginx.org Cc: ?????; ????? Subject: Re: [nginx]access log and error log Your postrotate script contains single ticks (') instead of backticks (`), but you should really use the $( command ) construct if your /bin/sh is bash. INCORRECT (what you have now): if [ -f /var/run/nginx.pid ]; then kill -USR1 'cat /var/run/nginx.pid' fi CORRECT, BUT ARCHAIC: if [ -f /var/run/nginx.pid ]; then kill -USR1 `cat /var/run/nginx.pid` fi CORRECT AND MODERN: if [ -f /var/run/nginx.pid ]; then kill -USR1 $(cat /var/run/nginx.pid) fi On Mon, Aug 27, 2018 at 7:49 PM, ?????? > wrote: Hello, I?m suffering a problem with access.log and error.log for nginx.service on RHEL 7.4. The problem is either access.log and error.log are not recording the log. To solve this issue, I have tried multiple time changing logrotation setting for nginx. Below is the setting of logrotation for nginx.service.(/etc/logrotate.d/nginx) /var/log/nginx/*.log { daily missingok rotate 7 compress create 644 nginx adm sharedscripts postrotate if [ -f /var/run/nginx.pid ]; then kill -USR1 'cat /var/run/nginx.pid' fi endscript } By the way, the log will start recording on both log files after restarting the nginx.service. I would like to know the issue on here and how to solve it. Regards, Kentaro _______________________________________________ nginx mailing list nginx at nginx.org http://mailman.nginx.org/mailman/listinfo/nginx -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists-nginx at swsystem.co.uk Tue Aug 28 12:32:35 2018 From: lists-nginx at swsystem.co.uk (Steve Wilson) Date: Tue, 28 Aug 2018 13:32:35 +0100 Subject: Resolver not working as expected In-Reply-To: <20180827133003.GK56558@mdounin.ru> References: <20180827133003.GK56558@mdounin.ru> Message-ID: <618cdda81fbe2a93172d75c46fc676dd@swsystem.co.uk> On 27/08/2018 14:30, Maxim Dounin wrote: > Hello! > > On Mon, Aug 27, 2018 at 06:56:01PM +0530, Sharan J wrote: > >> Hi, >> >> Sample conf: >> >> http{ >> resolver x.x.x.x; >> server { >> server_name _; >> location / { >> proxy_pass http://somedomain.com; >> } >> } >> >> I have nameservers configured in my resolv.conf. But, somedomain.com >> will >> be configured in x.x.x.x DNS server only. So, I have specified >> resolver in >> my nginx.conf. However, during startup/reload I get "host not found in >> upstream error". Why nginx is not considering resolver conf and >> searches >> only in the nameservers configured in resolv.conf? > > Names as written in the configuration are resolved while parsing > the configuration using the system resolver. DNS servers defined > by the "resolver" directive are only used for dynamic / run-time > name resolution - for example, when variables are used in the > "proxy_pass" directive. I didn't want to hijack this thread but my question seems relevant. Is there a way to prevent dns resolution on a proxy_pass or upstream block during config parsing, or at least make it non-fatal? background/reason: I've a backend server with dynamic IP which doesn't always resolve. It will resolve when it's expected to be working but may not be available at config parsing time. From nginx-forum at forum.nginx.org Tue Aug 28 13:09:54 2018 From: nginx-forum at forum.nginx.org (Shliesce) Date: Tue, 28 Aug 2018 09:09:54 -0400 Subject: when exec nginx -t,same connection reset by peer Message-ID: <3c384b6f45bf4d72163205eb30f81cc9.NginxMailingListEnglish@forum.nginx.org> when i exec nginx -t,client receive connection by peer.In my prod,one server runs 5W qps, and cpu usage is close 25% Posted at Nginx Forum: https://forum.nginx.org/read.php?2,281006,281006#msg-281006 From nginx-forum at forum.nginx.org Tue Aug 28 13:10:57 2018 From: nginx-forum at forum.nginx.org (Shliesce) Date: Tue, 28 Aug 2018 09:10:57 -0400 Subject: when exec nginx -t,same connection reset by peer In-Reply-To: <3c384b6f45bf4d72163205eb30f81cc9.NginxMailingListEnglish@forum.nginx.org> References: <3c384b6f45bf4d72163205eb30f81cc9.NginxMailingListEnglish@forum.nginx.org> Message-ID: sorry,it is client receive connection reset by peer Posted at Nginx Forum: https://forum.nginx.org/read.php?2,281006,281007#msg-281007 From nginx-forum at forum.nginx.org Tue Aug 28 14:40:32 2018 From: nginx-forum at forum.nginx.org (buchireddy@insuranceinbox.com) Date: Tue, 28 Aug 2018 10:40:32 -0400 Subject: ip restriction Message-ID: <59bd2867dab3d46051c0abec1cba3e9b.NginxMailingListEnglish@forum.nginx.org> IP restriction not working in nginx. shows 403 for allowed ip and denied ip address also. is the http_access_module enabled by default i am using nginx 1.14.0 Posted at Nginx Forum: https://forum.nginx.org/read.php?2,281008,281008#msg-281008 From mdounin at mdounin.ru Tue Aug 28 15:47:47 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 28 Aug 2018 18:47:47 +0300 Subject: nginx-1.15.3 Message-ID: <20180828154746.GR56558@mdounin.ru> Changes with nginx 1.15.3 28 Aug 2018 *) Feature: now TLSv1.3 can be used with BoringSSL. *) Feature: the "ssl_early_data" directive, currently available with BoringSSL. *) Feature: the "keepalive_timeout" and "keepalive_requests" directives in the "upstream" block. *) Bugfix: the ngx_http_dav_module did not truncate destination file when copying a file over an existing one with the COPY method. *) Bugfix: the ngx_http_dav_module used zero access rights on the destination file and did not preserve file modification time when moving a file between different file systems with the MOVE method. *) Bugfix: the ngx_http_dav_module used default access rights when copying a file with the COPY method. *) Workaround: some clients might not work when using HTTP/2; the bug had appeared in 1.13.5. *) Bugfix: nginx could not be built with LibreSSL 2.8.0. -- Maxim Dounin http://nginx.org/ From maxim at nginx.com Tue Aug 28 16:37:35 2018 From: maxim at nginx.com (Maxim Konovalov) Date: Tue, 28 Aug 2018 19:37:35 +0300 Subject: nginx.conf-2018 Message-ID: <76d41c7d-f8b3-5577-8fdd-86809253ccde@nginx.com> Hello, As some of you probably know we are doing a conference in Atlanta in October 8 - 11 this year. You can find its full agenda here: https://www.nginx.com/nginxconf/2018/agenda/ I'd like to draw special attention to nginx dev training that Roman and Valentin are doing on the day 0 and day 3. This training is rather unique, guys will provide in-depth talk about nginx code development and also give some practical examples. I hope you enjoy the rest of the conf as well. I must admit it is not cheap. Good news is that I provide a cheat^Wdiscount code for you. Thanks, Maxim -- Maxim Konovalov From kworthington at gmail.com Tue Aug 28 17:07:44 2018 From: kworthington at gmail.com (Kevin Worthington) Date: Tue, 28 Aug 2018 17:07:44 +0000 Subject: [nginx-announce] nginx-1.15.3 In-Reply-To: <20180828154751.GS56558@mdounin.ru> References: <20180828154751.GS56558@mdounin.ru> Message-ID: Hello Nginx users, Now available: Nginx 1.15.3 for Windows https://kevinworthington.com/nginxwin1153 (32-bit and 64-bit versions) These versions are to support legacy users who are already using Cygwin based builds of Nginx. Officially supported native Windows binaries are at nginx.org. Announcements are also available here: Twitter http://twitter.com/kworthington Google+ https://plus.google.com/+KevinWorthington/ Thank you, Kevin -- Kevin Worthington kworthington *@* (gmail] [dot} {com) https://kevinworthington.com/ https://twitter.com/kworthington https://plus.google.com/+KevinWorthington/ On Tue, Aug 28, 2018 at 3:47 PM, Maxim Dounin wrote: > Changes with nginx 1.15.3 28 Aug > 2018 > > *) Feature: now TLSv1.3 can be used with BoringSSL. > > *) Feature: the "ssl_early_data" directive, currently available with > BoringSSL. > > *) Feature: the "keepalive_timeout" and "keepalive_requests" directives > in the "upstream" block. > > *) Bugfix: the ngx_http_dav_module did not truncate destination file > when copying a file over an existing one with the COPY method. > > *) Bugfix: the ngx_http_dav_module used zero access rights on the > destination file and did not preserve file modification time when > moving a file between different file systems with the MOVE method. > > *) Bugfix: the ngx_http_dav_module used default access rights when > copying a file with the COPY method. > > *) Workaround: some clients might not work when using HTTP/2; the bug > had appeared in 1.13.5. > > *) Bugfix: nginx could not be built with LibreSSL 2.8.0. > > > -- > Maxim Dounin > http://nginx.org/ > _______________________________________________ > nginx-announce mailing list > nginx-announce at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-announce > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsharan15 at gmail.com Wed Aug 29 06:55:37 2018 From: jsharan15 at gmail.com (Sharan J) Date: Wed, 29 Aug 2018 12:25:37 +0530 Subject: Resolver not working as expected In-Reply-To: <20180827133003.GK56558@mdounin.ru> References: <20180827133003.GK56558@mdounin.ru> Message-ID: Hi, Thanks for the response :) But, if I remove all the DNS servers from my resolv.conf then, the nameserver configured by the "resolver" directive is been used while parsing the configuration. Is this an expected behavior? Thanks, Shanthu On Mon, Aug 27, 2018 at 7:00 PM Maxim Dounin wrote: > Hello! > > On Mon, Aug 27, 2018 at 06:56:01PM +0530, Sharan J wrote: > > > Hi, > > > > Sample conf: > > > > http{ > > resolver x.x.x.x; > > server { > > server_name _; > > location / { > > proxy_pass http://somedomain.com; > > } > > } > > > > I have nameservers configured in my resolv.conf. But, somedomain.com > will > > be configured in x.x.x.x DNS server only. So, I have specified resolver > in > > my nginx.conf. However, during startup/reload I get "host not found in > > upstream error". Why nginx is not considering resolver conf and searches > > only in the nameservers configured in resolv.conf? > > Names as written in the configuration are resolved while parsing > the configuration using the system resolver. DNS servers defined > by the "resolver" directive are only used for dynamic / run-time > name resolution - for example, when variables are used in the > "proxy_pass" directive. > > -- > Maxim Dounin > http://mdounin.ru/ > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Wed Aug 29 11:19:47 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 29 Aug 2018 14:19:47 +0300 Subject: Resolver not working as expected In-Reply-To: References: <20180827133003.GK56558@mdounin.ru> Message-ID: <20180829111946.GV56558@mdounin.ru> Hello! On Wed, Aug 29, 2018 at 12:25:37PM +0530, Sharan J wrote: > But, if I remove all the DNS servers from my resolv.conf then, the > nameserver configured by the "resolver" directive is been used while > parsing the configuration. Is this an expected behavior? This depends on your system configuration. nginx itself does not use the DNS servers defined by the "resolver" directive during configuration parsing, it uses getaddrinfo() and/or gethostbyname() library functions. Note that resolv.conf is not the only configuration file to configure system name resolution - on most systems, there is also /etc/nsswitch.conf and /etc/hosts to consider. -- Maxim Dounin http://mdounin.ru/ From kaushalshriyan at gmail.com Wed Aug 29 15:45:26 2018 From: kaushalshriyan at gmail.com (Kaushal Shriyan) Date: Wed, 29 Aug 2018 21:15:26 +0530 Subject: nginx prevent file download Message-ID: Hi, I am running nginx webserver and i have set the below location block in nginx.conf configuration file to prevent a file to download. When i hit http://example.com/web.config on the browser, the web.config file gets downloaded. It is not working. location ~* \.(config)$ { deny all; } I will appreciate if somebody can pitch in for help. Thanks in Advance. Best Regards, Kaushal -------------- next part -------------- An HTML attachment was scrubbed... URL: From nginx-forum at forum.nginx.org Wed Aug 29 23:14:01 2018 From: nginx-forum at forum.nginx.org (jarstewa) Date: Wed, 29 Aug 2018 19:14:01 -0400 Subject: Throttle requests with limit_req rate based on header from response to auth subrequest Message-ID: I'm hoping to use the limit_req directive with different rates based on a header that is returned from the auth subrequest. I got some ideas from https://www.ruby-forum.com/topic/4418040 but am running into a few problems. Here is my configuration: > user nginx; > worker_processes auto; > error_log /var/log/nginx/error.log warn; > pid /var/run/nginx.pid; > events { > worker_connections 10000; > } > > worker_rlimit_nofile 10000; > > http { > log_subrequest on; > > log_format main escape=json '{ "timestamp": "$time_local", "client": "$remote_addr",' > ' "method": "$request_method", "uri": "$uri",' > ' "request_length": $request_length,' > ' "status": $status, "bytes_sent": $bytes_sent,' > ' "upstream_status": "$upstream_status",' > ' "request_id": "$request_id",' > ' "request_uri": "$request_uri",' > ' "tier": "$tier",' > ' "upstream_http_tier": "$upstream_http_tier",' > ' "2X_key": "$2X_key",' > ' "3X_key": "$3X_key",' > ' "2X_key_from_upstream": "$2X_key_from_upstream",' > ' "3X_key_from_upstream": "$3X_key_from_upstream",' > ' "origin": "$http_origin"}' ; > > access_log /var/log/nginx/access.log main; > sendfile on; > tcp_nopush on; > tcp_nodelay on; > keepalive_timeout 65; > types_hash_max_size 2048; > include /etc/nginx/mime.types; > default_type application/octet-stream; > proxy_buffering on; > proxy_buffers 8 64k; > proxy_cache_path /dev/shm/nginx/auth use_temp_path=off levels=1:2 keys_zone=auth_cache:1024m inactive=30m max_size=1g; > proxy_cache_path /dev/shm/nginx/manifests use_temp_path=off levels=1:2 keys_zone=manifest_cache:100m inactive=30s max_size=10g; > proxy_cache_methods GET HEAD; > proxy_cache_lock on; > proxy_cache_use_stale updating; > proxy_bind 0.0.0.0; > proxy_ignore_headers Expires; > proxy_pass_header Server; > > > map $request_uri $endpoint_id { > default "unknown"; > ~^/out/v\d+/(?P } > > # Mappings based on the tier header from the /auth request > map $tier $2X_key {~02x $endpoint_id; default "";} > map $tier $3X_key {~03x $endpoint_id; default "";} > map $upstream_http_tier $2X_key_from_upstream {~02x $endpoint_id; default "";} > map $upstream_http_tier $3X_key_from_upstream {~03x $endpoint_id; default "";} > > # Throttle zones based on the results of the above mapping > limit_req_zone $2X_key zone=2x_zone:20m rate=10r/s; > limit_req_zone $3X_key zone=3x_zone:20m rate=100r/s; > limit_req_zone $2X_key_from_upstream zone=2x_zone_from_upstream:20m rate=10r/s; > limit_req_zone $3X_key_from_upstream zone=3x_zone_from_upstream:20m rate=100r/s; > > > server { > listen 80 default_server; > listen [::]:80 default_server; > server_name default_backend; > server_tokens off; > access_log /var/log/nginx/access.log main; > > root /var/www/html; > > error_page 401 /error_pages/401.html; > error_page 403 /error_pages/403.html; > error_page 404 /error_pages/404.html; > error_page 429 /error_pages/429.html; > error_page 500 501 502 503 504 /error_pages/5xx.html; > > set $upstream_server http://my_server:80; > > proxy_http_version 1.1; > proxy_set_header Connection ""; > proxy_connect_timeout 10; > proxy_send_timeout 30; > proxy_read_timeout 30; > > proxy_cache_valid 404 412 1s; > > location ~* \.(m3u8|mpd|isml?/manifest)$ { > auth_request /auth; > > # Capture the tier header from auth request > auth_request_set $tier $upstream_http_tier; > > # Throttling based on mappings from tier > limit_req zone=2x_zone burst=10 nodelay; > limit_req zone=3x_zone burst=10 nodelay; > limit_req zone=2x_zone_from_upstream burst=10 nodelay; > limit_req zone=3x_zone_from_upstream burst=10 nodelay; > limit_req_status 429; > > proxy_pass $upstream_server; > proxy_cache manifest_cache; > set $cache_key "${endpoint_id}"; > proxy_cache_key $cache_key; > proxy_cache_valid 200 301 302 2s; > > access_log /var/log/nginx/access.log main; > } > > location /auth { > internal; > > set $auth_type 1; > proxy_pass_request_body off; > proxy_pass $upstream_server/auth?endpoint=$endpoint_id; > > proxy_cache auth_cache; > set $auth_cache_key "${endpoint_id}"; > proxy_cache_key $auth_cache_key; > proxy_cache_valid 200 301 302 5m; > proxy_cache_valid 400 401 403 404 5m; > > access_log /var/log/nginx/access.log main; > } > } > } > I'm expecting the following line to capture the "tier" header that comes back from the auth subrequest (sometimes it will be "02x", and sometimes "03x"): > auth_request_set $tier $upstream_http_tier; Then, that value will be passed into several mappings that should either return "" or a value, depending on which throttle zone should be applied. (There are variants using both $tier and $upstream_http_tier from attempts at troubleshooting.) > map $tier $2X_key {~02x $endpoint_id; default "";} > map $tier $3X_key {~03x $endpoint_id; default "";} > map $upstream_http_tier $2X_key_from_upstream {~02x $endpoint_id; default "";} > map $upstream_http_tier $3X_key_from_upstream {~03x $endpoint_id; default "";} Finally, I expect only the limit_req directive with a non-empty key to be applied: > limit_req_zone $2X_key zone=2x_zone:20m rate=10r/s; > limit_req_zone $3X_key zone=3x_zone:20m rate=100r/s; > limit_req_zone $2X_key_from_upstream zone=2x_zone_from_upstream:20m rate=10r/s; > limit_req_zone $3X_key_from_upstream zone=3x_zone_from_upstream:20m rate=100r/s; However, it's look like at least two things are not behaving as I expect. 1) The only throttling that I see is coming from 2x_zone, even when the header comes back as "03x". 2018/08/29 22:37:31 [error] 12626#0: *1596735 limiting requests, excess: 10.010 by zone "2x_zone", client: 10.0.136.178, server: default_backend, request: "GET /out/v1/04b1719535444a1d84389aeb0a1fb912/throttleCanary.m3u8 HTTP/1.1", host: "myserver" 2) Mapping based on the variable captured from auth_request_set don't appear to be set as I expect on the main request. Here's what I see in the access log for an auth request and a main request: { "timestamp": "29/Aug/2018:22:37:31 +0000", "client": "10.0.136.178", "method": "GET", "uri": "/auth", "status": 200, "bytes_sent": 0, "upstream_status": "", "server_name": "default_backend", "request_id": "d6545f5758c2b3944438c80f2e964678", "request_uri": "/out/v1/04b1719535444a1d84389aeb0a1fb912/throttleCanary.m3u8", "tier": "", "upstream_http_tier": "02x", "2X_key": "", "3X_key": "", "2X_key_from_upstream": "04b1719535444a1d84389aeb0a1fb912", "3X_key_from_upstream": "", "origin": "" } { "timestamp": "29/Aug/2018:22:37:31 +0000", "client": "10.0.136.178", "method": "GET", "uri": "/out/v1/04b1719535444a1d84389aeb0a1fb912/throttleCanary.m3u8", "status": 200, "bytes_sent": 652, "upstream_status": "", "server_name": "default_backend", "request_id": "d6545f5758c2b3944438c80f2e964678", "request_uri": "/out/v1/04b1719535444a1d84389aeb0a1fb912/throttleCanary.m3u8", "tier": "02x", "upstream_http_tier": "", "2X_key": "", "3X_key": "", "2X_key_from_upstream": "04b1719535444a1d84389aeb0a1fb912", "3X_key_from_upstream": "", "origin": "" } Notice that in the main request, "tier" is "02x" as expected, but the mapping based on $tier (2X_key) is empty while the mapping based on $http_upstream_tier (2X_key_from_upstream) has a value. Moreover, since only 2X_key_from_upstream has a value, I would expect the throttle based on that key (2x_zone_from_upstream) to take effect, not 2x_zone whose key is empty. I would really appreciate any help explaining my misunderstanding or advice with how better to implement what I'm trying to do. Thanks, Jared Posted at Nginx Forum: https://forum.nginx.org/read.php?2,281034,281034#msg-281034 From francis at daoine.org Thu Aug 30 07:45:51 2018 From: francis at daoine.org (Francis Daly) Date: Thu, 30 Aug 2018 08:45:51 +0100 Subject: nginx prevent file download In-Reply-To: References: Message-ID: <20180830074551.GA3537@daoine.org> On Wed, Aug 29, 2018 at 09:15:26PM +0530, Kaushal Shriyan wrote: Hi there, > When i hit > http://example.com/web.config on the browser, the web.config file gets > downloaded. It is not working. > > location ~* \.(config)$ { > deny all; > } It works for me. Are you sure that the config file that you think you are using is actually being used? Do you have any other "location" blocks defined that nginx will choose to process the "/web.config" request, instead of this one? If you test by temporarily making this the only location{} block in your config, do you see it work or fail? f -- Francis Daly francis at daoine.org From francis at daoine.org Thu Aug 30 08:09:16 2018 From: francis at daoine.org (Francis Daly) Date: Thu, 30 Aug 2018 09:09:16 +0100 Subject: Throttle requests with limit_req rate based on header from response to auth subrequest In-Reply-To: References: Message-ID: <20180830080916.GB3537@daoine.org> On Wed, Aug 29, 2018 at 07:14:01PM -0400, jarstewa wrote: Hi there, I do not know the answer, and I have not tested the code you provided. But, one suggestion which might be quick for you to test: what happens if you change all of your variable names so that they do not start with a digit? As in: rename $2X_key to be (for example) $a2X_key. It is possible that "$2X_key" will be expanded as "X_key" when $2 has no value. > > # Throttle zones based on the results of the above mapping > > limit_req_zone $2X_key zone=2x_zone:20m rate=10r/s; > > limit_req_zone $3X_key zone=3x_zone:20m rate=100r/s; > > limit_req_zone $2X_key_from_upstream zone=2x_zone_from_upstream:20m > rate=10r/s; > > limit_req_zone $3X_key_from_upstream zone=3x_zone_from_upstream:20m > rate=100r/s; If the first limit_req_zone argument is true in each case, the lowest rates is on the first one, so that is the one that will always take effect. f -- Francis Daly francis at daoine.org From eylonsa at securingsam.com Thu Aug 30 13:34:29 2018 From: eylonsa at securingsam.com (Eylon Saadon) Date: Thu, 30 Aug 2018 16:34:29 +0300 Subject: mirror delay Message-ID: Hi, I'm using the mirror module in my "production" nginx in order to mirror real traffic to a test envrionment. I don't want this mirroring to affect the latency of the production environment, but it looks like the nginx is waiting for the response from the test environment. is there a way to avoid this? I just want the request to get to the test environment and let it process it. but it shouldn't wait fo r the response from the test environment in order to respond to the request -- Thanks, Eylon Saadon -------------- next part -------------- An HTML attachment was scrubbed... URL: From arut at nginx.com Thu Aug 30 13:52:27 2018 From: arut at nginx.com (Roman Arutyunyan) Date: Thu, 30 Aug 2018 16:52:27 +0300 Subject: mirror delay In-Reply-To: References: Message-ID: <20180830135227.GF43355@Romans-MacBook-Air.local> Hi, On Thu, Aug 30, 2018 at 04:34:29PM +0300, Eylon Saadon wrote: > Hi, > I'm using the mirror module in my "production" nginx in order to mirror > real traffic to a test envrionment. > I don't want this mirroring to affect the latency of the production > environment, but it looks like the nginx is waiting for the response from > the test environment. > is there a way to avoid this? I just want the request to get to the test > environment and let it process it. but it shouldn't wait fo r the response > from the test environment in order to respond to the request Usually a mirror subrequest does not affect the main request. However there are two issues with mirroring: - the next request on the same connection will not be processed until all mirror subrequests finish. Try disabling keepalive and see if it helps. - if you use sendfile and tcp_nopush, it's possible that the response is not pushed properly because of a mirror subrequest, which may result in a delay. Turn off sendfile and see if it helps. -- Roman Arutyunyan From eylonsa at securingsam.com Thu Aug 30 14:19:53 2018 From: eylonsa at securingsam.com (Eylon Saadon) Date: Thu, 30 Aug 2018 17:19:53 +0300 Subject: mirror delay In-Reply-To: <20180830135227.GF43355@Romans-MacBook-Air.local> References: <20180830135227.GF43355@Romans-MacBook-Air.local> Message-ID: hi, thanks for the quick response! I'm not using sendfile or tcp_nopush. just to make sure. I should disable the keepalive for the mirror location. and do it like so? server { resolver 8.8.8.8; listen 80; location / { proxy_set_header Host $host; proxy_pass http://server:9000; } location /mirror { internal; keepalive_timeout 0; proxy_pass https://test_server$request_uri; } } Thanks, eylon saadon On Thu, Aug 30, 2018 at 4:52 PM Roman Arutyunyan wrote: > Hi, > > On Thu, Aug 30, 2018 at 04:34:29PM +0300, Eylon Saadon wrote: > > Hi, > > I'm using the mirror module in my "production" nginx in order to mirror > > real traffic to a test envrionment. > > I don't want this mirroring to affect the latency of the production > > environment, but it looks like the nginx is waiting for the response from > > the test environment. > > is there a way to avoid this? I just want the request to get to the test > > environment and let it process it. but it shouldn't wait fo r the > response > > from the test environment in order to respond to the request > > Usually a mirror subrequest does not affect the main request. However > there > are two issues with mirroring: > > - the next request on the same connection will not be processed until all > mirror subrequests finish. Try disabling keepalive and see if it helps. > > - if you use sendfile and tcp_nopush, it's possible that the response is > not > pushed properly because of a mirror subrequest, which may result in a > delay. > Turn off sendfile and see if it helps. > > -- > Roman Arutyunyan > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx > -- Thanks, Eylon Saadon -------------- next part -------------- An HTML attachment was scrubbed... URL: From arut at nginx.com Thu Aug 30 14:28:01 2018 From: arut at nginx.com (Roman Arutyunyan) Date: Thu, 30 Aug 2018 17:28:01 +0300 Subject: mirror delay In-Reply-To: References: <20180830135227.GF43355@Romans-MacBook-Air.local> Message-ID: <20180830142801.GG43355@Romans-MacBook-Air.local> Hi, On Thu, Aug 30, 2018 at 05:19:53PM +0300, Eylon Saadon wrote: > hi, > thanks for the quick response! > I'm not using sendfile or tcp_nopush. > just to make sure. I should disable the keepalive for the mirror location. > and do it like so? No, for the primary location. This will help us understand the reason why you have the delay. > server { > > resolver 8.8.8.8; > > listen 80; > > location / { > proxy_set_header Host $host; > proxy_pass http://server:9000; > } > > location /mirror { > internal; > keepalive_timeout 0; > proxy_pass https://test_server$request_uri; > } > } > > > Thanks, > > eylon saadon > > > On Thu, Aug 30, 2018 at 4:52 PM Roman Arutyunyan wrote: > > > Hi, > > > > On Thu, Aug 30, 2018 at 04:34:29PM +0300, Eylon Saadon wrote: > > > Hi, > > > I'm using the mirror module in my "production" nginx in order to mirror > > > real traffic to a test envrionment. > > > I don't want this mirroring to affect the latency of the production > > > environment, but it looks like the nginx is waiting for the response from > > > the test environment. > > > is there a way to avoid this? I just want the request to get to the test > > > environment and let it process it. but it shouldn't wait fo r the > > response > > > from the test environment in order to respond to the request > > > > Usually a mirror subrequest does not affect the main request. However > > there > > are two issues with mirroring: > > > > - the next request on the same connection will not be processed until all > > mirror subrequests finish. Try disabling keepalive and see if it helps. > > > > - if you use sendfile and tcp_nopush, it's possible that the response is > > not > > pushed properly because of a mirror subrequest, which may result in a > > delay. > > Turn off sendfile and see if it helps. > > > > -- > > Roman Arutyunyan > > _______________________________________________ > > nginx mailing list > > nginx at nginx.org > > http://mailman.nginx.org/mailman/listinfo/nginx > > > > > -- > Thanks, > Eylon Saadon > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx -- Roman Arutyunyan From roger at netskrt.io Thu Aug 30 16:09:44 2018 From: roger at netskrt.io (Roger Fischer) Date: Thu, 30 Aug 2018 09:09:44 -0700 Subject: Ignore Certificate Errors Message-ID: Hello, is there a way to make NGINX more forgiving on TLS certificate errors? Or would that have to be done in OpenSSL instead? When I use openssl s_client, I get the following errors from the upstream server: 140226185430680:error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01:rsa_pk1.c:103: 140226185430680:error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:705: 140226185430680:error:1408D07B:SSL routines:ssl3_get_key_exchange:bad signature:s3_clnt.c:2010: This causes NGINX (reverse proxy) to return 502 Bad Gateway to the browser. The NGINX error log shows: 2018/08/29 09:09:59 [crit] 11633#11633: *28 SSL_do_handshake() failed (SSL: error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed error:1408D07B:SSL routines:ssl3_get_key_exchange:bad signature) while SSL handshaking to upstream, client: 192.168.1.66, server: s5.example.com, request: "GET /xyz I have added ?proxy_ssl_verify off;?, but that did not make any difference. Surprisingly, the browser (directly to the upstream server) does not complain about the TLS error. Is there anything else I can do either in NGINX or openssl to suppress the 502 Bad Gateway? Thanks? Roger PS: I don?t have control over the upstream server, so I can?t fix the root cause (faulty certificate). -------------- next part -------------- An HTML attachment was scrubbed... URL: From nginx-forum at forum.nginx.org Thu Aug 30 16:53:47 2018 From: nginx-forum at forum.nginx.org (jarstewa) Date: Thu, 30 Aug 2018 12:53:47 -0400 Subject: Throttle requests with limit_req rate based on header from response to auth subrequest In-Reply-To: <20180830080916.GB3537@daoine.org> References: <20180830080916.GB3537@daoine.org> Message-ID: Francis Daly Wrote: ------------------------------------------------------- > On Wed, Aug 29, 2018 at 07:14:01PM -0400, jarstewa wrote: > > Hi there, > > I do not know the answer, and I have not tested the code you provided. > > But, one suggestion which might be quick for you to test: > > what happens if you change all of your variable names so that they do > not start with a digit? > > As in: rename $2X_key to be (for example) $a2X_key. > > It is possible that "$2X_key" will be expanded as "X_key" when $2 has > no value. > Thanks for the suggestion. I modified my example to have names likes $key_two rather than $2X_key and I do see the same behavior. > If the first limit_req_zone argument is true in each case, the lowest > rates is > on the first one, so that is the one that will always take effect. Hm, it seems to me that the first limit_req_zone argument is "", based on the value of the variable that I see in the logs, so I don't know why that limit is taking effect. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,281034,281049#msg-281049 From mdounin at mdounin.ru Thu Aug 30 18:13:45 2018 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 30 Aug 2018 21:13:45 +0300 Subject: Ignore Certificate Errors In-Reply-To: References: Message-ID: <20180830181344.GD56558@mdounin.ru> Hello! On Thu, Aug 30, 2018 at 09:09:44AM -0700, Roger Fischer wrote: > Hello, > > is there a way to make NGINX more forgiving on TLS certificate errors? Or would that have to be done in OpenSSL instead? > > When I use openssl s_client, I get the following errors from the upstream server: > > 140226185430680:error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01:rsa_pk1.c:103: > 140226185430680:error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:705: > 140226185430680:error:1408D07B:SSL routines:ssl3_get_key_exchange:bad signature:s3_clnt.c:2010: > > This causes NGINX (reverse proxy) to return 502 Bad Gateway to the browser. > > The NGINX error log shows: > > 2018/08/29 09:09:59 [crit] 11633#11633: *28 SSL_do_handshake() failed (SSL: error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed error:1408D07B:SSL routines:ssl3_get_key_exchange:bad signature) while SSL handshaking to upstream, client: 192.168.1.66, server: s5.example.com, request: "GET /xyz > > I have added ?proxy_ssl_verify off;?, but that did not make any difference. > > Surprisingly, the browser (directly to the upstream server) does not complain about the TLS error. > > Is there anything else I can do either in NGINX or openssl to suppress the 502 Bad Gateway? > > Thanks? > > Roger > > PS: I don?t have control over the upstream server, so I can?t fix the root cause (faulty certificate). As per the error message, the problem seems to be not with the cerifitcate, but with the key exchange during the SSL handshake. For some reason signature verification after the key exchange fails due to wrong padding. Most likely the problem is specific to some ciphers, so forcing a different cipher with proxy_ssl_ciphers could help, see http://nginx.org/r/proxy_ssl_ciphers. -- Maxim Dounin http://mdounin.ru/ From nginx-forum at forum.nginx.org Thu Aug 30 23:47:43 2018 From: nginx-forum at forum.nginx.org (jarstewa) Date: Thu, 30 Aug 2018 19:47:43 -0400 Subject: Throttle requests with limit_req rate based on header from response to auth subrequest In-Reply-To: References: Message-ID: Digging into this some more today, I've continued to find what seems to be odd behavior. If I remove all of the limit_req directives, then the mapped variables based on the upstream are always present: { "upstream_http_tier": "", "tier": "02x", "http_tier": "", "key_two": "", "key_three": "", "key_two_from_upstream": "match", "key_three_from_upstream": "nonempty", } Whereas if I add the limit_req directives that reference "key_*" back in, then those keys are no longer populated correctly: { "upstream_http_tier": "", "tier": "02x", "http_tier": "", "key_two": "", "key_three": "", "key_two_from_upstream": "", "key_three_from_upstream": "", } Posted at Nginx Forum: https://forum.nginx.org/read.php?2,281034,281056#msg-281056 From nginx-forum at forum.nginx.org Fri Aug 31 00:13:33 2018 From: nginx-forum at forum.nginx.org (jarstewa) Date: Thu, 30 Aug 2018 20:13:33 -0400 Subject: Throttle requests with limit_req rate based on header from response to auth subrequest In-Reply-To: References: Message-ID: Hmm, I notice this from the map documentation: > Since variables are evaluated only when they are used, the mere declaration even of a large number of ?map? variables does not add any extra costs to request processing. Here is what I suspect: 1) The limit_req directive is being processed before the auth subrequest 2) Therefore, when the limit_req is present, the mapped variables are empty since the header from the auth request is not present. But when limit_req is removed, the mapped variables are evaluated by the access_log, which happens after the auth subrequest has been made. Can anyone kindly confirm or reject this theory? Thanks again, Jared Posted at Nginx Forum: https://forum.nginx.org/read.php?2,281034,281057#msg-281057 From nginx-forum at forum.nginx.org Fri Aug 31 09:02:21 2018 From: nginx-forum at forum.nginx.org (traquila) Date: Fri, 31 Aug 2018 05:02:21 -0400 Subject: Serve multiple requests from a single proxy request Message-ID: <7c4d9816717bcfd034e64c564e18084f.NginxMailingListEnglish@forum.nginx.org> Hello, I'm wondering if nginx is able to serve multiple requests from a single proxy request before it completes. I am using the following configuration: proxy_cache_lock on; proxy_cache_lock_timeout 5s; proxy_cache ram; proxy_pass myUpstream; My upstream uses chunked transfer encoding and serve the request in 10 sec. Now if I try to send 2 requests to nginx, the first one starts responding immediately but the second will starts 5 sec later (lock timeout) and then perform a second request to my upstream. Is there a way to configure nginx to immediately respond to multiple requests with a single request to my upstream? Thank you in advance, Traquila Posted at Nginx Forum: https://forum.nginx.org/read.php?2,281058,281058#msg-281058 From francis at daoine.org Fri Aug 31 11:09:08 2018 From: francis at daoine.org (Francis Daly) Date: Fri, 31 Aug 2018 12:09:08 +0100 Subject: Throttle requests with limit_req rate based on header from response to auth subrequest In-Reply-To: References: Message-ID: <20180831110908.GD3537@daoine.org> On Thu, Aug 30, 2018 at 08:13:33PM -0400, jarstewa wrote: Hi there, For what it's worth: > 1) The limit_req directive is being processed before the auth subrequest > 2) Therefore, when the limit_req is present, the mapped variables are empty > since the header from the auth request is not present. But when limit_req > is removed, the mapped variables are evaluated by the access_log, which > happens after the auth subrequest has been made. > > Can anyone kindly confirm or reject this theory? I'm pretty sure that that is what is happening. In the mail thread you linked to originally, the limit_req part was being calculated based on something in the request. You are trying to calculate it based on something not in the request. I don't have a suggested solution. Cheers, f -- Francis Daly francis at daoine.org From arut at nginx.com Fri Aug 31 12:07:13 2018 From: arut at nginx.com (Roman Arutyunyan) Date: Fri, 31 Aug 2018 15:07:13 +0300 Subject: Serve multiple requests from a single proxy request In-Reply-To: <7c4d9816717bcfd034e64c564e18084f.NginxMailingListEnglish@forum.nginx.org> References: <7c4d9816717bcfd034e64c564e18084f.NginxMailingListEnglish@forum.nginx.org> Message-ID: <20180831120713.GI43355@Romans-MacBook-Air.local> Hi, On Fri, Aug 31, 2018 at 05:02:21AM -0400, traquila wrote: > Hello, > I'm wondering if nginx is able to serve multiple requests from a single > proxy request before it completes. > > I am using the following configuration: > > proxy_cache_lock on; > proxy_cache_lock_timeout 5s; > proxy_cache ram; > proxy_pass myUpstream; > > My upstream uses chunked transfer encoding and serve the request in 10 sec. > Now if I try to send 2 requests to nginx, the first one starts responding > immediately but the second will starts 5 sec later (lock timeout) and then > perform a second request to my upstream. This is all normal considering you have proxy_cache_lock enabled. Your second request waits until the first request completes and fills up the cache entry. This cache entry is supposed to be used to serve the second request. But it expires after 5 seconds and makes another request to the upstream. > Is there a way to configure nginx to immediately respond to multiple > requests with a single request to my upstream? There is no proxy/cache option to enable that behavior. However, if your response is big enough, you can try using the slice module: http://nginx.org/en/docs/http/ngx_http_slice_module.html With this module you can slice the response in pieces and cache each one separately. While the same logic as above is true for the slices too, the fact that they are usually much smaller than the entire response makes it look like the response is proxied to multiple client simultaneously. -- Roman Arutyunyan From nginx-forum at forum.nginx.org Fri Aug 31 15:29:42 2018 From: nginx-forum at forum.nginx.org (traquila) Date: Fri, 31 Aug 2018 11:29:42 -0400 Subject: Serve multiple requests from a single proxy request In-Reply-To: <20180831120713.GI43355@Romans-MacBook-Air.local> References: <20180831120713.GI43355@Romans-MacBook-Air.local> Message-ID: Thank you for your answer. This means nginx is not compatible with CMAF and low latency streaming. I tried the slice module and read its code but it does not cover my needs. I guess I have to develop a new proxy module. thanks Traquila Roman Arutyunyan Wrote: ------------------------------------------------------- > Hi, > > On Fri, Aug 31, 2018 at 05:02:21AM -0400, traquila wrote: > > Hello, > > I'm wondering if nginx is able to serve multiple requests from a > single > > proxy request before it completes. > > > > I am using the following configuration: > > > > proxy_cache_lock on; > > proxy_cache_lock_timeout 5s; > > proxy_cache ram; > > proxy_pass myUpstream; > > > > My upstream uses chunked transfer encoding and serve the request in > 10 sec. > > Now if I try to send 2 requests to nginx, the first one starts > responding > > immediately but the second will starts 5 sec later (lock timeout) > and then > > perform a second request to my upstream. > > This is all normal considering you have proxy_cache_lock enabled. > Your second > request waits until the first request completes and fills up the cache > entry. > This cache entry is supposed to be used to serve the second request. > But it > expires after 5 seconds and makes another request to the upstream. > > > Is there a way to configure nginx to immediately respond to multiple > > requests with a single request to my upstream? > > There is no proxy/cache option to enable that behavior. However, if > your > response is big enough, you can try using the slice module: > > http://nginx.org/en/docs/http/ngx_http_slice_module.html > > With this module you can slice the response in pieces and cache each > one > separately. While the same logic as above is true for the slices too, > the fact that they are usually much smaller than the entire response > makes > it look like the response is proxied to multiple client > simultaneously. > > -- > Roman Arutyunyan > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx Posted at Nginx Forum: https://forum.nginx.org/read.php?2,281058,281062#msg-281062