From mdounin at mdounin.ru Sun Aug 1 01:32:53 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Sun, 1 Aug 2021 04:32:53 +0300 Subject: Abnormal delays in download In-Reply-To: <7b912201-e833-507e-6194-1b2013af6c1c@couderc.eu> References: <7b912201-e833-507e-6194-1b2013af6c1c@couderc.eu> Message-ID: Hello! On Fri, Jul 30, 2021 at 10:40:00AM +0200, Pierre Couderc wrote: > I am trying to download a 12M pdf file and it takes more than 4 minutes > with http nginx while it takes a few seconds with scp. > > I have asked detailed log here https://paste.debian.net/1206029/ > > I see timeouts but I am not able to interpret them correctly. > > nginx is in lxd container in a nearly not used? server (htop load > average 0.08 ...). > > I use nearly defaut debian configuration of nginx. The logs provided suggest that the client is limiting download rather than nginx. Further, the client seems to do at least 17 various range requests to the file which is being downloaded, thus downloading many more than the file itself, but this is probably not the issue. Rather, the issue seems that the client does not proceed with the file download while doing these requests. Try disabling Firefox PDF Viewer to see if it helps (Preferences -> General -> Applications -> Portable Document Format (PDF), set to "Save File", see [1]). Alternatively, try using curl (or wget, or fetch, or whatever you prefer) to download the file instead of trying to view it in the browser. [1] https://support.mozilla.org/en-US/kb/view-pdf-files-firefox-or-choose-another-viewer#w_disable-the-built-in-pdf-viewer-and-use-another-viewer -- Maxim Dounin http://mdounin.ru/ From pierre at couderc.eu Sun Aug 1 20:38:48 2021 From: pierre at couderc.eu (Pierre Couderc) Date: Sun, 1 Aug 2021 22:38:48 +0200 Subject: Abnormal delays in download In-Reply-To: References: <7b912201-e833-507e-6194-1b2013af6c1c@couderc.eu> Message-ID: On 8/1/21 3:32 AM, Maxim Dounin wrote: > Hello! > > On Fri, Jul 30, 2021 at 10:40:00AM +0200, Pierre Couderc wrote: > >> I am trying to download a 12M pdf file and it takes more than 4 minutes >> with http nginx while it takes a few seconds with scp. >> >> I have asked detailed log here https://paste.debian.net/1206029/ >> >> I see timeouts but I am not able to interpret them correctly. >> >> nginx is in lxd container in a nearly not used? server (htop load >> average 0.08 ...). >> >> I use nearly defaut debian configuration of nginx. > The logs provided suggest that the client is limiting download > rather than nginx. Further, the client seems to do at least 17 > various range requests to the file which is being downloaded, thus > downloading many more than the file itself, but this is probably > not the issue. Rather, the issue seems that the client does not > proceed with the file download while doing these requests. > > Try disabling Firefox PDF Viewer to see if it helps (Preferences > -> General -> Applications -> Portable Document Format (PDF), set > to "Save File", see [1]). Alternatively, try using curl (or wget, > or fetch, or whatever you prefer) to download the file instead of > trying to view it in the browser. > > [1] https://support.mozilla.org/en-US/kb/view-pdf-files-firefox-or-choose-another-viewer#w_disable-the-built-in-pdf-viewer-and-use-another-viewer > Thank you very much, I had not the idea it could be a client such as firefox, which I suppose bulletproof... So I followed your idea and tried with wget : about? 2 minutes 40s.. This is too much long too as with scp (in wifi at 2meters from the server) it is only 3 seconds... ;) Log is here (shorter ;) : https://paste.debian.net/1206278/ Thank you again... PC From mdounin at mdounin.ru Sun Aug 1 23:19:59 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 2 Aug 2021 02:19:59 +0300 Subject: Abnormal delays in download In-Reply-To: References: <7b912201-e833-507e-6194-1b2013af6c1c@couderc.eu> Message-ID: Hello! On Sun, Aug 01, 2021 at 10:38:48PM +0200, Pierre Couderc wrote: > On 8/1/21 3:32 AM, Maxim Dounin wrote: > > Hello! > > > > On Fri, Jul 30, 2021 at 10:40:00AM +0200, Pierre Couderc wrote: > > > >> I am trying to download a 12M pdf file and it takes more than 4 minutes > >> with http nginx while it takes a few seconds with scp. > >> > >> I have asked detailed log here https://paste.debian.net/1206029/ > >> > >> I see timeouts but I am not able to interpret them correctly. > >> > >> nginx is in lxd container in a nearly not used? server (htop load > >> average 0.08 ...). > >> > >> I use nearly defaut debian configuration of nginx. > > The logs provided suggest that the client is limiting download > > rather than nginx. Further, the client seems to do at least 17 > > various range requests to the file which is being downloaded, thus > > downloading many more than the file itself, but this is probably > > not the issue. Rather, the issue seems that the client does not > > proceed with the file download while doing these requests. > > > > Try disabling Firefox PDF Viewer to see if it helps (Preferences > > -> General -> Applications -> Portable Document Format (PDF), set > > to "Save File", see [1]). Alternatively, try using curl (or wget, > > or fetch, or whatever you prefer) to download the file instead of > > trying to view it in the browser. > > > > [1] https://support.mozilla.org/en-US/kb/view-pdf-files-firefox-or-choose-another-viewer#w_disable-the-built-in-pdf-viewer-and-use-another-viewer > > > Thank you very much, I had not the idea it could be a client such as > firefox, which I suppose bulletproof... > > So I followed your idea and tried with wget : about? 2 minutes 40s.. > > This is too much long too as with scp (in wifi at 2meters from the > server) it is only 3 seconds... ;) > > Log is here (shorter ;) : https://paste.debian.net/1206278/ No additional range requests now, though still looks limited by the client. Any antivirus/firewall on the client (or on the server)? What happens if you download a random binary file of similar size instead of a pdf file? -- Maxim Dounin http://mdounin.ru/ From pierre at couderc.eu Mon Aug 2 12:46:03 2021 From: pierre at couderc.eu (Pierre Couderc) Date: Mon, 2 Aug 2021 14:46:03 +0200 Subject: Abnormal delays in download In-Reply-To: References: <7b912201-e833-507e-6194-1b2013af6c1c@couderc.eu> Message-ID: <3146c51b-b935-cf41-4897-e8e86cf6cf7c@couderc.eu> On 8/2/21 1:19 AM, Maxim Dounin wrote: > Hello! > > On Sun, Aug 01, 2021 at 10:38:48PM +0200, Pierre Couderc wrote: > >> On 8/1/21 3:32 AM, Maxim Dounin wrote: >>> Hello! >>> >>> On Fri, Jul 30, 2021 at 10:40:00AM +0200, Pierre Couderc wrote: >>> >>>> I am trying to download a 12M pdf file and it takes more than 4 minutes >>>> with http nginx while it takes a few seconds with scp. >>>> >>>> I have asked detailed log here https://paste.debian.net/1206029/ >>>> >>>> I see timeouts but I am not able to interpret them correctly. >>>> >>>> nginx is in lxd container in a nearly not used? server (htop load >>>> average 0.08 ...). >>>> >>>> I use nearly defaut debian configuration of nginx. >>> The logs provided suggest that the client is limiting download >>> rather than nginx. Further, the client seems to do at least 17 >>> various range requests to the file which is being downloaded, thus >>> downloading many more than the file itself, but this is probably >>> not the issue. Rather, the issue seems that the client does not >>> proceed with the file download while doing these requests. >>> >>> Try disabling Firefox PDF Viewer to see if it helps (Preferences >>> -> General -> Applications -> Portable Document Format (PDF), set >>> to "Save File", see [1]). Alternatively, try using curl (or wget, >>> or fetch, or whatever you prefer) to download the file instead of >>> trying to view it in the browser. >>> >>> [1] https://support.mozilla.org/en-US/kb/view-pdf-files-firefox-or-choose-another-viewer#w_disable-the-built-in-pdf-viewer-and-use-another-viewer >>> >> Thank you very much, I had not the idea it could be a client such as >> firefox, which I suppose bulletproof... >> >> So I followed your idea and tried with wget : about? 2 minutes 40s.. >> >> This is too much long too as with scp (in wifi at 2meters from the >> server) it is only 3 seconds... ;) >> >> Log is here (shorter ;) : https://paste.debian.net/1206278/ > No additional range requests now, though still looks limited by > the client. Any antivirus/firewall on the client (or on the > server)? No. I have not soon understood the interest of a firewall : either the port is closed and? I do not see the interest, either it is open and I can only trust the application which manages it... All my clients are debian and have no antivirus or firewall and tests are done in the room where is the server. nginx is in a lxd container with snap. > What happens if you download a random binary file of > similar size instead of a pdf file? > It does not change, after copying ggg.pdf to ggg using wget : https://paste.debian.net/1206329/ Time out seem to occur : 2021/08/02 14:39:10 [debug] 17204#17204: *22637 event timer add: 26: 60000:7599423246 2021/08/02 14:39:15 [debug] 17204#17204: *22637 http run request: "/ggg?" I have tried too with another client computer with equivalent results... Thanks again. PC From mdounin at mdounin.ru Mon Aug 2 14:04:40 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 2 Aug 2021 17:04:40 +0300 Subject: Abnormal delays in download In-Reply-To: <3146c51b-b935-cf41-4897-e8e86cf6cf7c@couderc.eu> References: <7b912201-e833-507e-6194-1b2013af6c1c@couderc.eu> <3146c51b-b935-cf41-4897-e8e86cf6cf7c@couderc.eu> Message-ID: Hello! On Mon, Aug 02, 2021 at 02:46:03PM +0200, Pierre Couderc wrote: > On 8/2/21 1:19 AM, Maxim Dounin wrote: > > Hello! > > > > On Sun, Aug 01, 2021 at 10:38:48PM +0200, Pierre Couderc wrote: > > > >> On 8/1/21 3:32 AM, Maxim Dounin wrote: > >>> Hello! > >>> > >>> On Fri, Jul 30, 2021 at 10:40:00AM +0200, Pierre Couderc wrote: > >>> > >>>> I am trying to download a 12M pdf file and it takes more than 4 minutes > >>>> with http nginx while it takes a few seconds with scp. > >>>> > >>>> I have asked detailed log here https://paste.debian.net/1206029/ > >>>> > >>>> I see timeouts but I am not able to interpret them correctly. > >>>> > >>>> nginx is in lxd container in a nearly not used? server (htop load > >>>> average 0.08 ...). > >>>> > >>>> I use nearly defaut debian configuration of nginx. > >>> The logs provided suggest that the client is limiting download > >>> rather than nginx. Further, the client seems to do at least 17 > >>> various range requests to the file which is being downloaded, thus > >>> downloading many more than the file itself, but this is probably > >>> not the issue. Rather, the issue seems that the client does not > >>> proceed with the file download while doing these requests. > >>> > >>> Try disabling Firefox PDF Viewer to see if it helps (Preferences > >>> -> General -> Applications -> Portable Document Format (PDF), set > >>> to "Save File", see [1]). Alternatively, try using curl (or wget, > >>> or fetch, or whatever you prefer) to download the file instead of > >>> trying to view it in the browser. > >>> > >>> [1] https://support.mozilla.org/en-US/kb/view-pdf-files-firefox-or-choose-another-viewer#w_disable-the-built-in-pdf-viewer-and-use-another-viewer > >>> > >> Thank you very much, I had not the idea it could be a client such as > >> firefox, which I suppose bulletproof... > >> > >> So I followed your idea and tried with wget : about? 2 minutes 40s.. > >> > >> This is too much long too as with scp (in wifi at 2meters from the > >> server) it is only 3 seconds... ;) > >> > >> Log is here (shorter ;) : https://paste.debian.net/1206278/ > > No additional range requests now, though still looks limited by > > the client. Any antivirus/firewall on the client (or on the > > server)? > > No. I have not soon understood the interest of a firewall : either the > port is closed and? I do not see the interest, either it is open and I > can only trust the application which manages it... > > All my clients are debian and have no antivirus or firewall and tests > are done in the room where is the server. > > nginx is in a lxd container with snap. Ok, so it does not seem to be the client issue, yet for some reason nginx cannot proceed with sending data to the client as if network speed was severely limited. Is sshd you test against in the same lxd container? Any resource limits applied to the container? What happens when you run nginx on the host system instead? [...] > Time out seem to occur : > > 2021/08/02 14:39:10 [debug] 17204#17204: *22637 event timer add: 26: > 60000:7599423246 > 2021/08/02 14:39:15 [debug] 17204#17204: *22637 http run request: "/ggg?" These aren't timeouts, but debug log lines: first says that nginx armed a timer for 60 seconds, and second one is about starting request processing 5 seconds later. Debug logging is not configured at the global level and hence there is no debug logs about event processing, but it is nevertheless clear that the request processing was started due to an event reported by epoll() rather than due to a timeout. -- Maxim Dounin http://mdounin.ru/ From pierre at couderc.eu Mon Aug 2 14:48:39 2021 From: pierre at couderc.eu (Pierre Couderc) Date: Mon, 2 Aug 2021 16:48:39 +0200 Subject: Abnormal delays in download In-Reply-To: References: <7b912201-e833-507e-6194-1b2013af6c1c@couderc.eu> <3146c51b-b935-cf41-4897-e8e86cf6cf7c@couderc.eu> Message-ID: On 8/2/21 4:04 PM, Maxim Dounin wrote: > Hello! > > On Mon, Aug 02, 2021 at 02:46:03PM +0200, Pierre Couderc wrote: > >> On 8/2/21 1:19 AM, Maxim Dounin wrote: >>> Hello! >>> >>> On Sun, Aug 01, 2021 at 10:38:48PM +0200, Pierre Couderc wrote: >>> >>>> On 8/1/21 3:32 AM, Maxim Dounin wrote: >>>>> Hello! >>>>> >>>>> On Fri, Jul 30, 2021 at 10:40:00AM +0200, Pierre Couderc wrote: >>>>> >>>>>> I am trying to download a 12M pdf file and it takes more than 4 minutes >>>>>> with http nginx while it takes a few seconds with scp. >>>>>> >>>>>> I have asked detailed log here https://paste.debian.net/1206029/ >>>>>> >>>>>> I see timeouts but I am not able to interpret them correctly. >>>>>> >>>>>> nginx is in lxd container in a nearly not used? server (htop load >>>>>> average 0.08 ...). >>>>>> >>>>>> I use nearly defaut debian configuration of nginx. >>>>> The logs provided suggest that the client is limiting download >>>>> rather than nginx. Further, the client seems to do at least 17 >>>>> various range requests to the file which is being downloaded, thus >>>>> downloading many more than the file itself, but this is probably >>>>> not the issue. Rather, the issue seems that the client does not >>>>> proceed with the file download while doing these requests. >>>>> >>>>> Try disabling Firefox PDF Viewer to see if it helps (Preferences >>>>> -> General -> Applications -> Portable Document Format (PDF), set >>>>> to "Save File", see [1]). Alternatively, try using curl (or wget, >>>>> or fetch, or whatever you prefer) to download the file instead of >>>>> trying to view it in the browser. >>>>> >>>>> [1] https://support.mozilla.org/en-US/kb/view-pdf-files-firefox-or-choose-another-viewer#w_disable-the-built-in-pdf-viewer-and-use-another-viewer >>>>> >>>> Thank you very much, I had not the idea it could be a client such as >>>> firefox, which I suppose bulletproof... >>>> >>>> So I followed your idea and tried with wget : about? 2 minutes 40s.. >>>> >>>> This is too much long too as with scp (in wifi at 2meters from the >>>> server) it is only 3 seconds... ;) >>>> >>>> Log is here (shorter ;) : https://paste.debian.net/1206278/ >>> No additional range requests now, though still looks limited by >>> the client. Any antivirus/firewall on the client (or on the >>> server)? >> No. I have not soon understood the interest of a firewall : either the >> port is closed and? I do not see the interest, either it is open and I >> can only trust the application which manages it... >> >> All my clients are debian and have no antivirus or firewall and tests >> are done in the room where is the server. >> >> nginx is in a lxd container with snap. > Ok, so it does not seem to be the client issue, yet for some > reason nginx cannot proceed with sending data to the client as if > network speed was severely limited. > > Is sshd you test against in the same lxd container? Any resource > limits applied to the container? What happens when you run nginx > on the host system instead? Thank you, you put me on new ways, I have many things to check 1- network problems : check if packets are routed through local router 2- if yes, local router has Qos (for SIP phones): : check how it works In fact it seems that before nginx, I may have to debug network problems... > [...] > >> Time out seem to occur : >> >> 2021/08/02 14:39:10 [debug] 17204#17204: *22637 event timer add: 26: >> 60000:7599423246 >> 2021/08/02 14:39:15 [debug] 17204#17204: *22637 http run request: "/ggg?" > These aren't timeouts, but debug log lines: first says that nginx > armed a timer for 60 seconds, and second one is about starting > request processing 5 seconds later. Debug logging is not > configured at the global level and hence there is no debug logs > about event processing, but it is nevertheless clear that the > request processing was started due to an event reported by epoll() > rather than due to a timeout. > Thank you very much ! PC From kaushalshriyan at gmail.com Mon Aug 2 15:42:59 2021 From: kaushalshriyan at gmail.com (Kaushal Shriyan) Date: Mon, 2 Aug 2021 21:12:59 +0530 Subject: 404 Not Found while accessing the file Message-ID: Hi, I am running nginx version: nginx/1.20.1 on CentOS Linux release 7.9.2009 (Core). When I am accessing https://dsmadeveloperportal.mydomain.com/sites/default/files/2020-08/dsmaAccount%20and%20Transaction%20API%20Specification_2_1.yml, I am encountering 404 Not Found while accessing it. On the server, the file is there #ls -l "/var/www/html/dacdeveloperportal/web/sites/default/files/2020-08/dsmaAccount and Transaction API Specification_2_1.yml" -rwxrwxrwx 1 nginx nginx 418867 Aug 2 05:21 /var/www/html/dacdeveloperportal/web/sites/default/files/2020-08/dsmaAccount and Transaction API Specification_2_1.yml [root at dsmadeveloperportal web]#pwd /var/www/html/dsmadeveloperportal/web [root at dsmadeveloperportal web]# ls -l total 44 -rw-rw-r-- 1 nginx nginx 385 Jul 12 14:21 autoload.php drwxr-xr-x 12 nginx nginx 4096 Jul 28 11:52 core -rw-r--r-- 1 nginx nginx 1507 Jul 28 11:53 example.gitignore -rw-rw-r-- 1 nginx nginx 549 Jul 12 14:21 index.php -rw-r--r-- 1 nginx nginx 95 Jul 28 11:53 INSTALL.txt drwxrwxr-x 6 nginx nginx 77 Jul 28 11:53 libraries drwxrwxr-x 4 nginx nginx 69 Jul 28 11:53 modules drwxrwxr-x 2 nginx nginx 40 Jul 28 11:53 profiles -rw-r--r-- 1 nginx nginx 5924 Jul 28 11:53 README.txt -rw-rw-r-- 1 nginx nginx 1594 Jul 12 14:21 robots.txt drwxrwxr-x 3 nginx nginx 130 Jul 28 11:53 sites drwxrwxr-x 4 nginx nginx 69 Jul 28 11:53 themes -rw-rw-r-- 1 nginx nginx 848 Jul 12 14:21 update.php -rw-rw-r-- 1 nginx nginx 4566 Jul 12 14:21 web.config [root at dacdeveloperportal web]# *nginx config* > #cat /etc/nginx/conf.d/dsmadeveloperportal.conf > server { > listen 80 default_server; > server_name dsmadeveloperportal.mydomain.com; > return 301 https://$server_name$request_uri; } server { listen 443 ssl; server_name dsmadeveloperportal.mydomain.com; ssl_protocols TLSv1.3 TLSv1.2; ssl_stapling on; ssl_stapling_verify on; ssl_certificate /etc/ssl/certs/ dsmadeveloperportal.mydomain.com/fullchain1.pem; ssl_certificate_key /etc/ssl/certs/ dsmadeveloperportal.mydomain.com/privkey1.pem; ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384; root /var/www/html/dsmadeveloperportal/web; ssl_prefer_server_ciphers on; ssl_dhparam /etc/ssl/certs/ dsmadeveloperportal.mydomain.com/dhparam.pem; add_header Strict-Transport-Security: max-age=63072000; proxy_busy_buffers_size 512k; proxy_buffers 4 512k; proxy_buffer_size 256k; index index.php index.html index.htm; location = /favicon.ico { log_not_found off; access_log off; } location = /robots.txt { allow all; log_not_found off; access_log off; } location ~ \..*/.*\.php$ { return 403; } #location ~ ^/sites/.*/private/ { # return 403; #} # Block access to scripts in site files directory location ~ ^/sites/[^/]+/files/.*\.php$ { deny all; } # Allow "Well-Known URIs" as per RFC 5785 location ~* ^/.well-known/ { allow all; } # Block access to "hidden" files and directories whose names begin with a # period. This includes directories used by version control systems such # as Subversion or Git to store control files. location ~ (^|/)\. { return 403; } location / { # try_files $uri @rewrite; # For Drupal <= 6 try_files $uri /index.php?$query_string; # For Drupal >= 7 } location @rewrite { rewrite ^/(.*)$ /index.php?q=$1; } # Don't allow direct access to PHP files in the vendor directory. location ~ /vendor/.*\.php$ { deny all; return 404; } # Protect files and directories from prying eyes. location ~* \.(engine|inc|install|make|module|profile|po|sh|.*sql|theme|twig|tpl(\.php)?|xtmpl|yml)(~|\.sw[op]|\.bak|\.orig|\.save)?$|^(\.(?!well-known).*|Entries.*|Repository|Root|Tag|Template|composer\.(json|lock)|web\.config)$|^#.*#$|\.php(~|\.sw[op]|\.bak|\.orig|\.save)$ { deny all; return 404; } location ~ \.php$ { include fastcgi.conf; fastcgi_pass 127.0.0.1:9000; fastcgi_read_timeout 300s; fastcgi_buffers 16 16k; fastcgi_buffer_size 32k; } location ^~ /core/install.php { deny all; } } Please suggest. Thanks in advance and I look forward to hearing from you. Best Regards, Kaushal -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Mon Aug 2 17:48:12 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 2 Aug 2021 20:48:12 +0300 Subject: 404 Not Found while accessing the file In-Reply-To: References: Message-ID: Hello! On Mon, Aug 02, 2021 at 09:12:59PM +0530, Kaushal Shriyan wrote: > I am running nginx version: nginx/1.20.1 on CentOS Linux release 7.9.2009 > (Core). When I am accessing > https://dsmadeveloperportal.mydomain.com/sites/default/files/2020-08/dsmaAccount%20and%20Transaction%20API%20Specification_2_1.yml, > I am encountering 404 Not Found while accessing it. On the server, the > file is there > > #ls -l > "/var/www/html/dacdeveloperportal/web/sites/default/files/2020-08/dsmaAccount > and Transaction API Specification_2_1.yml" > -rwxrwxrwx 1 nginx nginx 418867 Aug 2 05:21 > /var/www/html/dacdeveloperportal/web/sites/default/files/2020-08/dsmaAccount > and Transaction API Specification_2_1.yml [...] > root /var/www/html/dsmadeveloperportal/web; Note this does not match the file location, "dsma" vs "dac". Note well that additional useful information can be found in nginx error log. Looking into error log usually greatly simplifies debugging problems like this. -- Maxim Dounin http://mdounin.ru/ From arut at nginx.com Tue Aug 3 09:14:59 2021 From: arut at nginx.com (Roman Arutyunyan) Date: Tue, 3 Aug 2021 12:14:59 +0300 Subject: Resurrecting the Async Open Discussion In-Reply-To: <013d30d9438514e306a56e666544e907.NginxMailingListEnglish@forum.nginx.org> References: <013d30d9438514e306a56e666544e907.NginxMailingListEnglish@forum.nginx.org> Message-ID: <20210803091459.62vhkwkms6qglvtp@Romans-MacBook-Pro.local> Hi, On Sun, Jul 25, 2021 at 07:33:20AM -0400, NCviQwilt wrote: > Hello, > > This is a follow up on the discussion started 3 years ago at: > https://forum.nginx.org/read.php?29,280794,280794 > I can't respond at that forum or topic, so I hope this is the right place. Actually, it would be really nice to continue the discussion in that thread. You don't need the forum for that. Just write to nginx-devel mailing list: http://mailman.nginx.org/mailman/listinfo/nginx-devel > I'm working at Qwilt, and we too have experienced the 999th percentile > problem as discussed in Cloudflare's blog post. > We were happy to see an official patch from Nginx Dev, but sad to see the > old discussion has since died with no official release. The reason why it died is we got no feedback from users showing the patches worked for them. A few people showed interest, but eventually sent no feedback. > I've started working on applying the patch from the old discussion, but the > integration isn't smooth since the source code has progressed since then > (especially with regards to the static module). I've updated the series. Not it applies to the latest nginx source code. http://mailman.nginx.org/pipermail/nginx-devel/2021-August/014194.html > We'll keep working on it, > but we're wondering if there are any plans for future official support for > this feature? > Otherwise any Nginx upgrade would be difficult. The best way to make sure the feature is supported is to commit the patches. As I said before, we need more feedback for that. Until then we'll try to keep the patches up-to-date. But this may become harder as the source code changes. > We'd be more than happy to help with the testing of the patch. We'll be happy to get feedback from you. > Best regards, > > Noam > > Posted at Nginx Forum: https://forum.nginx.org/read.php?2,292104,292104#msg-292104 > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx -- Roman Arutyunyan From nginx-forum at forum.nginx.org Tue Aug 3 13:13:07 2021 From: nginx-forum at forum.nginx.org (SannisDev) Date: Tue, 03 Aug 2021 09:13:07 -0400 Subject: Nginx with big geo maps affects whole system performance on reload Message-ID: <4090b516ab04167a71481efb0aa2246a.NginxMailingListEnglish@forum.nginx.org> I'm running Nginx 1.20.1 on CentOS 8.4.2105 with a quite large geo maps, including country/isp/connection for both IPv4 and IPv6, with raw size about 80Mb. Hardware is: 2x Intel(R) Xeon(R) Silver 4116 CPU @ 2.10GHz, 64Gb RAM On nginx testconf or reload I'm experiencing increase of running nginx latency, 99th percentile growth from 10ms to seconds. After some investigation I found that most hot function in testconf/reload are (sudo perf record -F 999999 /local/nginx/sbin/nginx -t): ``` 24.92% nginx nginx-1.20.1-cf9ac595_20210611_just_upgrade [.] ngx_conf_read_token 14.84% nginx nginx-1.20.1-cf9ac595_20210611_just_upgrade [.] ngx_radix128tree_insert 13.16% nginx nginx-1.20.1-cf9ac595_20210611_just_upgrade [.] ngx_http_geo 6.19% nginx [kernel.kallsyms] [k] inet_csk_bind_conflict 5.69% nginx nginx-1.20.1-cf9ac595_20210611_just_upgrade [.] ngx_radix32tree_insert 5.40% nginx nginx-1.20.1-cf9ac595_20210611_just_upgrade [.] ngx_ptocidr ``` Also perf stat slows high percent of cache misses (sudo perf stat -d /local/nginx/sbin/nginx -t): ``` Performance counter stats for '/local/nginx/sbin/nginx -t': 7,119.99 msec task-clock # 0.999 CPUs utilized 360 context-switches # 0.051 K/sec 1 cpu-migrations # 0.000 K/sec 289,666 page-faults # 0.041 M/sec 16,938,524,088 cycles # 2.379 GHz (62.49%) 26,748,500,575 instructions # 1.58 insn per cycle (74.99%) 7,737,642,525 branches # 1086.749 M/sec (74.99%) 35,255,267 branch-misses # 0.46% of all branches (75.00%) 5,309,612,140 L1-dcache-loads # 745.733 M/sec (75.01%) 91,351,291 L1-dcache-load-misses # 1.72% of all L1-dcache accesses (75.01%) 11,559,050 LLC-loads # 1.623 M/sec (50.00%) 7,370,770 LLC-load-misses # 63.77% of all LL-cache accesses (49.99%) 7.125064873 seconds time elapsed 5.849075000 seconds user 1.212134000 seconds sys ``` I suppose that this cache misses is the root cause of system slowdown on testconf/reload. Maybe someone have same problem and have some suggestions how to improve this situation? Posted at Nginx Forum: https://forum.nginx.org/read.php?2,292129,292129#msg-292129 From mdounin at mdounin.ru Tue Aug 3 14:41:27 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 3 Aug 2021 17:41:27 +0300 Subject: Nginx with big geo maps affects whole system performance on reload In-Reply-To: <4090b516ab04167a71481efb0aa2246a.NginxMailingListEnglish@forum.nginx.org> References: <4090b516ab04167a71481efb0aa2246a.NginxMailingListEnglish@forum.nginx.org> Message-ID: Hello! On Tue, Aug 03, 2021 at 09:13:07AM -0400, SannisDev wrote: > I'm running Nginx 1.20.1 on CentOS 8.4.2105 with a quite large geo maps, > including country/isp/connection for both IPv4 and IPv6, with raw size about > 80Mb. > Hardware is: 2x Intel(R) Xeon(R) Silver 4116 CPU @ 2.10GHz, 64Gb RAM > > On nginx testconf or reload I'm experiencing increase of running nginx > latency, 99th percentile growth from 10ms to seconds. [...] > I suppose that this cache misses is the root cause of system slowdown on > testconf/reload. > > Maybe someone have same problem and have some suggestions how to improve > this situation? Loading large geo map can be quite expensive, that's expected: nginx has to parse all the records provided in text and build corresponding radix tree. Make sure you have enough spare CPU and free memory to actually do the work without affecting other tasks on the server. In particular, in my practice it is a good idea to make sure the number of nginx worker processes is less than number of available cores, or you'll risk getting high latencies when doing CPU-intensive tasks like parsing geo maps or compressing logs. Also it might be a good idea to switch to using ranges and a separate include file, so nginx will be able to cache parsing results in a binary file. -- Maxim Dounin http://mdounin.ru/ From kaushalshriyan at gmail.com Tue Aug 3 15:43:06 2021 From: kaushalshriyan at gmail.com (Kaushal Shriyan) Date: Tue, 3 Aug 2021 21:13:06 +0530 Subject: 404 Not Found while accessing the file In-Reply-To: References: Message-ID: On Mon, Aug 2, 2021 at 11:18 PM Maxim Dounin wrote: > Hello! > > On Mon, Aug 02, 2021 at 09:12:59PM +0530, Kaushal Shriyan wrote: > > > I am running nginx version: nginx/1.20.1 on CentOS Linux release 7.9.2009 > > (Core). When I am accessing > > > https://dsmadeveloperportal.mydomain.com/sites/default/files/2020-08/dsmaAccount%20and%20Transaction%20API%20Specification_2_1.yml > , > > I am encountering 404 Not Found while accessing it. On the server, the > > file is there > > > > #ls -l > > > "/var/www/html/dacdeveloperportal/web/sites/default/files/2020-08/dsmaAccount > > and Transaction API Specification_2_1.yml" > > -rwxrwxrwx 1 nginx nginx 418867 Aug 2 05:21 > > > /var/www/html/dacdeveloperportal/web/sites/default/files/2020-08/dsmaAccount > > and Transaction API Specification_2_1.yml > > [...] > > > root /var/www/html/dsmadeveloperportal/web; > > Note this does not match the file location, "dsma" vs "dac". > > Note well that additional useful information can be found in nginx > error log. Looking into error log usually greatly simplifies > debugging problems like this. > > -- > Maxim Dounin > http://mdounin.ru/ > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx Hi Maxim, Thanks for the email. Apologies for sharing the incorrect file path, there was a typo error in the file path. Please ignore it. The correct file path is as below:- #ls -l "/var/www/html/dsmadeveloperportal/web/sites/default/files/2021-08/Account%20and%20Transaction%20API%20Specification_2_1.yml" -rwxrwxrwx 1 nginx nginx 418867 Aug 2 05:21 /var/www/html/dsmadeveloperportal/web/sites/default/files/2021-08/Account%20and%20Transaction%20API%20Specification_2_1.yml [root at dsmadeveloperportal web]#pwd /var/www/html/dsmadeveloperportal/web [root at dsmadeveloperportal web]# ls -l total 44 -rw-rw-r-- 1 nginx nginx 385 Jul 12 14:21 autoload.php drwxr-xr-x 12 nginx nginx 4096 Jul 28 11:52 core -rw-r--r-- 1 nginx nginx 1507 Jul 28 11:53 example.gitignore -rw-rw-r-- 1 nginx nginx 549 Jul 12 14:21 index.php -rw-r--r-- 1 nginx nginx 95 Jul 28 11:53 INSTALL.txt drwxrwxr-x 6 nginx nginx 77 Jul 28 11:53 libraries drwxrwxr-x 4 nginx nginx 69 Jul 28 11:53 modules drwxrwxr-x 2 nginx nginx 40 Jul 28 11:53 profiles -rw-r--r-- 1 nginx nginx 5924 Jul 28 11:53 README.txt -rw-rw-r-- 1 nginx nginx 1594 Jul 12 14:21 robots.txt drwxrwxr-x 3 nginx nginx 130 Jul 28 11:53 sites drwxrwxr-x 4 nginx nginx 69 Jul 28 11:53 themes -rw-rw-r-- 1 nginx nginx 848 Jul 12 14:21 update.php -rw-rw-r-- 1 nginx nginx 4566 Jul 12 14:21 web.config [root at dacdeveloperportal web]# I have attached the Nginx web server debug logs along with the Nginx config files for your reference. Please let me know if you need any additional information. Thanks in advance and I look forward to hearing from you. Please comment. Best Regards, Kaushal -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- 2021/08/03 09:55:07 [debug] 3406#3406: accept on 0.0.0.0:443, ready: 0 2021/08/03 09:55:07 [debug] 3405#3405: accept on 0.0.0.0:443, ready: 0 2021/08/03 09:55:07 [debug] 3406#3406: accept() not ready (11: Resource temporarily unavailable) 2021/08/03 09:55:07 [debug] 3405#3405: posix_memalign: 0000564F51CCF980:512 @16 2021/08/03 09:55:07 [debug] 3405#3405: *6 accept: 14.98.153.6:65126 fd:15 2021/08/03 09:55:07 [debug] 3405#3405: *6 event timer add: 15: 60000:67098843 2021/08/03 09:55:07 [debug] 3405#3405: *6 reusable connection: 1 2021/08/03 09:55:07 [debug] 3405#3405: *6 epoll add event: fd:15 op:1 ev:80002001 2021/08/03 09:55:07 [debug] 3405#3405: *6 http check ssl handshake 2021/08/03 09:55:07 [debug] 3405#3405: *6 http recv(): 1 2021/08/03 09:55:07 [debug] 3405#3405: *6 https ssl handshake: 0x16 2021/08/03 09:55:07 [debug] 3405#3405: *6 tcp_nodelay 2021/08/03 09:55:07 [debug] 3405#3405: *6 reusable connection: 0 2021/08/03 09:55:07 [debug] 3405#3405: *6 SSL server name: "dsmadeveloperportal.mydomain.com" 2021/08/03 09:55:07 [debug] 3405#3405: *6 SSL ALPN supported by client: h2 2021/08/03 09:55:07 [debug] 3405#3405: *6 SSL ALPN supported by client: http/1.1 2021/08/03 09:55:07 [debug] 3405#3405: *6 SSL ALPN selected: http/1.1 2021/08/03 09:55:07 [debug] 3405#3405: *6 SSL certificate status callback 2021/08/03 09:55:07 [debug] 3405#3405: *6 SSL_do_handshake: -1 2021/08/03 09:55:07 [debug] 3405#3405: *6 SSL_get_error: 2 2021/08/03 09:55:07 [debug] 3406#3406: accept on 0.0.0.0:443, ready: 0 2021/08/03 09:55:07 [debug] 3405#3405: accept on 0.0.0.0:443, ready: 0 2021/08/03 09:55:07 [debug] 3406#3406: posix_memalign: 0000564F51D20F70:512 @16 2021/08/03 09:55:07 [debug] 3405#3405: accept() not ready (11: Resource temporarily unavailable) 2021/08/03 09:55:07 [debug] 3406#3406: *7 accept: 14.98.153.6:65127 fd:11 2021/08/03 09:55:07 [debug] 3406#3406: *7 event timer add: 11: 60000:67098847 2021/08/03 09:55:07 [debug] 3406#3406: *7 reusable connection: 1 2021/08/03 09:55:07 [debug] 3406#3406: *7 epoll add event: fd:11 op:1 ev:80002001 2021/08/03 09:55:07 [debug] 3406#3406: *7 http check ssl handshake 2021/08/03 09:55:07 [debug] 3406#3406: *7 http recv(): 1 2021/08/03 09:55:07 [debug] 3406#3406: *7 https ssl handshake: 0x16 2021/08/03 09:55:07 [debug] 3406#3406: *7 tcp_nodelay 2021/08/03 09:55:07 [debug] 3406#3406: *7 reusable connection: 0 2021/08/03 09:55:07 [debug] 3406#3406: *7 SSL server name: "dsmadeveloperportal.mydomain.com" 2021/08/03 09:55:07 [debug] 3406#3406: *7 SSL ALPN supported by client: h2 2021/08/03 09:55:07 [debug] 3406#3406: *7 SSL ALPN supported by client: http/1.1 2021/08/03 09:55:07 [debug] 3406#3406: *7 SSL ALPN selected: http/1.1 2021/08/03 09:55:07 [debug] 3406#3406: *7 SSL certificate status callback 2021/08/03 09:55:07 [debug] 3406#3406: *7 SSL_do_handshake: -1 2021/08/03 09:55:07 [debug] 3406#3406: *7 SSL_get_error: 2 2021/08/03 09:55:07 [debug] 3405#3405: *6 SSL handshake handler: 0 2021/08/03 09:55:07 [debug] 3405#3405: *6 SSL_do_handshake: 1 2021/08/03 09:55:07 [debug] 3405#3405: *6 SSL: TLSv1.3, cipher: "TLS_AES_256_GCM_SHA384 TLSv1.3 Kx=any Au=any Enc=AESGCM(256) Mac=AEAD" 2021/08/03 09:55:07 [debug] 3405#3405: *6 reusable connection: 1 2021/08/03 09:55:07 [debug] 3405#3405: *6 http wait request handler 2021/08/03 09:55:07 [debug] 3405#3405: *6 malloc: 0000564F51CDA060:1024 2021/08/03 09:55:07 [debug] 3405#3405: *6 SSL_read: -1 2021/08/03 09:55:07 [debug] 3405#3405: *6 SSL_get_error: 2 2021/08/03 09:55:07 [debug] 3405#3405: *6 free: 0000564F51CDA060 2021/08/03 09:55:07 [debug] 3405#3405: *6 http wait request handler 2021/08/03 09:55:07 [debug] 3405#3405: *6 malloc: 0000564F51CDA060:1024 2021/08/03 09:55:07 [debug] 3405#3405: *6 SSL_read: 1024 2021/08/03 09:55:07 [debug] 3405#3405: *6 SSL_read: avail:0 2021/08/03 09:55:07 [debug] 3405#3405: *6 reusable connection: 0 2021/08/03 09:55:07 [debug] 3405#3405: *6 posix_memalign: 0000564F51CD8D70:4096 @16 2021/08/03 09:55:07 [debug] 3405#3405: *6 http process request line 2021/08/03 09:55:07 [debug] 3405#3405: *6 http request line: "GET /sites/default/files/2021-08/Account%20and%20Transaction%20API%20Specification_2_1.yml HTTP/1.1" 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'2F:/' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:1 in:'73:s' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'69:i' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'74:t' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'65:e' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'73:s' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'2F:/' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:1 in:'64:d' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'65:e' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'66:f' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'61:a' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'75:u' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'6C:l' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'74:t' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'2F:/' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:1 in:'66:f' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'69:i' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'6C:l' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'65:e' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'73:s' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'2F:/' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:1 in:'32:2' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'30:0' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'32:2' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'31:1' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'2D:-' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'30:0' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'38:8' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'2F:/' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:1 in:'41:A' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'63:c' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'63:c' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'6F:o' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'75:u' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'6E:n' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'74:t' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'25:%' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:4 in:'32:2' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:5 in:'30:0' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'20: ' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'61:a' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'6E:n' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'64:d' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'25:%' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:4 in:'32:2' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:5 in:'30:0' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'20: ' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'54:T' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'72:r' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'61:a' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'6E:n' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'73:s' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'61:a' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'63:c' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'74:t' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'69:i' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'6F:o' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'6E:n' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'25:%' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:4 in:'32:2' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:5 in:'30:0' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'20: ' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'41:A' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'50:P' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'49:I' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'25:%' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:4 in:'32:2' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:5 in:'30:0' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'20: ' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'53:S' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'70:p' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'65:e' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'63:c' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'69:i' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'66:f' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'69:i' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'63:c' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'61:a' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'74:t' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'69:i' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'6F:o' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'6E:n' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'5F:_' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'32:2' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'5F:_' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'31:1' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'2E:.' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'79:y' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'6D:m' 2021/08/03 09:55:07 [debug] 3405#3405: *6 s:0 in:'6C:l' 2021/08/03 09:55:07 [debug] 3405#3405: *6 http uri: "/sites/default/files/2021-08/Account and Transaction API Specification_2_1.yml" 2021/08/03 09:55:07 [debug] 3405#3405: *6 http args: "" 2021/08/03 09:55:07 [debug] 3405#3405: *6 http exten: "yml" 2021/08/03 09:55:07 [debug] 3405#3405: *6 posix_memalign: 0000564F51CCD660:4096 @16 2021/08/03 09:55:07 [debug] 3405#3405: *6 http process request header line 2021/08/03 09:55:07 [debug] 3405#3405: *6 http header: "Host: dsmadeveloperportal.mydomain.com" 2021/08/03 09:55:07 [debug] 3405#3405: *6 http header: "Connection: keep-alive" 2021/08/03 09:55:07 [debug] 3405#3405: *6 http header: "sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"" 2021/08/03 09:55:07 [debug] 3405#3405: *6 http header: "sec-ch-ua-mobile: ?0" 2021/08/03 09:55:07 [debug] 3405#3405: *6 http header: "Upgrade-Insecure-Requests: 1" 2021/08/03 09:55:07 [debug] 3405#3405: *6 http header: "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36" 2021/08/03 09:55:07 [debug] 3405#3405: *6 http header: "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9" 2021/08/03 09:55:07 [debug] 3405#3405: *6 http header: "Sec-Fetch-Site: none" 2021/08/03 09:55:07 [debug] 3405#3405: *6 http header: "Sec-Fetch-Mode: navigate" 2021/08/03 09:55:07 [debug] 3405#3405: *6 http header: "Sec-Fetch-User: ?1" 2021/08/03 09:55:07 [debug] 3405#3405: *6 http header: "Sec-Fetch-Dest: document" 2021/08/03 09:55:07 [debug] 3405#3405: *6 http header: "Accept-Encoding: gzip, deflate, br" 2021/08/03 09:55:07 [debug] 3405#3405: *6 http header: "Accept-Language: en-GB,en-US;q=0.9,en;q=0.8" 2021/08/03 09:55:07 [debug] 3405#3405: *6 http alloc large header buffer 2021/08/03 09:55:07 [debug] 3405#3405: *6 posix_memalign: 0000564F51CDA470:512 @16 2021/08/03 09:55:07 [debug] 3405#3405: *6 malloc: 0000564F51D9CC40:8192 2021/08/03 09:55:07 [debug] 3405#3405: *6 http large header alloc: 0000564F51D9CC40 8192 2021/08/03 09:55:07 [debug] 3405#3405: *6 http large header copy: 269 2021/08/03 09:55:07 [debug] 3405#3405: *6 SSL_read: 417 2021/08/03 09:55:07 [debug] 3405#3405: *6 SSL_read: -1 2021/08/03 09:55:07 [debug] 3405#3405: *6 SSL_get_error: 2 2021/08/03 09:55:07 [debug] 3405#3405: *6 http header: "Cookie: experimentation_subject_id=eyJfcmFpbHMiOnsibWVzc2FnZSI6IklqWXdOemt6TWpRNUxUQTRNakF0TkRObU55MDRaVGxqTFRJMFpqY3hPV1V3TXpJNE1pST0iLCJleHAiOm51bGwsInB1ciI6ImNvb2tpZS5leHBlcmltZW50YXRpb25fc3ViamVjdF9pZCJ9fQ%3D%3D--3d42b9c34822d75f78ddb657a93af64e73760eb9; AMCVS_2AB511DA57FB93B60A495C21%40AdobeOrg=1; AMCV_2AB511DA57FB93B60A495C21%40AdobeOrg=2096510701%7CMCIDTS%7C18843%7CMCMID%7C13117440465946190474636317212457833781%7CMCAID%7CNONE%7CMCOPTOUT-1627965451s%7CNONE%7CvVersion%7C2.0.0; utag_main=v_id:017b09df8ec90003eb1c1c75d7e90107900370710093c$_sn:1$_se:4$_ss:0$_st:1627980352491$ses_id:1627977923627%3Bexp-session$_pn:4%3Bexp-session$vapi_domain:mydomain.com" 2021/08/03 09:55:07 [debug] 3405#3405: *6 http header: "sec-gpc: 1" 2021/08/03 09:55:07 [debug] 3405#3405: *6 http header done 2021/08/03 09:55:07 [debug] 3405#3405: *6 event timer del: 15: 67098843 2021/08/03 09:55:07 [debug] 3405#3405: *6 generic phase: 0 2021/08/03 09:55:07 [debug] 3405#3405: *6 rewrite phase: 1 2021/08/03 09:55:07 [debug] 3405#3405: *6 test location: "/" 2021/08/03 09:55:07 [debug] 3405#3405: *6 test location: "favicon.ico" 2021/08/03 09:55:07 [debug] 3405#3405: *6 test location: "robots.txt" 2021/08/03 09:55:07 [debug] 3405#3405: *6 test location: ~ "\..*/.*\.php$" 2021/08/03 09:55:07 [debug] 3405#3405: *6 test location: ~ "^/sites/[^/]+/files/.*\.php$" 2021/08/03 09:55:07 [debug] 3405#3405: *6 test location: ~ "^/.well-known/" 2021/08/03 09:55:07 [debug] 3405#3405: *6 test location: ~ "(^|/)\." 2021/08/03 09:55:07 [debug] 3405#3405: *6 test location: ~ "/vendor/.*\.php$" 2021/08/03 09:55:07 [debug] 3405#3405: *6 test location: ~ "\.(engine|inc|install|make|module|profile|po|sh|.*sql|theme|twig|tpl(\.php)?|xtmpl|yml)(~|\.sw[op]|\.bak|\.orig|\.save)?$|^(\.(?!well-known).*|Entries.*|Repository|Root|Tag|Template|composer\.(json|lock)|web\.config)$|^#.*#$|\.php(~|\.sw[op]|\.bak|\.orig|\.save)$" 2021/08/03 09:55:07 [debug] 3405#3405: *6 using configuration "\.(engine|inc|install|make|module|profile|po|sh|.*sql|theme|twig|tpl(\.php)?|xtmpl|yml)(~|\.sw[op]|\.bak|\.orig|\.save)?$|^(\.(?!well-known).*|Entries.*|Repository|Root|Tag|Template|composer\.(json|lock)|web\.config)$|^#.*#$|\.php(~|\.sw[op]|\.bak|\.orig|\.save)$" 2021/08/03 09:55:07 [debug] 3405#3405: *6 http cl:-1 max:1048576 2021/08/03 09:55:07 [debug] 3405#3405: *6 rewrite phase: 3 2021/08/03 09:55:07 [debug] 3405#3405: *6 http finalize request: 404, "/sites/default/files/2021-08/Account and Transaction API Specification_2_1.yml?" a:1, c:1 2021/08/03 09:55:07 [debug] 3405#3405: *6 http special response: 404, "/sites/default/files/2021-08/Account and Transaction API Specification_2_1.yml?" 2021/08/03 09:55:07 [debug] 3405#3405: *6 http set discard body 2021/08/03 09:55:07 [debug] 3405#3405: *6 HTTP/1.1 404 Not Found Server: nginx/1.20.1 Date: Tue, 03 Aug 2021 09:55:07 GMT Content-Type: text/html Content-Length: 555 Connection: keep-alive 2021/08/03 09:55:07 [debug] 3405#3405: *6 write new buf t:1 f:0 0000564F51CCDB88, pos 0000564F51CCDB88, size: 155 file: 0, size: 0 2021/08/03 09:55:07 [debug] 3405#3405: *6 http write filter: l:0 f:0 s:155 2021/08/03 09:55:07 [debug] 3405#3405: *6 http output filter "/sites/default/files/2021-08/Account and Transaction API Specification_2_1.yml?" 2021/08/03 09:55:07 [debug] 3405#3405: *6 http copy filter: "/sites/default/files/2021-08/Account and Transaction API Specification_2_1.yml?" 2021/08/03 09:55:07 [debug] 3405#3405: *6 http postpone filter "/sites/default/files/2021-08/Account and Transaction API Specification_2_1.yml?" 0000564F51CD9D60 2021/08/03 09:55:07 [debug] 3405#3405: *6 write old buf t:1 f:0 0000564F51CCDB88, pos 0000564F51CCDB88, size: 155 file: 0, size: 0 2021/08/03 09:55:07 [debug] 3405#3405: *6 write new buf t:0 f:0 0000000000000000, pos 0000564F4FF066C0, size: 100 file: 0, size: 0 2021/08/03 09:55:07 [debug] 3405#3405: *6 write new buf t:0 f:0 0000000000000000, pos 0000564F4FF06E20, size: 53 file: 0, size: 0 2021/08/03 09:55:07 [debug] 3405#3405: *6 write new buf t:0 f:0 0000000000000000, pos 0000564F4FF06C00, size: 402 file: 0, size: 0 2021/08/03 09:55:07 [debug] 3405#3405: *6 http write filter: l:1 f:0 s:710 2021/08/03 09:55:07 [debug] 3405#3405: *6 http write filter limit 0 2021/08/03 09:55:07 [debug] 3405#3405: *6 malloc: 0000564F51D98AF0:16384 2021/08/03 09:55:07 [debug] 3405#3405: *6 SSL buf copy: 155 2021/08/03 09:55:07 [debug] 3405#3405: *6 SSL buf copy: 100 2021/08/03 09:55:07 [debug] 3405#3405: *6 SSL buf copy: 53 2021/08/03 09:55:07 [debug] 3405#3405: *6 SSL buf copy: 402 2021/08/03 09:55:07 [debug] 3405#3405: *6 SSL to write: 710 2021/08/03 09:55:07 [debug] 3405#3405: *6 SSL_write: 710 2021/08/03 09:55:07 [debug] 3405#3405: *6 http write filter 0000000000000000 2021/08/03 09:55:07 [debug] 3405#3405: *6 http copy filter: 0 "/sites/default/files/2021-08/Account and Transaction API Specification_2_1.yml?" 2021/08/03 09:55:07 [debug] 3405#3405: *6 http finalize request: 0, "/sites/default/files/2021-08/Account and Transaction API Specification_2_1.yml?" a:1, c:1 2021/08/03 09:55:07 [debug] 3405#3405: *6 set http keepalive handler 2021/08/03 09:55:07 [debug] 3405#3405: *6 http close request 2021/08/03 09:55:07 [debug] 3405#3405: *6 http log handler ==> /var/log/nginx/dsmadeveloperportal-access.log <== 14.98.153.6 - - [03/Aug/2021:09:55:07 +0000] "GET /sites/default/files/2021-08/Account%20and%20Transaction%20API%20Specification_2_1.yml HTTP/1.1" 404 555 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36" ==> /var/log/nginx/dsmadeveloperportal-error.log <== 2021/08/03 09:55:07 [debug] 3405#3405: *6 free: 0000564F51CD8D70, unused: 0 2021/08/03 09:55:07 [debug] 3405#3405: *6 free: 0000564F51CCD660, unused: 1827 2021/08/03 09:55:07 [debug] 3405#3405: *6 free: 0000564F51CDA060 2021/08/03 09:55:07 [debug] 3405#3405: *6 hc free: 0000000000000000 2021/08/03 09:55:07 [debug] 3405#3405: *6 hc busy: 0000564F51CDA4F0 1 2021/08/03 09:55:07 [debug] 3405#3405: *6 free: 0000564F51D9CC40 2021/08/03 09:55:07 [debug] 3405#3405: *6 free: 0000564F51D98AF0 2021/08/03 09:55:07 [debug] 3405#3405: *6 reusable connection: 1 2021/08/03 09:55:07 [debug] 3405#3405: *6 event timer add: 15: 65000:67103897 2021/08/03 09:55:07 [debug] 3406#3406: *7 SSL handshake handler: 0 2021/08/03 09:55:07 [debug] 3406#3406: *7 SSL_do_handshake: 1 2021/08/03 09:55:07 [debug] 3406#3406: *7 SSL: TLSv1.3, cipher: "TLS_AES_256_GCM_SHA384 TLSv1.3 Kx=any Au=any Enc=AESGCM(256) Mac=AEAD" 2021/08/03 09:55:07 [debug] 3406#3406: *7 reusable connection: 1 2021/08/03 09:55:07 [debug] 3406#3406: *7 http wait request handler 2021/08/03 09:55:07 [debug] 3406#3406: *7 malloc: 0000564F51D2AD10:1024 2021/08/03 09:55:07 [debug] 3406#3406: *7 SSL_read: -1 2021/08/03 09:55:07 [debug] 3406#3406: *7 SSL_get_error: 2 2021/08/03 09:55:07 [debug] 3406#3406: *7 free: 0000564F51D2AD10 2021/08/03 09:56:07 [debug] 3406#3406: *7 event timer del: 11: 67098847 2021/08/03 09:56:07 [debug] 3406#3406: *7 http wait request handler 2021/08/03 09:56:07 [info] 3406#3406: *7 client timed out (110: Connection timed out) while waiting for request, client: 14.98.153.6, server: 0.0.0.0:443 2021/08/03 09:56:07 [debug] 3406#3406: *7 close http connection: 11 2021/08/03 09:56:07 [debug] 3406#3406: *7 SSL_shutdown: 1 2021/08/03 09:56:07 [debug] 3406#3406: *7 reusable connection: 0 2021/08/03 09:56:07 [debug] 3406#3406: *7 free: 0000000000000000 2021/08/03 09:56:07 [debug] 3406#3406: *7 free: 0000564F51D20F70, unused: 0 2021/08/03 09:56:12 [debug] 3405#3405: *6 event timer del: 15: 67103897 2021/08/03 09:56:12 [debug] 3405#3405: *6 http keepalive handler 2021/08/03 09:56:12 [debug] 3405#3405: *6 close http connection: 15 2021/08/03 09:56:12 [debug] 3405#3405: *6 SSL_shutdown: 1 2021/08/03 09:56:12 [debug] 3405#3405: *6 reusable connection: 0 2021/08/03 09:56:12 [debug] 3405#3405: *6 free: 0000000000000000 2021/08/03 09:56:12 [debug] 3405#3405: *6 free: 0000000000000000 2021/08/03 09:56:12 [debug] 3405#3405: *6 free: 0000000000000000 2021/08/03 09:56:12 [debug] 3405#3405: *6 free: 0000564F51CCF980, unused: 0 2021/08/03 09:56:12 [debug] 3405#3405: *6 free: 0000564F51CDA470, unused: 272 -------------- next part -------------- cat /etc/nginx/conf.d/dsmadeveloperportal.conf server { listen 80 default_server; server_name dsmadeveloperportal.mydomain.com; return 301 https://$server_name$request_uri; } server { listen 443 ssl; server_name dsmadeveloperportal.mydomain.com; ssl_protocols TLSv1.3 TLSv1.2; ssl_stapling on; ssl_stapling_verify on; ssl_certificate /etc/ssl/certs/dsmadeveloperportal.mydomain.com/fullchain1.pem; ssl_certificate_key /etc/ssl/certs/dsmadeveloperportal.mydomain.com/privkey1.pem; access_log /var/log/nginx/dsmadeveloperportal-access.log; error_log /var/log/nginx/dsmadeveloperportal-error.log debug; ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384; root /var/www/html/dsmadeveloperportal/web; ssl_prefer_server_ciphers on; ssl_dhparam /etc/ssl/certs/dsmadeveloperportal.mydomain.com/dhparam.pem; add_header Strict-Transport-Security: max-age=63072000; proxy_busy_buffers_size 512k; proxy_buffers 4 512k; proxy_buffer_size 256k; index index.php index.html index.htm; location = /favicon.ico { log_not_found off; access_log off; } location = /robots.txt { allow all; log_not_found off; access_log off; } location ~ \..*/.*\.php$ { return 403; } #location ~ ^/sites/.*/private/ { # return 403; #} # Block access to scripts in site files directory location ~ ^/sites/[^/]+/files/.*\.php$ { deny all; } # Allow "Well-Known URIs" as per RFC 5785 location ~* ^/.well-known/ { allow all; } # Block access to "hidden" files and directories whose names begin with a # period. This includes directories used by version control systems such # as Subversion or Git to store control files. location ~ (^|/)\. { return 403; } location / { # try_files $uri @rewrite; # For Drupal <= 6 try_files $uri /index.php?$query_string; # For Drupal >= 7 } location @rewrite { rewrite ^/(.*)$ /index.php?q=$1; } # Don't allow direct access to PHP files in the vendor directory. location ~ /vendor/.*\.php$ { deny all; return 404; } # Protect files and directories from prying eyes. location ~* \.(engine|inc|install|make|module|profile|po|sh|.*sql|theme|twig|tpl(\.php)?|xtmpl|yml)(~|\.sw[op]|\.bak|\.orig|\.save)?$|^(\.(?!well-known).*|Entries.*|Repository|Root|Tag|Template|composer\.(json|lock)|web\.config)$|^#.*#$|\.php(~|\.sw[op]|\.bak|\.orig|\.save)$ { deny all; return 404; } location ~ \.php$ { include fastcgi.conf; fastcgi_pass 127.0.0.1:9000; fastcgi_read_timeout 300s; fastcgi_buffers 16 16k; fastcgi_buffer_size 32k; } location ^~ /core/install.php { deny all; } } From nginx-forum at forum.nginx.org Tue Aug 3 17:00:50 2021 From: nginx-forum at forum.nginx.org (SannisDev) Date: Tue, 03 Aug 2021 13:00:50 -0400 Subject: Nginx with big geo maps affects whole system performance on reload In-Reply-To: References: Message-ID: <78df9cfdfd840f9a592ab7734ac2712e.NginxMailingListEnglish@forum.nginx.org> Thanks for your quick answer! >> I'm running Nginx 1.20.1 on CentOS 8.4.2105 with a quite large geo maps, >> including country/isp/connection for both IPv4 and IPv6, with raw size about >> 80Mb. >> Hardware is: 2x Intel(R) Xeon(R) Silver 4116 CPU @ 2.10GHz, 64Gb RAM >> >> On nginx testconf or reload I'm experiencing increase of running nginx >> latency, 99th percentile growth from 10ms to seconds. [...] > Loading large geo map can be quite expensive, that's expected: > nginx has to parse all the records provided in text and build > corresponding radix tree. Make sure you have enough spare CPU and > free memory to actually do the work without affecting other tasks > on the server. In particular, in my practice it is a good idea to > make sure the number of nginx worker processes is less than number > of available cores, or you'll risk getting high latencies when > doing CPU-intensive tasks like parsing geo maps or compressing > logs. I followed this path but did not reach the end. Currently I set workers processes twice less than physical cores number, but this does not affect situation. Next step should be start nginx under taskset and set workers affinity to other cores but this sounds hacky for me. > Also it might be a good idea to switch to using ranges and a > separate include file, so nginx will be able to cache parsing > results in a binary file. According to [1] ranges does not support IPv6 for now. Is there a chance to request this via feature request? [1]: https://github.com/nginx/nginx/blob/master/src/http/modules/ngx_http_geo_module.c#L732 Posted at Nginx Forum: https://forum.nginx.org/read.php?2,292129,292140#msg-292140 From mdounin at mdounin.ru Tue Aug 3 17:47:15 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 3 Aug 2021 20:47:15 +0300 Subject: Nginx with big geo maps affects whole system performance on reload In-Reply-To: <78df9cfdfd840f9a592ab7734ac2712e.NginxMailingListEnglish@forum.nginx.org> References: <78df9cfdfd840f9a592ab7734ac2712e.NginxMailingListEnglish@forum.nginx.org> Message-ID: Hello! On Tue, Aug 03, 2021 at 01:00:50PM -0400, SannisDev wrote: > Thanks for your quick answer! > > >> I'm running Nginx 1.20.1 on CentOS 8.4.2105 with a quite large geo maps, > >> including country/isp/connection for both IPv4 and IPv6, with raw size > about > >> 80Mb. > >> Hardware is: 2x Intel(R) Xeon(R) Silver 4116 CPU @ 2.10GHz, 64Gb RAM > >> > >> On nginx testconf or reload I'm experiencing increase of running nginx > >> latency, 99th percentile growth from 10ms to seconds. > > [...] > > > Loading large geo map can be quite expensive, that's expected: > > nginx has to parse all the records provided in text and build > > corresponding radix tree. Make sure you have enough spare CPU and > > free memory to actually do the work without affecting other tasks > > on the server. In particular, in my practice it is a good idea to > > make sure the number of nginx worker processes is less than number > > of available cores, or you'll risk getting high latencies when > > doing CPU-intensive tasks like parsing geo maps or compressing > > logs. > > I followed this path but did not reach the end. > Currently I set workers processes twice less than physical cores number, > but this does not affect situation. Next step should be start nginx under > taskset > and set workers affinity to other cores but this sounds hacky for me. The worker_cpu_affinity directive can be used to control which cores are used by nginx (http://nginx.org/r/worker_cpu_affinity). But if you are still seeing latency spikes on configuration testing with nginx workers using only half of the available CPU cores (cores, not threads, correct?) this might indicate there is something wrong with scheduler configuration and it may need tuning. > > Also it might be a good idea to switch to using ranges and a > > separate include file, so nginx will be able to cache parsing > > results in a binary file. > > According to [1] ranges does not support IPv6 for now. > Is there a chance to request this via feature request? It's always a good idea to express the interest if there is one. Existing feature request is here: https://trac.nginx.org/nginx/ticket/1306 -- Maxim Dounin http://mdounin.ru/ From mdounin at mdounin.ru Tue Aug 3 18:07:01 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 3 Aug 2021 21:07:01 +0300 Subject: 404 Not Found while accessing the file In-Reply-To: References: Message-ID: Hello! On Tue, Aug 03, 2021 at 09:13:06PM +0530, Kaushal Shriyan wrote: > On Mon, Aug 2, 2021 at 11:18 PM Maxim Dounin wrote: > > > On Mon, Aug 02, 2021 at 09:12:59PM +0530, Kaushal Shriyan wrote: > > > > > I am running nginx version: nginx/1.20.1 on CentOS Linux release 7.9.2009 > > > (Core). When I am accessing > > > > > https://dsmadeveloperportal.mydomain.com/sites/default/files/2020-08/dsmaAccount%20and%20Transaction%20API%20Specification_2_1.yml > > , > > > I am encountering 404 Not Found while accessing it. On the server, the > > > file is there > > > > > > #ls -l > > > > > "/var/www/html/dacdeveloperportal/web/sites/default/files/2020-08/dsmaAccount > > > and Transaction API Specification_2_1.yml" > > > -rwxrwxrwx 1 nginx nginx 418867 Aug 2 05:21 > > > > > /var/www/html/dacdeveloperportal/web/sites/default/files/2020-08/dsmaAccount > > > and Transaction API Specification_2_1.yml > > > > [...] > > > > > root /var/www/html/dsmadeveloperportal/web; > > > > Note this does not match the file location, "dsma" vs "dac". > > > > Note well that additional useful information can be found in nginx > > error log. Looking into error log usually greatly simplifies > > debugging problems like this. [...] > Thanks for the email. Apologies for sharing the incorrect file path, there > was a typo error in the file path. Please ignore it. > The correct file path is as below:- [...] > 2021/08/03 09:55:07 [debug] 3405#3405: *6 http uri: "/sites/default/files/2021-08/Account and Transaction API Specification_2_1.yml" [...] > 2021/08/03 09:55:07 [debug] 3405#3405: *6 test location: ~ "\.(engine|inc|install|make|module|profile|po|sh|.*sql|theme|twig|tpl(\.php)?|xtmpl|yml)(~|\.sw[op]|\.bak|\.orig|\.save)?$|^(\.(?!well-known).*|Entries.*|Repository|Root|Tag|Template|composer\.(json|lock)|web\.config)$|^#.*#$|\.php(~|\.sw[op]|\.bak|\.orig|\.save)$" > 2021/08/03 09:55:07 [debug] 3405#3405: *6 using configuration "\.(engine|inc|install|make|module|profile|po|sh|.*sql|theme|twig|tpl(\.php)?|xtmpl|yml)(~|\.sw[op]|\.bak|\.orig|\.save)?$|^(\.(?!well-known).*|Entries.*|Repository|Root|Tag|Template|composer\.(json|lock)|web\.config)$|^#.*#$|\.php(~|\.sw[op]|\.bak|\.orig|\.save)$" > 2021/08/03 09:55:07 [debug] 3405#3405: *6 http cl:-1 max:1048576 > 2021/08/03 09:55:07 [debug] 3405#3405: *6 rewrite phase: 3 > 2021/08/03 09:55:07 [debug] 3405#3405: *6 http finalize request: 404, "/sites/default/files/2021-08/Account and Transaction API Specification_2_1.yml?" a:1, c:1 As it is immediately clear from the debug log, the request is matched by the regexp location in your configuration which is expected to match all requests to ".yml" files and return the 404 error. Quoting the configuration: > # Protect files and directories from prying eyes. > location ~* \.(engine|inc|install|make|module|profile|po|sh|.*sql|theme|twig|tpl(\.php)?|xtmpl|yml)(~|\.sw[op]|\.bak|\.orig|\.save)?$|^(\.(?!well-known).*|Entries.*|Repository|Root|Tag|Template|composer\.(json|lock)|web\.config)$|^#.*#$|\.php(~|\.sw[op]|\.bak|\.orig|\.save)$ { > deny all; > return 404; > } The most simple solution would be to remove this location - given the complexity, it is expected to be a constant source of issues like this. Alternatively, you can modify the location to allow all ".yml" files or this particular file. This is not something I can recommend though. -- Maxim Dounin http://mdounin.ru/ From mouseless at free.fr Tue Aug 3 19:14:02 2021 From: mouseless at free.fr (Vincent M.) Date: Tue, 3 Aug 2021 21:14:02 +0200 Subject: nginx getimagesize from local server: 403 Message-ID: <4f3ad3b5-e780-f1df-4697-0b2abaf954f9@free.fr> Hello all, I have a 403 error when trying to use the PHP function getimagesize on nginx based web server: list($imageWidth, $imageHeight) = getimagesize($imagesSource['Main']); Where $imageSource['Main'] is like: https://mywebsite/include/php/render/canvas/VR/2/image/U35p/ver//bgColor/ffffff/widthMM/300/heightMM/388/aube-sunny.jpg It's an image generated by PHP-imagick module and is not locally available. This image can be opened by a browser with no problem. To generate this image which go through 404. Nginx is configured like that: error_page 404 = /url_rewriting.php; And in the file url_rewriting.php, we analyse the URL to see if it's real 404 or an image which must be generated: if((isset($urlHash[2]) && $urlHash[2] === 'render')) { ... ??? header('Status: 200 OK', false, 200); ??? chdir('include/php/render/'.$_GET['render']); ??? require ('include/php/render/'.$_GET['render'].'/render_img.php'); ??? exit; } All is working fine but not the getimagesize from my own server. What is strange is that it's working fine on my development environment which has the same versions of nginx and PHP and pretty the same configuration. It might a configuration issue or a right issue on a certain file but I don't know where to check. Thank you, Vincent. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kaushalshriyan at gmail.com Tue Aug 3 22:42:21 2021 From: kaushalshriyan at gmail.com (Kaushal Shriyan) Date: Wed, 4 Aug 2021 04:12:21 +0530 Subject: 404 Not Found while accessing the file In-Reply-To: References: Message-ID: On Tue, Aug 3, 2021 at 11:37 PM Maxim Dounin wrote: > Hello! > > On Tue, Aug 03, 2021 at 09:13:06PM +0530, Kaushal Shriyan wrote: > > > On Mon, Aug 2, 2021 at 11:18 PM Maxim Dounin wrote: > > > > > On Mon, Aug 02, 2021 at 09:12:59PM +0530, Kaushal Shriyan wrote: > > > > > > > I am running nginx version: nginx/1.20.1 on CentOS Linux release > 7.9.2009 > > > > (Core). When I am accessing > > > > > > > > https://dsmadeveloperportal.mydomain.com/sites/default/files/2020-08/dsmaAccount%20and%20Transaction%20API%20Specification_2_1.yml > > > , > > > > I am encountering 404 Not Found while accessing it. On the server, > the > > > > file is there > > > > > > > > #ls -l > > > > > > > > "/var/www/html/dacdeveloperportal/web/sites/default/files/2020-08/dsmaAccount > > > > and Transaction API Specification_2_1.yml" > > > > -rwxrwxrwx 1 nginx nginx 418867 Aug 2 05:21 > > > > > > > > /var/www/html/dacdeveloperportal/web/sites/default/files/2020-08/dsmaAccount > > > > and Transaction API Specification_2_1.yml > > > > > > [...] > > > > > > > root /var/www/html/dsmadeveloperportal/web; > > > > > > Note this does not match the file location, "dsma" vs "dac". > > > > > > Note well that additional useful information can be found in nginx > > > error log. Looking into error log usually greatly simplifies > > > debugging problems like this. > > [...] > > > Thanks for the email. Apologies for sharing the incorrect file path, > there > > was a typo error in the file path. Please ignore it. > > The correct file path is as below:- > > [...] > > > 2021/08/03 09:55:07 [debug] 3405#3405: *6 http uri: > "/sites/default/files/2021-08/Account and Transaction API > Specification_2_1.yml" > > [...] > > > 2021/08/03 09:55:07 [debug] 3405#3405: *6 test location: ~ > "\.(engine|inc|install|make|module|profile|po|sh|.*sql|theme|twig|tpl(\.php)?|xtmpl|yml)(~|\.sw[op]|\.bak|\.orig|\.save)?$|^(\.(?!well-known).*|Entries.*|Repository|Root|Tag|Template|composer\.(json|lock)|web\.config)$|^#.*#$|\.php(~|\.sw[op]|\.bak|\.orig|\.save)$" > > 2021/08/03 09:55:07 [debug] 3405#3405: *6 using configuration > "\.(engine|inc|install|make|module|profile|po|sh|.*sql|theme|twig|tpl(\.php)?|xtmpl|yml)(~|\.sw[op]|\.bak|\.orig|\.save)?$|^(\.(?!well-known).*|Entries.*|Repository|Root|Tag|Template|composer\.(json|lock)|web\.config)$|^#.*#$|\.php(~|\.sw[op]|\.bak|\.orig|\.save)$" > > 2021/08/03 09:55:07 [debug] 3405#3405: *6 http cl:-1 max:1048576 > > 2021/08/03 09:55:07 [debug] 3405#3405: *6 rewrite phase: 3 > > 2021/08/03 09:55:07 [debug] 3405#3405: *6 http finalize request: 404, > "/sites/default/files/2021-08/Account and Transaction API > Specification_2_1.yml?" a:1, c:1 > > As it is immediately clear from the debug log, the request is > matched by the regexp location in your configuration which is > expected to match all requests to ".yml" files and return the 404 > error. Quoting the configuration: > > > # Protect files and directories from prying eyes. > > location ~* > \.(engine|inc|install|make|module|profile|po|sh|.*sql|theme|twig|tpl(\.php)?|xtmpl|yml)(~|\.sw[op]|\.bak|\.orig|\.save)?$|^(\.(?!well-known).*|Entries.*|Repository|Root|Tag|Template|composer\.(json|lock)|web\.config)$|^#.*#$|\.php(~|\.sw[op]|\.bak|\.orig|\.save)$ > { > > deny all; > > return 404; > > } > > The most simple solution would be to remove this location - given > the complexity, it is expected to be a constant source of issues > like this. > > Alternatively, you can modify the location to allow all ".yml" > files or this particular file. This is not something I can > recommend though. > Thanks Maxim and it worked like a charm. Is there a way to set the default permission to 700 whenever the user uploads the file in the nginx docroot? Is there an Nginx config that can set the file permission to 700 once the user uploads? Please comment. Thanks in advance. Best Regards, Kaushal -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Wed Aug 4 00:28:02 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 4 Aug 2021 03:28:02 +0300 Subject: 404 Not Found while accessing the file In-Reply-To: References: Message-ID: Hello! On Wed, Aug 04, 2021 at 04:12:21AM +0530, Kaushal Shriyan wrote: [...] > Thanks Maxim and it worked like a charm. Thanks for the confirmation. > Is there a way to set the default > permission to 700 whenever the user uploads the file in the nginx docroot? > Is there an Nginx config that can set the file permission to 700 once the > user uploads? Handling user uploads is not something nginx does. You have to set permissions where uploading happens. -- Maxim Dounin http://mdounin.ru/ From mouseless at free.fr Wed Aug 4 11:50:00 2021 From: mouseless at free.fr (Vincent M.) Date: Wed, 4 Aug 2021 13:50:00 +0200 Subject: nginx getimagesize from local server: 403 In-Reply-To: <4f3ad3b5-e780-f1df-4697-0b2abaf954f9@free.fr> References: <4f3ad3b5-e780-f1df-4697-0b2abaf954f9@free.fr> Message-ID: <55d31639-8b7b-7f2e-487a-aa0f796d31a7@free.fr> I found the reason! I have check and my files and folders are already with rights: 0755 for folders and 0644 for files. I checked also the owner of the website and unfortunately I did some test and did not resolved the issue. I have put a file aaa.jpg on my website root folder: Locally this code is working fine: list($imageWidth, $imageHeight) = getimagesize('http://local-mywebsite.com/aaa.jpg'); echo $imageWidth; So I change the code to: list($imageWidth, $imageHeight) = getimagesize('https://mywebsite.com/aaa.jpg'); echo $imageWidth; and I get the error: 2021/08/04 11:33:00 [error] 2196869#2196869: *388196 FastCGI sent in stderr: "PHP message: PHP Warning: getimagesize(https://mywebsite.com/aaa.jpg): failed to open stream: HTTP request failed! HTTP/1.1 403 Forbidden I change the right to 0777 to file and still have the 403. I changed the owner of the file aaa.jpg to www-data and changed the owner of the website root folder to www-data as well but still have the 403. On /etc/nginx/nginx.conf file I have: user www-data; I didn't know where to look at! And unexpectedly I remembered that yesterday I added some rules to nginx to block bad bots. Which I didn't do locally. I added this configuration rules: https://github.com/mitchellkrogza/nginx-badbot-blocker And took a look at it and notified this rules: ## Add here all hosts that should be spared any referrer checking. ## Whitelist all your own IPs in this section, each IP followed by a 0; geo $bad_referer { ??? 127.0.0.1 0; ??? 111.111.111.111 0; } I added the comment for 127.0.0.1 and it's now working fine. What is the line 111.1111.111.111 0 for ? Do you have any advise on this rules added from https://github.com/mitchellkrogza/nginx-badbot-blocker ? Thanks :) Le 03/08/2021 ? 21:14, Vincent M. a ?crit?: > > Hello all, > > I have a 403 error when trying to use the PHP function getimagesize on > nginx based web server: > > list($imageWidth, $imageHeight) = getimagesize($imagesSource['Main']); > > Where $imageSource['Main'] is like: > > https://mywebsite/include/php/render/canvas/VR/2/image/U35p/ver//bgColor/ffffff/widthMM/300/heightMM/388/aube-sunny.jpg > > It's an image generated by PHP-imagick module and is not locally > available. This image can be opened by a browser with no problem. > > To generate this image which go through 404. Nginx is configured like > that: > > error_page 404 = /url_rewriting.php; > And in the file url_rewriting.php, we analyse the URL to see if it's > real 404 or an image which must be generated: > > if((isset($urlHash[2]) && $urlHash[2] === 'render')) { > ... > ??? header('Status: 200 OK', false, 200); > ??? chdir('include/php/render/'.$_GET['render']); > ??? require ('include/php/render/'.$_GET['render'].'/render_img.php'); > ??? exit; > } > All is working fine but not the getimagesize from my own server. > > What is strange is that it's working fine on my development > environment which has the same versions of nginx and PHP and pretty > the same configuration. > > It might a configuration issue or a right issue on a certain file but > I don't know where to check. > > Thank you, > Vincent. > > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx -------------- next part -------------- An HTML attachment was scrubbed... URL: From nginx-forum at forum.nginx.org Wed Aug 4 15:53:06 2021 From: nginx-forum at forum.nginx.org (mumu) Date: Wed, 04 Aug 2021 11:53:06 -0400 Subject: Using IPv6 wildcard address in server_name Message-ID: <1e2139c1a25d33c99d446cf8b5137cfa.NginxMailingListEnglish@forum.nginx.org> Hi, there is wildcard support for name-based virtual hosts. Is there anything similar for IP-based virtual host with IPv4/IPv6 addresses ? Posted at Nginx Forum: https://forum.nginx.org/read.php?2,292153,292153#msg-292153 From nginx-forum at forum.nginx.org Thu Aug 5 07:50:45 2021 From: nginx-forum at forum.nginx.org (nginx_newbie1) Date: Thu, 05 Aug 2021 03:50:45 -0400 Subject: How to disable request pipelining on nginx upstream In-Reply-To: <1811807.ET9GBBa5to@vbart-laptop> References: <1811807.ET9GBBa5to@vbart-laptop> Message-ID: <90d831eb00a6c5f1a609f7e189369f51.NginxMailingListEnglish@forum.nginx.org> > No, it's not possible. > > As already were said twice, nginx dosn't support pipelining on the > upstream > side. > > wbr, Valentin V. Bartenev Has there been any progress on this in the last few years? I am trying to use Nginx as a proxy to our legacy server but the hosting company has a limit of 50 connections per IP address. Theoretically, if Nginx can fetch 50 requests concurrently and we have a load of 500 users on the client-side, how can I manage that from a single server? Is there any setting that could help in this situation? Posted at Nginx Forum: https://forum.nginx.org/read.php?2,269248,292161#msg-292161 From nginx-forum at forum.nginx.org Thu Aug 5 07:57:22 2021 From: nginx-forum at forum.nginx.org (nginx_newbie1) Date: Thu, 05 Aug 2021 03:57:22 -0400 Subject: How to disable request pipelining on nginx upstream In-Reply-To: <90d831eb00a6c5f1a609f7e189369f51.NginxMailingListEnglish@forum.nginx.org> References: <1811807.ET9GBBa5to@vbart-laptop> <90d831eb00a6c5f1a609f7e189369f51.NginxMailingListEnglish@forum.nginx.org> Message-ID: <2deea967ac59c16da79e5d00a39e6961.NginxMailingListEnglish@forum.nginx.org> Our upstream actually supports HTTP/2 connections. So, another question would be, if I use HTTP when fetching data from upstream. I will be able to have more than 50 requests concurrently on upstream? Posted at Nginx Forum: https://forum.nginx.org/read.php?2,269248,292162#msg-292162 From n5d9xq3ti233xiyif2vp at protonmail.ch Thu Aug 5 10:01:28 2021 From: n5d9xq3ti233xiyif2vp at protonmail.ch (Laura Smith) Date: Thu, 05 Aug 2021 10:01:28 +0000 Subject: NGINX Docs - where to submit change suggestions ? Message-ID: Hi Any ideas where change suggestions for docs should be submitted ? Specifically this page: http://nginx.org/en/linux_packages.html#Debian The instructions presented are not inline with Debian best-practices. As per https://wiki.debian.org/DebianRepository/UseThirdParty: "The key MUST be downloaded over a secure mechanism like HTTPS to a location only writable by root, which SHOULD be /usr/share/keyrings. The key MUST NOT be placed in /etc/apt/trusted.gpg.d or loaded by apt-key add. A sources.list entry SHOULD have the signed-by option set. The signed-by entry MUST point to a file, and not a fingerprint." From thresh at nginx.com Thu Aug 5 10:41:14 2021 From: thresh at nginx.com (Konstantin Pavlov) Date: Thu, 5 Aug 2021 13:41:14 +0300 Subject: NGINX Docs - where to submit change suggestions ? In-Reply-To: References: Message-ID: Hi Laura, 05.08.2021 13:01, Laura Smith wrote: > Hi > > Any ideas where change suggestions for docs should be submitted ? Specifically this page: http://nginx.org/en/linux_packages.html#Debian Here is fine. > The instructions presented are not inline with Debian best-practices. > > As per https://wiki.debian.org/DebianRepository/UseThirdParty: > "The key MUST be downloaded over a secure mechanism like HTTPS to a location only writable by root, which SHOULD be /usr/share/keyrings. The key MUST NOT be placed in /etc/apt/trusted.gpg.d or loaded by apt-key add. A sources.list entry SHOULD have the signed-by option set. The signed-by entry MUST point to a file, and not a fingerprint." Yeah, I think it makes sense to rework it. I'll prepare the patches - thanks for the notification! Relevant reading: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=861695 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=877012 Have a good day, -- Konstantin Pavlov https://www.nginx.com/ From mouseless at free.fr Fri Aug 6 14:20:56 2021 From: mouseless at free.fr (Vincent M.) Date: Fri, 6 Aug 2021 16:20:56 +0200 Subject: Same root folder for multiple website different robots.txt Message-ID: Hello, I have a first website with this configuration: server { ??????? server_name mywebsite.com; ??????? root /var/www/website/prod; ... } and a second one like this with the same root folder: server { ??????? server_name pro.mywebsite.com; ??????? root /var/www/website/prod; ... } And the only problem is they have the same robots.txt file. Is it possible to tell the second server to link the file robots.txt to another file like robots-pro.txt? And by the way the url http://pro.mywebsite.com/robots.txt would open the file robots-pro.txt. Thanks, Vincent. -------------- next part -------------- An HTML attachment was scrubbed... URL: From francis at daoine.org Fri Aug 6 16:21:07 2021 From: francis at daoine.org (Francis Daly) Date: Fri, 6 Aug 2021 17:21:07 +0100 Subject: Same root folder for multiple website different robots.txt In-Reply-To: References: Message-ID: <20210806162107.GJ11167@daoine.org> On Fri, Aug 06, 2021 at 04:20:56PM +0200, Vincent M. wrote: Hi there, > Is it possible to tell the second server to link the file robots.txt to > another file like robots-pro.txt? Either of location = /robots.txt { alias /var/www/website/prod/robots-pro.txt; } location = /robots.txt { try_files /robots-pro.txt =404; } should work. In the first case, you list the absolute file name. In the second, you list the file within the normal "root" directory. Cheers, f -- Francis Daly francis at daoine.org From francis at daoine.org Fri Aug 6 16:25:11 2021 From: francis at daoine.org (Francis Daly) Date: Fri, 6 Aug 2021 17:25:11 +0100 Subject: Using IPv6 wildcard address in server_name In-Reply-To: <1e2139c1a25d33c99d446cf8b5137cfa.NginxMailingListEnglish@forum.nginx.org> References: <1e2139c1a25d33c99d446cf8b5137cfa.NginxMailingListEnglish@forum.nginx.org> Message-ID: <20210806162511.GK11167@daoine.org> On Wed, Aug 04, 2021 at 11:53:06AM -0400, mumu wrote: Hi there, > Hi, there is wildcard support for name-based virtual hosts. Is there > anything similar for IP-based virtual host with IPv4/IPv6 addresses ? Does the "default_server" option on the "listen" directive do what you want? If not, could you explain in more detail what it is that you want? Thanks, f -- Francis Daly francis at daoine.org From mouseless at free.fr Sat Aug 7 11:35:14 2021 From: mouseless at free.fr (Vincent M.) Date: Sat, 7 Aug 2021 13:35:14 +0200 Subject: Same root folder for multiple website different robots.txt In-Reply-To: <20210806162107.GJ11167@daoine.org> References: <20210806162107.GJ11167@daoine.org> Message-ID: Le 06/08/2021 ? 18:21, Francis Daly a ?crit?: > On Fri, Aug 06, 2021 at 04:20:56PM +0200, Vincent M. wrote: > > Hi there, > >> Is it possible to tell the second server to link the file robots.txt to >> another file like robots-pro.txt? > Either of > > location = /robots.txt { alias /var/www/website/prod/robots-pro.txt; } > > location = /robots.txt { try_files /robots-pro.txt =404; } > > should work. > > In the first case, you list the absolute file name. In the second, > you list the file within the normal "root" directory. > > Cheers, > > f Yes, helped me a lot. For the main domain I did to block robots-denyall.txt: ??????? location /robots-denyall.txt { ??????????????? return 404; ??????? } And for the subdomain which I don't want to be referenced: ??????? location = /robots.txt { ??????????????? alias /var/www/mywebsite/prod/robots-denyall.txt; ??????? } Thank you :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From nginx-forum at forum.nginx.org Wed Aug 11 02:17:52 2021 From: nginx-forum at forum.nginx.org (Viete) Date: Tue, 10 Aug 2021 22:17:52 -0400 Subject: Modern alternative to CGI/FastCGI Message-ID: <69ef7838d7bbbd33affbdf701fef0499.NginxMailingListEnglish@forum.nginx.org> Hello, I am using nginx and trying to add REST API. I need to have C++ backend respond to the REST API calls. What are my modern options other than CGI/FastCGI old technology that seems to be unmaintained anymore. Thank you! Posted at Nginx Forum: https://forum.nginx.org/read.php?2,292183,292183#msg-292183 From teward at thomas-ward.net Wed Aug 11 03:15:38 2021 From: teward at thomas-ward.net (Thomas Ward) Date: Tue, 10 Aug 2021 23:15:38 -0400 Subject: Modern alternative to CGI/FastCGI In-Reply-To: <69ef7838d7bbbd33affbdf701fef0499.NginxMailingListEnglish@forum.nginx.org> References: <69ef7838d7bbbd33affbdf701fef0499.NginxMailingListEnglish@forum.nginx.org> Message-ID: Why not build your C++ backend to have a web listener that you can handle responses directly with?? Typically speaking, a lot of REST APIs I see are designed to accept HTTP and then have nginx or similar reverse_proxy requests to them to the API is (indirectly) exposed to port 80 (HTTP) or 443 (HTTPS), and then nginx facilitates the communication between the requesting client and the backend, without requiring CGI / FastCGI to do it. (This however is an application development question, moreso than an nginx one, as to how you would go about handing the communication to the client via NGINX, though proxy_pass and straight HTTP request/URL handling on your application is going to be simpler than trying to find a 'scripting' solution like CGI or FastCGI; you'll find a lot of current web frameworks like Django, etc. behave this way, accepting a proxy_pass or similar (or a uwsgi integration) to handle the handoff between NGINX and the application.) Thomas On 8/10/21 10:17 PM, Viete wrote: > Hello, I am using nginx and trying to add REST API. I need to have C++ > backend respond to the REST API calls. What are my modern options other than > CGI/FastCGI old technology that seems to be unmaintained anymore. > > Thank you! > > Posted at Nginx Forum: https://forum.nginx.org/read.php?2,292183,292183#msg-292183 > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lance at wordkeeper.com Thu Aug 12 00:43:12 2021 From: lance at wordkeeper.com (Lance Dockins) Date: Wed, 11 Aug 2021 19:43:12 -0500 Subject: NJS Questions and/or Bugs References: <654d0f53-892f-4497-9528-65b0628c481d@Spark> Message-ID: We?ve been experimenting with NJS in its current form (as compared with the more established LuaJIT integration provided by OpenResty) and it?s surfaced a series of questions about objects and features in NJS that the documentation, examples, recordings, and 3rd party examples don?t seem to answer very well.??I?m not sure if those should be asked here or in another division of Nginx but I wanted to lay them out. Request Data ? Is there already an NJS object or are there plans for an object that parses and maps POST key/value pairs into a JSON object? ? Is there already an NJS object or are there plans for an object that parses out the cookies into key/value pairs of a JSON object? We?re been experimenting with args, variables, rawVariables, etc and as a general rule, those objects seem to always come back as empty objects except for the args array which seems to always include the parsed output of the query string in a key/value pairing.??That might also be a fallacy of our testing methodology since we?re using JSON.stringify to visualize the output in the browser.??The documentation doesn?t fully flesh out what would be contained in these r and s values so it?s hard to really determine what they might or should contain without trying something like that.??Thus far, the only way we have found to retrieve cookies or POST vars from the request is by writing our own parsing logic in JS and passing the requestText to those to parse them out in various ways depending on whether they are url encoded, form/multipart, cookies, etc. Note that we are testing this via the js_content tie in. Variables ? What exactly should r.variables and r.rawVariables contain???The assumption that I?ve been making is that those correspond to the variables defined in the Nginx configuration or Nginx variables that were set by NJS (making them editable by NJS as well but also available to Nginx conf).??Thus far we just get empty objects in response to that when outputting that via the method I named above.??But we most definitely have a TON of Nginx variables that we?ve manually set in the conf aside from the core Nginx variables Access Filters ? Are there any plans to add an access phase filter???I know that this can be simulated via the js_content tie in but that phase runs later than the access phase in Nginx anyway and requires you to either do a subrequest or internalRedirect over to a named backend route in order to both access filter the request AND also support the intended functionality of the location block where that is used.??That?s both more cumbersome and slower than it would be to just use the regular Nginx content phase and configuration for content and use NJS for access control in the access phase of the request. Obviously LuaJIT and OpenResty are much more mature and there?s an argument to be had for LuaJIT being more performant but it?s not as well supported as NJS and JS is a much friendlier ecosystem to than LuaJIT so we prefer NJS/JS if possible.??None of that is even accounting for all of the more complex dependency management that you have to do with LuaJIT if you don?t just use a straight OpenResty build.??So ideally we?d prefer to use NJS if it is workable.??That said, at the moment, though, there appear to be a few key things in the basic use and scripting of the request that LuaJIT/OpenResty implementations have that NJS doesn?t even though they seem like they are really things that should be in any request scripting system like: 1. Filters for the different phase of the Nginx request 2. Direct variable access to the Nginx variables (either read only or writable depending on the context within which they were created) 3. Array like access to the request arguments (e.g. cookies, post args, query string args, etc).??Right now the only version of this that we?ve found reliably working in the js_content phase is query string related stuff (in the args array) Apologies if there are holes in my understanding here or confusion here - even so much as if this is directed to the wrong ?forum? within Nginx.??NJS looks awesome but the lack of clarity in the documentation and the tendency we?ve seen to get an empty object back out of JSON.stringify from what appears like it should be a fully filed out object has made it hard to get much headway into this. -- Lance Dockins -------------- next part -------------- An HTML attachment was scrubbed... URL: From osa at freebsd.org.ru Thu Aug 12 01:49:24 2021 From: osa at freebsd.org.ru (Sergey A. Osokin) Date: Thu, 12 Aug 2021 04:49:24 +0300 Subject: NJS Questions and/or Bugs In-Reply-To: References: <654d0f53-892f-4497-9528-65b0628c481d@Spark> Message-ID: Hi Lance, hope you're doing well. On Wed, Aug 11, 2021 at 07:43:12PM -0500, Lance Dockins wrote: > We?ve been experimenting with NJS in its current form (as compared > with the more established LuaJIT integration provided by OpenResty) > and it?s surfaced a series of questions about objects and features > in NJS that the documentation, examples, recordings, and 3rd party > examples don?t seem to answer very well.??I?m not sure if those > should be asked here or in another division of Nginx but I wanted > to lay them out. > > Request Data > > ? Is there already an NJS object or are there plans for an object > that parses and maps POST key/value pairs into a JSON object? It's already implemented, here's an example of code: var jObj = JSON.stringify(r.args); > ? Is there already an NJS object or are there plans for an object > that parses out the cookies into key/value pairs of a JSON object? Incoming headers of a request are represented in r.headersIn read-only object, so it's possible to implement some additional logic, like the following one: for (var h in r.headersIn) { // where: // h is the header's name of a request // r.headersIn[h] is the header's value } Hope that helps. -- Sergey Osokin From lance at wordkeeper.com Thu Aug 12 02:05:32 2021 From: lance at wordkeeper.com (Lance Dockins) Date: Wed, 11 Aug 2021 21:05:32 -0500 Subject: NJS Questions and/or Bugs In-Reply-To: References: <654d0f53-892f-4497-9528-65b0628c481d@Spark> Message-ID: <51294409-4c46-4812-b362-5d3961b42547@Spark> Hi Sergey, Thank you.??Perhaps something is going wrong with the implementation of r.args that we?re using.??I?m just getting an empty JS object for POST vars.??Args only seems to work for query string vars for me. Here?s the key config info. In the location block for the route that I?m testing: add_header Content-Type text/plain; js_content dbadmin.access; The dbadmin.access function just has this at the top of the function right now: r.return(200,?JSON.stringify(r.args)); And here is a screenshot of a POST to the location route that I?m using.??Assuming that the POST vars were organized into a JSON object, I would assume an output in JSON in key/value pairs.??But what I?m getting is an empty object. https://share.wordkeeper.com/AFXzvfi2kO1x024Mc9sm I?m using the latest version of NJS 0.6.1 and the latest version of Nginx 1.21.1.??Is this an indication that there is a bug somewhere in NJS or does this seem more like I?m misunderstanding or misconfguring things? As I mentioned, I can get the post args into an array by implementing my own JS functions for parsing the POST body via r.requestText and something similar for parsing/processing cookies.??It just seemed like this should be working better - in regards to the POST especially. -- Lance Dockins On Aug 11, 2021, 8:50 PM -0500, Sergey A. Osokin , wrote: > Hi Lance, > > hope you're doing well. > > On Wed, Aug 11, 2021 at 07:43:12PM -0500, Lance Dockins wrote: > > We?ve been experimenting with NJS in its current form (as compared > > with the more established LuaJIT integration provided by OpenResty) > > and it?s surfaced a series of questions about objects and features > > in NJS that the documentation, examples, recordings, and 3rd party > > examples don?t seem to answer very well.??I?m not sure if those > > should be asked here or in another division of Nginx but I wanted > > to lay them out. > > > > Request Data > > > > ? Is there already an NJS object or are there plans for an object > > that parses and maps POST key/value pairs into a JSON object? > > It's already implemented, here's an example of code: > > var jObj = JSON.stringify(r.args); > > > ? Is there already an NJS object or are there plans for an object > > that parses out the cookies into key/value pairs of a JSON object? > > Incoming headers of a request are represented in r.headersIn read-only > object, so it's possible to implement some additional logic, like the > following one: > > for (var h in r.headersIn) { > // where: > // h is the header's name of a request > // r.headersIn[h] is the header's value > } > > Hope that helps. > > -- > Sergey Osokin > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx -------------- next part -------------- An HTML attachment was scrubbed... URL: From osa at freebsd.org.ru Thu Aug 12 02:19:53 2021 From: osa at freebsd.org.ru (Sergey A. Osokin) Date: Thu, 12 Aug 2021 05:19:53 +0300 Subject: NJS Questions and/or Bugs In-Reply-To: <51294409-4c46-4812-b362-5d3961b42547@Spark> References: <654d0f53-892f-4497-9528-65b0628c481d@Spark> <51294409-4c46-4812-b362-5d3961b42547@Spark> Message-ID: Hi Lance, On Wed, Aug 11, 2021 at 09:05:32PM -0500, Lance Dockins wrote: > Hi Sergey, > > Thank you.??Perhaps something is going wrong with the implementation > of r.args that we?re using.??I?m just getting an empty JS object for > POST vars.??Args only seems to work for query string vars for me. > > Here?s the key config info. > > In the location block for the route that I?m testing: > > add_header Content-Type text/plain; > js_content dbadmin.access; > > The dbadmin.access function just has this at the top of the function right now: > r.return(200,?JSON.stringify(r.args)); Could you please share a couple of things: - whole dbadmin.js file; - a server block from nginx configuration file with a full location description For development and testing I'd recommend to use: - curl command line utility, i.e. something like: % curl -v --data-binary @test.bin http://127.0.0.1:8080/test?foo=bar\&baz=1 - enable debugging as described at http://nginx.org/en/docs/debugging_log.html so you'll be able to see what's going on with a request processing -- Sergey From lance at wordkeeper.com Thu Aug 12 02:31:41 2021 From: lance at wordkeeper.com (Lance Dockins) Date: Wed, 11 Aug 2021 21:31:41 -0500 Subject: NJS Questions and/or Bugs In-Reply-To: References: <654d0f53-892f-4497-9528-65b0628c481d@Spark> <51294409-4c46-4812-b362-5d3961b42547@Spark> Message-ID: <58e705d4-9d08-402e-ae14-529eaa601608@Spark> Hi Sergey, Sure.??The tool that I?m using for HTTP testing is just a GUI interface directly to HTTP so it works the same as a browser would since it?s just straight HTTP.??It supports curl mode too but that?s just a alternative wrapper for HTTP and none of the methods that we?re using to interface with this via POST (including direct browser POST) are producing different results. I realize that the implementation in the access function right now makes no sense.??This is just a stripped down version of where this will eventually go specifically for the sake of testing POST args access.??I get the same results even if we remove the r.internalRedirect statement. dbadmin.js function?access(r){ ?? ?r.return(200,?JSON.stringify(r.args)); ?? ?r.internalRedirect('@php'); } export?default?{access} The location block for the config file: # Pass PHP requests to PHP-FPM location ~ \.php$ { limit_req zone=post_limit burst=5 nodelay; limit_req_status 429; add_header Content-Type text/plain; js_content dbadmin.access; } location @php{ set $no_cache 1; include includes/fastcgi.conf; fastcgi_pass unix:/var/run/php-fpm/dbadmin.sock; # fastcgi_cache dbadmin; } The relevant import directive from nginx.conf js_import scripts/security/dbadmin.js; Incidentally, this config also illustrates why I was asking about access phase filters.??In LuaJIT/OpenResty, we can tap into the access phase to deny a request without having to define a separate location block to internalRedirect to.? ?It seems like the only way to simulate this with NJS right now is to define a location, use js_content to hand off to NJS for content, and then internalRedirect or sub request to a different location block to allow requests through.??It?s simpler if there is just a way to control access in the access phase without having to use a workaround in the content phase to try to add access control first and then run the content phase normally. In any case, I also wanted to confirm that the curl method of doing the POST doesn?t really change anything as well.??Here?s a screenshot along with the corresponding curl request that the app runs. https://share.wordkeeper.com/n4SEbeW3mUqZXtejjXsS -- Lance Dockins Minister of Magic WordKeeper Office: 405.585.2500 Cell: 405.306.7401 https://wordkeeper.com On Aug 11, 2021, 9:20 PM -0500, Sergey A. Osokin , wrote: > Hi Lance, > > On Wed, Aug 11, 2021 at 09:05:32PM -0500, Lance Dockins wrote: > > Hi Sergey, > > > > Thank you.??Perhaps something is going wrong with the implementation > > of r.args that we?re using.??I?m just getting an empty JS object for > > POST vars.??Args only seems to work for query string vars for me. > > > > Here?s the key config info. > > > > In the location block for the route that I?m testing: > > > > add_header Content-Type text/plain; > > js_content dbadmin.access; > > > > The dbadmin.access function just has this at the top of the function right now: > > r.return(200,?JSON.stringify(r.args)); > > Could you please share a couple of things: > > - whole dbadmin.js file; > - a server block from nginx configuration file with a full location > description > > For development and testing I'd recommend to use: > - curl command line utility, i.e. something like: > % curl -v --data-binary @test.bin http://127.0.0.1:8080/test?foo=bar\&baz=1 > > - enable debugging as described at http://nginx.org/en/docs/debugging_log.html > so you'll be able to see what's going on with a request processing > > -- > Sergey > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx -------------- next part -------------- An HTML attachment was scrubbed... URL: From lance at wordkeeper.com Thu Aug 12 02:38:50 2021 From: lance at wordkeeper.com (Lance Dockins) Date: Wed, 11 Aug 2021 21:38:50 -0500 Subject: NJS Questions and/or Bugs In-Reply-To: <58e705d4-9d08-402e-ae14-529eaa601608@Spark> References: <654d0f53-892f-4497-9528-65b0628c481d@Spark> <51294409-4c46-4812-b362-5d3961b42547@Spark> <58e705d4-9d08-402e-ae14-529eaa601608@Spark> Message-ID: Oh? and to reiterate, I get similarly empty object responses when access the variables and rawVariables objects.??Most of them seem to be empty objects, actually. -- Lance Dockins On Aug 11, 2021, 9:31 PM -0500, Lance Dockins , wrote: > Hi Sergey, > > Sure.??The tool that I?m using for HTTP testing is just a GUI interface directly to HTTP so it works the same as a browser would since it?s just straight HTTP.??It supports curl mode too but that?s just a alternative wrapper for HTTP and none of the methods that we?re using to interface with this via POST (including direct browser POST) are producing different results. > > I realize that the implementation in the access function right now makes no sense.??This is just a stripped down version of where this will eventually go specifically for the sake of testing POST args access.??I get the same results even if we remove the r.internalRedirect statement. > > dbadmin.js > function?access(r){??? ?r.return(200,?JSON.stringify(r.args)); > ?? ?r.internalRedirect('@php');} > export?default?{access} > The location block for the config file:# Pass PHP requests to PHP-FPM > location ~ \.php$ { > limit_req zone=post_limit burst=5 nodelay; limit_req_status 429; > add_header Content-Type text/plain; js_content dbadmin.access;} > > location @php{ > set $no_cache 1; include includes/fastcgi.conf; > fastcgi_pass unix:/var/run/php-fpm/dbadmin.sock; # fastcgi_cache dbadmin;} > > The relevant import directive from nginx.conf > js_import scripts/security/dbadmin.js; > > Incidentally, this config also illustrates why I was asking about access phase filters.??In LuaJIT/OpenResty, we can tap into the access phase to deny a request without having to define a separate location block to internalRedirect to.? ?It seems like the only way to simulate this with NJS right now is to define a location, use js_content to hand off to NJS for content, and then internalRedirect or sub request to a different location block to allow requests through.??It?s simpler if there is just a way to control access in the access phase without having to use a workaround in the content phase to try to add access control first and then run the content phase normally. > In any case, I also wanted to confirm that the curl method of doing the POST doesn?t really change anything as well.??Here?s a screenshot along with the corresponding curl request that the app runs.https://share.wordkeeper.com/n4SEbeW3mUqZXtejjXsS > > -- > Lance Dockins > > On Aug 11, 2021, 9:20 PM -0500, Sergey A. Osokin , wrote: > > Hi Lance, > > > > On Wed, Aug 11, 2021 at 09:05:32PM -0500, Lance Dockins wrote: > > > Hi Sergey, > > > > > > Thank you.??Perhaps something is going wrong with the implementation > > > of r.args that we?re using.??I?m just getting an empty JS object for > > > POST vars.??Args only seems to work for query string vars for me. > > > > > > Here?s the key config info. > > > > > > In the location block for the route that I?m testing: > > > > > > add_header Content-Type text/plain; > > > js_content dbadmin.access; > > > > > > The dbadmin.access function just has this at the top of the function right now: > > > r.return(200,?JSON.stringify(r.args)); > > > > Could you please share a couple of things: > > > > - whole dbadmin.js file; > > - a server block from nginx configuration file with a full location > > description > > > > For development and testing I'd recommend to use: > > - curl command line utility, i.e. something like: > > % curl -v --data-binary @test.bin http://127.0.0.1:8080/test?foo=bar\&baz=1 > > > > - enable debugging as described at http://nginx.org/en/docs/debugging_log.html > > so you'll be able to see what's going on with a request processing > > > > -- > > Sergey > > _______________________________________________ > > nginx mailing list > > nginx at nginx.org > > http://mailman.nginx.org/mailman/listinfo/nginx -------------- next part -------------- An HTML attachment was scrubbed... URL: From osa at freebsd.org.ru Thu Aug 12 02:40:42 2021 From: osa at freebsd.org.ru (Sergey A. Osokin) Date: Thu, 12 Aug 2021 05:40:42 +0300 Subject: NJS Questions and/or Bugs In-Reply-To: <58e705d4-9d08-402e-ae14-529eaa601608@Spark> References: <654d0f53-892f-4497-9528-65b0628c481d@Spark> <51294409-4c46-4812-b362-5d3961b42547@Spark> <58e705d4-9d08-402e-ae14-529eaa601608@Spark> Message-ID: On Wed, Aug 11, 2021 at 09:31:41PM -0500, Lance Dockins wrote: > Hi Sergey, > ... > > dbadmin.js > function?access(r){ > ?? ?r.return(200,?JSON.stringify(r.args)); > > ?? ?r.internalRedirect('@php'); Two points: - missing return directive here; - r.internalRedirect will never happend because of r.return above. > } > > export?default?{access} Here's my code that works as expected: function argskeyvalue(r) { var jObj = JSON.stringify(r.args) + '\n'; try { var parseJSON = JSON.parse(jObj); } catch (err) { r.return(400, 'Cannot parse jObj'); return; } r.return(200, jObj); return; } export default { argskeyvalue }; As you can see it creates a JSON object jObj, then test it's parseable and if so return 200. Hope that helps. -- Sergey From osa at freebsd.org.ru Thu Aug 12 02:42:23 2021 From: osa at freebsd.org.ru (Sergey A. Osokin) Date: Thu, 12 Aug 2021 05:42:23 +0300 Subject: NJS Questions and/or Bugs In-Reply-To: References: <654d0f53-892f-4497-9528-65b0628c481d@Spark> <51294409-4c46-4812-b362-5d3961b42547@Spark> <58e705d4-9d08-402e-ae14-529eaa601608@Spark> Message-ID: On Wed, Aug 11, 2021 at 09:38:50PM -0500, Lance Dockins wrote: > Oh? and to reiterate, I get similarly empty object responses > when access the variables and rawVariables objects.??Most of > them seem to be empty objects, actually. I'd recommend to enable nginx debugging log as I mentioned earlier. Thank you. -- Sergey From lance at wordkeeper.com Thu Aug 12 02:48:48 2021 From: lance at wordkeeper.com (Lance Dockins) Date: Wed, 11 Aug 2021 21:48:48 -0500 Subject: NJS Questions and/or Bugs In-Reply-To: References: <654d0f53-892f-4497-9528-65b0628c481d@Spark> <51294409-4c46-4812-b362-5d3961b42547@Spark> <58e705d4-9d08-402e-ae14-529eaa601608@Spark> Message-ID: <1208b2ee-b512-4be5-b463-9da913b1f23b@Spark> I?ll try recompiling nginx with that to see if that changes anything. Also, to be clear, I know that r.internalRedirect won?t happen.??I actually stripped the access function down to almost it?s barest content to demonstrate the point and left that code in there just to demo what we do at the end of the route when we don?t just have a debug statement at the top of the function. Normally we either allow or deny the request and then just try to load the originally intended content for that block (though we have to internalRedirect to get that to work).??Even without that internalRedirect statement in the code, just returning the object data with no other code in the function is returning an empty JS object. In fact, even your code is returning nothing.??I just set it to this: function?access(r){ ?? ?var?jObj?=?JSON.stringify(r.args)?+?'\n'; ?? ?try?{ ?? ? ? ?var?parseJSON?=?JSON.parse(jObj); ?? ?}?catch?(err) { ?? ? ? ?r.return(400,?'Cannot parse jObj'); ?? ? ? ?return; ?? ?} ?? ?r.return(200,?jObj); ?? ?return; } export?default?{access} When I?submit a POST with vars, I still get an empty object in the response. -- Lance Dockins On Aug 11, 2021, 9:42 PM -0500, Sergey A. Osokin , wrote: > On Wed, Aug 11, 2021 at 09:38:50PM -0500, Lance Dockins wrote: > > Oh? and to reiterate, I get similarly empty object responses > > when access the variables and rawVariables objects.??Most of > > them seem to be empty objects, actually. > > I'd recommend to enable nginx debugging log as I mentioned earlier. > > Thank you. > > -- > Sergey > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx -------------- next part -------------- An HTML attachment was scrubbed... URL: From osa at freebsd.org.ru Thu Aug 12 02:57:29 2021 From: osa at freebsd.org.ru (Sergey A. Osokin) Date: Thu, 12 Aug 2021 05:57:29 +0300 Subject: NJS Questions and/or Bugs In-Reply-To: <1208b2ee-b512-4be5-b463-9da913b1f23b@Spark> References: <654d0f53-892f-4497-9528-65b0628c481d@Spark> <51294409-4c46-4812-b362-5d3961b42547@Spark> <58e705d4-9d08-402e-ae14-529eaa601608@Spark> <1208b2ee-b512-4be5-b463-9da913b1f23b@Spark> Message-ID: On Wed, Aug 11, 2021 at 09:48:48PM -0500, Lance Dockins wrote: > I?ll try recompiling nginx with that to see if that changes anything. I'd recommend to use official packages for Linux from the following URL, http://nginx.org/en/linux_packages.html > In fact, even your code is returning nothing. That's definitely something wrong on your side. Please simplify the nginx configuration, remove all unnecessary directives. ... > When I?submit a POST with vars, I still get an empty object in the response. A couple of suggestions: - Have you enabled debugging log? If so, could you share nginx-error.log. - How did you test the code? Here's my shell script: #!/bin/sh curl -v --data-binary @test.bin http://127.0.0.1:8080/test?foo=bar\&baz=1 #EOF Please report when all suggested points are done, thanks. -- Sergey From lance at wordkeeper.com Thu Aug 12 03:00:15 2021 From: lance at wordkeeper.com (Lance Dockins) Date: Wed, 11 Aug 2021 22:00:15 -0500 Subject: NJS Questions and/or Bugs In-Reply-To: References: <654d0f53-892f-4497-9528-65b0628c481d@Spark> <51294409-4c46-4812-b362-5d3961b42547@Spark> <58e705d4-9d08-402e-ae14-529eaa601608@Spark> <1208b2ee-b512-4be5-b463-9da913b1f23b@Spark> Message-ID: Unfortunately, we can?t really use standard packages.??We need extra modules.??I?m recompiling and enabling debug now.??The modules we?re using aren?t anything super abnormal and we might replace one of them with NJS at some point.??All I know for sure is that this seems to work just fine with OpenResty?s LuaJIT integration/module.??So it?s something that is different between NJS and that. Hopefully the debug will turn something up. -- Lance Dockins On Aug 11, 2021, 9:57 PM -0500, Sergey A. Osokin , wrote: > On Wed, Aug 11, 2021 at 09:48:48PM -0500, Lance Dockins wrote: > > I?ll try recompiling nginx with that to see if that changes anything. > > I'd recommend to use official packages for Linux from the following URL, > http://nginx.org/en/linux_packages.html > > > In fact, even your code is returning nothing. > > That's definitely something wrong on your side. Please simplify the > nginx configuration, remove all unnecessary directives. > > ... > > > When I?submit a POST with vars, I still get an empty object in the response. > > A couple of suggestions: > > - Have you enabled debugging log? If so, could you share nginx-error.log. > - How did you test the code? Here's my shell script: > > #!/bin/sh > > curl -v --data-binary @test.bin http://127.0.0.1:8080/test?foo=bar\&baz=1 > > #EOF > > Please report when all suggested points are done, thanks. > > -- > Sergey > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx -------------- next part -------------- An HTML attachment was scrubbed... URL: From lance at wordkeeper.com Thu Aug 12 03:44:41 2021 From: lance at wordkeeper.com (Lance Dockins) Date: Wed, 11 Aug 2021 22:44:41 -0500 Subject: NJS Questions and/or Bugs In-Reply-To: References: <654d0f53-892f-4497-9528-65b0628c481d@Spark> <51294409-4c46-4812-b362-5d3961b42547@Spark> <58e705d4-9d08-402e-ae14-529eaa601608@Spark> <1208b2ee-b512-4be5-b463-9da913b1f23b@Spark> Message-ID: Something is definitely wrong here but I?ve stripped most of the more advanced configuration out of the vhost conf, removed all modules other than NJS, and set the Nginx compile to pretty much the most vanilla compile option set that I can to rule out the possibility of 3rd party interference with this.??Nothing is having any impact on the args var output.??For POST, it?s always an empty object.??For GET, it?s always showing the query string vars in a key/value pair (if there are any).??The variables and rawVariables objects always seem to be empty but I?m not setting any vars with NJS right now.??I?m only setting those in Nginx. At this point, if you?re saying that it should be working, I?m inclined to think that I?ve discovered some sort of bug.??Debug didn?t turn up anything unexpected.??I?m not the most well versed person at parsing through debug logs but it all looked like expected stuff to me. Moreover, the config that I was using at the start of this conversation is a config that is working on multiple thousands of production websites on well over 100 servers that we care for with no issues whatsoever.??Not to mention that that config works just fine with LuaJIT.??The only difference was that we swapped out the LuaJIT hooks for NJS to see how that might work instead.? ?But like I said, at this point, even with no 3rd party modules, no patches of any sort, very basic compile settings, and no fancy use of Nginx settings (e.g. extensive use of IF), it?s just not working. So either I have stumbled across some flaw in NJS or there?s a misunderstanding somewhere in the discussion. -- Lance Dockins On Aug 11, 2021, 10:00 PM -0500, Lance Dockins , wrote: > Unfortunately, we can?t really use standard packages.??We need extra modules.??I?m recompiling and enabling debug now.??The modules we?re using aren?t anything super abnormal and we might replace one of them with NJS at some point.??All I know for sure is that this seems to work just fine with OpenResty?s LuaJIT integration/module.??So it?s something that is different between NJS and that. > > Hopefully the debug will turn something up. > > > -- > Lance Dockins > > On Aug 11, 2021, 9:57 PM -0500, Sergey A. Osokin , wrote: > > On Wed, Aug 11, 2021 at 09:48:48PM -0500, Lance Dockins wrote: > > > I?ll try recompiling nginx with that to see if that changes anything. > > > > I'd recommend to use official packages for Linux from the following URL, > > http://nginx.org/en/linux_packages.html > > > > > In fact, even your code is returning nothing. > > > > That's definitely something wrong on your side. Please simplify the > > nginx configuration, remove all unnecessary directives. > > > > ... > > > > > When I?submit a POST with vars, I still get an empty object in the response. > > > > A couple of suggestions: > > > > - Have you enabled debugging log? If so, could you share nginx-error.log. > > - How did you test the code? Here's my shell script: > > > > #!/bin/sh > > > > curl -v --data-binary @test.bin http://127.0.0.1:8080/test?foo=bar\&baz=1 > > > > #EOF > > > > Please report when all suggested points are done, thanks. > > > > -- > > Sergey > > _______________________________________________ > > nginx mailing list > > nginx at nginx.org > > http://mailman.nginx.org/mailman/listinfo/nginx -------------- next part -------------- An HTML attachment was scrubbed... URL: From lance at wordkeeper.com Thu Aug 12 03:47:54 2021 From: lance at wordkeeper.com (Lance Dockins) Date: Wed, 11 Aug 2021 22:47:54 -0500 Subject: NJS Questions and/or Bugs In-Reply-To: References: <654d0f53-892f-4497-9528-65b0628c481d@Spark> <51294409-4c46-4812-b362-5d3961b42547@Spark> <58e705d4-9d08-402e-ae14-529eaa601608@Spark> <1208b2ee-b512-4be5-b463-9da913b1f23b@Spark> Message-ID: Also, just to be clear, you?re saying that r.args should include parsed POST vars???Obviously it includes query string/GET vars.??That works.??But the values in the POST body do not show up in the args output.??And again, I basically see nothing coming up in the output for variables. This: # Pass PHP requests to PHP-FPM location ~ \.php$ { limit_req zone=post_limit burst=5 nodelay; limit_req_status 429; set $test 0; add_header Content-Type text/plain; js_content dbadmin.access; } function?access(r){ ?? ?var?jObj?=?JSON.stringify(r.args)?+?'\n'; ?? ?jObj?=?JSON.stringify(r.variables); ?? ?try?{ ?? ? ? ?var?parseJSON?=?JSON.parse(jObj); ?? ?}?catch?(err) { ?? ? ? ?r.return(400,?'Cannot parse jObj'); ?? ? ? ?return; ?? ?} ?? ?r.return(200,?jObj); ?? ?return; } export?default?{access} Shows nothing.??Just an empty JSON object like {} -- Lance Dockins Minister of Magic WordKeeper Office: 405.585.2500 Cell: 405.306.7401 https://wordkeeper.com On Aug 11, 2021, 10:44 PM -0500, Lance Dockins , wrote: > Something is definitely wrong here but I?ve stripped most of the more advanced configuration out of the vhost conf, removed all modules other than NJS, and set the Nginx compile to pretty much the most vanilla compile option set that I can to rule out the possibility of 3rd party interference with this.??Nothing is having any impact on the args var output.??For POST, it?s always an empty object.??For GET, it?s always showing the query string vars in a key/value pair (if there are any).??The variables and rawVariables objects always seem to be empty but I?m not setting any vars with NJS right now.??I?m only setting those in Nginx. > > At this point, if you?re saying that it should be working, I?m inclined to think that I?ve discovered some sort of bug.??Debug didn?t turn up anything unexpected.??I?m not the most well versed person at parsing through debug logs but it all looked like expected stuff to me. > > Moreover, the config that I was using at the start of this conversation is a config that is working on multiple thousands of production websites on well over 100 servers that we care for with no issues whatsoever.??Not to mention that that config works just fine with LuaJIT.??The only difference was that we swapped out the LuaJIT hooks for NJS to see how that might work instead.? ?But like I said, at this point, even with no 3rd party modules, no patches of any sort, very basic compile settings, and no fancy use of Nginx settings (e.g. extensive use of IF), it?s just not working. > > So either I have stumbled across some flaw in NJS or there?s a misunderstanding somewhere in the discussion. > > > -- > Lance Dockins > > On Aug 11, 2021, 10:00 PM -0500, Lance Dockins , wrote: > > Unfortunately, we can?t really use standard packages.??We need extra modules.??I?m recompiling and enabling debug now.??The modules we?re using aren?t anything super abnormal and we might replace one of them with NJS at some point.??All I know for sure is that this seems to work just fine with OpenResty?s LuaJIT integration/module.??So it?s something that is different between NJS and that. > > > > Hopefully the debug will turn something up. > > > > > > -- > > Lance Dockins > > > > On Aug 11, 2021, 9:57 PM -0500, Sergey A. Osokin , wrote: > > > On Wed, Aug 11, 2021 at 09:48:48PM -0500, Lance Dockins wrote: > > > > I?ll try recompiling nginx with that to see if that changes anything. > > > > > > I'd recommend to use official packages for Linux from the following URL, > > > http://nginx.org/en/linux_packages.html > > > > > > > In fact, even your code is returning nothing. > > > > > > That's definitely something wrong on your side. Please simplify the > > > nginx configuration, remove all unnecessary directives. > > > > > > ... > > > > > > > When I?submit a POST with vars, I still get an empty object in the response. > > > > > > A couple of suggestions: > > > > > > - Have you enabled debugging log? If so, could you share nginx-error.log. > > > - How did you test the code? Here's my shell script: > > > > > > #!/bin/sh > > > > > > curl -v --data-binary @test.bin http://127.0.0.1:8080/test?foo=bar\&baz=1 > > > > > > #EOF > > > > > > Please report when all suggested points are done, thanks. > > > > > > -- > > > Sergey > > > _______________________________________________ > > > nginx mailing list > > > nginx at nginx.org > > > http://mailman.nginx.org/mailman/listinfo/nginx -------------- next part -------------- An HTML attachment was scrubbed... URL: From nginx-forum at forum.nginx.org Thu Aug 12 03:57:18 2021 From: nginx-forum at forum.nginx.org (Viete) Date: Wed, 11 Aug 2021 23:57:18 -0400 Subject: Modern alternative to CGI/FastCGI In-Reply-To: References: Message-ID: Thanks Thomas, that helps as I didn't know other frameworks use this idea. How would I go about also keeping a communication channel open for continuous data flow (for example, if I'd like to have nginx returning a stream of jpeg files (MJPEG) after the proxy_pass all)? Thank you! Posted at Nginx Forum: https://forum.nginx.org/read.php?2,292183,292208#msg-292208 From xeioex at nginx.com Thu Aug 12 06:14:42 2021 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Thu, 12 Aug 2021 09:14:42 +0300 Subject: NJS Questions and/or Bugs In-Reply-To: References: <654d0f53-892f-4497-9528-65b0628c481d@Spark> Message-ID: Hi Lance, Thanks for your detailed feedback. >What exactly should r.variables and r.rawVariables contain?? ... r.variables and r.rawVariables are special objects, because they are created on the fly. There is no way to get all the possible variable names from nginx. So Object.keys(r.variables) is empty list. Therefore dumping and iterating over r.variables returns nothing. But, if you have a $foo variable, r.variables.foo would return the $foo content as a string. > That might also be a fallacy of our testing methodology since we?re using JSON.stringify to visualize the output in the browser. The JSON.stringify() is no the best, use njs.dump(). http://nginx.org/en/docs/njs/reference.html#njs >Is there already an NJS object or are there plans for an object that parses and maps POST key/value pairs into a JSON object? Take a look at https://github.com/nginx/njs/issues/48#issuecomment-415745451 >Filters for the different phase of the Nginx request We are thinking about the adding phases handlers. >Direct variable access to the Nginx variables (either read only or writable depending on the context within which they were created) this is already in place >Array like access to the request arguments (e.g. cookies, post args, query string args, etc).? Right now the only version of this that we?ve found reliably working in the js_content phase is query string related stuff (in the args array) Feel free to add a feature request here https://github.com/nginx/njs Although this is not a feature requested often (for some reason). From thresh at nginx.com Thu Aug 12 10:21:36 2021 From: thresh at nginx.com (Konstantin Pavlov) Date: Thu, 12 Aug 2021 13:21:36 +0300 Subject: NGINX Docs - where to submit change suggestions ? In-Reply-To: References: Message-ID: <812ec882-9932-9340-cbdf-761b9928ccec@nginx.com> Hi Laura, 05.08.2021 13:41, Konstantin Pavlov wrote: > Hi Laura, > > 05.08.2021 13:01, Laura Smith wrote: >> Hi >> >> Any ideas where change suggestions for docs should be submitted ? Specifically this page: http://nginx.org/en/linux_packages.html#Debian > > Here is fine. > >> The instructions presented are not inline with Debian best-practices. The page has been updated to conform to best practices for Debian and Ubuntu. Thanks! -- Konstantin Pavlov https://www.nginx.com/ From n5d9xq3ti233xiyif2vp at protonmail.ch Thu Aug 12 12:35:43 2021 From: n5d9xq3ti233xiyif2vp at protonmail.ch (Laura Smith) Date: Thu, 12 Aug 2021 12:35:43 +0000 Subject: NGINX Docs - where to submit change suggestions ? In-Reply-To: <812ec882-9932-9340-cbdf-761b9928ccec@nginx.com> References: <812ec882-9932-9340-cbdf-761b9928ccec@nginx.com> Message-ID: ??????? Original Message ??????? On Thursday, August 12th, 2021 at 11:21 AM, Konstantin Pavlov wrote: > Hi Laura, > > 05.08.2021 13:41, Konstantin Pavlov wrote: > > > Hi Laura, > > > > 05.08.2021 13:01, Laura Smith wrote: > > > > > Hi > > > > > > Any ideas where change suggestions for docs should be submitted ? Specifically this page: http://nginx.org/en/linux_packages.html#Debian > > > > Here is fine. > > > > > The instructions presented are not inline with Debian best-practices. > > The page has been updated to conform to best practices for Debian and > > Ubuntu. > > Thanks! > > ------------------------------------------------------------------------------------------ > > Konstantin Pavlov > > https://www.nginx.com/ Super, thanks ! From lance at wordkeeper.com Thu Aug 12 12:38:25 2021 From: lance at wordkeeper.com (Lance Dockins) Date: Thu, 12 Aug 2021 07:38:25 -0500 Subject: NJS Questions and/or Bugs In-Reply-To: References: <654d0f53-892f-4497-9528-65b0628c481d@Spark> Message-ID: <9a6b4919-a32b-49c7-94b4-426f60c5658b@Spark> Thank you Dmitry.??This helps a ton. I will note that the post key/value pair solution only works for URL encoded post values.??If the enctype is set to multipart/form-data, you get get the data without writing your own data parser.??I already wrote one for URL encoded types and was working on one for multipart form data at the time that I wrote this.??Parsing the data with JS is just not the most elegant solution - particularly since some sort of parsing is going on already.??Maybe I am also just spoiled from using fastcgi to PHP where all of that is pre-parsed.? ?But it seems like a fairly straightforward request data access feature.??I guess I will just have to make due for now. -- Lance Dockins On Aug 12, 2021, 1:14 AM -0500, Dmitry Volyntsev , wrote: > Hi Lance, > > Thanks for your detailed feedback. > > > What exactly should r.variables and r.rawVariables contain?? ... > > r.variables and r.rawVariables are special objects, because they are > created on the fly. There is no way to get all the possible variable > names from nginx. So Object.keys(r.variables) is empty list. Therefore > dumping and iterating over r.variables returns nothing. But, if you have > a $foo variable, r.variables.foo would return the $foo content as a string. > > > That might also be a fallacy of our testing methodology since we?re > using JSON.stringify to visualize the output in the browser. > > The JSON.stringify() is no the best, use njs.dump(). > http://nginx.org/en/docs/njs/reference.html#njs > > > Is there already an NJS object or are there plans for an object that > parses and maps POST key/value pairs into a JSON object? > > Take a look at https://github.com/nginx/njs/issues/48#issuecomment-415745451 > > > Filters for the different phase of the Nginx request > > We are thinking about the adding phases handlers. > > > Direct variable access to the Nginx variables (either read only or > writable depending on the context within which they were created) > > this is already in place > > > Array like access to the request arguments (e.g. cookies, post args, > query string args, etc).? Right now the only version of this that we?ve > found reliably working in the js_content phase is query string related > stuff (in the args array) > > Feel free to add a feature request here https://github.com/nginx/njs > Although this is not a feature requested often (for some reason). -------------- next part -------------- An HTML attachment was scrubbed... URL: From nginx-forum at forum.nginx.org Fri Aug 13 09:02:04 2021 From: nginx-forum at forum.nginx.org (pcsystemd) Date: Fri, 13 Aug 2021 05:02:04 -0400 Subject: Nginx image cache Message-ID: <84627528d96128cc72acd2d7abdd2f69.NginxMailingListEnglish@forum.nginx.org> Hello, We are using Nginx to do proxypath with images that are on an Amazon AWS server. We have disabled the cache for the following locations: location ~ /utle/af/ { # no cache expires -1; rewrite utle/af/(.*) /$1 break; resolver 127.0.0.1 valid=10s; proxy_pass https://hfufofjfh.cloudfront.net; proxy_set_header Host $proxy_host; } location ~ /ytlo/tt/ { # no cache expires -1; rewrite ytlo/tt/(.*) /$1 break; resolver 127.0.0.1 valid=10s; proxy_pass https://hfufofjfh.cloudfront.net; proxy_set_header Host $proxy_host; } but sometimes we no longer have images displayed on our websites and we do not have to restart nginx to find the images again! However when I consult the Nginx logs I do not see a 404 or 403 error, etc. suddenly we do not manage to know where is the concern? whether it's the AWS cache side or our Nginx side! Any idea to suggest to us? Thanks a lot Posted at Nginx Forum: https://forum.nginx.org/read.php?2,292221,292221#msg-292221 From nginx-forum at forum.nginx.org Sat Aug 14 21:07:08 2021 From: nginx-forum at forum.nginx.org (Nekilc) Date: Sat, 14 Aug 2021 17:07:08 -0400 Subject: http/3 SSL_ERROR_RX_RECORD_TOO_LONG Message-ID: <0620abe8468cc5aa3d64e1ef67b55341.NginxMailingListEnglish@forum.nginx.org> I can access it normally through the ip address, but access through the domain name will result in SSL_ERROR_RX_RECORD_TOO_LONG demo: https://hi-fun.me https://119.3.191.228 config: # HTTPS server server { listen 80; server_name hi-fun.me; return 301 https://$server_name$request_uri; } server { listen 443 http3 quic reuseport; listen 443 ssl http2; server_name localhost; quic_retry on; ssl_protocols TLSv1.2 TLSv1.3; http3_max_table_capacity 50; http3_max_blocked_streams 30; http3_max_concurrent_pushes 30; http3_push 10; http3_push_preload on; ssl_certificate /home/app/hi-fun.me.cert; ssl_certificate_key /home/app/hi-fun.me.key; ssl_ciphers AESGCM:ALL:!aNULL:!ADH:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM:+SSLv3:; ssl_session_cache shared:SSL:1m; ssl_session_timeout 5m; ssl_early_data on; ssl_prefer_server_ciphers on; add_header Alt-Svc 'h3=":443"; ma=86400'; # Debug 0-RTT. add_header X-Early-Data $tls1_3_early_data; add_header QUIC-Status $quic; add_header x-frame-options "deny"; add_header Strict-Transport-Security "max-age=31536000" always; location / { root /home/app; index index.html index.htm; try_files $uri $uri/ @router; } location @router { rewrite ^.*$ /index.html last; # ?????uri ????????uri?vue?? } } map $ssl_early_data $tls1_3_early_data { "~." $ssl_early_data; default ""; } Posted at Nginx Forum: https://forum.nginx.org/read.php?2,292225,292225#msg-292225 From nginx-forum at forum.nginx.org Sun Aug 15 14:49:42 2021 From: nginx-forum at forum.nginx.org (Nekilc) Date: Sun, 15 Aug 2021 10:49:42 -0400 Subject: =?UTF-8?B?5Zue5aSN77yaaHR0cC8zIFNTTCBFUlJPUiBSWCBSRUNPUkQgVE9PIExPTkc=?= In-Reply-To: <0620abe8468cc5aa3d64e1ef67b55341.NginxMailingListEnglish@forum.nginx.org> References: <0620abe8468cc5aa3d64e1ef67b55341.NginxMailingListEnglish@forum.nginx.org> Message-ID: All right, I've found the cause of the problem. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,292225,292230#msg-292230 From yichun at openresty.com Mon Aug 16 05:40:18 2021 From: yichun at openresty.com (Yichun Zhang) Date: Sun, 15 Aug 2021 22:40:18 -0700 Subject: [ANN] OpenResty 1.19.9.1 released Message-ID: Hi folks, I am happy to announce the new formal release, 1.19.9.1, of our OpenResty web platform based on NGINX and LuaJIT. It is the first OpenResty version based on Nginx core 1.19.1. The full announcement, download links, and change logs can be found below: https://openresty.org/en/ann-1019009001.html You can download the software packages here: https://openresty.org/en/download.html We recently added official APK package repository for Alpine 3.14: https://openresty.org/en/linux-packages.html OpenResty is a high performance and dynamic web platform based on our enhanced version of Nginx core, our enhanced version of LuaJIT, and many powerful Nginx modules and Lua libraries. See OpenResty's homepage for details: https://openresty.org/en/ Commercial support is provided by OpenResty Inc. for OpenResty users: https://openresty.com/ Enjoy! Best, Yichun -- Yichun Zhang Founder and CEO of OpenResty Inc. https://openresty.com/ From yichun at openresty.com Mon Aug 16 05:44:52 2021 From: yichun at openresty.com (Yichun Zhang) Date: Sun, 15 Aug 2021 22:44:52 -0700 Subject: [ANN] OpenResty 1.19.9.1 released In-Reply-To: References: Message-ID: Oops, sorry, there was a typo in my previous email. It should be "It is the first OpenResty version based on Nginx core 1.19.9." Best, Yichun From ali12111381 at outlook.com Wed Aug 18 11:45:07 2021 From: ali12111381 at outlook.com (Ali Ghanbari) Date: Wed, 18 Aug 2021 11:45:07 +0000 Subject: I can't use mw3 multiplayer!! Message-ID: It says this: [cid:image003.png at 01D7944C.345CC7C0] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 564C933B9B394109BA7755C842355E69.png Type: image/png Size: 131839 bytes Desc: 564C933B9B394109BA7755C842355E69.png URL: From vbart at nginx.com Fri Aug 20 04:12:13 2021 From: vbart at nginx.com (Valentin V. Bartenev) Date: Fri, 20 Aug 2021 07:12:13 +0300 Subject: Unit 1.25.0 release Message-ID: <2207026.ElGaqSPkdT@vbart-laptop> Hi, I'm glad to announce a new release of NGINX Unit. This one is much awaited not only because the last one occurred quite some time ago, but also because it contains some sought-after features that were requested quite often. :: Obtaining The Originating Client IP Address :: When Unit operates behind a reverse proxy, it receives all incoming connections from a proxy machine address. As a result, the originating IP address of a client cannot be determined from the IP protocol. To overcome this, a special HTTP request header field can be used to carry the client IP address information over one to several proxies. Such header fields are usually called "X-Forwarded-For", but variations exist as well ("X-Real-IP", "X-Remote-Addr", etc..). Before, Unit could not use information from such header fields otherwise than just pass them on "as is." With this release, functionality similar to the real-ip nginx module became available. Now, in any listener object, you can specify a "client_ip" option, configuring trusted proxy addresses and the header field name, to obtain the client IP address: { "listeners": { "*:80": { "client_ip": { "header": "X-Forwarded-For", "recursive": true, "source": [ "10.0.0.0/8", "150.172.238.0/24" ] } } } } Unit will use the address obtained from this header to the same effect as if a direct connection was made from the client. For instance, it will be reflected in any logs, used for source address matching in routing, and provided to the application via a relevant request environment (e. g. $_SERVER['REMOTE_ADDR'] in PHP). See more details in the documentation: - https://unit.nginx.org/configuration/#originating-ip-identification :: Control API to Restart Application Processes :: Unit dynamic configuration is pretty smart and granular. If it detects no changes to an application during reconfiguration, it won't touch the application's processes. However, sometimes our users need to restart a specific application, and the only good way to do that was to intentionally introduce a change to the application's configuration. Usually, a dummy "environment" option was used for this: curl -X PUT -d '"$RANDOM"' --unix-socket /var/run/control.unit.sock \ /config/applications//environment/gen While it worked well, the solution can't be called elegant; it was more like a workaround. But now, Unit has a special section in the control API that allows restarting any configured application with a basic GET request: curl --unix-socket /var/run/control.unit.sock \ /control/applications//restart See here for the details of app process management in Unit: - https://unit.nginx.org/configuration/#process-management :: TLS Sessions Cache and Tickets :: A full TLS handshake can be quite expensive; to save server resources and reduce latency in subsequent client connections, two ways are commonly used: TLS sessions cache and TLS session tickets. The main difference between the two is who stores the session information: the server (cache) or the client (tickets). Now, Unit allows you to configure either or both: { "tls": { "certificate": "bundle", "session": { "cache_size": 10000, "timeout": 600, "tickets": true } } } For tickets, it doesn't only allow enabling or disabling them; you can specify shared ticket keys between multiple servers and rotate them. See more sophisticated configurations in the docs: - https://unit.nginx.org/configuration/#ssl-tls-configuration We will proceed to improve the client-side protocol support to be on par with nginx in this regard or even go further. To be specific, HTTP/2 and HTTP/3 are definitely on our shortlist. :: Ruby Process and Thread Start/Stop Hooks :: Earlier this year, one of our users opened a feature request on Unit's GitHub: https://github.com/nginx/unit/issues/535; we were asked to support hooks to be triggered on process or thread start/stop, as does another popular Ruby web server, Puma. These are usually used to instantiate a database connection or to perform some other initialization or cleanup work. A few months later, we've fulfilled the request. Here we go: - https://unit.nginx.org/configuration/#ruby That's why I always ask you not to hesitate and instead open a feature request for any crazy idea you may have on our GitHub issue tracker: - https://github.com/nginx/unit/issues We'd like to hear from you, we'd like to know your cases, your issues, anything you're struggling with or are missing and would want to see in Unit. Sure, not all requests are handled fast. There's plenty of them pending for years already. It's different case by case; sometimes, we're just busy with other important tasks, sometimes the feature depends on other missing parts, which also depend on other ones, and so on. Sometimes, it just takes a while to find a good solution, to design a good architecture, or to find a proper method of configuring something. Anyway, all your requests are collected and carefully examined; perhaps, it's your idea that will be implemented next. Please go and open a ticket if in doubt. The full changelog for the release: Changes with Unit 1.25.0 19 Aug 2021 *) Feature: client IP address replacement from a specified HTTP header field. *) Feature: TLS sessions cache. *) Feature: TLS session tickets. *) Feature: application restart control. *) Feature: process and thread lifecycle hooks in Ruby. *) Bugfix: the router process could crash on TLS connection open when multiple listeners with TLS certificates were configured; the bug had appeared in 1.23.0. *) Bugfix: TLS connections were rejected for configurations with multiple certificate bundles in a listener if the client did not use SNI. *) Bugfix: the router process could crash with frequent multithreaded application reconfiguration. *) Bugfix: compatibility issues with some Python ASGI apps, notably based on the Starlette framework. *) Bugfix: a descriptor and memory leak occurred in the router process when an app process stopped or crashed. *) Bugfix: the controller or router process could crash if the configuration contained a full-form IPv6 in a listener address. *) Bugfix: the router process crashed when a request was passed to an empty "routes" or "upstreams" using a variable "pass" option. *) Bugfix: the router process crashed while matching a request to an empty array of source or destination address patterns. In the meantime, there are several other features currently at different stages of development and implementation: - Variable support in the static file serving options - Custom variables from regexp captures in the "match" object - Simple request rewrites using variables - More variables to access request and connection information - A statistics API - Unit CLI utility tool - App prototype processes to reduce memory usage, share the PHP opcache, and improve the handling of apps isolation - njs integration (https://nginx.org/en/docs/njs/index.html) - .NET Core language module prototype Some of them bound to appear in the next release. Stay tuned! wbr, Valentin V. Bartenev From lywjk at outlook.com Tue Aug 24 08:37:47 2021 From: lywjk at outlook.com (=?gb2312?B?zfUgvrK/rQ==?=) Date: Tue, 24 Aug 2021 08:37:47 +0000 Subject: Segmentation fault sometimes In-Reply-To: References: Message-ID: Hi, The ?nginx -V? info: ```` nginx version: nginx/1.20.0 built by gcc 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) built with OpenSSL 1.1.1k 25 Mar 2021 TLS SNI support enabled configure arguments: --prefix=/usr/local/nginx --sbin-path=/usr/local/nginx/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --pid-path=/usr/local/nginx/nginx.pid --error-log-path=/usr/local/nginx/logs/error.log --http-client-body-temp-path=/usr/local/nginx/client_temp --http-proxy-temp-path=/usr/local/nginx/proxy_temp --http-fastcgi-temp-path=/usr/local/nginx/fastcgi_temp --http-uwsgi-temp-path=/usr/local/nginx/uwsgi_temp --http-scgi-temp-path=/usr/local/nginx/scgi_temp --user=nginx --group=nginx --add-module=../fastdfs-nginx-module-1.22/src --add-module=../nginx-client-module --add-module=../nginx-multiport-module --add-module=../nginx-toolkit-module --add-module=../nginx-rtmp-module --add-module=../nginx-ts-module --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-openssl=../openssl-1.1.1k --with-pcre=../pcre-8.44 --with-zlib=../zlib-1.2.11 --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-debug ```` The nginx compile with ?https://github.com/pingostack/pingos? project and ?https://github.com/happyfish100/fastdfs-nginx-module? project. Sometimes it crashed serveral times continuously. But then it works for a long time normally. It shows ```` #0 0x0000006e6f697461 in ?? () #1 0x0000000000464ed6 in ngx_log_error_core (level=level at entry=8, log=log at entry=0x3d27608, err=err at entry=0, fmt=fmt at entry=0x7a2816 "posix_memalign: %p:%uz @%uz") at src/core/ngx_log.c:168 #2 0x000000000048ab24 in ngx_memalign (alignment=alignment at entry=16, size=size at entry=4096, log=0x3d27608) at src/os/unix/ngx_alloc.c:65 #3 0x0000000000465aa2 in ngx_palloc_block (pool=0x3bf9b50, size=72) at src/core/ngx_palloc.c:186 #4 0x000000000046608e in ngx_pcalloc (pool=, size=size at entry=72) at src/core/ngx_palloc.c:302 #5 0x0000000000582c73 in ngx_rtmp_shared_alloc_frame (size=4096, cl=cl at entry=0x3d3c960, mandatory=mandatory at entry=0) at ../nginx-rtmp-module/ngx_rtmp_shared_module.c:225 #6 0x00000000005689e9 in ngx_rtmp_live_av (s=0x3fb8bf0, h=0x3bb1290, in=0x3d3c960) at ../nginx-rtmp-module/ngx_rtmp_live_module.c:773 #7 0x000000000055d1d8 in ngx_rtmp_receive_message (s=s at entry=0x3fb8bf0, h=h at entry=0x3bb1290, in=in at entry=0x3d3c960) at ../nginx-rtmp-module/ngx_rtmp_handler.c:874 #8 0x000000000055d991 in ngx_rtmp_recv (rev=) at ../nginx-rtmp-module/ngx_rtmp_handler.c:486 #9 0x0000000000491f11 in ngx_epoll_process_events (cycle=0x43bde90, timer=, flags=) at src/event/modules/ngx_epoll_module.c:901 #10 0x00000000004861d6 in ngx_process_events_and_timers (cycle=cycle at entry=0x43bde90) at src/event/ngx_event.c:247 #11 0x000000000048f9e5 in ngx_worker_process_cycle (cycle=cycle at entry=0x43bde90, data=data at entry=0x0) at src/os/unix/ngx_process_cycle.c:719 #12 0x000000000048df5f in ngx_spawn_process (cycle=cycle at entry=0x43bde90, proc=proc at entry=0x48f990 , data=data at entry=0x0, name=name at entry=0x7a3486 "worker process", respawn=respawn at entry=-4) at src/os/unix/ngx_process.c:199 ```` Or ```` (gdb) bt #0 0x0000000000464f13 in ngx_write_fd (n=100, buf=0x7ffd8d7ef2f0, fd=) at src/os/unix/ngx_files.h:147 #1 ngx_log_error_core (level=level at entry=8, log=log at entry=0x3b94c58, err=err at entry=0, fmt=fmt at entry=0x7a2816 "posix_memalign: %p:%uz @%uz") at src/core/ngx_log.c:183 #2 0x000000000048ab24 in ngx_memalign (alignment=alignment at entry=16, size=size at entry=4096, log=0x3b94c58) at src/os/unix/ngx_alloc.c:65 #3 0x0000000000465aa2 in ngx_palloc_block (pool=0x3c3c0b0, size=96) at src/core/ngx_palloc.c:186 #4 0x000000000046608e in ngx_pcalloc (pool=, size=size at entry=96) at src/core/ngx_palloc.c:302 ```` In gdb -------------- next part -------------- An HTML attachment was scrubbed... URL: From maxim at nginx.com Tue Aug 24 11:43:25 2021 From: maxim at nginx.com (Maxim Konovalov) Date: Tue, 24 Aug 2021 14:43:25 +0300 Subject: nginx brand survey, we are looking for your feedback Message-ID: Hi, On behalf of our marketing team I'd like to invite you, NGINX community members, users and developers, to participate in the NGINX Annual Community Survey. There are 20 questions there. It will take around 10 minutes of your valuable time to fill it: The NGINX Annual Community Survey is open, and we would like to hear from you. Over the past seven years, you've helped us improve our solutions and evolve our product roadmap. Please continue to share your experiences and ideas with us ? we value your feedback. The anonymous survey is here: https://survey.f5.com/v1/sites/nginx/SurveyResponse?at=1I0025DXE6KKG8JCV1KHEZ8U6WNP8QI5ONSB Upon completion, you will receive an exclusive 90% discount code for the NGINX swag store. Thank you for helping shape the future of NGINX. Regards, NGINX team -- Maxim Konovalov From nginx-forum at forum.nginx.org Thu Aug 26 12:25:25 2021 From: nginx-forum at forum.nginx.org (WusikiJeronii) Date: Thu, 26 Aug 2021 08:25:25 -0400 Subject: Can't upload a few files via nginx Message-ID: <93124237230fe4a009631b0d8cb81829.NginxMailingListEnglish@forum.nginx.org> If I upload 1,2,10 files uploading works, but I tried to upload 37 files and I get the error in the error.log: > 2021/08/23 16:37:04 [error] 19943#19943: *1 sendfile() failed (32: Broken pipe) while sending request to upstream, client: 127.0.0.1, server: localhost, request: "POST /upload/project HTTP/1.1", upstream: "http://127.0.0.1:8070/upload/project", host: "127.0.0.1", referrer: "http://127.0.0.1/projects" But if I try to upload these 37 files without Nginx uploading will be successful. I tried: * client_max_body_size 10240m; * keepalive_timeout 600; Doesn't work. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,292278,292278#msg-292278 From grzegorz.czesnik at hotmail.com Fri Aug 27 17:13:16 2021 From: grzegorz.czesnik at hotmail.com (Grzegorz Cze?nik) Date: Fri, 27 Aug 2021 17:13:16 +0000 Subject: Nginx PID Message-ID: Hi I installed a clean ubuntu 20.04.2 lts server. Next, I installed nginx (mainline) from the nginx repositories: https://nginx.org/en/linux_packages.html?&_ga=2.232266207.1750066394.1608933860-1476356162.1603899216#Ubuntu After typing sudo systemctl nginx status receives a message regarding the PID file: Aug 27 19:09:30 reverse-proxy systemd[1]: nginx.service: Can't open PID file /run/nginx.pid (yet?) after start: Operation not permitted Is it normal? Grzegorz -------------- next part -------------- An HTML attachment was scrubbed... URL: From teward at thomas-ward.net Fri Aug 27 17:49:40 2021 From: teward at thomas-ward.net (Thomas Ward) Date: Fri, 27 Aug 2021 13:49:40 -0400 Subject: Nginx PID In-Reply-To: References: Message-ID: <1b98c683-7852-564a-f171-39f044e1fc12@thomas-ward.net> A stock install of NGINX using that link and the repositories for mainline on 20.04 clean doesn't return this issue.? Did you attempt to start the service first before that, or did you install nginx from the Ubuntu repositories and then tried to install the nginx.org packages overtop those repository packages? Thomas On 8/27/21 1:13 PM, Grzegorz Cze?nik wrote: > > Hi > > I installed a clean ubuntu 20.04.2 lts server. > > Next, I installed nginx (mainline) from the nginx repositories: > https://nginx.org/en/linux_packages.html?&_ga=2.232266207.1750066394.1608933860-1476356162.1603899216#Ubuntu > > > > After typing sudo systemctl nginx status receives a message regarding > the PID file: > > Aug 27 19:09:30 reverse-proxy systemd[1]: nginx.service: Can't open > PID file /run/nginx.pid (yet?) after start: Operation not permitted > > Is it normal? > > Grzegorz > > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx -------------- next part -------------- An HTML attachment was scrubbed... URL: From teward at thomas-ward.net Fri Aug 27 17:51:04 2021 From: teward at thomas-ward.net (Thomas Ward) Date: Fri, 27 Aug 2021 13:51:04 -0400 Subject: Nginx PID In-Reply-To: References: Message-ID: A stock install of NGINX using that link and the repositories for mainline on 20.04 clean doesn't return this issue.? Did you attempt to start the service first before that, or did you install nginx from the Ubuntu repositories and then tried to install the nginx.org packages overtop those repository packages? Thomas On 8/27/21 1:13 PM, Grzegorz Cze?nik wrote: > > Hi > > I installed a clean ubuntu 20.04.2 lts server. > > Next, I installed nginx (mainline) from the nginx repositories: > https://nginx.org/en/linux_packages.html?&_ga=2.232266207.1750066394.1608933860-1476356162.1603899216#Ubuntu > > > > After typing sudo systemctl nginx status receives a message regarding > the PID file: > > Aug 27 19:09:30 reverse-proxy systemd[1]: nginx.service: Can't open > PID file /run/nginx.pid (yet?) after start: Operation not permitted > > Is it normal? > > Grzegorz > > > _______________________________________________ > nginx mailing list > nginx at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- _______________________________________________ nginx mailing list nginx at nginx.org http://mailman.nginx.org/mailman/listinfo/nginx From grzegorz.czesnik at hotmail.com Fri Aug 27 17:54:09 2021 From: grzegorz.czesnik at hotmail.com (Grzegorz Cze?nik) Date: Fri, 27 Aug 2021 17:54:09 +0000 Subject: Nginx PID In-Reply-To: <1b98c683-7852-564a-f171-39f044e1fc12@thomas-ward.net> References: <1b98c683-7852-564a-f171-39f044e1fc12@thomas-ward.net> Message-ID: No, it's a clean, clean install, I haven't installed nginx not in another repository. I just took steps from the front yard - nothing else. From: nginx [mailto:nginx-bounces at nginx.org] On Behalf Of Thomas Ward Sent: Friday, August 27, 2021 7:50 PM To: nginx at nginx.org Subject: Re: Nginx PID A stock install of NGINX using that link and the repositories for mainline on 20.04 clean doesn't return this issue. Did you attempt to start the service first before that, or did you install nginx from the Ubuntu repositories and then tried to install the nginx.org packages overtop those repository packages? Thomas On 8/27/21 1:13 PM, Grzegorz Cze?nik wrote: Hi I installed a clean ubuntu 20.04.2 lts server. Next, I installed nginx (mainline) from the nginx repositories: https://nginx.org/en/linux_packages.html?&_ga=2.232266207.1750066394.1608933860-1476356162.1603899216#Ubuntu After typing sudo systemctl nginx status receives a message regarding the PID file: Aug 27 19:09:30 reverse-proxy systemd[1]: nginx.service: Can't open PID file /run/nginx.pid (yet?) after start: Operation not permitted Is it normal? Grzegorz _______________________________________________ nginx mailing list nginx at nginx.org http://mailman.nginx.org/mailman/listinfo/nginx -------------- next part -------------- An HTML attachment was scrubbed... URL: From grzegorz.czesnik at hotmail.com Fri Aug 27 18:40:48 2021 From: grzegorz.czesnik at hotmail.com (Grzegorz Cze?nik) Date: Fri, 27 Aug 2021 18:40:48 +0000 Subject: Nginx PID In-Reply-To: <1b98c683-7852-564a-f171-39f044e1fc12@thomas-ward.net> References: <1b98c683-7852-564a-f171-39f044e1fc12@thomas-ward.net> Message-ID: And.... After giving the command: gpg --dry-run --quiet --import --import-options import-show /usr/share/keyrings/nginx-archive-keyring.gpg I have received a message: pub rsa2048 2011-08-19 [SC] [expires: 2024-06-14] 573BFD6B3D8FBC641079A6ABABF5BD827BD9BF62 uid nginx signing key gpg: keyblock resource '/home/nginx/.gnupg/pubring.kbx': No such file or directory From: nginx [mailto:nginx-bounces at nginx.org] On Behalf Of Thomas Ward Sent: Friday, August 27, 2021 7:50 PM To: nginx at nginx.org Subject: Re: Nginx PID A stock install of NGINX using that link and the repositories for mainline on 20.04 clean doesn't return this issue. Did you attempt to start the service first before that, or did you install nginx from the Ubuntu repositories and then tried to install the nginx.org packages overtop those repository packages? Thomas On 8/27/21 1:13 PM, Grzegorz Cze?nik wrote: Hi I installed a clean ubuntu 20.04.2 lts server. Next, I installed nginx (mainline) from the nginx repositories: https://nginx.org/en/linux_packages.html?&_ga=2.232266207.1750066394.1608933860-1476356162.1603899216#Ubuntu After typing sudo systemctl nginx status receives a message regarding the PID file: Aug 27 19:09:30 reverse-proxy systemd[1]: nginx.service: Can't open PID file /run/nginx.pid (yet?) after start: Operation not permitted Is it normal? Grzegorz _______________________________________________ nginx mailing list nginx at nginx.org http://mailman.nginx.org/mailman/listinfo/nginx -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvambani1991 at gmail.com Fri Aug 27 20:12:41 2021 From: rvambani1991 at gmail.com (Rajen Patel) Date: Fri, 27 Aug 2021 13:12:41 -0700 Subject: Need help regarding DSR with NGINX - not working as expected Message-ID: here are my setup details for the k8s cluster: Nginx running as UDP proxy nginx.conf: | user root; worker_processes 1; events { worker_connections 10240; } http { server { listen 80; server_name localhost; location / { root /usr/share/nginx/html; #Change this line index index.html index.htm; } } } stream { server { listen *4729 udp;* proxy_pass j_close_stream_backend; proxy_responses 0; error_log stderr; proxy_bind $remote_addr:$remote_port transparent; } upstream juniper_close_stream_backend { server* 10.244.72.169:4729 *; } } --------------------------------------------------- nginx-> client service(*10.244.72.169:4729 *) *Nginx running as Kubernetes Daemonset as follows:* apiVersion: apps/v1 kind: DaemonSet metadata: name: nginx-daemonset template: metadata: creationTimestamp: null labels: app: nginx spec: containers: - image: nginx:1.14.2 imagePullPolicy: IfNotPresent name: nginx ports: - containerPort: 80 protocol: TCP resources: {} here nginx works fine and is able to preserve source IP and also forward spoofed IP. however client service is not able to receive any traffic. I believe there is some issue with iptables but don't know how to trace this issue. Your help is much appreciated. -------------- next part -------------- An HTML attachment was scrubbed... URL: From admin at onlymembers.co.nz Sat Aug 28 09:35:59 2021 From: admin at onlymembers.co.nz (OnlyMembers - Administration) Date: Sat, 28 Aug 2021 21:35:59 +1200 Subject: NGINX RTMP - Control via PHP? Message-ID: <37b8f88d-330e-60a4-ae83-4197785d5fd1@onlymembers.co.nz> Hi All Seeking guidance on best approach using FFMPEG/NGINX Plus/Opensource Youtube style live streaming scenario - multi broadcasters, multi viewers. Need to be able to scale up relatively easily. We would love to be able to push nginx-RTMP config additions via API if possible? from our PHP based backend Any tips on integrating FFMPEG, NGINX together for simple RTMP setup workflow would be epic. Not sure if microservices fits in our use case? Any other tips/info would be much appreciated :) Cheers -- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus From ali12111381 at outlook.com Sun Aug 29 07:59:09 2021 From: ali12111381 at outlook.com (Ali Ghanbari) Date: Sun, 29 Aug 2021 07:59:09 +0000 Subject: Segmentation fault sometimes In-Reply-To: References: Message-ID: is those was pictures?? Cuz I just see a lot of things that doesn?t have any meaning:/ Sent from Mail for Windows From: ? ?? Sent: Tuesday, August 24, 2021 1:08 PM To: nginx at nginx.org Subject: Segmentation fault sometimes Hi, The ?nginx -V? info: ```` nginx version: nginx/1.20.0 built by gcc 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) built with OpenSSL 1.1.1k 25 Mar 2021 TLS SNI support enabled configure arguments: --prefix=/usr/local/nginx --sbin-path=/usr/local/nginx/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --pid-path=/usr/local/nginx/nginx.pid --error-log-path=/usr/local/nginx/logs/error.log --http-client-body-temp-path=/usr/local/nginx/client_temp --http-proxy-temp-path=/usr/local/nginx/proxy_temp --http-fastcgi-temp-path=/usr/local/nginx/fastcgi_temp --http-uwsgi-temp-path=/usr/local/nginx/uwsgi_temp --http-scgi-temp-path=/usr/local/nginx/scgi_temp --user=nginx --group=nginx --add-module=../fastdfs-nginx-module-1.22/src --add-module=../nginx-client-module --add-module=../nginx-multiport-module --add-module=../nginx-toolkit-module --add-module=../nginx-rtmp-module --add-module=../nginx-ts-module --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-openssl=../openssl-1.1.1k --with-pcre=../pcre-8.44 --with-zlib=../zlib-1.2.11 --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-debug ```` The nginx compile with ?https://github.com/pingostack/pingos? project and ?https://github.com/happyfish100/fastdfs-nginx-module? project. Sometimes it crashed serveral times continuously. But then it works for a long time normally. It shows ```` #0 0x0000006e6f697461 in ?? () #1 0x0000000000464ed6 in ngx_log_error_core (level=level at entry=8, log=log at entry=0x3d27608, err=err at entry=0, fmt=fmt at entry=0x7a2816 "posix_memalign: %p:%uz @%uz") at src/core/ngx_log.c:168 #2 0x000000000048ab24 in ngx_memalign (alignment=alignment at entry=16, size=size at entry=4096, log=0x3d27608) at src/os/unix/ngx_alloc.c:65 #3 0x0000000000465aa2 in ngx_palloc_block (pool=0x3bf9b50, size=72) at src/core/ngx_palloc.c:186 #4 0x000000000046608e in ngx_pcalloc (pool=, size=size at entry=72) at src/core/ngx_palloc.c:302 #5 0x0000000000582c73 in ngx_rtmp_shared_alloc_frame (size=4096, cl=cl at entry=0x3d3c960, mandatory=mandatory at entry=0) at ../nginx-rtmp-module/ngx_rtmp_shared_module.c:225 #6 0x00000000005689e9 in ngx_rtmp_live_av (s=0x3fb8bf0, h=0x3bb1290, in=0x3d3c960) at ../nginx-rtmp-module/ngx_rtmp_live_module.c:773 #7 0x000000000055d1d8 in ngx_rtmp_receive_message (s=s at entry=0x3fb8bf0, h=h at entry=0x3bb1290, in=in at entry=0x3d3c960) at ../nginx-rtmp-module/ngx_rtmp_handler.c:874 #8 0x000000000055d991 in ngx_rtmp_recv (rev=) at ../nginx-rtmp-module/ngx_rtmp_handler.c:486 #9 0x0000000000491f11 in ngx_epoll_process_events (cycle=0x43bde90, timer=, flags=) at src/event/modules/ngx_epoll_module.c:901 #10 0x00000000004861d6 in ngx_process_events_and_timers (cycle=cycle at entry=0x43bde90) at src/event/ngx_event.c:247 #11 0x000000000048f9e5 in ngx_worker_process_cycle (cycle=cycle at entry=0x43bde90, data=data at entry=0x0) at src/os/unix/ngx_process_cycle.c:719 #12 0x000000000048df5f in ngx_spawn_process (cycle=cycle at entry=0x43bde90, proc=proc at entry=0x48f990 , data=data at entry=0x0, name=name at entry=0x7a3486 "worker process", respawn=respawn at entry=-4) at src/os/unix/ngx_process.c:199 ```` Or ```` (gdb) bt #0 0x0000000000464f13 in ngx_write_fd (n=100, buf=0x7ffd8d7ef2f0, fd=) at src/os/unix/ngx_files.h:147 #1 ngx_log_error_core (level=level at entry=8, log=log at entry=0x3b94c58, err=err at entry=0, fmt=fmt at entry=0x7a2816 "posix_memalign: %p:%uz @%uz") at src/core/ngx_log.c:183 #2 0x000000000048ab24 in ngx_memalign (alignment=alignment at entry=16, size=size at entry=4096, log=0x3b94c58) at src/os/unix/ngx_alloc.c:65 #3 0x0000000000465aa2 in ngx_palloc_block (pool=0x3c3c0b0, size=96) at src/core/ngx_palloc.c:186 #4 0x000000000046608e in ngx_pcalloc (pool=, size=size at entry=96) at src/core/ngx_palloc.c:302 ```` In gdb -------------- next part -------------- An HTML attachment was scrubbed... URL: From les at dunawaygroupllc.com Sun Aug 29 22:04:29 2021 From: les at dunawaygroupllc.com (Les Dunaway) Date: Sun, 29 Aug 2021 18:04:29 -0400 Subject: New install; no error messages; but website.conf in /etc/nginx/conf.d is being ignored Message-ID: When I send request, I get the default page Les Dunaway 770-490-7546 voice/text -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: nginx.conf Type: application/octet-stream Size: 2469 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: deadendjob.com.conf Type: application/octet-stream Size: 368 bytes Desc: not available URL: From osa at freebsd.org.ru Mon Aug 30 22:30:00 2021 From: osa at freebsd.org.ru (Sergey A. Osokin) Date: Tue, 31 Aug 2021 01:30:00 +0300 Subject: Segmentation fault sometimes In-Reply-To: References: Message-ID: Hi there, is it reproducible without the third-party module? Could you try to reproduce the issue with the recent version from the stable branch, i.e. 1.20.1. Thanks. -- Sergey Osokin On Tue, Aug 24, 2021 at 08:37:47AM +0000, ? ?? wrote: > Hi, > > The ?nginx -V? info: > ```` > nginx version: nginx/1.20.0 > built by gcc 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) > built with OpenSSL 1.1.1k 25 Mar 2021 > TLS SNI support enabled > configure arguments: --prefix=/usr/local/nginx --sbin-path=/usr/local/nginx/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --pid-path=/usr/local/nginx/nginx.pid --error-log-path=/usr/local/nginx/logs/error.log --http-client-body-temp-path=/usr/local/nginx/client_temp --http-proxy-temp-path=/usr/local/nginx/proxy_temp --http-fastcgi-temp-path=/usr/local/nginx/fastcgi_temp --http-uwsgi-temp-path=/usr/local/nginx/uwsgi_temp --http-scgi-temp-path=/usr/local/nginx/scgi_temp --user=nginx --group=nginx --add-module=../fastdfs-nginx-module-1.22/src --add-module=../nginx-client-module --add-module=../nginx-multiport-module --add-module=../nginx-toolkit-module --add-module=../nginx-rtmp-module --add-module=../nginx-ts-module --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-openssl=../openssl-1.1.1k --with-pcre=../pcre-8.44 --with-zlib=../zlib-1.2.11 --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-debug > ```` > > The nginx compile with ?https://github.com/pingostack/pingos? project and ?https://github.com/happyfish100/fastdfs-nginx-module? project. > > Sometimes it crashed serveral times continuously. But then it works for a long time normally. > > It shows > ```` > #0 0x0000006e6f697461 in ?? () > #1 0x0000000000464ed6 in ngx_log_error_core (level=level at entry=8, log=log at entry=0x3d27608, err=err at entry=0, fmt=fmt at entry=0x7a2816 "posix_memalign: %p:%uz @%uz") > at src/core/ngx_log.c:168 > #2 0x000000000048ab24 in ngx_memalign (alignment=alignment at entry=16, size=size at entry=4096, log=0x3d27608) at src/os/unix/ngx_alloc.c:65 > #3 0x0000000000465aa2 in ngx_palloc_block (pool=0x3bf9b50, size=72) at src/core/ngx_palloc.c:186 > #4 0x000000000046608e in ngx_pcalloc (pool=, size=size at entry=72) at src/core/ngx_palloc.c:302 > #5 0x0000000000582c73 in ngx_rtmp_shared_alloc_frame (size=4096, cl=cl at entry=0x3d3c960, mandatory=mandatory at entry=0) > at ../nginx-rtmp-module/ngx_rtmp_shared_module.c:225 > #6 0x00000000005689e9 in ngx_rtmp_live_av (s=0x3fb8bf0, h=0x3bb1290, in=0x3d3c960) at ../nginx-rtmp-module/ngx_rtmp_live_module.c:773 > #7 0x000000000055d1d8 in ngx_rtmp_receive_message (s=s at entry=0x3fb8bf0, h=h at entry=0x3bb1290, in=in at entry=0x3d3c960) > at ../nginx-rtmp-module/ngx_rtmp_handler.c:874 > #8 0x000000000055d991 in ngx_rtmp_recv (rev=) at ../nginx-rtmp-module/ngx_rtmp_handler.c:486 > #9 0x0000000000491f11 in ngx_epoll_process_events (cycle=0x43bde90, timer=, flags=) at src/event/modules/ngx_epoll_module.c:901 > #10 0x00000000004861d6 in ngx_process_events_and_timers (cycle=cycle at entry=0x43bde90) at src/event/ngx_event.c:247 > #11 0x000000000048f9e5 in ngx_worker_process_cycle (cycle=cycle at entry=0x43bde90, data=data at entry=0x0) at src/os/unix/ngx_process_cycle.c:719 > #12 0x000000000048df5f in ngx_spawn_process (cycle=cycle at entry=0x43bde90, proc=proc at entry=0x48f990 , data=data at entry=0x0, > name=name at entry=0x7a3486 "worker process", respawn=respawn at entry=-4) at src/os/unix/ngx_process.c:199 > ```` > Or > > ```` > (gdb) bt > #0 0x0000000000464f13 in ngx_write_fd (n=100, buf=0x7ffd8d7ef2f0, fd=) > at src/os/unix/ngx_files.h:147 > #1 ngx_log_error_core (level=level at entry=8, log=log at entry=0x3b94c58, err=err at entry=0, fmt=fmt at entry=0x7a2816 "posix_memalign: %p:%uz @%uz") > at src/core/ngx_log.c:183 > #2 0x000000000048ab24 in ngx_memalign (alignment=alignment at entry=16, size=size at entry=4096, log=0x3b94c58) at src/os/unix/ngx_alloc.c:65 > #3 0x0000000000465aa2 in ngx_palloc_block (pool=0x3c3c0b0, size=96) at src/core/ngx_palloc.c:186 > #4 0x000000000046608e in ngx_pcalloc (pool=, size=size at entry=96) at src/core/ngx_palloc.c:302 > ```` > > In gdb From francis at daoine.org Tue Aug 31 06:29:37 2021 From: francis at daoine.org (Francis Daly) Date: Tue, 31 Aug 2021 07:29:37 +0100 Subject: New install; no error messages; but website.conf in /etc/nginx/conf.d is being ignored In-Reply-To: References: Message-ID: <20210831062937.GA18734@daoine.org> On Sun, Aug 29, 2021 at 06:04:29PM -0400, Les Dunaway wrote: Hi there, > When I send request, I get the default page When your nginx is started, one specific nginx binary is used, and it might have a "-c" command-line argument to name the config file to use. If it does not have a "-c", then it uses the binary built-in default value. If you run that nginx binary (perhaps /usr/sbin/nginx) with the same "-c" argument, and an extra "-T", and pipe the output through "grep configuration", that should show you one line for each file that is actually being read. If that does not show your website.conf, then for some reason it is not being read at all. Alternatively: do your nginx logs show that the request that you send is getting to this nginx? If your browser resolves the domain name to a different address, then it will not be talking to this nginx; if your browser has cached a previous response to this request, then it may not be talking to this nginx. They might hint at the next place to look to find the solution. Good luck with it, f -- Francis Daly francis at daoine.org From ruv.donita at gmail.com Tue Aug 31 06:49:25 2021 From: ruv.donita at gmail.com (Dorin RuV) Date: Tue, 31 Aug 2021 08:49:25 +0200 Subject: $request_time in ngx_http_log_module Message-ID: Hi everybody, I'm currently working on adjusting the logging format to our nginx proxies. In order to comply to some logging endpoint standards I'm trying to find the right place where I could convert the $request_time value from "seconds with a millisecond resolution" to nanoseconds. Can it be done in nginx itself (without lua)? The following works but it's dirty: map $request_time $request_time_no_decimal { ~(.*)\.(.*) $1$2; } Best, Dorin -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Tue Aug 31 15:40:01 2021 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 31 Aug 2021 18:40:01 +0300 Subject: nginx-1.21.2 Message-ID: Changes with nginx 1.21.2 31 Aug 2021 *) Change: now nginx rejects HTTP/1.0 requests with the "Transfer-Encoding" header line. *) Change: export ciphers are no longer supported. *) Feature: OpenSSL 3.0 compatibility. *) Feature: the "Auth-SSL-Protocol" and "Auth-SSL-Cipher" header lines are now passed to the mail proxy authentication server. Thanks to Rob Mueller. *) Feature: request body filters API now permits buffering of the data being processed. *) Bugfix: backend SSL connections in the stream module might hang after an SSL handshake. *) Bugfix: the security level, which is available in OpenSSL 1.1.0 or newer, did not affect loading of the server certificates when set with "@SECLEVEL=N" in the "ssl_ciphers" directive. *) Bugfix: SSL connections with gRPC backends might hang if select, poll, or /dev/poll methods were used. *) Bugfix: when using HTTP/2 client request body was always written to disk if the "Content-Length" header line was not present in the request. -- Maxim Dounin http://nginx.org/ From xeioex at nginx.com Tue Aug 31 15:55:30 2021 From: xeioex at nginx.com (Dmitry Volyntsev) Date: Tue, 31 Aug 2021 18:55:30 +0300 Subject: njs-0.6.2 Message-ID: Hello, I'm glad to announce a new release of NGINX JavaScript module (njs). This release proceeds to extend the coverage of ECMAScript specifications. Notable new features: - Advanced Promise constructor methods : Promise.all(urls.map(u => ngx.fetch(u))) : .then(responses => r.return(200, JSON.stringify(responses))) Learn more about njs: - Overview and introduction: https://nginx.org/en/docs/njs/ - NGINX JavaScript in Your Web Server Configuration: https://youtu.be/Jc_L6UffFOs - Extending NGINX with Custom Code: https://youtu.be/0CVhq4AUU7M - Using node modules with njs: https://nginx.org/en/docs/njs/node_modules.html - Writing njs code using TypeScript definition files: https://nginx.org/en/docs/njs/typescript.html Feel free to try it and give us feedback on: - Github: https://github.com/nginx/njs/issues - Mailing list: https://mailman.nginx.org/mailman/listinfo/nginx-devel Changes with njs 0.6.2 31 Aug 2021 nginx modules: *) Bugfix: fixed CPU hog when js_filter is registered in both directions. Core: *) Feature: introduced AggregateError implementation. *) Feature: added remaining Promise constructor methods. The following methods were added: Promise.all(), Promise.allSettled(), Promise.any(), Promise.race(). *) Improvement: removed recursion from code generator. *) Bugfix: fixed rest parameter parsing without binding identifier. *) Bugfix: fixed resolve/reject callback for Promise.prototype.finally(). *) Bugfix: fixed %TypedArray%.prototype.join() with detached buffer. *) Bugfix: fixed memory leak in interactive shell.