From duret.tanguy at gmail.com Sat Nov 3 14:16:37 2012 From: duret.tanguy at gmail.com (Tengy Td) Date: Sat, 3 Nov 2012 15:16:37 +0100 Subject: Coordination between developers in the Nginx project In-Reply-To: References: Message-ID: Hello, I am a French student and I am currently realizing my final thesis in the field of Free/libre open source software. It would be a great help for me if you could answer a short online survey (it should take approximately 5 minutes). This survey is designed to reach a better understanding of the cooperation and coordination between developers in Free/libre open source projects. There is no right or wrong answers, therefore, feel free to answer spontaneously and to skip the questions you feel you do not want to answer. The link to the survey is: http://bit.ly/SHlnUc I would like to remind you that the participation is absolutely anonymous and voluntary, and you can quit it at any time. Your answers will be strictly confidential and will be used only for research purpose (no commercial use of any information you provided). If you find this survey interesting, you are welcome to share it with other developers! If you want to add something, or if you need further information about this survey, feel free to contact me at: d uret.tanguy at gmail.com Thank you in advance for your cooperation and your enthusiasm! :) Tanguy Duret -------------- next part -------------- An HTML attachment was scrubbed... URL: From agentzh at gmail.com Tue Nov 6 06:18:28 2012 From: agentzh at gmail.com (agentzh) Date: Mon, 5 Nov 2012 22:18:28 -0800 Subject: [PATCH] making ngx_http_upstream_test_connect catch "connection refused" with kqueue Message-ID: Hello! smallfish and I both ran into a small issue with the ngx_http_upstream_test_connect which does not catch errors like "Connection refused" when kqueue is used. The current logic only checks c->write->pending_eof but this error actually produced a read event on (at least) both Mac OS X and FreeBSD. I've attached a simple patch (for nginx 1.2.4) to fix this by checking both the read and write events. To reproduce the issue, please consider the following Nginx configuration snippet (assuming nothing is listening on the local port 1234): location = /t { proxy_pass http://127.0.0.1:1234/; } Accessing /t gives the following message in error.log: [error] 28474#0: *1 kevent() reported about an closed connection (61: Connection refused) while reading response header from upstream We can see that ngx_http_upstream missed it in ngx_http_upstream_test_connect but caught it when reading the (upstream) response header. After applying my patch, Nginx can now catch it in the context of connecting: [error] 34067#0: *1 kevent() reported that connect() failed (61: Connection refused) while connecting to upstream Best regards, -agentzh --- nginx-1.2.4/src/http/ngx_http_upstream.c 2012-08-06 10:34:08.000000000 -0700 +++ nginx-1.2.4-patched/src/http/ngx_http_upstream.c 2012-11-05 21:17:38.000000000 -0800 @@ -1808,10 +1808,22 @@ ngx_http_upstream_test_connect(ngx_conne #if (NGX_HAVE_KQUEUE) + ngx_event_t *ev; + if (ngx_event_flags & NGX_USE_KQUEUE_EVENT) { if (c->write->pending_eof) { + ev = c->write; + + } else if (c->read->pending_eof) { + ev = c->read; + + } else { + ev = NULL; + } + + if (ev) { c->log->action = "connecting to upstream"; - (void) ngx_connection_error(c, c->write->kq_errno, + (void) ngx_connection_error(c, ev->kq_errno, "kevent() reported that connect() failed"); return NGX_ERROR; } -------------- next part -------------- A non-text attachment was scrubbed... Name: nginx-1.2.4-upstream_test_connect_kqueue.patch Type: application/octet-stream Size: 854 bytes Desc: not available URL: From ru at nginx.com Tue Nov 6 17:05:53 2012 From: ru at nginx.com (Ruslan Ermilov) Date: Tue, 6 Nov 2012 21:05:53 +0400 Subject: [PATCH] making ngx_http_upstream_test_connect catch "connection refused" with kqueue In-Reply-To: References: Message-ID: <20121106170553.GA93490@lo0.su> On Mon, Nov 05, 2012 at 10:18:28PM -0800, agentzh wrote: > Hello! > > smallfish and I both ran into a small issue with the > ngx_http_upstream_test_connect which does not catch errors like > "Connection refused" when kqueue is used. > > The current logic only checks c->write->pending_eof but this error > actually produced a read event on (at least) both Mac OS X and > FreeBSD. > > I've attached a simple patch (for nginx 1.2.4) to fix this by checking > both the read and write events. > > To reproduce the issue, please consider the following Nginx > configuration snippet (assuming nothing is listening on the local port > 1234): > > location = /t { > proxy_pass http://127.0.0.1:1234/; > } > > Accessing /t gives the following message in error.log: > > [error] 28474#0: *1 kevent() reported about an closed connection > (61: Connection refused) while reading response header from upstream > > We can see that ngx_http_upstream missed it in > ngx_http_upstream_test_connect but caught it when reading the > (upstream) response header. > > After applying my patch, Nginx can now catch it in the context of connecting: > > [error] 34067#0: *1 kevent() reported that connect() failed (61: > Connection refused) while connecting to upstream > > Best regards, > -agentzh Your patch looks good. From mdounin at mdounin.ru Tue Nov 6 17:55:40 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 6 Nov 2012 21:55:40 +0400 Subject: [PATCH] making ngx_http_upstream_test_connect catch "connection refused" with kqueue In-Reply-To: References: Message-ID: <20121106175540.GQ40452@mdounin.ru> Hello! On Mon, Nov 05, 2012 at 10:18:28PM -0800, agentzh wrote: > Hello! > > smallfish and I both ran into a small issue with the > ngx_http_upstream_test_connect which does not catch errors like > "Connection refused" when kqueue is used. > > The current logic only checks c->write->pending_eof but this error > actually produced a read event on (at least) both Mac OS X and > FreeBSD. > > I've attached a simple patch (for nginx 1.2.4) to fix this by checking > both the read and write events. > > To reproduce the issue, please consider the following Nginx > configuration snippet (assuming nothing is listening on the local port > 1234): > > location = /t { > proxy_pass http://127.0.0.1:1234/; > } > > Accessing /t gives the following message in error.log: > > [error] 28474#0: *1 kevent() reported about an closed connection > (61: Connection refused) while reading response header from upstream > > We can see that ngx_http_upstream missed it in > ngx_http_upstream_test_connect but caught it when reading the > (upstream) response header. > > After applying my patch, Nginx can now catch it in the context of connecting: > > [error] 34067#0: *1 kevent() reported that connect() failed (61: > Connection refused) while connecting to upstream Thanks for catching this! > --- nginx-1.2.4/src/http/ngx_http_upstream.c 2012-08-06 10:34:08.000000000 -0700 > +++ nginx-1.2.4-patched/src/http/ngx_http_upstream.c 2012-11-05 > 21:17:38.000000000 -0800 > @@ -1808,10 +1808,22 @@ ngx_http_upstream_test_connect(ngx_conne > > #if (NGX_HAVE_KQUEUE) > > + ngx_event_t *ev; > + > if (ngx_event_flags & NGX_USE_KQUEUE_EVENT) { > if (c->write->pending_eof) { > + ev = c->write; > + > + } else if (c->read->pending_eof) { > + ev = c->read; > + > + } else { > + ev = NULL; > + } > + > + if (ev) { > c->log->action = "connecting to upstream"; > - (void) ngx_connection_error(c, c->write->kq_errno, > + (void) ngx_connection_error(c, ev->kq_errno, > "kevent() reported that connect() failed"); > return NGX_ERROR; > } I don't see a reason to introduce another local variable here, what about something like --- a/src/http/ngx_http_upstream.c +++ b/src/http/ngx_http_upstream.c @@ -1809,9 +1809,16 @@ ngx_http_upstream_test_connect(ngx_conne #if (NGX_HAVE_KQUEUE) if (ngx_event_flags & NGX_USE_KQUEUE_EVENT) { - if (c->write->pending_eof) { + if (c->write->pending_eof || c->read->pending_eof) { + if (c->write->pending_eof) { + err = c->write->kq_errno; + + } else { + err = c->read->kq_errno; + } + c->log->action = "connecting to upstream"; - (void) ngx_connection_error(c, c->write->kq_errno, + (void) ngx_connection_error(c, err, "kevent() reported that connect() failed"); return NGX_ERROR; } instead? -- Maxim Dounin http://nginx.com/support.html From ibobrik at gmail.com Tue Nov 6 18:04:22 2012 From: ibobrik at gmail.com (ivan babrou) Date: Tue, 6 Nov 2012 22:04:22 +0400 Subject: image_filter enhancement Message-ID: I should probably email my patch here to get some attention :) http://trac.nginx.org/nginx/ticket/241 -- Regards, Ian Babrou http://bobrik.name http://twitter.com/ibobrik skype:i.babrou -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Tue Nov 6 18:16:02 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 6 Nov 2012 22:16:02 +0400 Subject: image_filter enhancement In-Reply-To: References: Message-ID: <20121106181601.GS40452@mdounin.ru> Hello! On Tue, Nov 06, 2012 at 10:04:22PM +0400, ivan babrou wrote: > I should probably email my patch here to get some attention :) > > http://trac.nginx.org/nginx/ticket/241 Please post patch here, not links. -- Maxim Dounin http://nginx.com/support.html From ibobrik at gmail.com Tue Nov 6 18:22:57 2012 From: ibobrik at gmail.com (ivan babrou) Date: Tue, 6 Nov 2012 22:22:57 +0400 Subject: image_filter enhancement In-Reply-To: <20121106181601.GS40452@mdounin.ru> References: <20121106181601.GS40452@mdounin.ru> Message-ID: Patch adds ability to align image before crop. It adds directive image_filter_offset {left,center,right} {top,center,bottom}; For example, we could set: image_filter crop 128 128; image_filter_offset left top; And get cropped image aligned to top left corner. It's quite useful when dealing with photos of human beings :) On 6 November 2012 22:16, Maxim Dounin wrote: > Hello! > > On Tue, Nov 06, 2012 at 10:04:22PM +0400, ivan babrou wrote: > >> I should probably email my patch here to get some attention :) >> >> http://trac.nginx.org/nginx/ticket/241 > > Please post patch here, not links. > > -- > Maxim Dounin > http://nginx.com/support.html > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel -- Regards, Ian Babrou http://bobrik.name http://twitter.com/ibobrik skype:i.babrou -------------- next part -------------- A non-text attachment was scrubbed... Name: image_filter_alignment.patch Type: application/octet-stream Size: 4652 bytes Desc: not available URL: From agentzh at gmail.com Tue Nov 6 19:22:05 2012 From: agentzh at gmail.com (agentzh) Date: Tue, 6 Nov 2012 11:22:05 -0800 Subject: [PATCH] making ngx_http_upstream_test_connect catch "connection refused" with kqueue In-Reply-To: <20121106175540.GQ40452@mdounin.ru> References: <20121106175540.GQ40452@mdounin.ru> Message-ID: Hello! On Tue, Nov 6, 2012 at 9:55 AM, Maxim Dounin wrote: > > I don't see a reason to introduce another local variable here, > what about something like > > --- a/src/http/ngx_http_upstream.c > +++ b/src/http/ngx_http_upstream.c > @@ -1809,9 +1809,16 @@ ngx_http_upstream_test_connect(ngx_conne > #if (NGX_HAVE_KQUEUE) > > if (ngx_event_flags & NGX_USE_KQUEUE_EVENT) { > - if (c->write->pending_eof) { > + if (c->write->pending_eof || c->read->pending_eof) { > + if (c->write->pending_eof) { > + err = c->write->kq_errno; > + > + } else { > + err = c->read->kq_errno; > + } > + > c->log->action = "connecting to upstream"; > - (void) ngx_connection_error(c, c->write->kq_errno, > + (void) ngx_connection_error(c, err, > "kevent() reported that connect() failed"); > return NGX_ERROR; > } > > instead? > I'm fine with this new patch :) Best regards, -agentzh From crk_world at yahoo.com.cn Thu Nov 8 11:52:41 2012 From: crk_world at yahoo.com.cn (chen cw) Date: Thu, 8 Nov 2012 19:52:41 +0800 Subject: description for "crop" directive in ngx_http_image_filter_module's document is not correct In-Reply-To: <20121029152254.GA23202@vlpc> References: <20121029082846.GA27250@vlpc> <20121029152254.GA23202@vlpc> Message-ID: We are going to translate this passage into Chinese according to your explanation. Thank you On Mon, Oct 29, 2012 at 11:22 PM, Homutov Vladimir wrote: > On Mon, Oct 29, 2012 at 12:28:47PM +0400, Homutov Vladimir wrote: > > On Mon, Oct 29, 2012 at 11:47:02AM +0800, chen cw wrote: > > > Hi, > > > we think the description for "crop" directive in > > > ngx_http_image_filter_module's document is not correct, or at least not > > > clear. It says "proportionally reduces an image to the size of the > largest > > > side". we have tested and now believes it should be not "the largest > > > side", but "the smallest side". > > > > > > > Hi! > > > > The 'largest side' is related to the arguments of the 'crop' > > directive, not the source image size. > > > > With this in mind, the description is correct. > > > > For example, > > > > command 'crop 600x50' for source image 640x480 will: > > > > 1) proportionally reduce largest (600 - horizontal) side: 640->600 > > 2) crops extraneous edges by another side - vertical: 480->50 > > > > the result is that image got reduced horizontally a bit (proportionally) > > and cropped by vertical side (50 center pixels left). > > > > command 'crop 600x50' for 480x640 image will give 480x50: > > > > 1) proportionally reduce horizontal side (no actions, 480 < 600 > > requested) > > 2) leave 50 center pixels on another side > > > > command 'crop 50x600' for 640x480 image will give 50x480: > > > > 1) largest side is 600 (vertical), no actions (requested 600 < 480 > > actual) > > 2) another side is cropped to 50 pixels. > > actually, things are a bit more complicated: > > what is important, it is the correlation between width/height ratio > of the source image and the desired size. > > The 'largest side' phrase is misleading, as ratios are > compared, not sizes; thus largest ratio is important, not side. > > The effect is that crop chooses dimension (width or height) to be > resized to reach the best fit of source image in the destination frame. > > if the source image has w/h ratio less than requested frame, it will be > proportionally resized up to new width and extra height will be cut. > > if the source image has w/h ratio greater than or equal to requested > frame, it > will be proportionally resized to the new height and extra width will be > cut. > > Example: > > # source image:640x480 (w/h = 1.333) > image_filter crop 320 250; # (w/h = 1.28) > > since 1.33 > 1.28, source image is resized up to desired height > first (333x250) and then extra width cut (320x250). > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -- -- Charles Chen Software Engineer Server Platforms Team at Taobao.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From wandenberg at gmail.com Fri Nov 9 14:37:50 2012 From: wandenberg at gmail.com (Wandenberg Peixoto) Date: Fri, 9 Nov 2012 12:37:50 -0200 Subject: [PATCH] making ngx_http_parse_time support 1 digit day Message-ID: Hello! I was checking a strange behavior when parsing dates coming from different browsers with this function and observed that it not supports dates with only one digit, it expect that the date comes with 01, 02, 03 ... But as rfc822 and rfc2822 says the day can be 1, 2 , 3, ... 10, 15, ... So I am proposing this patch to be compliant with the rfc. I expect that this patch could be applied to nginx code. --- src/http/ngx_http_parse_time.c 2012-02-28 08:31:05.000000000 -0300 +++ src/http/ngx_http_parse_time.c 2012-11-09 11:38:38.664036971 -0200 @@ -54,12 +54,17 @@ ngx_http_parse_time(u_char *value, size_ } if (fmt != isoc) { - if (*p < '0' || *p > '9' || *(p + 1) < '0' || *(p + 1) > '9') { + if (*p < '0' || *p > '9' || ((*(p + 1) != ' ') && (*(p + 1) < '0' || *(p + 1) > '9'))) { return NGX_ERROR; } - day = (*p - '0') * 10 + *(p + 1) - '0'; - p += 2; + if (*(p + 1) == ' ') { + day = (*p - '0'); + p += 1; + } else { + day = (*p - '0') * 10 + *(p + 1) - '0'; + p += 2; + } if (*p == ' ') { if (end - p < 18) { Regards, Wandenberg -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ngx_http_parse_time_rfc822_one_or_two_digit_day.patch Type: application/octet-stream Size: 769 bytes Desc: not available URL: From mdounin at mdounin.ru Fri Nov 9 15:53:06 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 9 Nov 2012 19:53:06 +0400 Subject: [PATCH] making ngx_http_parse_time support 1 digit day In-Reply-To: References: Message-ID: <20121109155306.GE40452@mdounin.ru> Hello! On Fri, Nov 09, 2012 at 12:37:50PM -0200, Wandenberg Peixoto wrote: > I was checking a strange behavior when parsing dates coming from different > browsers with this function and observed that it not supports dates with > only one digit, it expect that the date comes with 01, 02, 03 ... > But as rfc822 and rfc2822 says the day can be 1, 2 , 3, ... 10, 15, ... > So I am proposing this patch to be compliant with the rfc. What nginx supports is specified here: http://tools.ietf.org/html/rfc2616#section-3.3.1 This includes support for 1-digit years as per asctime() format. Format which is subset of date format defined by RFC 1123 (RFC822) require two digits as per syntax. That is, nginx is perfectly RFC complaint - you just looking at wrong RFC. Just for reference, here are the syntax provided in RFC 2616: HTTP-date = rfc1123-date | rfc850-date | asctime-date rfc1123-date = wkday "," SP date1 SP time SP "GMT" rfc850-date = weekday "," SP date2 SP time SP "GMT" asctime-date = wkday SP date3 SP time SP 4DIGIT date1 = 2DIGIT SP month SP 4DIGIT ; day month year (e.g., 02 Jun 1982) date2 = 2DIGIT "-" month "-" 2DIGIT ; day-month-year (e.g., 02-Jun-82) date3 = month SP ( 2DIGIT | ( SP 1DIGIT )) ; month day (e.g., Jun 2) > I expect that this patch could be applied to nginx code. I don't think it's a good idea to relax parsing beyond what RFC 2616 requires unless there is really good reason to. [...] -- Maxim Dounin http://nginx.com/support.html From pgnet.dev at gmail.com Fri Nov 9 16:13:05 2012 From: pgnet.dev at gmail.com (pgndev) Date: Fri, 9 Nov 2012 08:13:05 -0800 Subject: IPv6 support in "geo" directive? Message-ID: I'm currently running 1.3.8. IPv6 is now widely deployed by our ISPs, and fully deployed across our networks. In my nginx.conf, using geo $TEST { default 0; 127.0.0.1/32 1; 2001:111:2222:333::4/64 1; } throws an error, nginx: [emerg] "geo" supports IPv4 only in /etc/nginx/nginx.conf:120 Is that still true -- no IPv6 geo support? Is there a plan for that to change anytime soon? If not, could we request that be added asap? We've a lot of conditionals in our configs that now fail without the IPv6 support. thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wandenberg at gmail.com Fri Nov 9 16:33:33 2012 From: wandenberg at gmail.com (Wandenberg Peixoto) Date: Fri, 9 Nov 2012 14:33:33 -0200 Subject: [PATCH] making ngx_http_parse_time support 1 digit day In-Reply-To: <20121109155306.GE40452@mdounin.ru> References: <20121109155306.GE40452@mdounin.ru> Message-ID: Hello Maxim, thanks for you explanation. Only one doubt, the rfc2616 on the same paragraph says "HTTP/1.1 clients and servers that parse the date value MUST accept all three formats (for compatibility with HTTP/1.0), though they MUST only generate the RFC 1123 format for representing HTTP-date values in header fields. See section 19.3for further information." This isn't the case for relax the parser? We are parsing dates. Regards On Fri, Nov 9, 2012 at 1:53 PM, Maxim Dounin wrote: > Hello! > > On Fri, Nov 09, 2012 at 12:37:50PM -0200, Wandenberg Peixoto wrote: > > > I was checking a strange behavior when parsing dates coming from > different > > browsers with this function and observed that it not supports dates with > > only one digit, it expect that the date comes with 01, 02, 03 ... > > But as rfc822 and rfc2822 says the day can be 1, 2 , 3, ... 10, 15, ... > > So I am proposing this patch to be compliant with the rfc. > > What nginx supports is specified here: > > http://tools.ietf.org/html/rfc2616#section-3.3.1 > > This includes support for 1-digit years as per asctime() format. > Format which is subset of date format defined by RFC 1123 (RFC822) > require two digits as per syntax. > > That is, nginx is perfectly RFC complaint - you just looking at > wrong RFC. > > Just for reference, here are the syntax provided in RFC 2616: > > HTTP-date = rfc1123-date | rfc850-date | asctime-date > rfc1123-date = wkday "," SP date1 SP time SP "GMT" > rfc850-date = weekday "," SP date2 SP time SP "GMT" > asctime-date = wkday SP date3 SP time SP 4DIGIT > date1 = 2DIGIT SP month SP 4DIGIT > ; day month year (e.g., 02 Jun 1982) > date2 = 2DIGIT "-" month "-" 2DIGIT > ; day-month-year (e.g., 02-Jun-82) > date3 = month SP ( 2DIGIT | ( SP 1DIGIT )) > ; month day (e.g., Jun 2) > > > I expect that this patch could be applied to nginx code. > > I don't think it's a good idea to relax parsing beyond what > RFC 2616 requires unless there is really good reason to. > > [...] > > -- > Maxim Dounin > http://nginx.com/support.html > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgnet.dev at gmail.com Fri Nov 9 16:37:39 2012 From: pgnet.dev at gmail.com (pgndev) Date: Fri, 9 Nov 2012 08:37:39 -0800 Subject: IPv6 support in "geo" directive? In-Reply-To: <60DBB6B6-77DB-43CC-8198-FC79335101C0@nginx.com> References: <60DBB6B6-77DB-43CC-8198-FC79335101C0@nginx.com> Message-ID: Hi > We can certainly discuss that as part of our commercial offerings (sponsored development). > > Let me know if that might be of any interest. > > Many thanks. Is that to suggest that it's not currently planned/intended for implementation in the community version? Or that it will be done more quickly if it's "sponsored"? Is there an alternative to IPv6 src-address conditionals in nginx, rather than using $geo? On Fri, Nov 9, 2012 at 8:20 AM, Andrew Alexeev wrote: > Hi, > > On Nov 9, 2012, at 8:13 PM, pgndev wrote: > > I'm currently running 1.3.8. > > IPv6 is now widely deployed by our ISPs, and fully deployed across our > networks. > > In my nginx.conf, using > > geo $TEST { default 0; 127.0.0.1/32 1; > 2001:111:2222:333::4/64 1; } > > throws an error, > > nginx: [emerg] "geo" supports IPv4 only in /etc/nginx/nginx.conf:120 > > Is that still true -- no IPv6 geo support? > > Is there a plan for that to change anytime soon? > > If not, could we request that be added asap? We've a lot of conditionals > in our configs that now fail without the IPv6 support. > > > We can certainly discuss that as part of our commercial offerings > (sponsored development). > > Let me know if that might be of any interest. > > Many thanks. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Fri Nov 9 17:39:25 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 9 Nov 2012 21:39:25 +0400 Subject: [PATCH] making ngx_http_parse_time support 1 digit day In-Reply-To: References: <20121109155306.GE40452@mdounin.ru> Message-ID: <20121109173925.GF40452@mdounin.ru> Hello! On Fri, Nov 09, 2012 at 02:33:33PM -0200, Wandenberg Peixoto wrote: > Hello Maxim, > > thanks for you explanation. > Only one doubt, the rfc2616 on the same paragraph says > "HTTP/1.1 clients and servers that parse the date value MUST accept all > three formats (for compatibility with HTTP/1.0), though they MUST only > generate the RFC 1123 format for > representing HTTP-date values in header fields. See section > 19.3for further > information." > > This isn't the case for relax the parser? We are parsing dates. What nginx currently do is exactly as required by the quoted text: it accepts all three formats specified. > > Regards > > > On Fri, Nov 9, 2012 at 1:53 PM, Maxim Dounin wrote: > > > Hello! > > > > On Fri, Nov 09, 2012 at 12:37:50PM -0200, Wandenberg Peixoto wrote: > > > > > I was checking a strange behavior when parsing dates coming from > > different > > > browsers with this function and observed that it not supports dates with > > > only one digit, it expect that the date comes with 01, 02, 03 ... > > > But as rfc822 and rfc2822 says the day can be 1, 2 , 3, ... 10, 15, ... > > > So I am proposing this patch to be compliant with the rfc. > > > > What nginx supports is specified here: > > > > http://tools.ietf.org/html/rfc2616#section-3.3.1 > > > > This includes support for 1-digit years as per asctime() format. > > Format which is subset of date format defined by RFC 1123 (RFC822) > > require two digits as per syntax. > > > > That is, nginx is perfectly RFC complaint - you just looking at > > wrong RFC. > > > > Just for reference, here are the syntax provided in RFC 2616: > > > > HTTP-date = rfc1123-date | rfc850-date | asctime-date > > rfc1123-date = wkday "," SP date1 SP time SP "GMT" > > rfc850-date = weekday "," SP date2 SP time SP "GMT" > > asctime-date = wkday SP date3 SP time SP 4DIGIT > > date1 = 2DIGIT SP month SP 4DIGIT > > ; day month year (e.g., 02 Jun 1982) > > date2 = 2DIGIT "-" month "-" 2DIGIT > > ; day-month-year (e.g., 02-Jun-82) > > date3 = month SP ( 2DIGIT | ( SP 1DIGIT )) > > ; month day (e.g., Jun 2) > > > > > I expect that this patch could be applied to nginx code. > > > > I don't think it's a good idea to relax parsing beyond what > > RFC 2616 requires unless there is really good reason to. > > > > [...] > > > > -- > > Maxim Dounin > > http://nginx.com/support.html > > > > _______________________________________________ > > nginx-devel mailing list > > nginx-devel at nginx.org > > http://mailman.nginx.org/mailman/listinfo/nginx-devel > > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel -- Maxim Dounin http://nginx.com/support.html From wandenberg at gmail.com Fri Nov 9 17:46:29 2012 From: wandenberg at gmail.com (Wandenberg Peixoto) Date: Fri, 9 Nov 2012 15:46:29 -0200 Subject: [PATCH] making ngx_http_parse_time support 1 digit day In-Reply-To: <20121109173925.GF40452@mdounin.ru> References: <20121109155306.GE40452@mdounin.ru> <20121109173925.GF40452@mdounin.ru> Message-ID: OK. thanks for the answer. On Fri, Nov 9, 2012 at 3:39 PM, Maxim Dounin wrote: > Hello! > > On Fri, Nov 09, 2012 at 02:33:33PM -0200, Wandenberg Peixoto wrote: > > > Hello Maxim, > > > > thanks for you explanation. > > Only one doubt, the rfc2616 on the same paragraph says > > "HTTP/1.1 clients and servers that parse the date value MUST accept all > > three formats (for compatibility with HTTP/1.0), though they MUST only > > generate the RFC 1123 format for > > representing HTTP-date values in header fields. See section > > 19.3for further > > information." > > > > This isn't the case for relax the parser? We are parsing dates. > > What nginx currently do is exactly as required by the quoted text: > it accepts all three formats specified. > > > > > Regards > > > > > > On Fri, Nov 9, 2012 at 1:53 PM, Maxim Dounin wrote: > > > > > Hello! > > > > > > On Fri, Nov 09, 2012 at 12:37:50PM -0200, Wandenberg Peixoto wrote: > > > > > > > I was checking a strange behavior when parsing dates coming from > > > different > > > > browsers with this function and observed that it not supports dates > with > > > > only one digit, it expect that the date comes with 01, 02, 03 ... > > > > But as rfc822 and rfc2822 says the day can be 1, 2 , 3, ... 10, 15, > ... > > > > So I am proposing this patch to be compliant with the rfc. > > > > > > What nginx supports is specified here: > > > > > > http://tools.ietf.org/html/rfc2616#section-3.3.1 > > > > > > This includes support for 1-digit years as per asctime() format. > > > Format which is subset of date format defined by RFC 1123 (RFC822) > > > require two digits as per syntax. > > > > > > That is, nginx is perfectly RFC complaint - you just looking at > > > wrong RFC. > > > > > > Just for reference, here are the syntax provided in RFC 2616: > > > > > > HTTP-date = rfc1123-date | rfc850-date | asctime-date > > > rfc1123-date = wkday "," SP date1 SP time SP "GMT" > > > rfc850-date = weekday "," SP date2 SP time SP "GMT" > > > asctime-date = wkday SP date3 SP time SP 4DIGIT > > > date1 = 2DIGIT SP month SP 4DIGIT > > > ; day month year (e.g., 02 Jun 1982) > > > date2 = 2DIGIT "-" month "-" 2DIGIT > > > ; day-month-year (e.g., 02-Jun-82) > > > date3 = month SP ( 2DIGIT | ( SP 1DIGIT )) > > > ; month day (e.g., Jun 2) > > > > > > > I expect that this patch could be applied to nginx code. > > > > > > I don't think it's a good idea to relax parsing beyond what > > > RFC 2616 requires unless there is really good reason to. > > > > > > [...] > > > > > > -- > > > Maxim Dounin > > > http://nginx.com/support.html > > > > > > _______________________________________________ > > > nginx-devel mailing list > > > nginx-devel at nginx.org > > > http://mailman.nginx.org/mailman/listinfo/nginx-devel > > > > > > _______________________________________________ > > nginx-devel mailing list > > nginx-devel at nginx.org > > http://mailman.nginx.org/mailman/listinfo/nginx-devel > > > -- > Maxim Dounin > http://nginx.com/support.html > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From maxim at nginx.com Fri Nov 9 19:31:00 2012 From: maxim at nginx.com (Maxim Konovalov) Date: Fri, 09 Nov 2012 23:31:00 +0400 Subject: IPv6 support in "geo" directive? In-Reply-To: References: <60DBB6B6-77DB-43CC-8198-FC79335101C0@nginx.com> Message-ID: <509D59F4.8010208@nginx.com> On 11/9/12 8:37 PM, pgndev wrote: > Hi > > > We can certainly discuss that as part of our commercial offerings > (sponsored development). > > > > Let me know if that might be of any interest. > > > > Many thanks. > > Is that to suggest that it's not currently planned/intended for > implementation in the community version? Or that it will be done > more quickly if it's "sponsored"? > [...] IPv6 support in geo module is planned for 1.3 branch but no specific ETA. -- Maxim Konovalov +7 (910) 4293178 http://nginx.com/support.html From mdounin at mdounin.ru Mon Nov 12 17:03:39 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Mon, 12 Nov 2012 17:03:39 +0000 Subject: [nginx] svn commit: r4899 - in branches/stable-1.2/src: core http/modules/perl Message-ID: <20121112170339.53D4D3F9F47@mail.nginx.com> Author: mdounin Date: 2012-11-12 17:03:38 +0000 (Mon, 12 Nov 2012) New Revision: 4899 URL: http://trac.nginx.org/nginx/changeset/4899/nginx Log: Version bump. Modified: branches/stable-1.2/src/core/nginx.h branches/stable-1.2/src/http/modules/perl/nginx.pm Modified: branches/stable-1.2/src/core/nginx.h =================================================================== --- branches/stable-1.2/src/core/nginx.h 2012-10-30 13:35:18 UTC (rev 4898) +++ branches/stable-1.2/src/core/nginx.h 2012-11-12 17:03:38 UTC (rev 4899) @@ -9,8 +9,8 @@ #define _NGINX_H_INCLUDED_ -#define nginx_version 1002004 -#define NGINX_VERSION "1.2.4" +#define nginx_version 1002005 +#define NGINX_VERSION "1.2.5" #define NGINX_VER "nginx/" NGINX_VERSION #define NGINX_VAR "NGINX" Modified: branches/stable-1.2/src/http/modules/perl/nginx.pm =================================================================== --- branches/stable-1.2/src/http/modules/perl/nginx.pm 2012-10-30 13:35:18 UTC (rev 4898) +++ branches/stable-1.2/src/http/modules/perl/nginx.pm 2012-11-12 17:03:38 UTC (rev 4899) @@ -50,7 +50,7 @@ HTTP_INSUFFICIENT_STORAGE ); -our $VERSION = '1.2.4'; +our $VERSION = '1.2.5'; require XSLoader; XSLoader::load('nginx', $VERSION); From ibobrik at gmail.com Mon Nov 12 17:09:50 2012 From: ibobrik at gmail.com (ivan babrou) Date: Mon, 12 Nov 2012 21:09:50 +0400 Subject: image_filter enhancement In-Reply-To: <20121106181601.GS40452@mdounin.ru> References: <20121106181601.GS40452@mdounin.ru> Message-ID: Here's the patch. diff --git a/ngx_http_image_filter_module.c b/ngx_http_image_filter_module.c index c853c33..ef3c39c 100644 --- a/ngx_http_image_filter_module.c +++ b/ngx_http_image_filter_module.c @@ -32,6 +32,11 @@ #define NGX_HTTP_IMAGE_GIF 2 #define NGX_HTTP_IMAGE_PNG 3 +#define NGX_HTTP_IMAGE_OFFSET_CENTER 0 +#define NGX_HTTP_IMAGE_OFFSET_LEFT 1 +#define NGX_HTTP_IMAGE_OFFSET_RIGHT 2 +#define NGX_HTTP_IMAGE_OFFSET_TOP 3 +#define NGX_HTTP_IMAGE_OFFSET_BOTTOM 4 #define NGX_HTTP_IMAGE_BUFFERED 0x08 @@ -43,6 +48,8 @@ typedef struct { ngx_uint_t angle; ngx_uint_t jpeg_quality; ngx_uint_t sharpen; + ngx_uint_t ox; + ngx_uint_t oy; ngx_flag_t transparency; @@ -110,6 +117,8 @@ static char *ngx_http_image_filter_jpeg_quality(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); static char *ngx_http_image_filter_sharpen(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); +static char *ngx_http_image_filter_offset(ngx_conf_t *cf, ngx_command_t *cmd, + void *conf); static ngx_int_t ngx_http_image_filter_init(ngx_conf_t *cf); @@ -150,6 +159,13 @@ static ngx_command_t ngx_http_image_filter_commands[] = { offsetof(ngx_http_image_filter_conf_t, buffer_size), NULL }, + { ngx_string("image_filter_offset"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE2, + ngx_http_image_filter_offset, + NGX_HTTP_LOC_CONF_OFFSET, + 0, + NULL }, + ngx_null_command }; @@ -911,14 +927,12 @@ transparent: if ((ngx_uint_t) dx > ctx->max_width) { ox = dx - ctx->max_width; - } else { ox = 0; } if ((ngx_uint_t) dy > ctx->max_height) { oy = dy - ctx->max_height; - } else { oy = 0; } @@ -932,8 +946,17 @@ transparent: return NULL; } - ox /= 2; - oy /= 2; + if (conf->ox == NGX_HTTP_IMAGE_OFFSET_LEFT) { + ox = 0; + } else if (conf->ox == NGX_HTTP_IMAGE_OFFSET_CENTER) { + ox /= 2; + } + + if (conf->oy == NGX_HTTP_IMAGE_OFFSET_TOP) { + oy = 0; + } else if (conf->oy == NGX_HTTP_IMAGE_OFFSET_CENTER) { + oy /= 2; + } ngx_log_debug4(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "image crop: %d x %d @ %d x %d", @@ -1078,6 +1101,7 @@ ngx_http_image_out(ngx_http_request_t *r, ngx_uint_t type, gdImagePtr img, out = NULL; + switch (type) { case NGX_HTTP_IMAGE_JPEG: @@ -1175,6 +1199,8 @@ ngx_http_image_filter_create_conf(ngx_conf_t *cf) conf->angle = NGX_CONF_UNSET_UINT; conf->transparency = NGX_CONF_UNSET; conf->buffer_size = NGX_CONF_UNSET_SIZE; + conf->ox = NGX_CONF_UNSET_UINT; + conf->oy = NGX_CONF_UNSET_UINT; return conf; } @@ -1223,6 +1249,9 @@ ngx_http_image_filter_merge_conf(ngx_conf_t *cf, void *parent, void *child) ngx_conf_merge_size_value(conf->buffer_size, prev->buffer_size, 1 * 1024 * 1024); + ngx_conf_merge_uint_value(conf->ox, prev->ox, NGX_HTTP_IMAGE_OFFSET_CENTER); + ngx_conf_merge_uint_value(conf->oy, prev->oy, NGX_HTTP_IMAGE_OFFSET_CENTER); + return NGX_CONF_OK; } @@ -1473,6 +1502,38 @@ ngx_http_image_filter_sharpen(ngx_conf_t *cf, ngx_command_t *cmd, return NGX_CONF_OK; } +static char * +ngx_http_image_filter_offset(ngx_conf_t *cf, ngx_command_t *cmd, + void *conf) +{ + ngx_http_image_filter_conf_t *imcf = conf; + + ngx_str_t *value; + + value = cf->args->elts; + + if (ngx_strcmp(value[1].data, "center") == 0) { + imcf->ox = NGX_HTTP_IMAGE_OFFSET_CENTER; + } else if (ngx_strcmp(value[1].data, "left") == 0) { + imcf->ox = NGX_HTTP_IMAGE_OFFSET_LEFT; + } else if (ngx_strcmp(value[1].data, "right") == 0) { + imcf->ox = NGX_HTTP_IMAGE_OFFSET_RIGHT; + } else { + return NGX_CONF_ERROR; + } + + if (ngx_strcmp(value[2].data, "center") == 0) { + imcf->oy = NGX_HTTP_IMAGE_OFFSET_CENTER; + } else if (ngx_strcmp(value[2].data, "top") == 0) { + imcf->oy = NGX_HTTP_IMAGE_OFFSET_TOP; + } else if (ngx_strcmp(value[2].data, "bottom") == 0) { + imcf->oy = NGX_HTTP_IMAGE_OFFSET_BOTTOM; + } else { + return NGX_CONF_ERROR; + } + + return NGX_CONF_OK; +} static ngx_int_t ngx_http_image_filter_init(ngx_conf_t *cf) On 6 November 2012 22:16, Maxim Dounin wrote: > Hello! > > On Tue, Nov 06, 2012 at 10:04:22PM +0400, ivan babrou wrote: > >> I should probably email my patch here to get some attention :) >> >> http://trac.nginx.org/nginx/ticket/241 > > Please post patch here, not links. > > -- > Maxim Dounin > http://nginx.com/support.html > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel -- Regards, Ian Babrou http://bobrik.name http://twitter.com/ibobrik skype:i.babrou From mdounin at mdounin.ru Mon Nov 12 17:54:49 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Mon, 12 Nov 2012 17:54:49 +0000 Subject: [nginx] svn commit: r4900 - in branches/stable-1.2: . src/os/unix Message-ID: <20121112175449.8E6C83F9F4A@mail.nginx.com> Author: mdounin Date: 2012-11-12 17:54:49 +0000 (Mon, 12 Nov 2012) New Revision: 4900 URL: http://trac.nginx.org/nginx/changeset/4900/nginx Log: Merge of r4865: clearing of cpu_affinity after process spawn. This fixes unwanted/incorrect cpu_affinity use on dead worker processes respawn. While this is not ideal, it's expected to be better when previous situation where multiple processes were spawn with identical CPU affinity set. Reported by Charles Chen. Modified: branches/stable-1.2/ branches/stable-1.2/src/os/unix/ngx_process_cycle.c Index: branches/stable-1.2 =================================================================== --- branches/stable-1.2 2012-11-12 17:03:38 UTC (rev 4899) +++ branches/stable-1.2 2012-11-12 17:54:49 UTC (rev 4900) Property changes on: branches/stable-1.2 ___________________________________________________________________ Modified: svn:mergeinfo ## -1 +1 ## -/trunk:4611-4632,4636-4657,4671-4672,4674-4676,4682,4684-4699,4704-4706,4713,4736-4741,4754,4756-4771,4775,4777-4780,4782-4785,4795,4811-4820,4822-4824,4828-4835,4840-4844 +/trunk:4611-4632,4636-4657,4671-4672,4674-4676,4682,4684-4699,4704-4706,4713,4736-4741,4754,4756-4771,4775,4777-4780,4782-4785,4795,4811-4820,4822-4824,4828-4835,4840-4844,4865 \ No newline at end of property Modified: branches/stable-1.2/src/os/unix/ngx_process_cycle.c =================================================================== --- branches/stable-1.2/src/os/unix/ngx_process_cycle.c 2012-11-12 17:03:38 UTC (rev 4899) +++ branches/stable-1.2/src/os/unix/ngx_process_cycle.c 2012-11-12 17:54:49 UTC (rev 4900) @@ -371,6 +371,8 @@ ngx_pass_open_channel(cycle, &ch); } + + cpu_affinity = 0; } From mdounin at mdounin.ru Mon Nov 12 17:57:57 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Mon, 12 Nov 2012 17:57:57 +0000 Subject: [nginx] svn commit: r4901 - in branches/stable-1.2: . auto auto/lib/perl Message-ID: <20121112175757.BF2CF3F9F4A@mail.nginx.com> Author: mdounin Date: 2012-11-12 17:57:57 +0000 (Mon, 12 Nov 2012) New Revision: 4901 URL: http://trac.nginx.org/nginx/changeset/4901/nginx Log: Merge of r4866, r4867: configure fixes. *) Configure: help updated to list upstream keepalive and least_conn. Patch by Joshua Zhu. *) Configure: additional test for ExtUtils::Embed perl module presence. Now perl configure will correctly fail if ExtUtils::Embed perl module is not present in the system (found on Amazon Linux AMI, as of release 2012.03). Modified: branches/stable-1.2/ branches/stable-1.2/auto/lib/perl/conf branches/stable-1.2/auto/options Index: branches/stable-1.2 =================================================================== --- branches/stable-1.2 2012-11-12 17:54:49 UTC (rev 4900) +++ branches/stable-1.2 2012-11-12 17:57:57 UTC (rev 4901) Property changes on: branches/stable-1.2 ___________________________________________________________________ Modified: svn:mergeinfo ## -1 +1 ## -/trunk:4611-4632,4636-4657,4671-4672,4674-4676,4682,4684-4699,4704-4706,4713,4736-4741,4754,4756-4771,4775,4777-4780,4782-4785,4795,4811-4820,4822-4824,4828-4835,4840-4844,4865 +/trunk:4611-4632,4636-4657,4671-4672,4674-4676,4682,4684-4699,4704-4706,4713,4736-4741,4754,4756-4771,4775,4777-4780,4782-4785,4795,4811-4820,4822-4824,4828-4835,4840-4844,4865-4867 \ No newline at end of property Modified: branches/stable-1.2/auto/lib/perl/conf =================================================================== --- branches/stable-1.2/auto/lib/perl/conf 2012-11-12 17:54:49 UTC (rev 4900) +++ branches/stable-1.2/auto/lib/perl/conf 2012-11-12 17:57:57 UTC (rev 4901) @@ -12,7 +12,7 @@ if test -n "$NGX_PERL_VER"; then echo " + perl version: $NGX_PERL_VER" - if [ "`echo 'use 5.006001; print "OK"' | $NGX_PERL 2>&1`" != OK ]; then + if [ "`$NGX_PERL -e 'use 5.006001; print "OK"'`" != "OK" ]; then echo echo "$0: error: perl 5.6.1 or higher is required" echo @@ -20,6 +20,14 @@ exit 1; fi + if [ "`$NGX_PERL -MExtUtils::Embed -e 'print "OK"'`" != "OK" ]; then + echo + echo "$0: error: perl module ExtUtils::Embed is required" + echo + + exit 1; + fi + NGX_PERL_CFLAGS="$CFLAGS `$NGX_PERL -MExtUtils::Embed -e ccopts`" NGX_PM_CFLAGS=`$NGX_PERL -MExtUtils::Embed -e ccopts` Modified: branches/stable-1.2/auto/options =================================================================== --- branches/stable-1.2/auto/options 2012-11-12 17:54:49 UTC (rev 4900) +++ branches/stable-1.2/auto/options 2012-11-12 17:57:57 UTC (rev 4901) @@ -385,6 +385,10 @@ --without-http_browser_module disable ngx_http_browser_module --without-http_upstream_ip_hash_module disable ngx_http_upstream_ip_hash_module + --without-http_upstream_least_conn_module + disable ngx_http_upstream_least_conn_module + --without-http_upstream_keepalive_module + disable ngx_http_upstream_keepalive_module --with-http_perl_module enable ngx_http_perl_module --with-perl_modules_path=PATH set Perl modules path From mdounin at mdounin.ru Mon Nov 12 18:00:33 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Mon, 12 Nov 2012 18:00:33 +0000 Subject: [nginx] svn commit: r4902 - in branches/stable-1.2: . src/event Message-ID: <20121112180033.5A4BB3F9C3E@mail.nginx.com> Author: mdounin Date: 2012-11-12 18:00:32 +0000 (Mon, 12 Nov 2012) New Revision: 4902 URL: http://trac.nginx.org/nginx/changeset/4902/nginx Log: Merge of r4868, r4869: SSL minor fixes. *) SSL: fixed compression workaround to remove all methods. Previous code used sk_SSL_COMP_delete(ssl_comp_methods, i) while iterating stack from 0 to n, resulting in removal of only even compression methods. In real life this change is a nop, as there is only one compression method which is enabled by default in OpenSSL. *) SSL: added version checks for ssl compression workaround. The SSL_COMP_get_compression_methods() is only available as an API function in OpenSSL 0.9.8+, require it explicitly to unbreak build with OpenSSL 0.9.7. Modified: branches/stable-1.2/ branches/stable-1.2/src/event/ngx_event_openssl.c Index: branches/stable-1.2 =================================================================== --- branches/stable-1.2 2012-11-12 17:57:57 UTC (rev 4901) +++ branches/stable-1.2 2012-11-12 18:00:32 UTC (rev 4902) Property changes on: branches/stable-1.2 ___________________________________________________________________ Modified: svn:mergeinfo ## -1 +1 ## -/trunk:4611-4632,4636-4657,4671-4672,4674-4676,4682,4684-4699,4704-4706,4713,4736-4741,4754,4756-4771,4775,4777-4780,4782-4785,4795,4811-4820,4822-4824,4828-4835,4840-4844,4865-4867 +/trunk:4611-4632,4636-4657,4671-4672,4674-4676,4682,4684-4699,4704-4706,4713,4736-4741,4754,4756-4771,4775,4777-4780,4782-4785,4795,4811-4820,4822-4824,4828-4835,4840-4844,4865-4869 \ No newline at end of property Modified: branches/stable-1.2/src/event/ngx_event_openssl.c =================================================================== --- branches/stable-1.2/src/event/ngx_event_openssl.c 2012-11-12 17:57:57 UTC (rev 4901) +++ branches/stable-1.2/src/event/ngx_event_openssl.c 2012-11-12 18:00:32 UTC (rev 4902) @@ -94,23 +94,25 @@ OpenSSL_add_all_algorithms(); +#if OPENSSL_VERSION_NUMBER >= 0x0090800fL #ifndef SSL_OP_NO_COMPRESSION { /* * Disable gzip compression in OpenSSL prior to 1.0.0 version, * this saves about 522K per connection. */ - int i, n; + int n; STACK_OF(SSL_COMP) *ssl_comp_methods; ssl_comp_methods = SSL_COMP_get_compression_methods(); n = sk_SSL_COMP_num(ssl_comp_methods); - for (i = 0; i < n; i++) { - (void) sk_SSL_COMP_delete(ssl_comp_methods, i); + while (n--) { + (void) sk_SSL_COMP_pop(ssl_comp_methods); } } #endif +#endif ngx_ssl_connection_index = SSL_get_ex_new_index(0, NULL, NULL, NULL, NULL); From mdounin at mdounin.ru Mon Nov 12 18:39:52 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Mon, 12 Nov 2012 18:39:52 +0000 Subject: [nginx] svn commit: r4903 - in branches/stable-1.2: . auto misc src/core src/os/unix src/os/win32 Message-ID: <20121112183952.808A93F9EC2@mail.nginx.com> Author: mdounin Date: 2012-11-12 18:39:51 +0000 (Mon, 12 Nov 2012) New Revision: 4903 URL: http://trac.nginx.org/nginx/changeset/4903/nginx Log: Merge of r4870, r4871, r4890, r4895: minor fixes. *) Made sure to initialize the entire ngx_file_t structure. Found by Coverity. *) Correct plural form for "path" in the whole source base. *) Removed conditional compilation from waitpid() error test. There are reports that call to a signal handler for an exited process despite waitpid() already called for the process may happen on Linux as well. *) Style, parentheses instead of braces in misc/GNUMakefile. Modified: branches/stable-1.2/ branches/stable-1.2/auto/make branches/stable-1.2/misc/GNUmakefile branches/stable-1.2/src/core/ngx_cycle.c branches/stable-1.2/src/core/ngx_cycle.h branches/stable-1.2/src/core/ngx_file.c branches/stable-1.2/src/core/ngx_file.h branches/stable-1.2/src/os/unix/ngx_process.c branches/stable-1.2/src/os/unix/ngx_process_cycle.c branches/stable-1.2/src/os/win32/ngx_process_cycle.c Index: branches/stable-1.2 =================================================================== --- branches/stable-1.2 2012-11-12 18:00:32 UTC (rev 4902) +++ branches/stable-1.2 2012-11-12 18:39:51 UTC (rev 4903) Property changes on: branches/stable-1.2 ___________________________________________________________________ Modified: svn:mergeinfo ## -1 +1 ## -/trunk:4611-4632,4636-4657,4671-4672,4674-4676,4682,4684-4699,4704-4706,4713,4736-4741,4754,4756-4771,4775,4777-4780,4782-4785,4795,4811-4820,4822-4824,4828-4835,4840-4844,4865-4869 +/trunk:4611-4632,4636-4657,4671-4672,4674-4676,4682,4684-4699,4704-4706,4713,4736-4741,4754,4756-4771,4775,4777-4780,4782-4785,4795,4811-4820,4822-4824,4828-4835,4840-4844,4865-4871,4890,4895 \ No newline at end of property Modified: branches/stable-1.2/auto/make =================================================================== --- branches/stable-1.2/auto/make 2012-11-12 18:00:32 UTC (rev 4902) +++ branches/stable-1.2/auto/make 2012-11-12 18:39:51 UTC (rev 4903) @@ -49,7 +49,7 @@ ngx_all_srcs="$CORE_SRCS" -# the core dependences and include pathes +# the core dependences and include paths ngx_deps=`echo $CORE_DEPS $NGX_AUTO_CONFIG_H $NGX_PCH \ | sed -e "s/ *\([^ ][^ ]*\)/$ngx_regex_cont\1/g" \ @@ -69,7 +69,7 @@ END -# the http dependences and include pathes +# the http dependences and include paths if [ $HTTP = YES ]; then @@ -95,7 +95,7 @@ fi -# the mail dependences and include pathes +# the mail dependences and include paths if [ $MAIL = YES ]; then Modified: branches/stable-1.2/misc/GNUmakefile =================================================================== --- branches/stable-1.2/misc/GNUmakefile 2012-11-12 18:00:32 UTC (rev 4902) +++ branches/stable-1.2/misc/GNUmakefile 2012-11-12 18:39:51 UTC (rev 4903) @@ -86,7 +86,7 @@ win32: ./auto/configure \ --with-cc=cl \ - --builddir=${OBJS} \ + --builddir=$(OBJS) \ --with-debug \ --prefix= \ --conf-path=conf/nginx.conf \ @@ -100,8 +100,8 @@ --http-scgi-temp-path=temp/scgi_temp \ --http-uwsgi-temp-path=temp/uwsgi_temp \ --with-cc-opt=-DFD_SETSIZE=1024 \ - --with-pcre=${OBJS}/lib/${PCRE} \ - --with-zlib=${OBJS}/lib/${ZLIB} \ + --with-pcre=$(OBJS)/lib/$(PCRE) \ + --with-zlib=$(OBJS)/lib/$(ZLIB) \ --with-select_module \ --with-http_realip_module \ --with-http_addition_module \ @@ -114,7 +114,7 @@ --with-http_random_index_module \ --with-http_secure_link_module \ --with-mail \ - --with-openssl=${OBJS}/lib/${OPENSSL} \ + --with-openssl=$(OBJS)/lib/$(OPENSSL) \ --with-openssl-opt=enable-tlsext \ --with-http_ssl_module \ --with-mail_ssl_module \ Modified: branches/stable-1.2/src/core/ngx_cycle.c =================================================================== --- branches/stable-1.2/src/core/ngx_cycle.c 2012-11-12 18:00:32 UTC (rev 4902) +++ branches/stable-1.2/src/core/ngx_cycle.c 2012-11-12 18:39:51 UTC (rev 4903) @@ -118,18 +118,18 @@ } - n = old_cycle->pathes.nelts ? old_cycle->pathes.nelts : 10; + n = old_cycle->paths.nelts ? old_cycle->paths.nelts : 10; - cycle->pathes.elts = ngx_pcalloc(pool, n * sizeof(ngx_path_t *)); - if (cycle->pathes.elts == NULL) { + cycle->paths.elts = ngx_pcalloc(pool, n * sizeof(ngx_path_t *)); + if (cycle->paths.elts == NULL) { ngx_destroy_pool(pool); return NULL; } - cycle->pathes.nelts = 0; - cycle->pathes.size = sizeof(ngx_path_t *); - cycle->pathes.nalloc = n; - cycle->pathes.pool = pool; + cycle->paths.nelts = 0; + cycle->paths.size = sizeof(ngx_path_t *); + cycle->paths.nalloc = n; + cycle->paths.pool = pool; if (old_cycle->open_files.part.nelts) { @@ -334,7 +334,7 @@ } - if (ngx_create_pathes(cycle, ccf->user) != NGX_OK) { + if (ngx_create_paths(cycle, ccf->user) != NGX_OK) { goto failed; } @@ -1038,6 +1038,8 @@ ccf = (ngx_core_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_core_module); + ngx_memzero(&file, sizeof(ngx_file_t)); + file.name = ccf->pid; file.log = cycle->log; Modified: branches/stable-1.2/src/core/ngx_cycle.h =================================================================== --- branches/stable-1.2/src/core/ngx_cycle.h 2012-11-12 18:00:32 UTC (rev 4902) +++ branches/stable-1.2/src/core/ngx_cycle.h 2012-11-12 18:39:51 UTC (rev 4903) @@ -48,7 +48,7 @@ ngx_queue_t reusable_connections_queue; ngx_array_t listening; - ngx_array_t pathes; + ngx_array_t paths; ngx_list_t open_files; ngx_list_t shared_memory; Modified: branches/stable-1.2/src/core/ngx_file.c =================================================================== --- branches/stable-1.2/src/core/ngx_file.c 2012-11-12 18:00:32 UTC (rev 4902) +++ branches/stable-1.2/src/core/ngx_file.c 2012-11-12 18:39:51 UTC (rev 4903) @@ -412,8 +412,8 @@ path = *slot; - p = cf->cycle->pathes.elts; - for (i = 0; i < cf->cycle->pathes.nelts; i++) { + p = cf->cycle->paths.elts; + for (i = 0; i < cf->cycle->paths.nelts; i++) { if (p[i]->name.len == path->name.len && ngx_strcmp(p[i]->name.data, path->name.data) == 0) { @@ -457,7 +457,7 @@ } } - p = ngx_array_push(&cf->cycle->pathes); + p = ngx_array_push(&cf->cycle->paths); if (p == NULL) { return NGX_ERROR; } @@ -469,14 +469,14 @@ ngx_int_t -ngx_create_pathes(ngx_cycle_t *cycle, ngx_uid_t user) +ngx_create_paths(ngx_cycle_t *cycle, ngx_uid_t user) { ngx_err_t err; ngx_uint_t i; ngx_path_t **path; - path = cycle->pathes.elts; - for (i = 0; i < cycle->pathes.nelts; i++) { + path = cycle->paths.elts; + for (i = 0; i < cycle->paths.nelts; i++) { if (ngx_create_dir(path[i]->name.data, 0700) == NGX_FILE_ERROR) { err = ngx_errno; Modified: branches/stable-1.2/src/core/ngx_file.h =================================================================== --- branches/stable-1.2/src/core/ngx_file.h 2012-11-12 18:00:32 UTC (rev 4902) +++ branches/stable-1.2/src/core/ngx_file.h 2012-11-12 18:39:51 UTC (rev 4903) @@ -130,7 +130,7 @@ ngx_int_t ngx_create_path(ngx_file_t *file, ngx_path_t *path); ngx_err_t ngx_create_full_path(u_char *dir, ngx_uint_t access); ngx_int_t ngx_add_path(ngx_conf_t *cf, ngx_path_t **slot); -ngx_int_t ngx_create_pathes(ngx_cycle_t *cycle, ngx_uid_t user); +ngx_int_t ngx_create_paths(ngx_cycle_t *cycle, ngx_uid_t user); ngx_int_t ngx_ext_rename_file(ngx_str_t *src, ngx_str_t *to, ngx_ext_rename_file_t *ext); ngx_int_t ngx_copy_file(u_char *from, u_char *to, ngx_copy_file_t *cf); Modified: branches/stable-1.2/src/os/unix/ngx_process.c =================================================================== --- branches/stable-1.2/src/os/unix/ngx_process.c 2012-11-12 18:00:32 UTC (rev 4902) +++ branches/stable-1.2/src/os/unix/ngx_process.c 2012-11-12 18:39:51 UTC (rev 4903) @@ -474,8 +474,6 @@ return; } -#if (NGX_SOLARIS || NGX_FREEBSD) - /* * Solaris always calls the signal handler for each exited process * despite waitpid() may be already called for this process. @@ -491,8 +489,6 @@ return; } -#endif - ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, err, "waitpid() failed"); return; Modified: branches/stable-1.2/src/os/unix/ngx_process_cycle.c =================================================================== --- branches/stable-1.2/src/os/unix/ngx_process_cycle.c 2012-11-12 18:00:32 UTC (rev 4902) +++ branches/stable-1.2/src/os/unix/ngx_process_cycle.c 2012-11-12 18:39:51 UTC (rev 4903) @@ -386,8 +386,8 @@ manager = 0; loader = 0; - path = ngx_cycle->pathes.elts; - for (i = 0; i < ngx_cycle->pathes.nelts; i++) { + path = ngx_cycle->paths.elts; + for (i = 0; i < ngx_cycle->paths.nelts; i++) { if (path[i]->manager) { manager = 1; @@ -1341,8 +1341,8 @@ next = 60 * 60; - path = ngx_cycle->pathes.elts; - for (i = 0; i < ngx_cycle->pathes.nelts; i++) { + path = ngx_cycle->paths.elts; + for (i = 0; i < ngx_cycle->paths.nelts; i++) { if (path[i]->manager) { n = path[i]->manager(path[i]->data); @@ -1370,8 +1370,8 @@ cycle = (ngx_cycle_t *) ngx_cycle; - path = cycle->pathes.elts; - for (i = 0; i < cycle->pathes.nelts; i++) { + path = cycle->paths.elts; + for (i = 0; i < cycle->paths.nelts; i++) { if (ngx_terminate || ngx_quit) { break; Modified: branches/stable-1.2/src/os/win32/ngx_process_cycle.c =================================================================== --- branches/stable-1.2/src/os/win32/ngx_process_cycle.c 2012-11-12 18:00:32 UTC (rev 4902) +++ branches/stable-1.2/src/os/win32/ngx_process_cycle.c 2012-11-12 18:39:51 UTC (rev 4903) @@ -963,8 +963,8 @@ next = 60 * 60; - path = ngx_cycle->pathes.elts; - for (i = 0; i < ngx_cycle->pathes.nelts; i++) { + path = ngx_cycle->paths.elts; + for (i = 0; i < ngx_cycle->paths.nelts; i++) { if (path[i]->manager) { n = path[i]->manager(path[i]->data); @@ -1002,8 +1002,8 @@ cycle = (ngx_cycle_t *) ngx_cycle; - path = cycle->pathes.elts; - for (i = 0; i < cycle->pathes.nelts; i++) { + path = cycle->paths.elts; + for (i = 0; i < cycle->paths.nelts; i++) { if (ngx_terminate || ngx_quit) { break; From mdounin at mdounin.ru Mon Nov 12 18:47:08 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Mon, 12 Nov 2012 18:47:08 +0000 Subject: [nginx] svn commit: r4904 - in branches/stable-1.2: . src/core Message-ID: <20121112184708.493D83F9F4A@mail.nginx.com> Author: mdounin Date: 2012-11-12 18:47:07 +0000 (Mon, 12 Nov 2012) New Revision: 4904 URL: http://trac.nginx.org/nginx/changeset/4904/nginx Log: Merge of r4872, r4893: resolver cached addresses random rotation. Resolver: cached addresses are returned with random rotation now. This ensures balancing when working with dynamically resolved upstream servers with multiple addresses. Based on patch by Anton Jouline. Modified: branches/stable-1.2/ branches/stable-1.2/src/core/ngx_resolver.c Index: branches/stable-1.2 =================================================================== --- branches/stable-1.2 2012-11-12 18:39:51 UTC (rev 4903) +++ branches/stable-1.2 2012-11-12 18:47:07 UTC (rev 4904) Property changes on: branches/stable-1.2 ___________________________________________________________________ Modified: svn:mergeinfo ## -1 +1 ## -/trunk:4611-4632,4636-4657,4671-4672,4674-4676,4682,4684-4699,4704-4706,4713,4736-4741,4754,4756-4771,4775,4777-4780,4782-4785,4795,4811-4820,4822-4824,4828-4835,4840-4844,4865-4871,4890,4895 +/trunk:4611-4632,4636-4657,4671-4672,4674-4676,4682,4684-4699,4704-4706,4713,4736-4741,4754,4756-4771,4775,4777-4780,4782-4785,4795,4811-4820,4822-4824,4828-4835,4840-4844,4865-4872,4890,4893,4895 \ No newline at end of property Modified: branches/stable-1.2/src/core/ngx_resolver.c =================================================================== --- branches/stable-1.2/src/core/ngx_resolver.c 2012-11-12 18:39:51 UTC (rev 4903) +++ branches/stable-1.2/src/core/ngx_resolver.c 2012-11-12 18:47:07 UTC (rev 4904) @@ -88,6 +88,8 @@ static void ngx_resolver_free(ngx_resolver_t *r, void *p); static void ngx_resolver_free_locked(ngx_resolver_t *r, void *p); static void *ngx_resolver_dup(ngx_resolver_t *r, void *src, size_t size); +static in_addr_t *ngx_resolver_rotate(ngx_resolver_t *r, in_addr_t *src, + ngx_uint_t n); static u_char *ngx_resolver_log_error(ngx_log_t *log, u_char *buf, size_t len); @@ -445,8 +447,7 @@ if (naddrs != 1) { addr = 0; - addrs = ngx_resolver_dup(r, rn->u.addrs, - naddrs * sizeof(in_addr_t)); + addrs = ngx_resolver_rotate(r, rn->u.addrs, naddrs); if (addrs == NULL) { return NGX_ERROR; } @@ -2135,6 +2136,32 @@ } +static in_addr_t * +ngx_resolver_rotate(ngx_resolver_t *r, in_addr_t *src, ngx_uint_t n) +{ + void *dst, *p; + ngx_uint_t j; + + dst = ngx_resolver_alloc(r, n * sizeof(in_addr_t)); + + if (dst == NULL) { + return dst; + } + + j = ngx_random() % n; + + if (j == 0) { + ngx_memcpy(dst, src, n * sizeof(in_addr_t)); + return dst; + } + + p = ngx_cpymem(dst, &src[j], (n - j) * sizeof(in_addr_t)); + ngx_memcpy(p, src, j * sizeof(in_addr_t)); + + return dst; +} + + char * ngx_resolver_strerror(ngx_int_t err) { From mdounin at mdounin.ru Tue Nov 13 10:42:17 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Tue, 13 Nov 2012 10:42:17 +0000 Subject: [nginx] svn commit: r4905 - in branches/stable-1.2: . src/event src/http src/http/modules Message-ID: <20121113104217.5EE363F9C3E@mail.nginx.com> Author: mdounin Date: 2012-11-13 10:42:16 +0000 (Tue, 13 Nov 2012) New Revision: 4905 URL: http://trac.nginx.org/nginx/changeset/4905/nginx Log: Merge of r4885: ssl_verify_client optional_no_ca. SSL: the "ssl_verify_client" directive parameter "optional_no_ca". This parameter allows to don't require certificate to be signed by a trusted CA, e.g. if CA certificate isn't known in advance, like in WebID protocol. Note that it doesn't add any security unless the certificate is actually checked to be trusted by some external means (e.g. by a backend). Patch by Mike Kazantsev, Eric O'Connor. Modified: branches/stable-1.2/ branches/stable-1.2/src/event/ngx_event_openssl.h branches/stable-1.2/src/http/modules/ngx_http_ssl_module.c branches/stable-1.2/src/http/ngx_http_request.c Index: branches/stable-1.2 =================================================================== --- branches/stable-1.2 2012-11-12 18:47:07 UTC (rev 4904) +++ branches/stable-1.2 2012-11-13 10:42:16 UTC (rev 4905) Property changes on: branches/stable-1.2 ___________________________________________________________________ Modified: svn:mergeinfo ## -1 +1 ## -/trunk:4611-4632,4636-4657,4671-4672,4674-4676,4682,4684-4699,4704-4706,4713,4736-4741,4754,4756-4771,4775,4777-4780,4782-4785,4795,4811-4820,4822-4824,4828-4835,4840-4844,4865-4872,4890,4893,4895 +/trunk:4611-4632,4636-4657,4671-4672,4674-4676,4682,4684-4699,4704-4706,4713,4736-4741,4754,4756-4771,4775,4777-4780,4782-4785,4795,4811-4820,4822-4824,4828-4835,4840-4844,4865-4872,4885,4890,4893,4895 \ No newline at end of property Modified: branches/stable-1.2/src/event/ngx_event_openssl.h =================================================================== --- branches/stable-1.2/src/event/ngx_event_openssl.h 2012-11-12 18:47:07 UTC (rev 4904) +++ branches/stable-1.2/src/event/ngx_event_openssl.h 2012-11-13 10:42:16 UTC (rev 4905) @@ -120,7 +120,14 @@ #define ngx_ssl_get_server_conf(ssl_ctx) \ SSL_CTX_get_ex_data(ssl_ctx, ngx_ssl_server_conf_index) +#define ngx_ssl_verify_error_optional(n) \ + (n == X509_V_ERR_DEPTH_ZERO_SELF_SIGNED_CERT \ + || n == X509_V_ERR_SELF_SIGNED_CERT_IN_CHAIN \ + || n == X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY \ + || n == X509_V_ERR_CERT_UNTRUSTED \ + || n == X509_V_ERR_UNABLE_TO_VERIFY_LEAF_SIGNATURE) + ngx_int_t ngx_ssl_get_protocol(ngx_connection_t *c, ngx_pool_t *pool, ngx_str_t *s); ngx_int_t ngx_ssl_get_cipher_name(ngx_connection_t *c, ngx_pool_t *pool, Modified: branches/stable-1.2/src/http/modules/ngx_http_ssl_module.c =================================================================== --- branches/stable-1.2/src/http/modules/ngx_http_ssl_module.c 2012-11-12 18:47:07 UTC (rev 4904) +++ branches/stable-1.2/src/http/modules/ngx_http_ssl_module.c 2012-11-13 10:42:16 UTC (rev 4905) @@ -48,6 +48,7 @@ { ngx_string("off"), 0 }, { ngx_string("on"), 1 }, { ngx_string("optional"), 2 }, + { ngx_string("optional_no_ca"), 3 }, { ngx_null_string, 0 } }; @@ -466,7 +467,7 @@ if (conf->verify) { - if (conf->client_certificate.len == 0) { + if (conf->client_certificate.len == 0 && conf->verify != 3) { ngx_log_error(NGX_LOG_EMERG, cf->log, 0, "no ssl_client_certificate for ssl_client_verify"); return NGX_CONF_ERROR; Modified: branches/stable-1.2/src/http/ngx_http_request.c =================================================================== --- branches/stable-1.2/src/http/ngx_http_request.c 2012-11-12 18:47:07 UTC (rev 4904) +++ branches/stable-1.2/src/http/ngx_http_request.c 2012-11-13 10:42:16 UTC (rev 4905) @@ -1634,7 +1634,9 @@ if (sscf->verify) { rc = SSL_get_verify_result(c->ssl->connection); - if (rc != X509_V_OK) { + if (rc != X509_V_OK + && (sscf->verify != 3 || !ngx_ssl_verify_error_optional(rc))) + { ngx_log_error(NGX_LOG_INFO, c->log, 0, "client SSL certificate verify error: (%l:%s)", rc, X509_verify_cert_error_string(rc)); From mdounin at mdounin.ru Tue Nov 13 10:45:24 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Tue, 13 Nov 2012 10:45:24 +0000 Subject: [nginx] svn commit: r4906 - in branches/stable-1.2: . src/http src/http/modules Message-ID: <20121113104524.F30FF3F9C4E@mail.nginx.com> Author: mdounin Date: 2012-11-13 10:45:23 +0000 (Tue, 13 Nov 2012) New Revision: 4906 URL: http://trac.nginx.org/nginx/changeset/4906/nginx Log: Merge of r4886, r4887, r4894: log variables generalization: *) Log: $apache_bytes_sent removed. It was renamed to $body_bytes_sent in nginx 0.3.10 and the old name is deprecated since then. *) Variable $bytes_sent. It replicates variable $bytes_sent as previously available in log module only. Patch by Benjamin Gr?\195?\182ssing (with minor changes). *) Variables $connection and $connection_requests. Log module counterparts are removed as they aren't used often and there is no need to preserve them for efficiency. Modified: branches/stable-1.2/ branches/stable-1.2/src/http/modules/ngx_http_log_module.c branches/stable-1.2/src/http/ngx_http_variables.c Index: branches/stable-1.2 =================================================================== --- branches/stable-1.2 2012-11-13 10:42:16 UTC (rev 4905) +++ branches/stable-1.2 2012-11-13 10:45:23 UTC (rev 4906) Property changes on: branches/stable-1.2 ___________________________________________________________________ Modified: svn:mergeinfo ## -1 +1 ## -/trunk:4611-4632,4636-4657,4671-4672,4674-4676,4682,4684-4699,4704-4706,4713,4736-4741,4754,4756-4771,4775,4777-4780,4782-4785,4795,4811-4820,4822-4824,4828-4835,4840-4844,4865-4872,4885,4890,4893,4895 +/trunk:4611-4632,4636-4657,4671-4672,4674-4676,4682,4684-4699,4704-4706,4713,4736-4741,4754,4756-4771,4775,4777-4780,4782-4785,4795,4811-4820,4822-4824,4828-4835,4840-4844,4865-4872,4885-4887,4890,4893-4895 \ No newline at end of property Modified: branches/stable-1.2/src/http/modules/ngx_http_log_module.c =================================================================== --- branches/stable-1.2/src/http/modules/ngx_http_log_module.c 2012-11-13 10:42:16 UTC (rev 4905) +++ branches/stable-1.2/src/http/modules/ngx_http_log_module.c 2012-11-13 10:45:23 UTC (rev 4906) @@ -78,10 +78,6 @@ static ssize_t ngx_http_log_script_write(ngx_http_request_t *r, ngx_http_log_script_t *script, u_char **name, u_char *buf, size_t len); -static u_char *ngx_http_log_connection(ngx_http_request_t *r, u_char *buf, - ngx_http_log_op_t *op); -static u_char *ngx_http_log_connection_requests(ngx_http_request_t *r, - u_char *buf, ngx_http_log_op_t *op); static u_char *ngx_http_log_pipe(ngx_http_request_t *r, u_char *buf, ngx_http_log_op_t *op); static u_char *ngx_http_log_time(ngx_http_request_t *r, u_char *buf, @@ -194,9 +190,6 @@ static ngx_http_log_var_t ngx_http_log_vars[] = { - { ngx_string("connection"), NGX_ATOMIC_T_LEN, ngx_http_log_connection }, - { ngx_string("connection_requests"), NGX_INT_T_LEN, - ngx_http_log_connection_requests }, { ngx_string("pipe"), 1, ngx_http_log_pipe }, { ngx_string("time_local"), sizeof("28/Sep/1970:12:00:00 +0600") - 1, ngx_http_log_time }, @@ -209,8 +202,6 @@ { ngx_string("bytes_sent"), NGX_OFF_T_LEN, ngx_http_log_bytes_sent }, { ngx_string("body_bytes_sent"), NGX_OFF_T_LEN, ngx_http_log_body_bytes_sent }, - { ngx_string("apache_bytes_sent"), NGX_OFF_T_LEN, - ngx_http_log_body_bytes_sent }, { ngx_string("request_length"), NGX_SIZE_T_LEN, ngx_http_log_request_length }, @@ -502,22 +493,6 @@ static u_char * -ngx_http_log_connection(ngx_http_request_t *r, u_char *buf, - ngx_http_log_op_t *op) -{ - return ngx_sprintf(buf, "%uA", r->connection->number); -} - - -static u_char * -ngx_http_log_connection_requests(ngx_http_request_t *r, u_char *buf, - ngx_http_log_op_t *op) -{ - return ngx_sprintf(buf, "%ui", r->connection->requests); -} - - -static u_char * ngx_http_log_pipe(ngx_http_request_t *r, u_char *buf, ngx_http_log_op_t *op) { if (r->pipeline) { @@ -1143,12 +1118,6 @@ goto invalid; } - if (ngx_strncmp(var.data, "apache_bytes_sent", 17) == 0) { - ngx_conf_log_error(NGX_LOG_WARN, cf, 0, - "use \"$body_bytes_sent\" instead of " - "\"$apache_bytes_sent\""); - } - for (v = ngx_http_log_vars; v->name.len; v++) { if (v->name.len == var.len Modified: branches/stable-1.2/src/http/ngx_http_variables.c =================================================================== --- branches/stable-1.2/src/http/ngx_http_variables.c 2012-11-13 10:42:16 UTC (rev 4905) +++ branches/stable-1.2/src/http/ngx_http_variables.c 2012-11-13 10:45:23 UTC (rev 4906) @@ -69,6 +69,8 @@ ngx_http_variable_value_t *v, uintptr_t data); static ngx_int_t ngx_http_variable_remote_user(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data); +static ngx_int_t ngx_http_variable_bytes_sent(ngx_http_request_t *r, + ngx_http_variable_value_t *v, uintptr_t data); static ngx_int_t ngx_http_variable_body_bytes_sent(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data); static ngx_int_t ngx_http_variable_request_completion(ngx_http_request_t *r, @@ -95,6 +97,11 @@ static ngx_int_t ngx_http_variable_sent_transfer_encoding(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data); +static ngx_int_t ngx_http_variable_connection(ngx_http_request_t *r, + ngx_http_variable_value_t *v, uintptr_t data); +static ngx_int_t ngx_http_variable_connection_requests(ngx_http_request_t *r, + ngx_http_variable_value_t *v, uintptr_t data); + static ngx_int_t ngx_http_variable_nginx_version(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data); static ngx_int_t ngx_http_variable_hostname(ngx_http_request_t *r, @@ -212,6 +219,9 @@ { ngx_string("remote_user"), NULL, ngx_http_variable_remote_user, 0, 0, 0 }, + { ngx_string("bytes_sent"), NULL, ngx_http_variable_bytes_sent, + 0, 0, 0 }, + { ngx_string("body_bytes_sent"), NULL, ngx_http_variable_body_bytes_sent, 0, 0, 0 }, @@ -260,6 +270,12 @@ offsetof(ngx_http_request_t, limit_rate), NGX_HTTP_VAR_CHANGEABLE|NGX_HTTP_VAR_NOCACHEABLE, 0 }, + { ngx_string("connection"), NULL, + ngx_http_variable_connection, 0, 0, 0 }, + + { ngx_string("connection_requests"), NULL, + ngx_http_variable_connection_requests, 0, 0, 0 }, + { ngx_string("nginx_version"), NULL, ngx_http_variable_nginx_version, 0, 0, 0 }, @@ -1434,6 +1450,27 @@ static ngx_int_t +ngx_http_variable_bytes_sent(ngx_http_request_t *r, + ngx_http_variable_value_t *v, uintptr_t data) +{ + u_char *p; + + p = ngx_pnalloc(r->pool, NGX_OFF_T_LEN); + if (p == NULL) { + return NGX_ERROR; + } + + v->len = ngx_sprintf(p, "%O", r->connection->sent) - p; + v->valid = 1; + v->no_cacheable = 0; + v->not_found = 0; + v->data = p; + + return NGX_OK; +} + + +static ngx_int_t ngx_http_variable_body_bytes_sent(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data) { @@ -1788,6 +1825,48 @@ static ngx_int_t +ngx_http_variable_connection(ngx_http_request_t *r, + ngx_http_variable_value_t *v, uintptr_t data) +{ + u_char *p; + + p = ngx_pnalloc(r->pool, NGX_ATOMIC_T_LEN); + if (p == NULL) { + return NGX_ERROR; + } + + v->len = ngx_sprintf(p, "%uA", r->connection->number) - p; + v->valid = 1; + v->no_cacheable = 0; + v->not_found = 0; + v->data = p; + + return NGX_OK; +} + + +static ngx_int_t +ngx_http_variable_connection_requests(ngx_http_request_t *r, + ngx_http_variable_value_t *v, uintptr_t data) +{ + u_char *p; + + p = ngx_pnalloc(r->pool, NGX_INT_T_LEN); + if (p == NULL) { + return NGX_ERROR; + } + + v->len = ngx_sprintf(p, "%ui", r->connection->requests) - p; + v->valid = 1; + v->no_cacheable = 0; + v->not_found = 0; + v->data = p; + + return NGX_OK; +} + + +static ngx_int_t ngx_http_variable_nginx_version(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data) { From mdounin at mdounin.ru Tue Nov 13 11:00:37 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Tue, 13 Nov 2012 11:00:37 +0000 Subject: [nginx] svn commit: r4907 - in branches/stable-1.2: . src/core Message-ID: <20121113110037.8BC413F9C50@mail.nginx.com> Author: mdounin Date: 2012-11-13 11:00:37 +0000 (Tue, 13 Nov 2012) New Revision: 4907 URL: http://trac.nginx.org/nginx/changeset/4907/nginx Log: Merge of r4891: worker_processes auto. Core: the "auto" parameter of the "worker_processes" directive. The parameter will set the number of worker processes to the autodetected number of available CPU cores. Modified: branches/stable-1.2/ branches/stable-1.2/src/core/nginx.c Index: branches/stable-1.2 =================================================================== --- branches/stable-1.2 2012-11-13 10:45:23 UTC (rev 4906) +++ branches/stable-1.2 2012-11-13 11:00:37 UTC (rev 4907) Property changes on: branches/stable-1.2 ___________________________________________________________________ Modified: svn:mergeinfo ## -1 +1 ## -/trunk:4611-4632,4636-4657,4671-4672,4674-4676,4682,4684-4699,4704-4706,4713,4736-4741,4754,4756-4771,4775,4777-4780,4782-4785,4795,4811-4820,4822-4824,4828-4835,4840-4844,4865-4872,4885-4887,4890,4893-4895 +/trunk:4611-4632,4636-4657,4671-4672,4674-4676,4682,4684-4699,4704-4706,4713,4736-4741,4754,4756-4771,4775,4777-4780,4782-4785,4795,4811-4820,4822-4824,4828-4835,4840-4844,4865-4872,4885-4887,4890-4891,4893-4895 \ No newline at end of property Modified: branches/stable-1.2/src/core/nginx.c =================================================================== --- branches/stable-1.2/src/core/nginx.c 2012-11-13 10:45:23 UTC (rev 4906) +++ branches/stable-1.2/src/core/nginx.c 2012-11-13 11:00:37 UTC (rev 4907) @@ -21,6 +21,8 @@ static char *ngx_set_priority(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); static char *ngx_set_cpu_affinity(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); +static char *ngx_set_worker_processes(ngx_conf_t *cf, ngx_command_t *cmd, + void *conf); static ngx_conf_enum_t ngx_debug_points[] = { @@ -69,9 +71,9 @@ { ngx_string("worker_processes"), NGX_MAIN_CONF|NGX_DIRECT_CONF|NGX_CONF_TAKE1, - ngx_conf_set_num_slot, + ngx_set_worker_processes, 0, - offsetof(ngx_core_conf_t, worker_processes), + 0, NULL }, { ngx_string("debug_points"), @@ -1329,3 +1331,32 @@ return ccf->cpu_affinity[ccf->cpu_affinity_n - 1]; } + + +static char * +ngx_set_worker_processes(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) +{ + ngx_str_t *value; + ngx_core_conf_t *ccf; + + ccf = (ngx_core_conf_t *) conf; + + if (ccf->worker_processes != NGX_CONF_UNSET) { + return "is duplicate"; + } + + value = (ngx_str_t *) cf->args->elts; + + if (ngx_strcmp(value[1].data, "auto") == 0) { + ccf->worker_processes = ngx_ncpu; + return NGX_CONF_OK; + } + + ccf->worker_processes = ngx_atoi(value[1].data, value[1].len); + + if (ccf->worker_processes == NGX_ERROR) { + return "invalid value"; + } + + return NGX_CONF_OK; +} From mdounin at mdounin.ru Tue Nov 13 11:21:32 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Tue, 13 Nov 2012 11:21:32 +0000 Subject: [nginx] svn commit: r4908 - in branches/stable-1.2: . src/http Message-ID: <20121113112132.726E03FA204@mail.nginx.com> Author: mdounin Date: 2012-11-13 11:21:31 +0000 (Tue, 13 Nov 2012) New Revision: 4908 URL: http://trac.nginx.org/nginx/changeset/4908/nginx Log: Merge of r4892: keepalive memory usage optimization. The ngx_http_keepalive_handler() function is now trying to not keep c->buffer's memory for idle connections. This behaviour is consistent with the ngx_http_set_keepalive() function and it should decrease memory usage in some cases (especially if epoll/rtsig is used). Modified: branches/stable-1.2/ branches/stable-1.2/src/http/ngx_http_request.c Index: branches/stable-1.2 =================================================================== --- branches/stable-1.2 2012-11-13 11:00:37 UTC (rev 4907) +++ branches/stable-1.2 2012-11-13 11:21:31 UTC (rev 4908) Property changes on: branches/stable-1.2 ___________________________________________________________________ Modified: svn:mergeinfo ## -1 +1 ## -/trunk:4611-4632,4636-4657,4671-4672,4674-4676,4682,4684-4699,4704-4706,4713,4736-4741,4754,4756-4771,4775,4777-4780,4782-4785,4795,4811-4820,4822-4824,4828-4835,4840-4844,4865-4872,4885-4887,4890-4891,4893-4895 +/trunk:4611-4632,4636-4657,4671-4672,4674-4676,4682,4684-4699,4704-4706,4713,4736-4741,4754,4756-4771,4775,4777-4780,4782-4785,4795,4811-4820,4822-4824,4828-4835,4840-4844,4865-4872,4885-4887,4890-4895 \ No newline at end of property Modified: branches/stable-1.2/src/http/ngx_http_request.c =================================================================== --- branches/stable-1.2/src/http/ngx_http_request.c 2012-11-13 11:00:37 UTC (rev 4907) +++ branches/stable-1.2/src/http/ngx_http_request.c 2012-11-13 11:21:31 UTC (rev 4908) @@ -2745,6 +2745,20 @@ ngx_http_close_connection(c); } + /* + * Like ngx_http_set_keepalive() we are trying to not hold + * c->buffer's memory for a keepalive connection. + */ + + if (ngx_pfree(c->pool, b->start) == NGX_OK) { + + /* + * the special note that c->buffer's memory was freed + */ + + b->pos = NULL; + } + return; } From mdounin at mdounin.ru Tue Nov 13 11:24:14 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Tue, 13 Nov 2012 11:24:14 +0000 Subject: [nginx] svn commit: r4909 - in branches/stable-1.2: . src/event src/http Message-ID: <20121113112414.6C5673F9F35@mail.nginx.com> Author: mdounin Date: 2012-11-13 11:24:14 +0000 (Tue, 13 Nov 2012) New Revision: 4909 URL: http://trac.nginx.org/nginx/changeset/4909/nginx Log: Merge of r4896: event pipe: fixed handling of buf_to_file data. Input filter might free a buffer if there is no data in it, and in case of first buffer (used for cache header and request header, aka p->buf_to_file) this resulted in cache corruption. Buffer memory was reused to read upstream response before headers were written to disk. Fix is to avoid moving pointers in ngx_event_pipe_add_free_buf() to a buffer start if we were asked to free a buffer used by p->buf_to_file. This fixes occasional cache file corruption, usually resulted in "cache file ... has md5 collision" alerts. Reported by Anatoli Marinov. Modified: branches/stable-1.2/ branches/stable-1.2/src/event/ngx_event_pipe.c branches/stable-1.2/src/http/ngx_http_upstream.c Index: branches/stable-1.2 =================================================================== --- branches/stable-1.2 2012-11-13 11:21:31 UTC (rev 4908) +++ branches/stable-1.2 2012-11-13 11:24:14 UTC (rev 4909) Property changes on: branches/stable-1.2 ___________________________________________________________________ Modified: svn:mergeinfo ## -1 +1 ## -/trunk:4611-4632,4636-4657,4671-4672,4674-4676,4682,4684-4699,4704-4706,4713,4736-4741,4754,4756-4771,4775,4777-4780,4782-4785,4795,4811-4820,4822-4824,4828-4835,4840-4844,4865-4872,4885-4887,4890-4895 +/trunk:4611-4632,4636-4657,4671-4672,4674-4676,4682,4684-4699,4704-4706,4713,4736-4741,4754,4756-4771,4775,4777-4780,4782-4785,4795,4811-4820,4822-4824,4828-4835,4840-4844,4865-4872,4885-4887,4890-4896 \ No newline at end of property Modified: branches/stable-1.2/src/event/ngx_event_pipe.c =================================================================== --- branches/stable-1.2/src/event/ngx_event_pipe.c 2012-11-13 11:21:31 UTC (rev 4908) +++ branches/stable-1.2/src/event/ngx_event_pipe.c 2012-11-13 11:24:14 UTC (rev 4909) @@ -946,8 +946,15 @@ return NGX_ERROR; } - b->pos = b->start; - b->last = b->start; + if (p->buf_to_file && b->start == p->buf_to_file->start) { + b->pos = p->buf_to_file->last; + b->last = p->buf_to_file->last; + + } else { + b->pos = b->start; + b->last = b->start; + } + b->shadow = NULL; cl->buf = b; Modified: branches/stable-1.2/src/http/ngx_http_upstream.c =================================================================== --- branches/stable-1.2/src/http/ngx_http_upstream.c 2012-11-13 11:21:31 UTC (rev 4908) +++ branches/stable-1.2/src/http/ngx_http_upstream.c 2012-11-13 11:24:14 UTC (rev 4909) @@ -2287,6 +2287,7 @@ return; } + p->buf_to_file->start = u->buffer.start; p->buf_to_file->pos = u->buffer.start; p->buf_to_file->last = u->buffer.pos; p->buf_to_file->temporary = 1; From mdounin at mdounin.ru Tue Nov 13 13:34:59 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Tue, 13 Nov 2012 13:34:59 +0000 Subject: [nginx] svn commit: r4910 - branches/stable-1.2/docs/xml/nginx Message-ID: <20121113133459.A298C3F9EA7@mail.nginx.com> Author: mdounin Date: 2012-11-13 13:34:59 +0000 (Tue, 13 Nov 2012) New Revision: 4910 URL: http://trac.nginx.org/nginx/changeset/4910/nginx Log: nginx-1.2.5-RELEASE Modified: branches/stable-1.2/docs/xml/nginx/changes.xml Modified: branches/stable-1.2/docs/xml/nginx/changes.xml =================================================================== --- branches/stable-1.2/docs/xml/nginx/changes.xml 2012-11-13 11:24:14 UTC (rev 4909) +++ branches/stable-1.2/docs/xml/nginx/changes.xml 2012-11-13 13:34:59 UTC (rev 4910) @@ -5,6 +5,75 @@ + + + + +???????? optional_no_ca ????????? ssl_verify_client.
+??????? ??????? ????????? ? Eric O'Connor. +
+ +the "optional_no_ca" parameter of the "ssl_verify_client" directive.
+Thanks to Mike Kazantsev and Eric O'Connor. +
+
+ + + +?????????? $bytes_sent, $connection ? $connection_requests +?????? ????? ???????????? ?? ?????? ? ????????? log_format.
+??????? Benjamin Gr?ssing. +
+ +the $bytes_sent, $connection, and $connection_requests variables +can now be used not only in the "log_format" directive.
+Thanks to Benjamin Gr?ssing. +
+
+ + + +?????? resolver ????????? ??????? ?????? ??????? +???????????? ?????????????? ???????.
+??????? ?????? ??????. +
+ +resolver now randomly rotates addresses +returned from cache.
+Thanks to Anton Jouline. +
+
+ + + +???????? auto ????????? worker_processes. + + +the "auto" parameter of the "worker_processes" directive. + + + + + +????????? "cache file ... has md5 collision". + + +"cache file ... has md5 collision" alert. + + + + + +????????????? ? OpenSSL 0.9.7. + + +OpenSSL 0.9.7 compatibility. + + + +
+ + From mdounin at mdounin.ru Tue Nov 13 13:35:19 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Tue, 13 Nov 2012 13:35:19 +0000 Subject: [nginx] svn commit: r4911 - in tags: . release-1.2.5 Message-ID: <20121113133519.B38C73F9F47@mail.nginx.com> Author: mdounin Date: 2012-11-13 13:35:19 +0000 (Tue, 13 Nov 2012) New Revision: 4911 URL: http://trac.nginx.org/nginx/changeset/4911/nginx Log: release-1.2.5 tag Added: tags/release-1.2.5/ Index: tags/release-1.2.5 =================================================================== --- branches/stable-1.2 2012-11-13 13:34:59 UTC (rev 4910) +++ tags/release-1.2.5 2012-11-13 13:35:19 UTC (rev 4911) Property changes on: tags/release-1.2.5 ___________________________________________________________________ Added: svn:ignore ## -0,0 +1,14 ## +access.log +client_body_temp +fastcgi_temp +proxy_temp +scgi_temp +uwsgi_temp +GNUmakefile +Makefile +makefile +nginx +nginx.conf +nginx-*.tar.gz +objs* +tmp Added: svn:mergeinfo ## -0,0 +1 ## +/trunk:4611-4632,4636-4657,4671-4672,4674-4676,4682,4684-4699,4704-4706,4713,4736-4741,4754,4756-4771,4775,4777-4780,4782-4785,4795,4811-4820,4822-4824,4828-4835,4840-4844,4865-4872,4885-4887,4890-4896 \ No newline at end of property From mdounin at mdounin.ru Wed Nov 14 23:47:32 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 15 Nov 2012 03:47:32 +0400 Subject: image_filter enhancement In-Reply-To: References: <20121106181601.GS40452@mdounin.ru> Message-ID: <20121114234732.GC40452@mdounin.ru> Hello! On Mon, Nov 12, 2012 at 09:09:50PM +0400, ivan babrou wrote: > Here's the patch. [...] > @@ -150,6 +159,13 @@ static ngx_command_t ngx_http_image_filter_commands[] = { > offsetof(ngx_http_image_filter_conf_t, buffer_size), > NULL }, > > + { ngx_string("image_filter_offset"), > + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE2, > + ngx_http_image_filter_offset, > + NGX_HTTP_LOC_CONF_OFFSET, > + 0, > + NULL }, > + > ngx_null_command > }; >From here it looks like the patch needs more work. Though functionality proposed is probably needed, as it's second patch for the crop offset posted this year. See here for the one posted previously by Maxim Bublis: http://mailman.nginx.org/pipermail/nginx-devel/2012-April/002156.html Maxim's aproach is a bit better in some aspectes as a) it calls crop offset as crop offset, and b) it allows to configure crop offsets via variables. On the other hand, his patch asks for cleaner code and more user-friendly configuration, as "left top" is much easier to understand compared to "0 0". Overral, it will be probably fine to have both Maxim Bublis's and your patches merged to have powerfull yet simple to configure functionality. From user perspective it would be fine to have syntax similar to background-position CSS property (http://www.w3.org/TR/CSS21/colors.html#propdef-background-position). > > @@ -911,14 +927,12 @@ transparent: > > if ((ngx_uint_t) dx > ctx->max_width) { > ox = dx - ctx->max_width; > - > } else { > ox = 0; > } > > if ((ngx_uint_t) dy > ctx->max_height) { > oy = dy - ctx->max_height; > - > } else { > oy = 0; > } Just a side note: please avoid unrelated changes. [...] > @@ -1078,6 +1101,7 @@ ngx_http_image_out(ngx_http_request_t *r, > ngx_uint_t type, gdImagePtr img, > > out = NULL; > > + > switch (type) { > > case NGX_HTTP_IMAGE_JPEG: Same here. [...] -- Maxim Dounin http://nginx.com/support.html From ru at nginx.com Fri Nov 16 07:49:42 2012 From: ru at nginx.com (ru at nginx.com) Date: Fri, 16 Nov 2012 07:49:42 +0000 Subject: [nginx] svn commit: r4912 - in trunk/src: core http/modules/perl Message-ID: <20121116074942.1B81E3F9FA6@mail.nginx.com> Author: ru Date: 2012-11-16 07:49:41 +0000 (Fri, 16 Nov 2012) New Revision: 4912 URL: http://trac.nginx.org/nginx/changeset/4912/nginx Log: Version bump. Modified: trunk/src/core/nginx.h trunk/src/http/modules/perl/nginx.pm Modified: trunk/src/core/nginx.h =================================================================== --- trunk/src/core/nginx.h 2012-11-13 13:35:19 UTC (rev 4911) +++ trunk/src/core/nginx.h 2012-11-16 07:49:41 UTC (rev 4912) @@ -9,8 +9,8 @@ #define _NGINX_H_INCLUDED_ -#define nginx_version 1003008 -#define NGINX_VERSION "1.3.8" +#define nginx_version 1003009 +#define NGINX_VERSION "1.3.9" #define NGINX_VER "nginx/" NGINX_VERSION #define NGINX_VAR "NGINX" Modified: trunk/src/http/modules/perl/nginx.pm =================================================================== --- trunk/src/http/modules/perl/nginx.pm 2012-11-13 13:35:19 UTC (rev 4911) +++ trunk/src/http/modules/perl/nginx.pm 2012-11-16 07:49:41 UTC (rev 4912) @@ -50,7 +50,7 @@ HTTP_INSUFFICIENT_STORAGE ); -our $VERSION = '1.3.8'; +our $VERSION = '1.3.9'; require XSLoader; XSLoader::load('nginx', $VERSION); From ru at nginx.com Fri Nov 16 09:25:52 2012 From: ru at nginx.com (ru at nginx.com) Date: Fri, 16 Nov 2012 09:25:52 +0000 Subject: [nginx] svn commit: r4913 - trunk/src/os/unix Message-ID: <20121116092552.AB9183F9C0C@mail.nginx.com> Author: ru Date: 2012-11-16 09:25:52 +0000 (Fri, 16 Nov 2012) New Revision: 4913 URL: http://trac.nginx.org/nginx/changeset/4913/nginx Log: Fixed setting of CPU affinity on respawn of dead worker processes. Worker processes are now made aware of their sequential number needed to select CPU affinity mask. This replaces a workaround from r4865. Modified: trunk/src/os/unix/ngx_process_cycle.c Modified: trunk/src/os/unix/ngx_process_cycle.c =================================================================== --- trunk/src/os/unix/ngx_process_cycle.c 2012-11-16 07:49:41 UTC (rev 4912) +++ trunk/src/os/unix/ngx_process_cycle.c 2012-11-16 09:25:52 UTC (rev 4913) @@ -20,7 +20,7 @@ static ngx_uint_t ngx_reap_children(ngx_cycle_t *cycle); static void ngx_master_process_exit(ngx_cycle_t *cycle); static void ngx_worker_process_cycle(ngx_cycle_t *cycle, void *data); -static void ngx_worker_process_init(ngx_cycle_t *cycle, ngx_uint_t priority); +static void ngx_worker_process_init(ngx_cycle_t *cycle, ngx_int_t worker); static void ngx_worker_process_exit(ngx_cycle_t *cycle); static void ngx_channel_handler(ngx_event_t *ev); #if (NGX_THREADS) @@ -62,7 +62,6 @@ #endif -uint64_t cpu_affinity; static u_char master_process[] = "master process"; @@ -360,19 +359,15 @@ for (i = 0; i < n; i++) { - cpu_affinity = ngx_get_cpu_affinity(i); + ngx_spawn_process(cycle, ngx_worker_process_cycle, + (void *) (intptr_t) i, "worker process", type); - ngx_spawn_process(cycle, ngx_worker_process_cycle, NULL, - "worker process", type); - ch.pid = ngx_processes[ngx_process_slot].pid; ch.slot = ngx_process_slot; ch.fd = ngx_processes[ngx_process_slot].channel[0]; ngx_pass_open_channel(cycle, &ch); } - - cpu_affinity = 0; } @@ -726,12 +721,14 @@ static void ngx_worker_process_cycle(ngx_cycle_t *cycle, void *data) { + ngx_int_t worker = (intptr_t) data; + ngx_uint_t i; ngx_connection_t *c; ngx_process = NGX_PROCESS_WORKER; - ngx_worker_process_init(cycle, 1); + ngx_worker_process_init(cycle, worker); ngx_setproctitle("worker process"); @@ -837,9 +834,10 @@ static void -ngx_worker_process_init(ngx_cycle_t *cycle, ngx_uint_t priority) +ngx_worker_process_init(ngx_cycle_t *cycle, ngx_int_t worker) { sigset_t set; + uint64_t cpu_affinity; ngx_int_t n; ngx_uint_t i; struct rlimit rlmt; @@ -853,7 +851,7 @@ ccf = (ngx_core_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_core_module); - if (priority && ccf->priority != 0) { + if (worker >= 0 && ccf->priority != 0) { if (setpriority(PRIO_PROCESS, 0, ccf->priority) == -1) { ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno, "setpriority(%d) failed", ccf->priority); @@ -917,8 +915,12 @@ } } - if (cpu_affinity) { - ngx_setaffinity(cpu_affinity, cycle->log); + if (worker >= 0) { + cpu_affinity = ngx_get_cpu_affinity(worker); + + if (cpu_affinity) { + ngx_setaffinity(cpu_affinity, cycle->log); + } } #if (NGX_HAVE_PR_SET_DUMPABLE) @@ -1298,7 +1300,7 @@ ngx_process = NGX_PROCESS_HELPER; - ngx_worker_process_init(cycle, 0); + ngx_worker_process_init(cycle, -1); ngx_close_listening_sockets(cycle); From ru at nginx.com Fri Nov 16 09:37:15 2012 From: ru at nginx.com (ru at nginx.com) Date: Fri, 16 Nov 2012 09:37:15 +0000 Subject: [nginx] svn commit: r4914 - trunk/src/http Message-ID: <20121116093715.4CCDD3F9FEB@mail.nginx.com> Author: ru Date: 2012-11-16 09:37:14 +0000 (Fri, 16 Nov 2012) New Revision: 4914 URL: http://trac.nginx.org/nginx/changeset/4914/nginx Log: Variables $request_time and $msec. Log module counterparts are preserved for efficiency. Modified: trunk/src/http/ngx_http_variables.c Modified: trunk/src/http/ngx_http_variables.c =================================================================== --- trunk/src/http/ngx_http_variables.c 2012-11-16 09:25:52 UTC (rev 4913) +++ trunk/src/http/ngx_http_variables.c 2012-11-16 09:37:14 UTC (rev 4914) @@ -79,6 +79,8 @@ ngx_http_variable_value_t *v, uintptr_t data); static ngx_int_t ngx_http_variable_request_body_file(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data); +static ngx_int_t ngx_http_variable_request_time(ngx_http_request_t *r, + ngx_http_variable_value_t *v, uintptr_t data); static ngx_int_t ngx_http_variable_status(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data); @@ -108,6 +110,8 @@ ngx_http_variable_value_t *v, uintptr_t data); static ngx_int_t ngx_http_variable_pid(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data); +static ngx_int_t ngx_http_variable_msec(ngx_http_request_t *r, + ngx_http_variable_value_t *v, uintptr_t data); /* * TODO: @@ -237,6 +241,9 @@ ngx_http_variable_request_body_file, 0, 0, 0 }, + { ngx_string("request_time"), NULL, ngx_http_variable_request_time, + 0, NGX_HTTP_VAR_NOCACHEABLE, 0 }, + { ngx_string("status"), NULL, ngx_http_variable_status, 0, NGX_HTTP_VAR_NOCACHEABLE, 0 }, @@ -285,6 +292,9 @@ { ngx_string("pid"), NULL, ngx_http_variable_pid, 0, 0, 0 }, + { ngx_string("msec"), NULL, ngx_http_variable_msec, + 0, NGX_HTTP_VAR_NOCACHEABLE, 0 }, + #if (NGX_HAVE_TCP_INFO) { ngx_string("tcpinfo_rtt"), NULL, ngx_http_variable_tcpinfo, 0, NGX_HTTP_VAR_NOCACHEABLE, 0 }, @@ -1825,6 +1835,35 @@ static ngx_int_t +ngx_http_variable_request_time(ngx_http_request_t *r, + ngx_http_variable_value_t *v, uintptr_t data) +{ + u_char *p; + ngx_time_t *tp; + ngx_msec_int_t ms; + + p = ngx_pnalloc(r->pool, NGX_TIME_T_LEN + 4); + if (p == NULL) { + return NGX_ERROR; + } + + tp = ngx_timeofday(); + + ms = (ngx_msec_int_t) + ((tp->sec - r->start_sec) * 1000 + (tp->msec - r->start_msec)); + ms = ngx_max(ms, 0); + + v->len = ngx_sprintf(p, "%T.%03M", ms / 1000, ms % 1000) - p; + v->valid = 1; + v->no_cacheable = 0; + v->not_found = 0; + v->data = p; + + return NGX_OK; +} + + +static ngx_int_t ngx_http_variable_connection(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data) { @@ -1915,6 +1954,30 @@ } +static ngx_int_t +ngx_http_variable_msec(ngx_http_request_t *r, + ngx_http_variable_value_t *v, uintptr_t data) +{ + u_char *p; + ngx_time_t *tp; + + p = ngx_pnalloc(r->pool, NGX_TIME_T_LEN + 4); + if (p == NULL) { + return NGX_ERROR; + } + + tp = ngx_timeofday(); + + v->len = ngx_sprintf(p, "%T.%03M", tp->sec, tp->msec) - p; + v->valid = 1; + v->no_cacheable = 0; + v->not_found = 0; + v->data = p; + + return NGX_OK; +} + + void * ngx_http_map_find(ngx_http_request_t *r, ngx_http_map_t *map, ngx_str_t *match) { From mdounin at mdounin.ru Fri Nov 16 11:02:22 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 16 Nov 2012 15:02:22 +0400 Subject: [PATCH 00 of 13] chunked request body support Message-ID: Hello! Here is patch series with chunked request body support, and some minor related fixes/enhancements. Review and testing appreciated. Full patch is available here: http://nginx.org/patches/chunked/ -- Maxim Dounin From mdounin at mdounin.ru Fri Nov 16 11:02:23 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 16 Nov 2012 15:02:23 +0400 Subject: [PATCH 01 of 13] Dav: fixed segfault on PUT if body was already read (ticket #238) In-Reply-To: References: Message-ID: # HG changeset patch # User Maxim Dounin # Date 1352393278 -14400 # Node ID c26606971d58dd27df5f72b9fcf90bc883038d76 # Parent 1b2abbd52edc283f69c7513a6cb5406f7913ecae Dav: fixed segfault on PUT if body was already read (ticket #238). If request body reading happens with different options it's possible that there will be no r->request_body->temp_file available (or even no r->request_body available if body was discarded). Return internal server error in this case instead of committing suicide by dereferencing a null pointer. diff --git a/src/http/modules/ngx_http_dav_module.c b/src/http/modules/ngx_http_dav_module.c --- a/src/http/modules/ngx_http_dav_module.c +++ b/src/http/modules/ngx_http_dav_module.c @@ -209,6 +209,11 @@ ngx_http_dav_put_handler(ngx_http_reques ngx_ext_rename_file_t ext; ngx_http_dav_loc_conf_t *dlcf; + if (r->request_body == NULL || r->request_body->temp_file == NULL) { + ngx_http_finalize_request(r, NGX_HTTP_INTERNAL_SERVER_ERROR); + return; + } + ngx_http_map_uri_to_path(r, &path, &root, 0); path.len--; From mdounin at mdounin.ru Fri Nov 16 11:02:24 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 16 Nov 2012 15:02:24 +0400 Subject: [PATCH 02 of 13] Core: added debug logging of writev() in ngx_write_chain_to_file() In-Reply-To: References: Message-ID: <52c683ed3d912230d21e.1353063744@vm-bsd.mdounin.ru> # HG changeset patch # User Maxim Dounin # Date 1352393278 -14400 # Node ID 52c683ed3d912230d21e3c058517ed3b3acdaee7 # Parent c26606971d58dd27df5f72b9fcf90bc883038d76 Core: added debug logging of writev() in ngx_write_chain_to_file(). diff --git a/src/os/unix/ngx_files.c b/src/os/unix/ngx_files.c --- a/src/os/unix/ngx_files.c +++ b/src/os/unix/ngx_files.c @@ -241,6 +241,9 @@ ngx_write_chain_to_file(ngx_file_t *file return NGX_ERROR; } + ngx_log_debug2(NGX_LOG_DEBUG_CORE, file->log, 0, + "writev: %d, %z", file->fd, n); + file->sys_offset += n; file->offset += n; total += n; From mdounin at mdounin.ru Fri Nov 16 11:02:25 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 16 Nov 2012 15:02:25 +0400 Subject: [PATCH 03 of 13] Request body: fixed "501 Not Implemented" error handling In-Reply-To: References: Message-ID: <72ea28f498c763865667.1353063745@vm-bsd.mdounin.ru> # HG changeset patch # User Maxim Dounin # Date 1352393278 -14400 # Node ID 72ea28f498c7638656676c32e17abcf695b1ff32 # Parent 52c683ed3d912230d21e3c058517ed3b3acdaee7 Request body: fixed "501 Not Implemented" error handling. It is not about "Method" but a generic message, and is expected to be used e.g. if specified Transfer-Encoding is not supported. Fixed message to match RFC 2616. Additionally, disable keepalive on such errors as we won't be able to read request body correctly if we don't understand Transfer-Encoding used. diff --git a/src/http/ngx_http_header_filter_module.c b/src/http/ngx_http_header_filter_module.c --- a/src/http/ngx_http_header_filter_module.c +++ b/src/http/ngx_http_header_filter_module.c @@ -112,7 +112,7 @@ static ngx_str_t ngx_http_status_lines[] #define NGX_HTTP_OFF_5XX (NGX_HTTP_LAST_4XX - 400 + NGX_HTTP_OFF_4XX) ngx_string("500 Internal Server Error"), - ngx_string("501 Method Not Implemented"), + ngx_string("501 Not Implemented"), ngx_string("502 Bad Gateway"), ngx_string("503 Service Temporarily Unavailable"), ngx_string("504 Gateway Time-out"), diff --git a/src/http/ngx_http_special_response.c b/src/http/ngx_http_special_response.c --- a/src/http/ngx_http_special_response.c +++ b/src/http/ngx_http_special_response.c @@ -260,9 +260,9 @@ static char ngx_http_error_500_page[] = static char ngx_http_error_501_page[] = "" CRLF -"501 Method Not Implemented" CRLF +"501 Not Implemented" CRLF "" CRLF -"

501 Method Not Implemented

" CRLF +"

501 Not Implemented

" CRLF ; @@ -384,6 +384,7 @@ ngx_http_special_response_handler(ngx_ht case NGX_HTTPS_CERT_ERROR: case NGX_HTTPS_NO_CERT: case NGX_HTTP_INTERNAL_SERVER_ERROR: + case NGX_HTTP_NOT_IMPLEMENTED: r->keepalive = 0; } } From mdounin at mdounin.ru Fri Nov 16 11:02:26 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 16 Nov 2012 15:02:26 +0400 Subject: [PATCH 04 of 13] Request body: $request_body variable generalization In-Reply-To: References: Message-ID: # HG changeset patch # User Maxim Dounin # Date 1352393278 -14400 # Node ID a1c71119ec4c40d0410a6575c583bd65ac6cb690 # Parent 72ea28f498c7638656676c32e17abcf695b1ff32 Request body: $request_body variable generalization. The $request_body variable was assuming there can't be more than two buffers. While this is currently true due to request body reading implementation details, this is not a good thing to depend on and may change in the future. diff --git a/src/http/ngx_http_variables.c b/src/http/ngx_http_variables.c --- a/src/http/ngx_http_variables.c +++ b/src/http/ngx_http_variables.c @@ -1767,7 +1767,7 @@ ngx_http_variable_request_body(ngx_http_ { u_char *p; size_t len; - ngx_buf_t *buf, *next; + ngx_buf_t *buf; ngx_chain_t *cl; if (r->request_body == NULL @@ -1792,8 +1792,13 @@ ngx_http_variable_request_body(ngx_http_ return NGX_OK; } - next = cl->next->buf; - len = (buf->last - buf->pos) + (next->last - next->pos); + len = buf->last - buf->pos; + cl = cl->next; + + for ( /* void */ ; cl; cl = cl->next) { + buf = cl->buf; + len += buf->last - buf->pos; + } p = ngx_pnalloc(r->pool, len); if (p == NULL) { @@ -1801,9 +1806,12 @@ ngx_http_variable_request_body(ngx_http_ } v->data = p; - - p = ngx_cpymem(p, buf->pos, buf->last - buf->pos); - ngx_memcpy(p, next->pos, next->last - next->pos); + cl = r->request_body->bufs; + + for ( /* void */ ; cl; cl = cl->next) { + buf = cl->buf; + p = ngx_cpymem(p, buf->pos, buf->last - buf->pos); + } v->len = len; v->valid = 1; From mdounin at mdounin.ru Fri Nov 16 11:02:27 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 16 Nov 2012 15:02:27 +0400 Subject: [PATCH 05 of 13] Request body: code duplication reduced, no functional changes In-Reply-To: References: Message-ID: # HG changeset patch # User Maxim Dounin # Date 1352393278 -14400 # Node ID dd6b3c1ecee2ca6fdd21b4645dec00639b22a0ae # Parent a1c71119ec4c40d0410a6575c583bd65ac6cb690 Request body: code duplication reduced, no functional changes. The r->request_body_in_file_only with empty body case is now handled in ngx_http_write_request_body(). diff --git a/src/http/ngx_http_request_body.c b/src/http/ngx_http_request_body.c --- a/src/http/ngx_http_request_body.c +++ b/src/http/ngx_http_request_body.c @@ -33,7 +33,6 @@ ngx_http_read_client_request_body(ngx_ht ssize_t size; ngx_buf_t *b; ngx_chain_t *cl, **next; - ngx_temp_file_t *tf; ngx_http_request_body_t *rb; ngx_http_core_loc_conf_t *clcf; @@ -65,30 +64,7 @@ ngx_http_read_client_request_body(ngx_ht if (r->headers_in.content_length_n == 0) { if (r->request_body_in_file_only) { - tf = ngx_pcalloc(r->pool, sizeof(ngx_temp_file_t)); - if (tf == NULL) { - return NGX_HTTP_INTERNAL_SERVER_ERROR; - } - - tf->file.fd = NGX_INVALID_FILE; - tf->file.log = r->connection->log; - tf->path = clcf->client_body_temp_path; - tf->pool = r->pool; - tf->warn = "a client request body is buffered to a temporary file"; - tf->log_level = r->request_body_file_log_level; - tf->persistent = r->request_body_in_persistent_file; - tf->clean = r->request_body_in_clean_file; - - if (r->request_body_file_group_access) { - tf->access = 0660; - } - - rb->temp_file = tf; - - if (ngx_create_temp_file(&tf->file, tf->path, tf->pool, - tf->persistent, tf->clean, tf->access) - != NGX_OK) - { + if (ngx_http_write_request_body(r, NULL) != NGX_OK) { return NGX_HTTP_INTERNAL_SERVER_ERROR; } } @@ -419,6 +395,19 @@ ngx_http_write_request_body(ngx_http_req } rb->temp_file = tf; + + if (body == NULL) { + /* empty body with r->request_body_in_file_only */ + + if (ngx_create_temp_file(&tf->file, tf->path, tf->pool, + tf->persistent, tf->clean, tf->access) + != NGX_OK) + { + return NGX_ERROR; + } + + return NGX_OK; + } } n = ngx_write_chain_to_temp_file(rb->temp_file, body); From mdounin at mdounin.ru Fri Nov 16 11:02:28 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 16 Nov 2012 15:02:28 +0400 Subject: [PATCH 06 of 13] Request body: fixed socket leak on errors In-Reply-To: References: Message-ID: <49ecf88ba2e6bf559f5c.1353063748@vm-bsd.mdounin.ru> # HG changeset patch # User Maxim Dounin # Date 1352393422 -14400 # Node ID 49ecf88ba2e6bf559f5cc307e466b3284555ae09 # Parent dd6b3c1ecee2ca6fdd21b4645dec00639b22a0ae Request body: fixed socket leak on errors. The r->main->count reference counter was always incremented in ngx_http_read_client_request_body(), while it is only needs to be incremented on positive returns. diff --git a/src/http/ngx_http_request_body.c b/src/http/ngx_http_request_body.c --- a/src/http/ngx_http_request_body.c +++ b/src/http/ngx_http_request_body.c @@ -31,6 +31,7 @@ ngx_http_read_client_request_body(ngx_ht { size_t preread; ssize_t size; + ngx_int_t rc; ngx_buf_t *b; ngx_chain_t *cl, **next; ngx_http_request_body_t *rb; @@ -44,12 +45,14 @@ ngx_http_read_client_request_body(ngx_ht } if (ngx_http_test_expect(r) != NGX_OK) { - return NGX_HTTP_INTERNAL_SERVER_ERROR; + rc = NGX_HTTP_INTERNAL_SERVER_ERROR; + goto done; } rb = ngx_pcalloc(r->pool, sizeof(ngx_http_request_body_t)); if (rb == NULL) { - return NGX_HTTP_INTERNAL_SERVER_ERROR; + rc = NGX_HTTP_INTERNAL_SERVER_ERROR; + goto done; } r->request_body = rb; @@ -65,7 +68,8 @@ ngx_http_read_client_request_body(ngx_ht if (r->request_body_in_file_only) { if (ngx_http_write_request_body(r, NULL) != NGX_OK) { - return NGX_HTTP_INTERNAL_SERVER_ERROR; + rc = NGX_HTTP_INTERNAL_SERVER_ERROR; + goto done; } } @@ -95,7 +99,8 @@ ngx_http_read_client_request_body(ngx_ht b = ngx_calloc_buf(r->pool); if (b == NULL) { - return NGX_HTTP_INTERNAL_SERVER_ERROR; + rc = NGX_HTTP_INTERNAL_SERVER_ERROR; + goto done; } b->temporary = 1; @@ -106,7 +111,8 @@ ngx_http_read_client_request_body(ngx_ht rb->bufs = ngx_alloc_chain_link(r->pool); if (rb->bufs == NULL) { - return NGX_HTTP_INTERNAL_SERVER_ERROR; + rc = NGX_HTTP_INTERNAL_SERVER_ERROR; + goto done; } rb->bufs->buf = b; @@ -124,7 +130,8 @@ ngx_http_read_client_request_body(ngx_ht if (r->request_body_in_file_only) { if (ngx_http_write_request_body(r, rb->bufs) != NGX_OK) { - return NGX_HTTP_INTERNAL_SERVER_ERROR; + rc = NGX_HTTP_INTERNAL_SERVER_ERROR; + goto done; } } @@ -151,7 +158,8 @@ ngx_http_read_client_request_body(ngx_ht r->read_event_handler = ngx_http_read_client_request_body_handler; - return ngx_http_do_read_client_request_body(r); + rc = ngx_http_do_read_client_request_body(r); + goto done; } next = &rb->bufs->next; @@ -181,12 +189,14 @@ ngx_http_read_client_request_body(ngx_ht rb->buf = ngx_create_temp_buf(r->pool, size); if (rb->buf == NULL) { - return NGX_HTTP_INTERNAL_SERVER_ERROR; + rc = NGX_HTTP_INTERNAL_SERVER_ERROR; + goto done; } cl = ngx_alloc_chain_link(r->pool); if (cl == NULL) { - return NGX_HTTP_INTERNAL_SERVER_ERROR; + rc = NGX_HTTP_INTERNAL_SERVER_ERROR; + goto done; } cl->buf = rb->buf; @@ -211,7 +221,15 @@ ngx_http_read_client_request_body(ngx_ht r->read_event_handler = ngx_http_read_client_request_body_handler; - return ngx_http_do_read_client_request_body(r); + rc = ngx_http_do_read_client_request_body(r); + +done: + + if (rc >= NGX_HTTP_SPECIAL_RESPONSE) { + r->main->count--; + } + + return rc; } From mdounin at mdounin.ru Fri Nov 16 11:02:29 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 16 Nov 2012 15:02:29 +0400 Subject: [PATCH 07 of 13] Request body: properly handle events while discarding body In-Reply-To: References: Message-ID: # HG changeset patch # User Maxim Dounin # Date 1353062357 -14400 # Node ID cc692c8a60ad93fa9092140d601d57bd38764ff0 # Parent 49ecf88ba2e6bf559f5cc307e466b3284555ae09 Request body: properly handle events while discarding body. An attempt to call ngx_handle_read_event() before actually reading data from a socket might result in read event being disabled, which is wrong. Catched by body.t test on Solaris. diff --git a/src/http/ngx_http_request_body.c b/src/http/ngx_http_request_body.c --- a/src/http/ngx_http_request_body.c +++ b/src/http/ngx_http_request_body.c @@ -482,19 +482,21 @@ ngx_http_discard_request_body(ngx_http_r } } + if (ngx_http_read_discarded_request_body(r) == NGX_OK) { + r->lingering_close = 0; + return NGX_OK; + } + + /* == NGX_AGAIN */ + r->read_event_handler = ngx_http_discarded_request_body_handler; if (ngx_handle_read_event(rev, 0) != NGX_OK) { return NGX_HTTP_INTERNAL_SERVER_ERROR; } - if (ngx_http_read_discarded_request_body(r) == NGX_OK) { - r->lingering_close = 0; - - } else { - r->count++; - r->discard_body = 1; - } + r->count++; + r->discard_body = 1; return NGX_OK; } From mdounin at mdounin.ru Fri Nov 16 11:02:30 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 16 Nov 2012 15:02:30 +0400 Subject: [PATCH 08 of 13] Request body: chunked parsing moved to ngx_http_parse.c from proxy In-Reply-To: References: Message-ID: <9e14a7c2950b05c3f655.1353063750@vm-bsd.mdounin.ru> # HG changeset patch # User Maxim Dounin # Date 1352393278 -14400 # Node ID 9e14a7c2950b05c3f6550b4233ead43eefa57da5 # Parent cc692c8a60ad93fa9092140d601d57bd38764ff0 Request body: chunked parsing moved to ngx_http_parse.c from proxy. No functional changes. diff --git a/src/http/modules/ngx_http_proxy_module.c b/src/http/modules/ngx_http_proxy_module.c --- a/src/http/modules/ngx_http_proxy_module.c +++ b/src/http/modules/ngx_http_proxy_module.c @@ -81,13 +81,10 @@ typedef struct { typedef struct { ngx_http_status_t status; + ngx_http_chunked_t chunked; ngx_http_proxy_vars_t vars; size_t internal_body_length; - ngx_uint_t state; - off_t size; - off_t length; - ngx_uint_t head; /* unsigned head:1 */ } ngx_http_proxy_ctx_t; @@ -1252,7 +1249,7 @@ ngx_http_proxy_reinit_request(ngx_http_r ctx->status.count = 0; ctx->status.start = NULL; ctx->status.end = NULL; - ctx->state = 0; + ctx->chunked.state = 0; r->upstream->process_header = ngx_http_proxy_process_status_line; r->upstream->pipe->input_filter = ngx_http_proxy_copy_filter; @@ -1617,265 +1614,6 @@ ngx_http_proxy_copy_filter(ngx_event_pip } -static ngx_inline ngx_int_t -ngx_http_proxy_parse_chunked(ngx_http_request_t *r, ngx_buf_t *buf) -{ - u_char *pos, ch, c; - ngx_int_t rc; - ngx_http_proxy_ctx_t *ctx; - enum { - sw_chunk_start = 0, - sw_chunk_size, - sw_chunk_extension, - sw_chunk_extension_almost_done, - sw_chunk_data, - sw_after_data, - sw_after_data_almost_done, - sw_last_chunk_extension, - sw_last_chunk_extension_almost_done, - sw_trailer, - sw_trailer_almost_done, - sw_trailer_header, - sw_trailer_header_almost_done - } state; - - ctx = ngx_http_get_module_ctx(r, ngx_http_proxy_module); - - if (ctx == NULL) { - return NGX_ERROR; - } - - state = ctx->state; - - if (state == sw_chunk_data && ctx->size == 0) { - state = sw_after_data; - } - - rc = NGX_AGAIN; - - for (pos = buf->pos; pos < buf->last; pos++) { - - ch = *pos; - - ngx_log_debug2(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, - "http proxy chunked byte: %02Xd s:%d", ch, state); - - switch (state) { - - case sw_chunk_start: - if (ch >= '0' && ch <= '9') { - state = sw_chunk_size; - ctx->size = ch - '0'; - break; - } - - c = (u_char) (ch | 0x20); - - if (c >= 'a' && c <= 'f') { - state = sw_chunk_size; - ctx->size = c - 'a' + 10; - break; - } - - goto invalid; - - case sw_chunk_size: - if (ch >= '0' && ch <= '9') { - ctx->size = ctx->size * 16 + (ch - '0'); - break; - } - - c = (u_char) (ch | 0x20); - - if (c >= 'a' && c <= 'f') { - ctx->size = ctx->size * 16 + (c - 'a' + 10); - break; - } - - if (ctx->size == 0) { - - switch (ch) { - case CR: - state = sw_last_chunk_extension_almost_done; - break; - case LF: - state = sw_trailer; - break; - case ';': - case ' ': - case '\t': - state = sw_last_chunk_extension; - break; - default: - goto invalid; - } - - break; - } - - switch (ch) { - case CR: - state = sw_chunk_extension_almost_done; - break; - case LF: - state = sw_chunk_data; - break; - case ';': - case ' ': - case '\t': - state = sw_chunk_extension; - break; - default: - goto invalid; - } - - break; - - case sw_chunk_extension: - switch (ch) { - case CR: - state = sw_chunk_extension_almost_done; - break; - case LF: - state = sw_chunk_data; - } - break; - - case sw_chunk_extension_almost_done: - if (ch == LF) { - state = sw_chunk_data; - break; - } - goto invalid; - - case sw_chunk_data: - rc = NGX_OK; - goto data; - - case sw_after_data: - switch (ch) { - case CR: - state = sw_after_data_almost_done; - break; - case LF: - state = sw_chunk_start; - } - break; - - case sw_after_data_almost_done: - if (ch == LF) { - state = sw_chunk_start; - break; - } - goto invalid; - - case sw_last_chunk_extension: - switch (ch) { - case CR: - state = sw_last_chunk_extension_almost_done; - break; - case LF: - state = sw_trailer; - } - break; - - case sw_last_chunk_extension_almost_done: - if (ch == LF) { - state = sw_trailer; - break; - } - goto invalid; - - case sw_trailer: - switch (ch) { - case CR: - state = sw_trailer_almost_done; - break; - case LF: - goto done; - default: - state = sw_trailer_header; - } - break; - - case sw_trailer_almost_done: - if (ch == LF) { - goto done; - } - goto invalid; - - case sw_trailer_header: - switch (ch) { - case CR: - state = sw_trailer_header_almost_done; - break; - case LF: - state = sw_trailer; - } - break; - - case sw_trailer_header_almost_done: - if (ch == LF) { - state = sw_trailer; - break; - } - goto invalid; - - } - } - -data: - - ctx->state = state; - buf->pos = pos; - - switch (state) { - - case sw_chunk_start: - ctx->length = 3 /* "0" LF LF */; - break; - case sw_chunk_size: - ctx->length = 2 /* LF LF */ - + (ctx->size ? ctx->size + 4 /* LF "0" LF LF */ : 0); - break; - case sw_chunk_extension: - case sw_chunk_extension_almost_done: - ctx->length = 1 /* LF */ + ctx->size + 4 /* LF "0" LF LF */; - break; - case sw_chunk_data: - ctx->length = ctx->size + 4 /* LF "0" LF LF */; - break; - case sw_after_data: - case sw_after_data_almost_done: - ctx->length = 4 /* LF "0" LF LF */; - break; - case sw_last_chunk_extension: - case sw_last_chunk_extension_almost_done: - ctx->length = 2 /* LF LF */; - break; - case sw_trailer: - case sw_trailer_almost_done: - ctx->length = 1 /* LF */; - break; - case sw_trailer_header: - case sw_trailer_header_almost_done: - ctx->length = 2 /* LF LF */; - break; - - } - - return rc; - -done: - - return NGX_DONE; - -invalid: - - return NGX_ERROR; -} - - static ngx_int_t ngx_http_proxy_chunked_filter(ngx_event_pipe_t *p, ngx_buf_t *buf) { @@ -1901,7 +1639,7 @@ ngx_http_proxy_chunked_filter(ngx_event_ for ( ;; ) { - rc = ngx_http_proxy_parse_chunked(r, buf); + rc = ngx_http_parse_chunked(r, buf, &ctx->chunked); if (rc == NGX_OK) { @@ -1952,16 +1690,16 @@ ngx_http_proxy_chunked_filter(ngx_event_ ngx_log_debug2(NGX_LOG_DEBUG_EVENT, p->log, 0, "input buf #%d %p", b->num, b->pos); - if (buf->last - buf->pos >= ctx->size) { - - buf->pos += ctx->size; + if (buf->last - buf->pos >= ctx->chunked.size) { + + buf->pos += ctx->chunked.size; b->last = buf->pos; - ctx->size = 0; + ctx->chunked.size = 0; continue; } - ctx->size -= buf->last - buf->pos; + ctx->chunked.size -= buf->last - buf->pos; buf->pos = buf->last; b->last = buf->last; @@ -1982,7 +1720,7 @@ ngx_http_proxy_chunked_filter(ngx_event_ /* set p->length, minimal amount of data we want to see */ - p->length = ctx->length; + p->length = ctx->chunked.length; break; } @@ -1997,7 +1735,7 @@ ngx_http_proxy_chunked_filter(ngx_event_ ngx_log_debug2(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "http proxy chunked state %d, length %d", - ctx->state, p->length); + ctx->chunked.state, p->length); if (b) { b->shadow = buf; @@ -2094,7 +1832,7 @@ ngx_http_proxy_non_buffered_chunked_filt for ( ;; ) { - rc = ngx_http_proxy_parse_chunked(r, buf); + rc = ngx_http_parse_chunked(r, buf, &ctx->chunked); if (rc == NGX_OK) { @@ -2116,13 +1854,13 @@ ngx_http_proxy_non_buffered_chunked_filt b->pos = buf->pos; b->tag = u->output.tag; - if (buf->last - buf->pos >= ctx->size) { - buf->pos += ctx->size; + if (buf->last - buf->pos >= ctx->chunked.size) { + buf->pos += ctx->chunked.size; b->last = buf->pos; - ctx->size = 0; + ctx->chunked.size = 0; } else { - ctx->size -= buf->last - buf->pos; + ctx->chunked.size -= buf->last - buf->pos; buf->pos = buf->last; b->last = buf->last; } diff --git a/src/http/ngx_http.h b/src/http/ngx_http.h --- a/src/http/ngx_http.h +++ b/src/http/ngx_http.h @@ -18,6 +18,7 @@ typedef struct ngx_http_upstream_s ng typedef struct ngx_http_cache_s ngx_http_cache_t; typedef struct ngx_http_file_cache_s ngx_http_file_cache_t; typedef struct ngx_http_log_ctx_s ngx_http_log_ctx_t; +typedef struct ngx_http_chunked_s ngx_http_chunked_t; typedef ngx_int_t (*ngx_http_header_handler_pt)(ngx_http_request_t *r, ngx_table_elt_t *h, ngx_uint_t offset); @@ -52,6 +53,13 @@ struct ngx_http_log_ctx_s { }; +struct ngx_http_chunked_s { + ngx_uint_t state; + off_t size; + off_t length; +}; + + typedef struct { ngx_uint_t http_version; ngx_uint_t code; @@ -92,6 +100,8 @@ ngx_int_t ngx_http_arg(ngx_http_request_ ngx_str_t *value); void ngx_http_split_args(ngx_http_request_t *r, ngx_str_t *uri, ngx_str_t *args); +ngx_int_t ngx_http_parse_chunked(ngx_http_request_t *r, ngx_buf_t *b, + ngx_http_chunked_t *ctx); ngx_int_t ngx_http_find_server_conf(ngx_http_request_t *r); diff --git a/src/http/ngx_http_parse.c b/src/http/ngx_http_parse.c --- a/src/http/ngx_http_parse.c +++ b/src/http/ngx_http_parse.c @@ -1818,3 +1818,256 @@ ngx_http_split_args(ngx_http_request_t * args->len = 0; } } + + +ngx_int_t +ngx_http_parse_chunked(ngx_http_request_t *r, ngx_buf_t *b, + ngx_http_chunked_t *ctx) +{ + u_char *pos, ch, c; + ngx_int_t rc; + enum { + sw_chunk_start = 0, + sw_chunk_size, + sw_chunk_extension, + sw_chunk_extension_almost_done, + sw_chunk_data, + sw_after_data, + sw_after_data_almost_done, + sw_last_chunk_extension, + sw_last_chunk_extension_almost_done, + sw_trailer, + sw_trailer_almost_done, + sw_trailer_header, + sw_trailer_header_almost_done + } state; + + state = ctx->state; + + if (state == sw_chunk_data && ctx->size == 0) { + state = sw_after_data; + } + + rc = NGX_AGAIN; + + for (pos = b->pos; pos < b->last; pos++) { + + ch = *pos; + + ngx_log_debug2(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "http chunked byte: %02Xd s:%d", ch, state); + + switch (state) { + + case sw_chunk_start: + if (ch >= '0' && ch <= '9') { + state = sw_chunk_size; + ctx->size = ch - '0'; + break; + } + + c = (u_char) (ch | 0x20); + + if (c >= 'a' && c <= 'f') { + state = sw_chunk_size; + ctx->size = c - 'a' + 10; + break; + } + + goto invalid; + + case sw_chunk_size: + if (ch >= '0' && ch <= '9') { + ctx->size = ctx->size * 16 + (ch - '0'); + break; + } + + c = (u_char) (ch | 0x20); + + if (c >= 'a' && c <= 'f') { + ctx->size = ctx->size * 16 + (c - 'a' + 10); + break; + } + + if (ctx->size == 0) { + + switch (ch) { + case CR: + state = sw_last_chunk_extension_almost_done; + break; + case LF: + state = sw_trailer; + break; + case ';': + case ' ': + case '\t': + state = sw_last_chunk_extension; + break; + default: + goto invalid; + } + + break; + } + + switch (ch) { + case CR: + state = sw_chunk_extension_almost_done; + break; + case LF: + state = sw_chunk_data; + break; + case ';': + case ' ': + case '\t': + state = sw_chunk_extension; + break; + default: + goto invalid; + } + + break; + + case sw_chunk_extension: + switch (ch) { + case CR: + state = sw_chunk_extension_almost_done; + break; + case LF: + state = sw_chunk_data; + } + break; + + case sw_chunk_extension_almost_done: + if (ch == LF) { + state = sw_chunk_data; + break; + } + goto invalid; + + case sw_chunk_data: + rc = NGX_OK; + goto data; + + case sw_after_data: + switch (ch) { + case CR: + state = sw_after_data_almost_done; + break; + case LF: + state = sw_chunk_start; + } + break; + + case sw_after_data_almost_done: + if (ch == LF) { + state = sw_chunk_start; + break; + } + goto invalid; + + case sw_last_chunk_extension: + switch (ch) { + case CR: + state = sw_last_chunk_extension_almost_done; + break; + case LF: + state = sw_trailer; + } + break; + + case sw_last_chunk_extension_almost_done: + if (ch == LF) { + state = sw_trailer; + break; + } + goto invalid; + + case sw_trailer: + switch (ch) { + case CR: + state = sw_trailer_almost_done; + break; + case LF: + goto done; + default: + state = sw_trailer_header; + } + break; + + case sw_trailer_almost_done: + if (ch == LF) { + goto done; + } + goto invalid; + + case sw_trailer_header: + switch (ch) { + case CR: + state = sw_trailer_header_almost_done; + break; + case LF: + state = sw_trailer; + } + break; + + case sw_trailer_header_almost_done: + if (ch == LF) { + state = sw_trailer; + break; + } + goto invalid; + + } + } + +data: + + ctx->state = state; + b->pos = pos; + + switch (state) { + + case sw_chunk_start: + ctx->length = 3 /* "0" LF LF */; + break; + case sw_chunk_size: + ctx->length = 2 /* LF LF */ + + (ctx->size ? ctx->size + 4 /* LF "0" LF LF */ : 0); + break; + case sw_chunk_extension: + case sw_chunk_extension_almost_done: + ctx->length = 1 /* LF */ + ctx->size + 4 /* LF "0" LF LF */; + break; + case sw_chunk_data: + ctx->length = ctx->size + 4 /* LF "0" LF LF */; + break; + case sw_after_data: + case sw_after_data_almost_done: + ctx->length = 4 /* LF "0" LF LF */; + break; + case sw_last_chunk_extension: + case sw_last_chunk_extension_almost_done: + ctx->length = 2 /* LF LF */; + break; + case sw_trailer: + case sw_trailer_almost_done: + ctx->length = 1 /* LF */; + break; + case sw_trailer_header: + case sw_trailer_header_almost_done: + ctx->length = 2 /* LF LF */; + break; + + } + + return rc; + +done: + + return NGX_DONE; + +invalid: + + return NGX_ERROR; +} From mdounin at mdounin.ru Fri Nov 16 11:02:31 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 16 Nov 2012 15:02:31 +0400 Subject: [PATCH 09 of 13] Request body: adjust b->pos when chunked parsing done In-Reply-To: References: Message-ID: # HG changeset patch # User Maxim Dounin # Date 1352393278 -14400 # Node ID d04392abdec6cb1c7e3d09522288838ce90851b4 # Parent 9e14a7c2950b05c3f6550b4233ead43eefa57da5 Request body: adjust b->pos when chunked parsing done. This is a nop for the current code, though will allow to correctly parse pipelined requests. diff --git a/src/http/ngx_http_parse.c b/src/http/ngx_http_parse.c --- a/src/http/ngx_http_parse.c +++ b/src/http/ngx_http_parse.c @@ -2065,6 +2065,9 @@ data: done: + ctx->state = 0; + b->pos = pos + 1; + return NGX_DONE; invalid: From mdounin at mdounin.ru Fri Nov 16 11:02:32 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 16 Nov 2012 15:02:32 +0400 Subject: [PATCH 10 of 13] Request body: always use calculated size of a request body in proxy In-Reply-To: References: Message-ID: # HG changeset patch # User Maxim Dounin # Date 1352393278 -14400 # Node ID a3bad06d0ab3874b7479823f1c7240306792a110 # Parent d04392abdec6cb1c7e3d09522288838ce90851b4 Request body: always use calculated size of a request body in proxy. This allows to handle requests with chunked body, and also simplifies handling of various request body modifications. diff --git a/src/http/modules/ngx_http_proxy_module.c b/src/http/modules/ngx_http_proxy_module.c --- a/src/http/modules/ngx_http_proxy_module.c +++ b/src/http/modules/ngx_http_proxy_module.c @@ -83,7 +83,7 @@ typedef struct { ngx_http_status_t status; ngx_http_chunked_t chunked; ngx_http_proxy_vars_t vars; - size_t internal_body_length; + off_t internal_body_length; ngx_uint_t head; /* unsigned head:1 */ } ngx_http_proxy_ctx_t; @@ -555,6 +555,8 @@ static char ngx_http_proxy_version_11[] static ngx_keyval_t ngx_http_proxy_headers[] = { { ngx_string("Host"), ngx_string("$proxy_host") }, { ngx_string("Connection"), ngx_string("close") }, + { ngx_string("Content-Length"), ngx_string("$proxy_internal_body_length") }, + { ngx_string("Transfer-Encoding"), ngx_string("") }, { ngx_string("Keep-Alive"), ngx_string("") }, { ngx_string("Expect"), ngx_string("") }, { ngx_string("Upgrade"), ngx_string("") }, @@ -580,6 +582,8 @@ static ngx_str_t ngx_http_proxy_hide_he static ngx_keyval_t ngx_http_proxy_cache_headers[] = { { ngx_string("Host"), ngx_string("$proxy_host") }, { ngx_string("Connection"), ngx_string("close") }, + { ngx_string("Content-Length"), ngx_string("$proxy_internal_body_length") }, + { ngx_string("Transfer-Encoding"), ngx_string("") }, { ngx_string("Keep-Alive"), ngx_string("") }, { ngx_string("Expect"), ngx_string("") }, { ngx_string("Upgrade"), ngx_string("") }, @@ -1003,6 +1007,9 @@ ngx_http_proxy_create_request(ngx_http_r ctx->internal_body_length = body_len; len += body_len; + + } else { + ctx->internal_body_length = r->headers_in.content_length_n; } le.ip = plcf->headers_set_len->elts; @@ -2039,7 +2046,7 @@ ngx_http_proxy_internal_body_length_vari ctx = ngx_http_get_module_ctx(r, ngx_http_proxy_module); - if (ctx == NULL) { + if (ctx == NULL || ctx->internal_body_length < 0) { v->not_found = 1; return NGX_OK; } @@ -2048,13 +2055,13 @@ ngx_http_proxy_internal_body_length_vari v->no_cacheable = 0; v->not_found = 0; - v->data = ngx_pnalloc(r->connection->pool, NGX_SIZE_T_LEN); + v->data = ngx_pnalloc(r->connection->pool, NGX_OFF_T_LEN); if (v->data == NULL) { return NGX_ERROR; } - v->len = ngx_sprintf(v->data, "%uz", ctx->internal_body_length) - v->data; + v->len = ngx_sprintf(v->data, "%O", ctx->internal_body_length) - v->data; return NGX_OK; } @@ -2822,8 +2829,6 @@ ngx_http_proxy_merge_headers(ngx_conf_t } if (conf->headers_set_hash.buckets - && ((conf->body_source.data == NULL) - == (prev->body_source.data == NULL)) #if (NGX_HTTP_CACHE) && ((conf->upstream.cache == NULL) == (prev->upstream.cache == NULL)) #endif @@ -2906,16 +2911,6 @@ ngx_http_proxy_merge_headers(ngx_conf_t h++; } - if (conf->body_source.data) { - s = ngx_array_push(&headers_merged); - if (s == NULL) { - return NGX_ERROR; - } - - ngx_str_set(&s->key, "Content-Length"); - ngx_str_set(&s->value, "$proxy_internal_body_length"); - } - src = headers_merged.elts; for (i = 0; i < headers_merged.nelts; i++) { From mdounin at mdounin.ru Fri Nov 16 11:02:33 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 16 Nov 2012 15:02:33 +0400 Subject: [PATCH 11 of 13] Request body: $content_length variable to honor real body size In-Reply-To: References: Message-ID: # HG changeset patch # User Maxim Dounin # Date 1352393278 -14400 # Node ID ec54ef6b4826b3fa6d0f310cb4d80291dcdc7d3f # Parent a3bad06d0ab3874b7479823f1c7240306792a110 Request body: $content_length variable to honor real body size. This allows to handle requests with chunked body by fastcgi and uwsgi modules, and also simplifies handling of various request body modifications. diff --git a/src/http/ngx_http_variables.c b/src/http/ngx_http_variables.c --- a/src/http/ngx_http_variables.c +++ b/src/http/ngx_http_variables.c @@ -39,6 +39,8 @@ static ngx_int_t ngx_http_variable_tcpin ngx_http_variable_value_t *v, uintptr_t data); #endif +static ngx_int_t ngx_http_variable_content_length(ngx_http_request_t *r, + ngx_http_variable_value_t *v, uintptr_t data); static ngx_int_t ngx_http_variable_host(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data); static ngx_int_t ngx_http_variable_binary_remote_addr(ngx_http_request_t *r, @@ -153,8 +155,8 @@ static ngx_http_variable_t ngx_http_cor { ngx_string("http_cookie"), NULL, ngx_http_variable_headers, offsetof(ngx_http_request_t, headers_in.cookies), 0, 0 }, - { ngx_string("content_length"), NULL, ngx_http_variable_header, - offsetof(ngx_http_request_t, headers_in.content_length), 0, 0 }, + { ngx_string("content_length"), NULL, ngx_http_variable_content_length, + 0, 0, 0 }, { ngx_string("content_type"), NULL, ngx_http_variable_header, offsetof(ngx_http_request_t, headers_in.content_type), 0, 0 }, @@ -990,6 +992,39 @@ ngx_http_variable_tcpinfo(ngx_http_reque static ngx_int_t +ngx_http_variable_content_length(ngx_http_request_t *r, + ngx_http_variable_value_t *v, uintptr_t data) +{ + u_char *p; + + if (r->headers_in.content_length) { + v->len = r->headers_in.content_length->value.len; + v->data = r->headers_in.content_length->value.data; + v->valid = 1; + v->no_cacheable = 0; + v->not_found = 0; + + } else if (r->headers_in.content_length_n >= 0) { + p = ngx_pnalloc(r->pool, NGX_OFF_T_LEN); + if (p == NULL) { + return NGX_ERROR; + } + + v->len = ngx_sprintf(p, "%O", r->headers_in.content_length_n) - p; + v->data = p; + v->valid = 1; + v->no_cacheable = 0; + v->not_found = 0; + + } else { + v->not_found = 1; + } + + return NGX_OK; +} + + +static ngx_int_t ngx_http_variable_host(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data) { From mdounin at mdounin.ru Fri Nov 16 11:02:34 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 16 Nov 2012 15:02:34 +0400 Subject: [PATCH 12 of 13] Request body: recalculate size of a request body in scgi module In-Reply-To: References: Message-ID: <18c2ee91104005005fdc.1353063754@vm-bsd.mdounin.ru> # HG changeset patch # User Maxim Dounin # Date 1352393278 -14400 # Node ID 18c2ee91104005005fdc710fce2253400131c558 # Parent ec54ef6b4826b3fa6d0f310cb4d80291dcdc7d3f Request body: recalculate size of a request body in scgi module. This allows to handle requests with chunked body by scgi module, and also simplifies handling of various request body modifications. diff --git a/src/http/modules/ngx_http_scgi_module.c b/src/http/modules/ngx_http_scgi_module.c --- a/src/http/modules/ngx_http_scgi_module.c +++ b/src/http/modules/ngx_http_scgi_module.c @@ -533,10 +533,11 @@ ngx_http_scgi_create_key(ngx_http_reques static ngx_int_t ngx_http_scgi_create_request(ngx_http_request_t *r) { + off_t content_length_n; u_char ch, *key, *val, *lowcase_key; size_t len, key_len, val_len, allocated; ngx_buf_t *b; - ngx_str_t *content_length; + ngx_str_t content_length; ngx_uint_t i, n, hash, skip_empty, header_params; ngx_chain_t *cl, *body; ngx_list_part_t *part; @@ -545,12 +546,20 @@ ngx_http_scgi_create_request(ngx_http_re ngx_http_script_engine_t e, le; ngx_http_scgi_loc_conf_t *scf; ngx_http_script_len_code_pt lcode; - static ngx_str_t zero = ngx_string("0"); + u_char buffer[NGX_OFF_T_LEN]; - content_length = r->headers_in.content_length ? - &r->headers_in.content_length->value : &zero; + content_length_n = 0; + body = r->upstream->request_bufs; - len = sizeof("CONTENT_LENGTH") + content_length->len + 1; + while (body) { + content_length_n += ngx_buf_size(body->buf); + body = body->next; + } + + content_length.data = buffer; + content_length.len = ngx_sprintf(buffer, "%O", content_length_n) - buffer; + + len = sizeof("CONTENT_LENGTH") + content_length.len + 1; header_params = 0; ignored = NULL; @@ -672,11 +681,8 @@ ngx_http_scgi_create_request(ngx_http_re cl->buf = b; - b->last = ngx_snprintf(b->last, - NGX_SIZE_T_LEN + 1 + sizeof("CONTENT_LENGTH") - + NGX_OFF_T_LEN + 1, - "%ui:CONTENT_LENGTH%Z%V%Z", - len, content_length); + b->last = ngx_sprintf(b->last, "%ui:CONTENT_LENGTH%Z%V%Z", + len, &content_length); if (scf->params_len) { ngx_memzero(&e, sizeof(ngx_http_script_engine_t)); From mdounin at mdounin.ru Fri Nov 16 11:02:35 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 16 Nov 2012 15:02:35 +0400 Subject: [PATCH 13 of 13] Request body: chunked transfer encoding support In-Reply-To: References: Message-ID: # HG changeset patch # User Maxim Dounin # Date 1352393278 -14400 # Node ID afda241c3fc74023a5f041e6d0ccd95e0c92eed4 # Parent 18c2ee91104005005fdc710fce2253400131c558 Request body: chunked transfer encoding support. diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c +++ b/src/http/ngx_http_core_module.c @@ -848,7 +848,8 @@ ngx_http_handler(ngx_http_request_t *r) break; } - r->lingering_close = (r->headers_in.content_length_n > 0); + r->lingering_close = (r->headers_in.content_length_n > 0 + || r->headers_in.chunked); r->phase_handler = 0; } else { diff --git a/src/http/ngx_http_request.c b/src/http/ngx_http_request.c --- a/src/http/ngx_http_request.c +++ b/src/http/ngx_http_request.c @@ -1574,19 +1574,11 @@ ngx_http_process_request_header(ngx_http if (r->headers_in.content_length_n == NGX_ERROR) { ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, "client sent invalid \"Content-Length\" header"); - ngx_http_finalize_request(r, NGX_HTTP_LENGTH_REQUIRED); + ngx_http_finalize_request(r, NGX_HTTP_BAD_REQUEST); return NGX_ERROR; } } - if (r->method & NGX_HTTP_PUT && r->headers_in.content_length_n == -1) { - ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, - "client sent %V method without \"Content-Length\" header", - &r->method_name); - ngx_http_finalize_request(r, NGX_HTTP_LENGTH_REQUIRED); - return NGX_ERROR; - } - if (r->method & NGX_HTTP_TRACE) { ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, "client sent TRACE method"); @@ -1594,14 +1586,25 @@ ngx_http_process_request_header(ngx_http return NGX_ERROR; } - if (r->headers_in.transfer_encoding - && ngx_strcasestrn(r->headers_in.transfer_encoding->value.data, - "chunked", 7 - 1)) - { - ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, - "client sent \"Transfer-Encoding: chunked\" header"); - ngx_http_finalize_request(r, NGX_HTTP_LENGTH_REQUIRED); - return NGX_ERROR; + if (r->headers_in.transfer_encoding) { + if (r->headers_in.transfer_encoding->value.len == 7 + && ngx_strncasecmp(r->headers_in.transfer_encoding->value.data, + (u_char *) "chunked", 7) == 0) + { + r->headers_in.content_length = NULL; + r->headers_in.content_length_n = -1; + r->headers_in.chunked = 1; + + } else if (r->headers_in.transfer_encoding->value.len != 8 + || ngx_strncasecmp(r->headers_in.transfer_encoding->value.data, + (u_char *) "identity", 8) != 0) + { + ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, + "client sent unknown \"Transfer-Encoding\": \"%V\"", + &r->headers_in.transfer_encoding->value); + ngx_http_finalize_request(r, NGX_HTTP_NOT_IMPLEMENTED); + return NGX_ERROR; + } } if (r->headers_in.connection_type == NGX_HTTP_CONNECTION_KEEP_ALIVE) { diff --git a/src/http/ngx_http_request.h b/src/http/ngx_http_request.h --- a/src/http/ngx_http_request.h +++ b/src/http/ngx_http_request.h @@ -224,6 +224,7 @@ typedef struct { time_t keep_alive_n; unsigned connection_type:2; + unsigned chunked:1; unsigned msie:1; unsigned msie6:1; unsigned opera:1; @@ -276,7 +277,9 @@ typedef struct { ngx_chain_t *bufs; ngx_buf_t *buf; off_t rest; - ngx_chain_t *to_write; + ngx_chain_t *free; + ngx_chain_t *busy; + ngx_http_chunked_t *chunked; ngx_http_client_body_handler_pt post_handler; } ngx_http_request_body_t; diff --git a/src/http/ngx_http_request_body.c b/src/http/ngx_http_request_body.c --- a/src/http/ngx_http_request_body.c +++ b/src/http/ngx_http_request_body.c @@ -12,18 +12,21 @@ static void ngx_http_read_client_request_body_handler(ngx_http_request_t *r); static ngx_int_t ngx_http_do_read_client_request_body(ngx_http_request_t *r); -static ngx_int_t ngx_http_write_request_body(ngx_http_request_t *r, - ngx_chain_t *body); +static ngx_int_t ngx_http_write_request_body(ngx_http_request_t *r); static ngx_int_t ngx_http_read_discarded_request_body(ngx_http_request_t *r); +static ngx_int_t ngx_http_discard_request_body_filter(ngx_http_request_t *r, + ngx_buf_t *b); static ngx_int_t ngx_http_test_expect(ngx_http_request_t *r); +static ngx_int_t ngx_http_request_body_filter(ngx_http_request_t *r, + ngx_chain_t *in); +static ngx_int_t ngx_http_request_body_length_filter(ngx_http_request_t *r, + ngx_chain_t *in); +static ngx_int_t ngx_http_request_body_chunked_filter(ngx_http_request_t *r, + ngx_chain_t *in); +static ngx_int_t ngx_http_request_body_save_filter(ngx_http_request_t *r, + ngx_chain_t *in); -/* - * on completion ngx_http_read_client_request_body() adds to - * r->request_body->bufs one or two bufs: - * *) one memory buf that was preread in r->header_in; - * *) one memory or file buf that contains the rest of the body - */ ngx_int_t ngx_http_read_client_request_body(ngx_http_request_t *r, @@ -32,8 +35,7 @@ ngx_http_read_client_request_body(ngx_ht size_t preread; ssize_t size; ngx_int_t rc; - ngx_buf_t *b; - ngx_chain_t *cl, **next; + ngx_chain_t out; ngx_http_request_body_t *rb; ngx_http_core_loc_conf_t *clcf; @@ -55,19 +57,73 @@ ngx_http_read_client_request_body(ngx_ht goto done; } + /* + * set by ngx_pcalloc(): + * + * rb->bufs = NULL; + * rb->buf = NULL; + * rb->free = NULL; + * rb->busy = NULL; + * rb->chunked = NULL; + */ + + rb->rest = -1; + rb->post_handler = post_handler; + r->request_body = rb; - if (r->headers_in.content_length_n < 0) { + if (r->headers_in.content_length_n < 0 && !r->headers_in.chunked) { post_handler(r); return NGX_OK; } - clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module); + preread = r->header_in->last - r->header_in->pos; - if (r->headers_in.content_length_n == 0) { + if (preread) { + + /* there is the pre-read part of the request body */ + + ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "http client request body preread %uz", preread); + + out.buf = r->header_in; + out.next = NULL; + + rc = ngx_http_request_body_filter(r, &out); + + if (rc != NGX_OK) { + goto done; + } + + r->request_length += preread - (r->header_in->last - r->header_in->pos); + + if (!r->headers_in.chunked + && rb->rest > 0 + && rb->rest <= (off_t) (r->header_in->end - r->header_in->last)) + { + /* the whole request body may be placed in r->header_in */ + + rb->buf = r->header_in; + r->read_event_handler = ngx_http_read_client_request_body_handler; + + rc = ngx_http_do_read_client_request_body(r); + goto done; + } + + } else { + /* set rb->rest */ + + if (ngx_http_request_body_filter(r, NULL) != NGX_OK) { + rc = NGX_HTTP_INTERNAL_SERVER_ERROR; + goto done; + } + } + + if (rb->rest == 0) { + /* the whole request body was pre-read */ if (r->request_body_in_file_only) { - if (ngx_http_write_request_body(r, NULL) != NGX_OK) { + if (ngx_http_write_request_body(r) != NGX_OK) { rc = NGX_HTTP_INTERNAL_SERVER_ERROR; goto done; } @@ -78,102 +134,14 @@ ngx_http_read_client_request_body(ngx_ht return NGX_OK; } - rb->post_handler = post_handler; - - /* - * set by ngx_pcalloc(): - * - * rb->bufs = NULL; - * rb->buf = NULL; - * rb->rest = 0; - */ - - preread = r->header_in->last - r->header_in->pos; - - if (preread) { - - /* there is the pre-read part of the request body */ - - ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, - "http client request body preread %uz", preread); - - b = ngx_calloc_buf(r->pool); - if (b == NULL) { - rc = NGX_HTTP_INTERNAL_SERVER_ERROR; - goto done; - } - - b->temporary = 1; - b->start = r->header_in->pos; - b->pos = r->header_in->pos; - b->last = r->header_in->last; - b->end = r->header_in->end; - - rb->bufs = ngx_alloc_chain_link(r->pool); - if (rb->bufs == NULL) { - rc = NGX_HTTP_INTERNAL_SERVER_ERROR; - goto done; - } - - rb->bufs->buf = b; - rb->bufs->next = NULL; - - rb->buf = b; - - if ((off_t) preread >= r->headers_in.content_length_n) { - - /* the whole request body was pre-read */ - - r->header_in->pos += (size_t) r->headers_in.content_length_n; - r->request_length += r->headers_in.content_length_n; - b->last = r->header_in->pos; - - if (r->request_body_in_file_only) { - if (ngx_http_write_request_body(r, rb->bufs) != NGX_OK) { - rc = NGX_HTTP_INTERNAL_SERVER_ERROR; - goto done; - } - } - - post_handler(r); - - return NGX_OK; - } - - /* - * to not consider the body as pipelined request in - * ngx_http_set_keepalive() - */ - r->header_in->pos = r->header_in->last; - - r->request_length += preread; - - rb->rest = r->headers_in.content_length_n - preread; - - if (rb->rest <= (off_t) (b->end - b->last)) { - - /* the whole request body may be placed in r->header_in */ - - rb->to_write = rb->bufs; - - r->read_event_handler = ngx_http_read_client_request_body_handler; - - rc = ngx_http_do_read_client_request_body(r); - goto done; - } - - next = &rb->bufs->next; - - } else { - b = NULL; - rb->rest = r->headers_in.content_length_n; - next = &rb->bufs; - } + clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module); size = clcf->client_body_buffer_size; size += size >> 2; - if (rb->rest < size) { + /* TODO: honor r->request_body_in_single_buf */ + + if (!r->headers_in.chunked && rb->rest < size) { size = (ssize_t) rb->rest; if (r->request_body_in_single_buf) { @@ -182,9 +150,6 @@ ngx_http_read_client_request_body(ngx_ht } else { size = clcf->client_body_buffer_size; - - /* disable copying buffer for r->request_body_in_single_buf */ - b = NULL; } rb->buf = ngx_create_temp_buf(r->pool, size); @@ -193,32 +158,6 @@ ngx_http_read_client_request_body(ngx_ht goto done; } - cl = ngx_alloc_chain_link(r->pool); - if (cl == NULL) { - rc = NGX_HTTP_INTERNAL_SERVER_ERROR; - goto done; - } - - cl->buf = rb->buf; - cl->next = NULL; - - if (b && r->request_body_in_single_buf) { - size = b->last - b->pos; - ngx_memcpy(rb->buf->pos, b->pos, size); - rb->buf->last += size; - - next = &rb->bufs; - } - - *next = cl; - - if (r->request_body_in_file_only || r->request_body_in_single_buf) { - rb->to_write = rb->bufs; - - } else { - rb->to_write = rb->bufs->next ? rb->bufs->next : rb->bufs; - } - r->read_event_handler = ngx_http_read_client_request_body_handler; rc = ngx_http_do_read_client_request_body(r); @@ -255,9 +194,12 @@ ngx_http_read_client_request_body_handle static ngx_int_t ngx_http_do_read_client_request_body(ngx_http_request_t *r) { + off_t rest; size_t size; ssize_t n; + ngx_int_t rc; ngx_buf_t *b; + ngx_chain_t *cl, out; ngx_connection_t *c; ngx_http_request_body_t *rb; ngx_http_core_loc_conf_t *clcf; @@ -272,18 +214,44 @@ ngx_http_do_read_client_request_body(ngx for ( ;; ) { if (rb->buf->last == rb->buf->end) { - if (ngx_http_write_request_body(r, rb->to_write) != NGX_OK) { + /* pass buffer to request body filter chain */ + + out.buf = rb->buf; + out.next = NULL; + + rc = ngx_http_request_body_filter(r, &out); + + if (rc != NGX_OK) { + return rc; + } + + /* write to file */ + + if (ngx_http_write_request_body(r) != NGX_OK) { return NGX_HTTP_INTERNAL_SERVER_ERROR; } - rb->to_write = rb->bufs->next ? rb->bufs->next : rb->bufs; + /* update chains */ + + rc = ngx_http_request_body_filter(r, NULL); + + if (rc != NGX_OK) { + return rc; + } + + if (rb->busy != NULL) { + return NGX_HTTP_INTERNAL_SERVER_ERROR; + } + + rb->buf->pos = rb->buf->start; rb->buf->last = rb->buf->start; } size = rb->buf->end - rb->buf->last; + rest = rb->rest - (rb->buf->last - rb->buf->pos); - if ((off_t) size > rb->rest) { - size = (size_t) rb->rest; + if ((off_t) size > rest) { + size = (size_t) rest; } n = c->recv(c, rb->buf->last, size); @@ -306,9 +274,21 @@ ngx_http_do_read_client_request_body(ngx } rb->buf->last += n; - rb->rest -= n; r->request_length += n; + if (n == rest) { + /* pass buffer to request body filter chain */ + + out.buf = rb->buf; + out.next = NULL; + + rc = ngx_http_request_body_filter(r, &out); + + if (rc != NGX_OK) { + return rc; + } + } + if (rb->rest == 0) { break; } @@ -345,32 +325,24 @@ ngx_http_do_read_client_request_body(ngx /* save the last part */ - if (ngx_http_write_request_body(r, rb->to_write) != NGX_OK) { + if (ngx_http_write_request_body(r) != NGX_OK) { return NGX_HTTP_INTERNAL_SERVER_ERROR; } - b = ngx_calloc_buf(r->pool); - if (b == NULL) { + cl = ngx_chain_get_free_buf(r->pool, &rb->free); + if (cl == NULL) { return NGX_HTTP_INTERNAL_SERVER_ERROR; } + b = cl->buf; + + ngx_memzero(b, sizeof(ngx_buf_t)); + b->in_file = 1; - b->file_pos = 0; b->file_last = rb->temp_file->file.offset; b->file = &rb->temp_file->file; - if (rb->bufs->next) { - rb->bufs->next->buf = b; - - } else { - rb->bufs->buf = b; - } - } - - if (rb->bufs->next - && (r->request_body_in_file_only || r->request_body_in_single_buf)) - { - rb->bufs = rb->bufs->next; + rb->bufs = cl; } r->read_event_handler = ngx_http_block_reading; @@ -382,15 +354,19 @@ ngx_http_do_read_client_request_body(ngx static ngx_int_t -ngx_http_write_request_body(ngx_http_request_t *r, ngx_chain_t *body) +ngx_http_write_request_body(ngx_http_request_t *r) { ssize_t n; + ngx_chain_t *cl; ngx_temp_file_t *tf; ngx_http_request_body_t *rb; ngx_http_core_loc_conf_t *clcf; rb = r->request_body; + ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "http write client request body, bufs %p", rb->bufs); + if (rb->temp_file == NULL) { tf = ngx_pcalloc(r->pool, sizeof(ngx_temp_file_t)); if (tf == NULL) { @@ -414,7 +390,7 @@ ngx_http_write_request_body(ngx_http_req rb->temp_file = tf; - if (body == NULL) { + if (rb->bufs == NULL) { /* empty body with r->request_body_in_file_only */ if (ngx_create_temp_file(&tf->file, tf->path, tf->pool, @@ -428,7 +404,11 @@ ngx_http_write_request_body(ngx_http_req } } - n = ngx_write_chain_to_temp_file(rb->temp_file, body); + if (rb->bufs == NULL) { + return NGX_OK; + } + + n = ngx_write_chain_to_temp_file(rb->temp_file, rb->bufs); /* TODO: n == 0 or not complete and level event */ @@ -438,6 +418,14 @@ ngx_http_write_request_body(ngx_http_req rb->temp_file->offset += n; + /* mark all buffers as written */ + + for (cl = rb->bufs; cl; cl = cl->next) { + cl->buf->pos = cl->buf->last; + } + + rb->bufs = NULL; + return NGX_OK; } @@ -446,9 +434,10 @@ ngx_int_t ngx_http_discard_request_body(ngx_http_request_t *r) { ssize_t size; + ngx_int_t rc; ngx_event_t *rev; - if (r != r->main || r->discard_body) { + if (r != r->main || r->discard_body || r->request_body) { return NGX_OK; } @@ -464,20 +453,20 @@ ngx_http_discard_request_body(ngx_http_r ngx_del_timer(rev); } - if (r->headers_in.content_length_n <= 0 || r->request_body) { + if (r->headers_in.content_length_n <= 0 && !r->headers_in.chunked) { return NGX_OK; } size = r->header_in->last - r->header_in->pos; if (size) { - if (r->headers_in.content_length_n > size) { - r->header_in->pos += size; - r->headers_in.content_length_n -= size; + rc = ngx_http_discard_request_body_filter(r, r->header_in); - } else { - r->header_in->pos += (size_t) r->headers_in.content_length_n; - r->headers_in.content_length_n = 0; + if (rc != NGX_OK) { + return rc; + } + + if (r->headers_in.content_length_n == 0) { return NGX_OK; } } @@ -570,13 +559,19 @@ ngx_http_discarded_request_body_handler( static ngx_int_t ngx_http_read_discarded_request_body(ngx_http_request_t *r) { - size_t size; - ssize_t n; - u_char buffer[NGX_HTTP_DISCARD_BUFFER_SIZE]; + size_t size; + ssize_t n; + ngx_int_t rc; + ngx_buf_t b; + u_char buffer[NGX_HTTP_DISCARD_BUFFER_SIZE]; ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "http read discarded body"); + ngx_memzero(&b, sizeof(ngx_buf_t)); + + b.temporary = 1; + for ( ;; ) { if (r->headers_in.content_length_n == 0) { r->read_event_handler = ngx_http_block_reading; @@ -587,9 +582,8 @@ ngx_http_read_discarded_request_body(ngx return NGX_AGAIN; } - size = (r->headers_in.content_length_n > NGX_HTTP_DISCARD_BUFFER_SIZE) ? - NGX_HTTP_DISCARD_BUFFER_SIZE: - (size_t) r->headers_in.content_length_n; + size = (size_t) ngx_min(r->headers_in.content_length_n, + NGX_HTTP_DISCARD_BUFFER_SIZE); n = r->connection->recv(r->connection, buffer, size); @@ -606,12 +600,109 @@ ngx_http_read_discarded_request_body(ngx return NGX_OK; } - r->headers_in.content_length_n -= n; + b.pos = buffer; + b.last = buffer + n; + + rc = ngx_http_discard_request_body_filter(r, &b); + + if (rc != NGX_OK) { + r->connection->error = 1; + return NGX_OK; + } } } static ngx_int_t +ngx_http_discard_request_body_filter(ngx_http_request_t *r, ngx_buf_t *b) +{ + size_t size; + ngx_int_t rc; + ngx_http_request_body_t *rb; + + if (r->headers_in.chunked) { + + rb = r->request_body; + + if (rb == NULL) { + + rb = ngx_pcalloc(r->pool, sizeof(ngx_http_request_body_t)); + if (rb == NULL) { + return NGX_HTTP_INTERNAL_SERVER_ERROR; + } + + rb->chunked = ngx_pcalloc(r->pool, sizeof(ngx_http_chunked_t)); + if (rb == NULL) { + return NGX_HTTP_INTERNAL_SERVER_ERROR; + } + + r->request_body = rb; + } + + for ( ;; ) { + + rc = ngx_http_parse_chunked(r, b, rb->chunked); + + if (rc == NGX_OK) { + + /* a chunk has been parsed successfully */ + + size = b->last - b->pos; + + if ((off_t) size > rb->chunked->size) { + b->pos += rb->chunked->size; + rb->chunked->size = 0; + + } else { + rb->chunked->size -= size; + b->pos = b->last; + } + + continue; + } + + if (rc == NGX_DONE) { + + /* a whole response has been parsed successfully */ + + r->headers_in.content_length_n = 0; + break; + } + + if (rc == NGX_AGAIN) { + + /* set amount of data we want to see next time */ + + r->headers_in.content_length_n = rb->chunked->length; + break; + } + + /* invalid */ + + ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, + "client sent invalid chunked body"); + + return NGX_HTTP_BAD_REQUEST; + } + + } else { + size = b->last - b->pos; + + if ((off_t) size > r->headers_in.content_length_n) { + b->pos += r->headers_in.content_length_n; + r->headers_in.content_length_n = 0; + + } else { + b->pos = b->last; + r->headers_in.content_length_n -= size; + } + } + + return NGX_OK; +} + + +static ngx_int_t ngx_http_test_expect(ngx_http_request_t *r) { ngx_int_t n; @@ -651,3 +742,274 @@ ngx_http_test_expect(ngx_http_request_t return NGX_ERROR; } + + +static ngx_int_t +ngx_http_request_body_filter(ngx_http_request_t *r, ngx_chain_t *in) +{ + if (r->headers_in.chunked) { + return ngx_http_request_body_chunked_filter(r, in); + + } else { + return ngx_http_request_body_length_filter(r, in); + } +} + + +static ngx_int_t +ngx_http_request_body_length_filter(ngx_http_request_t *r, ngx_chain_t *in) +{ + size_t size; + ngx_int_t rc; + ngx_buf_t *b; + ngx_chain_t *cl, *tl, *out, **ll; + ngx_http_request_body_t *rb; + + rb = r->request_body; + + if (rb->rest == -1) { + ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "http request body content length filter"); + + rb->rest = r->headers_in.content_length_n; + } + + out = NULL; + ll = &out; + + for (cl = in; cl; cl = cl->next) { + + tl = ngx_chain_get_free_buf(r->pool, &rb->free); + if (tl == NULL) { + return NGX_HTTP_INTERNAL_SERVER_ERROR; + } + + b = tl->buf; + + ngx_memzero(b, sizeof(ngx_buf_t)); + + b->temporary = 1; + b->tag = (ngx_buf_tag_t) &ngx_http_read_client_request_body; + b->start = cl->buf->start; + b->pos = cl->buf->pos; + b->last = cl->buf->last; + b->end = cl->buf->end; + + size = cl->buf->last - cl->buf->pos; + + if ((off_t) size < rb->rest) { + cl->buf->pos = cl->buf->last; + rb->rest -= size; + + } else { + cl->buf->pos += rb->rest; + rb->rest = 0; + b->last = cl->buf->pos; + b->last_buf = 1; + } + + *ll = tl; + ll = &tl->next; + } + + rc = ngx_http_request_body_save_filter(r, out); + + ngx_chain_update_chains(r->pool, &rb->free, &rb->busy, &out, + (ngx_buf_tag_t) &ngx_http_read_client_request_body); + + return rc; +} + + +static ngx_int_t +ngx_http_request_body_chunked_filter(ngx_http_request_t *r, ngx_chain_t *in) +{ + size_t size; + ngx_int_t rc; + ngx_buf_t *b; + ngx_chain_t *cl, *out, *tl, **ll; + ngx_http_request_body_t *rb; + ngx_http_core_loc_conf_t *clcf; + + rb = r->request_body; + + if (rb->rest == -1) { + + ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "http request body chunked filter"); + + rb->chunked = ngx_pcalloc(r->pool, sizeof(ngx_http_chunked_t)); + if (rb->chunked == NULL) { + return NGX_HTTP_INTERNAL_SERVER_ERROR; + } + + r->headers_in.content_length_n = 0; + rb->rest = 3; + } + + out = NULL; + ll = &out; + + for (cl = in; cl; cl = cl->next) { + + for ( ;; ) { + + ngx_log_debug7(NGX_LOG_DEBUG_EVENT, r->connection->log, 0, + "http body chunked buf " + "t:%d f:%d %p, pos %p, size: %z file: %O, size: %z", + cl->buf->temporary, cl->buf->in_file, + cl->buf->start, cl->buf->pos, + cl->buf->last - cl->buf->pos, + cl->buf->file_pos, + cl->buf->file_last - cl->buf->file_pos); + + rc = ngx_http_parse_chunked(r, cl->buf, rb->chunked); + + if (rc == NGX_OK) { + + /* a chunk has been parsed successfully */ + + clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module); + + if (clcf->client_max_body_size + && clcf->client_max_body_size + < r->headers_in.content_length_n + rb->chunked->size) + { + ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, + "client intended to send too large chunked " + "body: %O bytes", + r->headers_in.content_length_n + + rb->chunked->size); + + r->lingering_close = 1; + + return NGX_HTTP_REQUEST_ENTITY_TOO_LARGE; + } + + tl = ngx_chain_get_free_buf(r->pool, &rb->free); + if (tl == NULL) { + return NGX_HTTP_INTERNAL_SERVER_ERROR; + } + + b = tl->buf; + + ngx_memzero(b, sizeof(ngx_buf_t)); + + b->temporary = 1; + b->tag = (ngx_buf_tag_t) &ngx_http_read_client_request_body; + b->start = cl->buf->start; + b->pos = cl->buf->pos; + b->last = cl->buf->last; + b->end = cl->buf->end; + + *ll = tl; + ll = &tl->next; + + size = cl->buf->last - cl->buf->pos; + + if ((off_t) size > rb->chunked->size) { + cl->buf->pos += rb->chunked->size; + r->headers_in.content_length_n += rb->chunked->size; + rb->chunked->size = 0; + + } else { + rb->chunked->size -= size; + r->headers_in.content_length_n += size; + cl->buf->pos = cl->buf->last; + } + + b->last = cl->buf->pos; + + continue; + } + + if (rc == NGX_DONE) { + + /* a whole response has been parsed successfully */ + + rb->rest = 0; + + tl = ngx_chain_get_free_buf(r->pool, &rb->free); + if (tl == NULL) { + return NGX_HTTP_INTERNAL_SERVER_ERROR; + } + + b = tl->buf; + + ngx_memzero(b, sizeof(ngx_buf_t)); + + b->last_buf = 1; + + *ll = tl; + ll = &tl->next; + + break; + } + + if (rc == NGX_AGAIN) { + + /* set rb->rest, amount of data we want to see next time */ + + rb->rest = rb->chunked->length; + + break; + } + + /* invalid */ + + ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, + "client sent invalid chunked body"); + + return NGX_HTTP_BAD_REQUEST; + } + } + + rc = ngx_http_request_body_save_filter(r, out); + + ngx_chain_update_chains(r->pool, &rb->free, &rb->busy, &out, + (ngx_buf_tag_t) &ngx_http_read_client_request_body); + + return rc; +} + + +static ngx_int_t +ngx_http_request_body_save_filter(ngx_http_request_t *r, ngx_chain_t *in) +{ + ngx_chain_t *cl; + ngx_http_request_body_t *rb; + + rb = r->request_body; + +#if (NGX_DEBUG) + + for (cl = rb->bufs; cl; cl = cl->next) { + ngx_log_debug7(NGX_LOG_DEBUG_EVENT, r->connection->log, 0, + "http body old buf t:%d f:%d %p, pos %p, size: %z " + "file: %O, size: %z", + cl->buf->temporary, cl->buf->in_file, + cl->buf->start, cl->buf->pos, + cl->buf->last - cl->buf->pos, + cl->buf->file_pos, + cl->buf->file_last - cl->buf->file_pos); + } + + for (cl = in; cl; cl = cl->next) { + ngx_log_debug7(NGX_LOG_DEBUG_EVENT, r->connection->log, 0, + "http body new buf t:%d f:%d %p, pos %p, size: %z " + "file: %O, size: %z", + cl->buf->temporary, cl->buf->in_file, + cl->buf->start, cl->buf->pos, + cl->buf->last - cl->buf->pos, + cl->buf->file_pos, + cl->buf->file_last - cl->buf->file_pos); + } + +#endif + + /* TODO: coalesce neighbouring buffers */ + + ngx_chain_add_copy(r->pool, &rb->bufs, in); + + return NGX_OK; +} From mdounin at mdounin.ru Fri Nov 16 11:48:55 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 16 Nov 2012 15:48:55 +0400 Subject: plans for chunked encoding In-Reply-To: <20121019142825.GB7781@w500> References: <20121019140704.GA7781@w500> <20121019141310.GM40452@mdounin.ru> <20121019142825.GB7781@w500> Message-ID: <20121116114855.GQ40452@mdounin.ru> Hello! On Fri, Oct 19, 2012 at 04:28:25PM +0200, Luka Perkov wrote: > Hi Maxim, > > On Fri, Oct 19, 2012 at 06:13:10PM +0400, Maxim Dounin wrote: > > > I'm wondering if there is any work done on this already ? If so I could > > > test some patches... > > > > I'll post it here once there is something to test. > > Thank you. JFYI, chunked request body patches are available here: http://mailman.nginx.org/pipermail/nginx-devel/2012-November/002961.html Review and testing appreciated. -- Maxim Dounin http://nginx.com/support.html From ru at nginx.com Fri Nov 16 12:18:05 2012 From: ru at nginx.com (ru at nginx.com) Date: Fri, 16 Nov 2012 12:18:05 +0000 Subject: [nginx] svn commit: r4915 - trunk/src/http Message-ID: <20121116121806.053633FAA0A@mail.nginx.com> Author: ru Date: 2012-11-16 12:18:05 +0000 (Fri, 16 Nov 2012) New Revision: 4915 URL: http://trac.nginx.org/nginx/changeset/4915/nginx Log: Upstream: honor the "down" flag for a single server. If an upstream block was defined with the only server marked as "down", e.g. upstream u { server 127.0.0.1:8080 down; } an attempt was made to contact the server despite the "down" flag. It is believed that immediate 502 response is better in such a case, and it's also consistent with what is currently done in case of multiple servers all marked as "down". Modified: trunk/src/http/ngx_http_upstream_round_robin.c Modified: trunk/src/http/ngx_http_upstream_round_robin.c =================================================================== --- trunk/src/http/ngx_http_upstream_round_robin.c 2012-11-16 09:37:14 UTC (rev 4914) +++ trunk/src/http/ngx_http_upstream_round_robin.c 2012-11-16 12:18:05 UTC (rev 4915) @@ -430,6 +430,10 @@ if (rrp->peers->single) { peer = &rrp->peers->peer[0]; + if (peer->down) { + goto failed; + } + } else { /* there are several peers */ From nginx at lukaperkov.net Fri Nov 16 13:17:58 2012 From: nginx at lukaperkov.net (Luka Perkov) Date: Fri, 16 Nov 2012 14:17:58 +0100 Subject: plans for chunked encoding In-Reply-To: <20121116114855.GQ40452@mdounin.ru> References: <20121019140704.GA7781@w500> <20121019141310.GM40452@mdounin.ru> <20121019142825.GB7781@w500> <20121116114855.GQ40452@mdounin.ru> Message-ID: <20121116131758.GA32599@w500> Hi Maxim, On Fri, Nov 16, 2012 at 03:48:55PM +0400, Maxim Dounin wrote: > > On Fri, Oct 19, 2012 at 06:13:10PM +0400, Maxim Dounin wrote: > > > > I'm wondering if there is any work done on this already ? If so I could > > > > test some patches... > > > > > > I'll post it here once there is something to test. > > > > Thank you. > > JFYI, chunked request body patches are available here: > > http://mailman.nginx.org/pipermail/nginx-devel/2012-November/002961.html > > Review and testing appreciated. Thank you for the patches. I'll test them this weekend (or early next week) and I'll give you my feedback after that. Regards, Luka From mdounin at mdounin.ru Fri Nov 16 18:29:19 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Fri, 16 Nov 2012 18:29:19 +0000 Subject: [nginx] svn commit: r4916 - trunk/src/http Message-ID: <20121116182920.09CF53FAA5D@mail.nginx.com> Author: mdounin Date: 2012-11-16 18:29:19 +0000 (Fri, 16 Nov 2012) New Revision: 4916 URL: http://trac.nginx.org/nginx/changeset/4916/nginx Log: Upstream: better detection of connect() failures with kqueue. Pending EOF might be reported on both read and write events, whichever comes first, so check both of them. Patch by Yichun Zhang (agentzh), slightly modified. Modified: trunk/src/http/ngx_http_upstream.c Modified: trunk/src/http/ngx_http_upstream.c =================================================================== --- trunk/src/http/ngx_http_upstream.c 2012-11-16 12:18:05 UTC (rev 4915) +++ trunk/src/http/ngx_http_upstream.c 2012-11-16 18:29:19 UTC (rev 4916) @@ -1809,9 +1809,16 @@ #if (NGX_HAVE_KQUEUE) if (ngx_event_flags & NGX_USE_KQUEUE_EVENT) { - if (c->write->pending_eof) { + if (c->write->pending_eof || c->read->pending_eof) { + if (c->write->pending_eof) { + err = c->write->kq_errno; + + } else { + err = c->read->kq_errno; + } + c->log->action = "connecting to upstream"; - (void) ngx_connection_error(c, c->write->kq_errno, + (void) ngx_connection_error(c, err, "kevent() reported that connect() failed"); return NGX_ERROR; } From mdounin at mdounin.ru Fri Nov 16 18:29:47 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 16 Nov 2012 22:29:47 +0400 Subject: [PATCH] making ngx_http_upstream_test_connect catch "connection refused" with kqueue In-Reply-To: References: <20121106175540.GQ40452@mdounin.ru> Message-ID: <20121116182947.GH40452@mdounin.ru> Hello! On Tue, Nov 06, 2012 at 11:22:05AM -0800, agentzh wrote: > Hello! > > On Tue, Nov 6, 2012 at 9:55 AM, Maxim Dounin wrote: > > > > I don't see a reason to introduce another local variable here, > > what about something like > > > > --- a/src/http/ngx_http_upstream.c > > +++ b/src/http/ngx_http_upstream.c > > @@ -1809,9 +1809,16 @@ ngx_http_upstream_test_connect(ngx_conne > > #if (NGX_HAVE_KQUEUE) > > > > if (ngx_event_flags & NGX_USE_KQUEUE_EVENT) { > > - if (c->write->pending_eof) { > > + if (c->write->pending_eof || c->read->pending_eof) { > > + if (c->write->pending_eof) { > > + err = c->write->kq_errno; > > + > > + } else { > > + err = c->read->kq_errno; > > + } > > + > > c->log->action = "connecting to upstream"; > > - (void) ngx_connection_error(c, c->write->kq_errno, > > + (void) ngx_connection_error(c, err, > > "kevent() reported that connect() failed"); > > return NGX_ERROR; > > } > > > > instead? > > > > I'm fine with this new patch :) Committed, thnx. -- Maxim Dounin http://nginx.com/support.html From mdounin at mdounin.ru Sat Nov 17 00:36:45 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Sat, 17 Nov 2012 00:36:45 +0000 Subject: [nginx] svn commit: r4917 - trunk/src/http Message-ID: <20121117003645.3B2C83FAA4A@mail.nginx.com> Author: mdounin Date: 2012-11-17 00:36:44 +0000 (Sat, 17 Nov 2012) New Revision: 4917 URL: http://trac.nginx.org/nginx/changeset/4917/nginx Log: Trailing whitespace fix. Modified: trunk/src/http/ngx_http_upstream.c Modified: trunk/src/http/ngx_http_upstream.c =================================================================== --- trunk/src/http/ngx_http_upstream.c 2012-11-16 18:29:19 UTC (rev 4916) +++ trunk/src/http/ngx_http_upstream.c 2012-11-17 00:36:44 UTC (rev 4917) @@ -1818,7 +1818,7 @@ } c->log->action = "connecting to upstream"; - (void) ngx_connection_error(c, err, + (void) ngx_connection_error(c, err, "kevent() reported that connect() failed"); return NGX_ERROR; } From dmitry.petroff at gmail.com Mon Nov 19 15:24:10 2012 From: dmitry.petroff at gmail.com (Dmitry Petrov) Date: Mon, 19 Nov 2012 18:24:10 +0300 Subject: =?UTF-8?B?U2VnbWVudGF0aW9uIGZhdWx0INCyIERBViDQvNC+0LTRg9C70LU=?= Message-ID: ????????????. ??? ????????????? iphon??????? ??????? ????????? ?? ?????????? ??????? ? DAV ?????? ?????? ????????? (???????? ??? nginx-1.2.1 ? ????? ?????????? printf-??, ????? ??????? ? 1.3.8 ?? ?? ?????): #0 0x08057494 in ngx_ext_rename_file (src=0x4, to=0xbfb3af54, ext=0xbfb3aedc) at src/core/ngx_file.c:545 #1 0x080bb8ee in ngx_http_dav_put_handler (r=0x8c71598) at src/http/modules/ngx_http_dav_module.c:262 #2 0x0809590e in ngx_http_read_client_request_body (r=0x8c71598, post_handler=0x80bb64b ) at src/http/ngx_http_request_body.c:44 #3 0x080bb5f1 in ngx_http_dav_handler (r=0x8c71598) at src/http/modules/ngx_http_dav_module.c:172 #4 0x08081103 in ngx_http_core_content_phase (r=0x8c71598, ph=0x8c86004) at src/http/ngx_http_core_module.c:1403 #5 0x0807febd in ngx_http_core_run_phases (r=0x8c71598) at src/http/ngx_http_core_module.c:877 #6 0x0808c40f in ngx_http_request_handler (ev=0x8caf230) at src/http/ngx_http_request.c:1864 #7 0x08077095 in ngx_epoll_process_events (cycle=0x8c72828, timer=75000, flags=1) at src/event/modules/ngx_epoll_module.c:710 #8 0x08069c6b in ngx_process_events_and_timers (cycle=0x8c72828) at src/event/ngx_event.c:247 #9 0x08075271 in ngx_worker_process_cycle (cycle=0x8c72828, data=0x0) at src/os/unix/ngx_process_cycle.c:806 #10 0x0807234f in ngx_spawn_process (cycle=0x8c72828, proc=0x8075124 , data=0x0, name=0x80e1e47 "worker process", respawn=-3) at src/os/unix/ngx_process.c:198 #11 0x08074484 in ngx_start_worker_processes (cycle=0x8c72828, n=1, type=-3) at src/os/unix/ngx_process_cycle.c:365 #12 0x08073c07 in ngx_master_process_cycle (cycle=0x8c72828) at src/os/unix/ngx_process_cycle.c:137 #13 0x0804ca97 in main (argc=5, argv=0xbfb3b4d4) at src/core/nginx.c:410 ??????????????????? ????????? ??????? ?? ???????, ???????????? ? ?????? ???????. ??????? ??????? ? src/http/ngx_http_request_body.c ? ?????? 42: ??? ???? ?????? <> ???? ?? ?????? PUT-???????? r->request_body ?????????? ?? ???????? ???????, ? r->request_body->temp_file ??? ????? NULL. ? ????? ??? ??????? ??????????? ???? ??????? ??? ?????????? post_handler, ??????? ? ????? ?????? ngx_http_dav_put_handler, ??? ??? ???????????? ??????, ?. ?. temp = &r->request_body->temp_file->file.name ???????? ???? ?? ????? ??????? ??????. ??? ???? ??? ?????????? ?????????????? ????????? ?? NULL r->request_body->temp_file (???? ??? 1.3.8 ???????????). ?? ???? ????????? ????????????? ???????? ???? ????, ?? ???????? ? ???????? ??????. -- Regards, Dmitry -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: nginx-1.3.8-dav-segfault-fix.diff Type: application/octet-stream Size: 699 bytes Desc: not available URL: From serphen at gmail.com Mon Nov 19 15:40:05 2012 From: serphen at gmail.com (Arnaud GRANAL) Date: Mon, 19 Nov 2012 17:40:05 +0200 Subject: =?UTF-8?B?UmU6IFNlZ21lbnRhdGlvbiBmYXVsdCDQsiBEQVYg0LzQvtC00YPQu9C1?= In-Reply-To: References: Message-ID: 2012/11/19 Dmitry Petrov > ????????????. > > Hey, > ??? ????????????? iphon??????? ??????? ????????? ?? ?????????? ??????? ? > DAV ?????? ?????? ????????? (???????? ??? nginx-1.2.1 ? ????? ?????????? > printf-??, ????? ??????? ? 1.3.8 ?? ?? ?????): > #0 0x08057494 in ngx_ext_rename_file (src=0x4, to=0xbfb3af54, > ext=0xbfb3aedc) at src/core/ngx_file.c:545 > #1 0x080bb8ee in ngx_http_dav_put_handler (r=0x8c71598) at > src/http/modules/ngx_http_dav_module.c:262 > #2 0x0809590e in ngx_http_read_client_request_body (r=0x8c71598, > post_handler=0x80bb64b ) at > src/http/ngx_http_request_body.c:44 > [...] > > ??? ???? ??? ?????????? ?????????????? ????????? ?? NULL > r->request_body->temp_file (???? ??? 1.3.8 ???????????). ?? ???? ????????? > ????????????? ???????? ???? ????, ?? ???????? ? ???????? ??????. > > I guess it was fixed in a recent commit by Maxim: # HG changeset patch # User Maxim Dounin # Date 1352393278 -14400 # Node ID c26606971d58dd27df5f72b9fcf90bc883038d76 # Parent 1b2abbd52edc283f69c7513a6cb5406f7913ecae Dav: fixed segfault on PUT if body was already read (ticket #238). If request body reading happens with different options it's possible that there will be no r->request_body->temp_file available (or even no r->request_body available if body was discarded). Return internal server error in this case instead of committing suicide by dereferencing a null pointer. diff --git a/src/http/modules/ngx_http_dav_module.c b/src/http/modules/ngx_http_dav_module.c --- a/src/http/modules/ngx_http_dav_module.c +++ b/src/http/modules/ngx_http_dav_module.c @@ -209,6 +209,11 @@ ngx_http_dav_put_handler(ngx_http_reques ngx_ext_rename_file_t ext; ngx_http_dav_loc_conf_t *dlcf; + if (r->request_body == NULL || r->request_body->temp_file == NULL) { + ngx_http_finalize_request(r, NGX_HTTP_INTERNAL_SERVER_ERROR); + return; + } + ngx_http_map_uri_to_path(r, &path, &root, 0); path.len--; Arnaud. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmitry.petroff at gmail.com Tue Nov 20 09:05:07 2012 From: dmitry.petroff at gmail.com (Dmitry Petrov) Date: Tue, 20 Nov 2012 12:05:07 +0300 Subject: =?UTF-8?B?UmU6IFNlZ21lbnRhdGlvbiBmYXVsdCDQsiBEQVYg0LzQvtC00YPQu9C1?= In-Reply-To: References: Message-ID: On Mon, Nov 19, 2012 at 7:40 PM, Arnaud GRANAL wrote: > I guess it was fixed in a recent commit by Maxim: > [..] > diff --git a/src/http/modules/ngx_http_dav_module.c > b/src/http/modules/ngx_http_dav_module.c > --- a/src/http/modules/ngx_http_dav_module.c > +++ b/src/http/modules/ngx_http_dav_module.c > @@ -209,6 +209,11 @@ ngx_http_dav_put_handler(ngx_http_reques > ngx_ext_rename_file_t ext; > ngx_http_dav_loc_conf_t *dlcf; > > + if (r->request_body == NULL || r->request_body->temp_file == NULL) { > + ngx_http_finalize_request(r, NGX_HTTP_INTERNAL_SERVER_ERROR); > + return; > + } > + > ngx_http_map_uri_to_path(r, &path, &root, 0); > > path.len--; > The problem is that I got this behavior on sequential valid PUT requests. After handling first request, nginx even doesn't try to read body of next request. It just calls ngx_http_dav_put_handler with r->response_body from previous request but with r->response_body->temp_file set to NULL. With extra check for temp_file not being NULL nginx actually reads second request body and handles it properly. -- Regards, Dmitry -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Tue Nov 20 09:21:56 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 20 Nov 2012 13:21:56 +0400 Subject: =?UTF-8?B?UmU6IFNlZ21lbnRhdGlvbiBmYXVsdCDQsiBEQVYg0LzQvtC00YPQu9C1?= In-Reply-To: References: Message-ID: <20121120092156.GN40452@mdounin.ru> Hello! On Tue, Nov 20, 2012 at 12:05:07PM +0300, Dmitry Petrov wrote: > On Mon, Nov 19, 2012 at 7:40 PM, Arnaud GRANAL wrote: > > > I guess it was fixed in a recent commit by Maxim: > > [..] > > diff --git a/src/http/modules/ngx_http_dav_module.c > > b/src/http/modules/ngx_http_dav_module.c > > --- a/src/http/modules/ngx_http_dav_module.c > > +++ b/src/http/modules/ngx_http_dav_module.c > > @@ -209,6 +209,11 @@ ngx_http_dav_put_handler(ngx_http_reques > > ngx_ext_rename_file_t ext; > > ngx_http_dav_loc_conf_t *dlcf; > > > > + if (r->request_body == NULL || r->request_body->temp_file == NULL) { > > + ngx_http_finalize_request(r, NGX_HTTP_INTERNAL_SERVER_ERROR); > > + return; > > + } > > + > > ngx_http_map_uri_to_path(r, &path, &root, 0); > > > > path.len--; > > > The problem is that I got this behavior on sequential valid PUT requests. > After handling first request, nginx even doesn't try to read body of next > request. It just calls ngx_http_dav_put_handler with r->response_body from It looks like this is a generic request body handling problem, which leads to incorrect body processing if an extra write event happens during request body reading. The patch quoted will fix resulting segfault in dav module (which may also happen in other conditions), but the underlying problem also needs to be fixed. If you have a test case to reproduce the problem it might be helpful. > previous request but with r->response_body->temp_file set to NULL. With > extra check for temp_file not being NULL nginx actually reads second > request body and handles it properly. The extra check you've added isn't correct as r->response_body->temp_file may be NULL for valid reasons, e.g. request body reading is done by a normal module like proxy, which doesn't require request body to be in file. -- Maxim Dounin http://nginx.com/support.html From igor at sysoev.ru Tue Nov 20 13:37:56 2012 From: igor at sysoev.ru (igor at sysoev.ru) Date: Tue, 20 Nov 2012 13:37:56 +0000 Subject: [nginx] svn commit: r4918 - in trunk/src: core os/unix Message-ID: <20121120133756.453FB3F9C48@mail.nginx.com> Author: is Date: 2012-11-20 13:37:55 +0000 (Tue, 20 Nov 2012) New Revision: 4918 URL: http://trac.nginx.org/nginx/changeset/4918/nginx Log: Fixed failure to start cache manager and cache loader processes if there were more than 512 listening sockets in configuration. Modified: trunk/src/core/ngx_connection.c trunk/src/os/unix/ngx_process_cycle.c Modified: trunk/src/core/ngx_connection.c =================================================================== --- trunk/src/core/ngx_connection.c 2012-11-17 00:36:44 UTC (rev 4917) +++ trunk/src/core/ngx_connection.c 2012-11-20 13:37:55 UTC (rev 4918) @@ -749,6 +749,8 @@ ls[i].fd = (ngx_socket_t) -1; } + + cycle->listening.nelts = 0; } Modified: trunk/src/os/unix/ngx_process_cycle.c =================================================================== --- trunk/src/os/unix/ngx_process_cycle.c 2012-11-17 00:36:44 UTC (rev 4917) +++ trunk/src/os/unix/ngx_process_cycle.c 2012-11-20 13:37:55 UTC (rev 4918) @@ -1296,14 +1296,19 @@ void *ident[4]; ngx_event_t ev; + /* + * Set correct process type since closing listening Unix domain socket + * in a master process also removes the Unix domain socket file. + */ + ngx_process = NGX_PROCESS_HELPER; + + ngx_close_listening_sockets(cycle); + + /* Set a moderate number of connections for a helper process. */ cycle->connection_n = 512; - ngx_process = NGX_PROCESS_HELPER; - ngx_worker_process_init(cycle, -1); - ngx_close_listening_sockets(cycle); - ngx_memzero(&ev, sizeof(ngx_event_t)); ev.handler = ctx->handler; ev.data = ident; From dmitry.petroff at gmail.com Tue Nov 20 14:29:58 2012 From: dmitry.petroff at gmail.com (Dmitry Petrov) Date: Tue, 20 Nov 2012 17:29:58 +0300 Subject: =?UTF-8?B?UmU6IFNlZ21lbnRhdGlvbiBmYXVsdCDQsiBEQVYg0LzQvtC00YPQu9C1?= In-Reply-To: <20121120092156.GN40452@mdounin.ru> References: <20121120092156.GN40452@mdounin.ru> Message-ID: Hello. Finally I was able to make a test that reproduces bug 100% of time. Here it is (nginx sources included): http://rghost.net/41694908 You need to: 1. tar zxf nginx-webdav-segfault.tar.gz 2. cd dav-bug/nginx-1.3.8 3. ./cf ? this will configure nginx with needed modules and /tmp as prefix 4. cd .. 5. In one console start nginx via 'make r' command. It will automatically use dav.conf which will start nginx on 0.0.0.0:8080 6. Attach to nginx process: gdb --pid `pgrep -n nginx` 7. Execute make ? this will launch test trying to connect to localhost:8080, upload some files etc. Make sure that 8181 port is free before running test case - it create dummy "HTTP server" that simply ignores all requests. I wasn't able to get this bug without proxy_pass'es between webdav PUT's. 8. Monitor console where you've started gdb - you will get SIGSEGV here The symptoms of the crash are the same as I described earlier: r->request_body is not NULL, set from some kind of previous request. r->request_body->temp_file is NULL. This test case just replaying dump recorded from Iphone HTTP session using TCP_CORK to achive exactly the same packets send (with scrambled payload ofc). Hope this helps. On Tue, Nov 20, 2012 at 1:21 PM, Maxim Dounin wrote: > Hello! > > On Tue, Nov 20, 2012 at 12:05:07PM +0300, Dmitry Petrov wrote: > > > On Mon, Nov 19, 2012 at 7:40 PM, Arnaud GRANAL > wrote: > > > > > I guess it was fixed in a recent commit by Maxim: > > > [..] > > > diff --git a/src/http/modules/ngx_http_dav_module.c > > > b/src/http/modules/ngx_http_dav_module.c > > > --- a/src/http/modules/ngx_http_dav_module.c > > > +++ b/src/http/modules/ngx_http_dav_module.c > > > @@ -209,6 +209,11 @@ ngx_http_dav_put_handler(ngx_http_reques > > > ngx_ext_rename_file_t ext; > > > ngx_http_dav_loc_conf_t *dlcf; > > > > > > + if (r->request_body == NULL || r->request_body->temp_file == > NULL) { > > > + ngx_http_finalize_request(r, NGX_HTTP_INTERNAL_SERVER_ERROR); > > > + return; > > > + } > > > + > > > ngx_http_map_uri_to_path(r, &path, &root, 0); > > > > > > path.len--; > > > > > The problem is that I got this behavior on sequential valid PUT requests. > > After handling first request, nginx even doesn't try to read body of next > > request. It just calls ngx_http_dav_put_handler with r->response_body > from > > It looks like this is a generic request body handling problem, > which leads to incorrect body processing if an extra write event > happens during request body reading. > > The patch quoted will fix resulting segfault in dav module (which > may also happen in other conditions), but the underlying problem > also needs to be fixed. If you have a test case to reproduce the > problem it might be helpful. > > > previous request but with r->response_body->temp_file set to NULL. With > > extra check for temp_file not being NULL nginx actually reads second > > request body and handles it properly. > > The extra check you've added isn't correct as > r->response_body->temp_file may be NULL for valid reasons, e.g. > request body reading is done by a normal module like proxy, which > doesn't require request body to be in file. > > -- > Maxim Dounin > http://nginx.com/support.html > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -- Regards, Dmitry -------------- next part -------------- An HTML attachment was scrubbed... URL: From flygoast at 126.com Tue Nov 20 23:53:04 2012 From: flygoast at 126.com (=?GBK?B?t+u5yw==?=) Date: Wed, 21 Nov 2012 07:53:04 +0800 (CST) Subject: nginx proxy cache support contional request( If-Modified-Since) Message-ID: <26a9a8ec.69515.13b203d45f7.Coremail.flygoast@126.com> Hi, I'm using nginx as a reverse proxy with cache. My content almost never changes. So I want to use a conditional GET request when fetch content from backend. However, nginx don't support it. Who have a such patch for it? Can give me? Thanks. PS, I want to know someday nginx will implement it or refuse to implement it? Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Wed Nov 21 00:51:38 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Wed, 21 Nov 2012 00:51:38 +0000 Subject: [nginx] svn commit: r4919 - trunk/src/http/modules Message-ID: <20121121005138.EEF813F9F96@mail.nginx.com> Author: mdounin Date: 2012-11-21 00:51:37 +0000 (Wed, 21 Nov 2012) New Revision: 4919 URL: http://trac.nginx.org/nginx/changeset/4919/nginx Log: Dav: fixed segfault on PUT if body was already read (ticket #238). If request body reading happens with different options it's possible that there will be no r->request_body->temp_file available (or even no r->request_body available if body was discarded). Return internal server error in this case instead of committing suicide by dereferencing a null pointer. Modified: trunk/src/http/modules/ngx_http_dav_module.c Modified: trunk/src/http/modules/ngx_http_dav_module.c =================================================================== --- trunk/src/http/modules/ngx_http_dav_module.c 2012-11-20 13:37:55 UTC (rev 4918) +++ trunk/src/http/modules/ngx_http_dav_module.c 2012-11-21 00:51:37 UTC (rev 4919) @@ -209,6 +209,11 @@ ngx_ext_rename_file_t ext; ngx_http_dav_loc_conf_t *dlcf; + if (r->request_body == NULL || r->request_body->temp_file == NULL) { + ngx_http_finalize_request(r, NGX_HTTP_INTERNAL_SERVER_ERROR); + return; + } + ngx_http_map_uri_to_path(r, &path, &root, 0); path.len--; From mdounin at mdounin.ru Wed Nov 21 00:52:35 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Wed, 21 Nov 2012 00:52:35 +0000 Subject: [nginx] svn commit: r4920 - trunk/src/os/unix Message-ID: <20121121005235.972C03F9F8A@mail.nginx.com> Author: mdounin Date: 2012-11-21 00:52:35 +0000 (Wed, 21 Nov 2012) New Revision: 4920 URL: http://trac.nginx.org/nginx/changeset/4920/nginx Log: Core: added debug logging of writev() in ngx_write_chain_to_file(). Modified: trunk/src/os/unix/ngx_files.c Modified: trunk/src/os/unix/ngx_files.c =================================================================== --- trunk/src/os/unix/ngx_files.c 2012-11-21 00:51:37 UTC (rev 4919) +++ trunk/src/os/unix/ngx_files.c 2012-11-21 00:52:35 UTC (rev 4920) @@ -241,6 +241,9 @@ return NGX_ERROR; } + ngx_log_debug2(NGX_LOG_DEBUG_CORE, file->log, 0, + "writev: %d, %z", file->fd, n); + file->sys_offset += n; file->offset += n; total += n; From mdounin at mdounin.ru Wed Nov 21 00:54:01 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Wed, 21 Nov 2012 00:54:01 +0000 Subject: [nginx] svn commit: r4921 - trunk/src/http Message-ID: <20121121005401.AAF073F9F8A@mail.nginx.com> Author: mdounin Date: 2012-11-21 00:54:01 +0000 (Wed, 21 Nov 2012) New Revision: 4921 URL: http://trac.nginx.org/nginx/changeset/4921/nginx Log: Request body: fixed "501 Not Implemented" error handling. It is not about "Method" but a generic message, and is expected to be used e.g. if specified Transfer-Encoding is not supported. Fixed message to match RFC 2616. Additionally, disable keepalive on such errors as we won't be able to read request body correctly if we don't understand Transfer-Encoding used. Modified: trunk/src/http/ngx_http_header_filter_module.c trunk/src/http/ngx_http_special_response.c Modified: trunk/src/http/ngx_http_header_filter_module.c =================================================================== --- trunk/src/http/ngx_http_header_filter_module.c 2012-11-21 00:52:35 UTC (rev 4920) +++ trunk/src/http/ngx_http_header_filter_module.c 2012-11-21 00:54:01 UTC (rev 4921) @@ -112,7 +112,7 @@ #define NGX_HTTP_OFF_5XX (NGX_HTTP_LAST_4XX - 400 + NGX_HTTP_OFF_4XX) ngx_string("500 Internal Server Error"), - ngx_string("501 Method Not Implemented"), + ngx_string("501 Not Implemented"), ngx_string("502 Bad Gateway"), ngx_string("503 Service Temporarily Unavailable"), ngx_string("504 Gateway Time-out"), Modified: trunk/src/http/ngx_http_special_response.c =================================================================== --- trunk/src/http/ngx_http_special_response.c 2012-11-21 00:52:35 UTC (rev 4920) +++ trunk/src/http/ngx_http_special_response.c 2012-11-21 00:54:01 UTC (rev 4921) @@ -260,9 +260,9 @@ static char ngx_http_error_501_page[] = "" CRLF -"501 Method Not Implemented" CRLF +"501 Not Implemented" CRLF "" CRLF -"

501 Method Not Implemented

" CRLF +"

501 Not Implemented

" CRLF ; @@ -384,6 +384,7 @@ case NGX_HTTPS_CERT_ERROR: case NGX_HTTPS_NO_CERT: case NGX_HTTP_INTERNAL_SERVER_ERROR: + case NGX_HTTP_NOT_IMPLEMENTED: r->keepalive = 0; } } From mdounin at mdounin.ru Wed Nov 21 00:55:06 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Wed, 21 Nov 2012 00:55:06 +0000 Subject: [nginx] svn commit: r4922 - trunk/src/http Message-ID: <20121121005506.C097B3F9F8A@mail.nginx.com> Author: mdounin Date: 2012-11-21 00:55:06 +0000 (Wed, 21 Nov 2012) New Revision: 4922 URL: http://trac.nginx.org/nginx/changeset/4922/nginx Log: Request body: $request_body variable generalization. The $request_body variable was assuming there can't be more than two buffers. While this is currently true due to request body reading implementation details, this is not a good thing to depend on and may change in the future. Modified: trunk/src/http/ngx_http_variables.c Modified: trunk/src/http/ngx_http_variables.c =================================================================== --- trunk/src/http/ngx_http_variables.c 2012-11-21 00:54:01 UTC (rev 4921) +++ trunk/src/http/ngx_http_variables.c 2012-11-21 00:55:06 UTC (rev 4922) @@ -1767,7 +1767,7 @@ { u_char *p; size_t len; - ngx_buf_t *buf, *next; + ngx_buf_t *buf; ngx_chain_t *cl; if (r->request_body == NULL @@ -1792,18 +1792,26 @@ return NGX_OK; } - next = cl->next->buf; - len = (buf->last - buf->pos) + (next->last - next->pos); + len = buf->last - buf->pos; + cl = cl->next; + for ( /* void */ ; cl; cl = cl->next) { + buf = cl->buf; + len += buf->last - buf->pos; + } + p = ngx_pnalloc(r->pool, len); if (p == NULL) { return NGX_ERROR; } v->data = p; + cl = r->request_body->bufs; - p = ngx_cpymem(p, buf->pos, buf->last - buf->pos); - ngx_memcpy(p, next->pos, next->last - next->pos); + for ( /* void */ ; cl; cl = cl->next) { + buf = cl->buf; + p = ngx_cpymem(p, buf->pos, buf->last - buf->pos); + } v->len = len; v->valid = 1; From mdounin at mdounin.ru Wed Nov 21 00:55:51 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Wed, 21 Nov 2012 00:55:51 +0000 Subject: [nginx] svn commit: r4923 - trunk/src/http Message-ID: <20121121005552.0CD5B3F9C14@mail.nginx.com> Author: mdounin Date: 2012-11-21 00:55:50 +0000 (Wed, 21 Nov 2012) New Revision: 4923 URL: http://trac.nginx.org/nginx/changeset/4923/nginx Log: Request body: code duplication reduced, no functional changes. The r->request_body_in_file_only with empty body case is now handled in ngx_http_write_request_body(). Modified: trunk/src/http/ngx_http_request_body.c Modified: trunk/src/http/ngx_http_request_body.c =================================================================== --- trunk/src/http/ngx_http_request_body.c 2012-11-21 00:55:06 UTC (rev 4922) +++ trunk/src/http/ngx_http_request_body.c 2012-11-21 00:55:50 UTC (rev 4923) @@ -33,7 +33,6 @@ ssize_t size; ngx_buf_t *b; ngx_chain_t *cl, **next; - ngx_temp_file_t *tf; ngx_http_request_body_t *rb; ngx_http_core_loc_conf_t *clcf; @@ -65,32 +64,9 @@ if (r->headers_in.content_length_n == 0) { if (r->request_body_in_file_only) { - tf = ngx_pcalloc(r->pool, sizeof(ngx_temp_file_t)); - if (tf == NULL) { + if (ngx_http_write_request_body(r, NULL) != NGX_OK) { return NGX_HTTP_INTERNAL_SERVER_ERROR; } - - tf->file.fd = NGX_INVALID_FILE; - tf->file.log = r->connection->log; - tf->path = clcf->client_body_temp_path; - tf->pool = r->pool; - tf->warn = "a client request body is buffered to a temporary file"; - tf->log_level = r->request_body_file_log_level; - tf->persistent = r->request_body_in_persistent_file; - tf->clean = r->request_body_in_clean_file; - - if (r->request_body_file_group_access) { - tf->access = 0660; - } - - rb->temp_file = tf; - - if (ngx_create_temp_file(&tf->file, tf->path, tf->pool, - tf->persistent, tf->clean, tf->access) - != NGX_OK) - { - return NGX_HTTP_INTERNAL_SERVER_ERROR; - } } post_handler(r); @@ -419,6 +395,19 @@ } rb->temp_file = tf; + + if (body == NULL) { + /* empty body with r->request_body_in_file_only */ + + if (ngx_create_temp_file(&tf->file, tf->path, tf->pool, + tf->persistent, tf->clean, tf->access) + != NGX_OK) + { + return NGX_ERROR; + } + + return NGX_OK; + } } n = ngx_write_chain_to_temp_file(rb->temp_file, body); From mdounin at mdounin.ru Wed Nov 21 00:57:16 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Wed, 21 Nov 2012 00:57:16 +0000 Subject: [nginx] svn commit: r4924 - trunk/src/http Message-ID: <20121121005716.8F86C3F9C46@mail.nginx.com> Author: mdounin Date: 2012-11-21 00:57:16 +0000 (Wed, 21 Nov 2012) New Revision: 4924 URL: http://trac.nginx.org/nginx/changeset/4924/nginx Log: Request body: fixed socket leak on errors. The r->main->count reference counter was always incremented in ngx_http_read_client_request_body(), while it is only needs to be incremented on positive returns. Modified: trunk/src/http/ngx_http_request_body.c Modified: trunk/src/http/ngx_http_request_body.c =================================================================== --- trunk/src/http/ngx_http_request_body.c 2012-11-21 00:55:50 UTC (rev 4923) +++ trunk/src/http/ngx_http_request_body.c 2012-11-21 00:57:16 UTC (rev 4924) @@ -31,6 +31,7 @@ { size_t preread; ssize_t size; + ngx_int_t rc; ngx_buf_t *b; ngx_chain_t *cl, **next; ngx_http_request_body_t *rb; @@ -44,12 +45,14 @@ } if (ngx_http_test_expect(r) != NGX_OK) { - return NGX_HTTP_INTERNAL_SERVER_ERROR; + rc = NGX_HTTP_INTERNAL_SERVER_ERROR; + goto done; } rb = ngx_pcalloc(r->pool, sizeof(ngx_http_request_body_t)); if (rb == NULL) { - return NGX_HTTP_INTERNAL_SERVER_ERROR; + rc = NGX_HTTP_INTERNAL_SERVER_ERROR; + goto done; } r->request_body = rb; @@ -65,7 +68,8 @@ if (r->request_body_in_file_only) { if (ngx_http_write_request_body(r, NULL) != NGX_OK) { - return NGX_HTTP_INTERNAL_SERVER_ERROR; + rc = NGX_HTTP_INTERNAL_SERVER_ERROR; + goto done; } } @@ -95,7 +99,8 @@ b = ngx_calloc_buf(r->pool); if (b == NULL) { - return NGX_HTTP_INTERNAL_SERVER_ERROR; + rc = NGX_HTTP_INTERNAL_SERVER_ERROR; + goto done; } b->temporary = 1; @@ -106,7 +111,8 @@ rb->bufs = ngx_alloc_chain_link(r->pool); if (rb->bufs == NULL) { - return NGX_HTTP_INTERNAL_SERVER_ERROR; + rc = NGX_HTTP_INTERNAL_SERVER_ERROR; + goto done; } rb->bufs->buf = b; @@ -124,7 +130,8 @@ if (r->request_body_in_file_only) { if (ngx_http_write_request_body(r, rb->bufs) != NGX_OK) { - return NGX_HTTP_INTERNAL_SERVER_ERROR; + rc = NGX_HTTP_INTERNAL_SERVER_ERROR; + goto done; } } @@ -151,7 +158,8 @@ r->read_event_handler = ngx_http_read_client_request_body_handler; - return ngx_http_do_read_client_request_body(r); + rc = ngx_http_do_read_client_request_body(r); + goto done; } next = &rb->bufs->next; @@ -181,12 +189,14 @@ rb->buf = ngx_create_temp_buf(r->pool, size); if (rb->buf == NULL) { - return NGX_HTTP_INTERNAL_SERVER_ERROR; + rc = NGX_HTTP_INTERNAL_SERVER_ERROR; + goto done; } cl = ngx_alloc_chain_link(r->pool); if (cl == NULL) { - return NGX_HTTP_INTERNAL_SERVER_ERROR; + rc = NGX_HTTP_INTERNAL_SERVER_ERROR; + goto done; } cl->buf = rb->buf; @@ -211,7 +221,15 @@ r->read_event_handler = ngx_http_read_client_request_body_handler; - return ngx_http_do_read_client_request_body(r); + rc = ngx_http_do_read_client_request_body(r); + +done: + + if (rc >= NGX_HTTP_SPECIAL_RESPONSE) { + r->main->count--; + } + + return rc; } From mdounin at mdounin.ru Wed Nov 21 00:57:57 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Wed, 21 Nov 2012 00:57:57 +0000 Subject: [nginx] svn commit: r4925 - trunk/src/http Message-ID: <20121121005757.337673F9C4E@mail.nginx.com> Author: mdounin Date: 2012-11-21 00:57:56 +0000 (Wed, 21 Nov 2012) New Revision: 4925 URL: http://trac.nginx.org/nginx/changeset/4925/nginx Log: Request body: properly handle events while discarding body. An attempt to call ngx_handle_read_event() before actually reading data from a socket might result in read event being disabled, which is wrong. Catched by body.t test on Solaris. Modified: trunk/src/http/ngx_http_request_body.c Modified: trunk/src/http/ngx_http_request_body.c =================================================================== --- trunk/src/http/ngx_http_request_body.c 2012-11-21 00:57:16 UTC (rev 4924) +++ trunk/src/http/ngx_http_request_body.c 2012-11-21 00:57:56 UTC (rev 4925) @@ -482,20 +482,22 @@ } } + if (ngx_http_read_discarded_request_body(r) == NGX_OK) { + r->lingering_close = 0; + return NGX_OK; + } + + /* == NGX_AGAIN */ + r->read_event_handler = ngx_http_discarded_request_body_handler; if (ngx_handle_read_event(rev, 0) != NGX_OK) { return NGX_HTTP_INTERNAL_SERVER_ERROR; } - if (ngx_http_read_discarded_request_body(r) == NGX_OK) { - r->lingering_close = 0; + r->count++; + r->discard_body = 1; - } else { - r->count++; - r->discard_body = 1; - } - return NGX_OK; } From mdounin at mdounin.ru Wed Nov 21 00:59:16 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Wed, 21 Nov 2012 00:59:16 +0000 Subject: [nginx] svn commit: r4926 - in trunk/src/http: . modules Message-ID: <20121121005916.93FEA3F9C46@mail.nginx.com> Author: mdounin Date: 2012-11-21 00:59:16 +0000 (Wed, 21 Nov 2012) New Revision: 4926 URL: http://trac.nginx.org/nginx/changeset/4926/nginx Log: Request body: chunked parsing moved to ngx_http_parse.c from proxy. No functional changes. Modified: trunk/src/http/modules/ngx_http_proxy_module.c trunk/src/http/ngx_http.h trunk/src/http/ngx_http_parse.c Modified: trunk/src/http/modules/ngx_http_proxy_module.c =================================================================== --- trunk/src/http/modules/ngx_http_proxy_module.c 2012-11-21 00:57:56 UTC (rev 4925) +++ trunk/src/http/modules/ngx_http_proxy_module.c 2012-11-21 00:59:16 UTC (rev 4926) @@ -81,13 +81,10 @@ typedef struct { ngx_http_status_t status; + ngx_http_chunked_t chunked; ngx_http_proxy_vars_t vars; size_t internal_body_length; - ngx_uint_t state; - off_t size; - off_t length; - ngx_uint_t head; /* unsigned head:1 */ } ngx_http_proxy_ctx_t; @@ -1252,7 +1249,7 @@ ctx->status.count = 0; ctx->status.start = NULL; ctx->status.end = NULL; - ctx->state = 0; + ctx->chunked.state = 0; r->upstream->process_header = ngx_http_proxy_process_status_line; r->upstream->pipe->input_filter = ngx_http_proxy_copy_filter; @@ -1617,265 +1614,6 @@ } -static ngx_inline ngx_int_t -ngx_http_proxy_parse_chunked(ngx_http_request_t *r, ngx_buf_t *buf) -{ - u_char *pos, ch, c; - ngx_int_t rc; - ngx_http_proxy_ctx_t *ctx; - enum { - sw_chunk_start = 0, - sw_chunk_size, - sw_chunk_extension, - sw_chunk_extension_almost_done, - sw_chunk_data, - sw_after_data, - sw_after_data_almost_done, - sw_last_chunk_extension, - sw_last_chunk_extension_almost_done, - sw_trailer, - sw_trailer_almost_done, - sw_trailer_header, - sw_trailer_header_almost_done - } state; - - ctx = ngx_http_get_module_ctx(r, ngx_http_proxy_module); - - if (ctx == NULL) { - return NGX_ERROR; - } - - state = ctx->state; - - if (state == sw_chunk_data && ctx->size == 0) { - state = sw_after_data; - } - - rc = NGX_AGAIN; - - for (pos = buf->pos; pos < buf->last; pos++) { - - ch = *pos; - - ngx_log_debug2(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, - "http proxy chunked byte: %02Xd s:%d", ch, state); - - switch (state) { - - case sw_chunk_start: - if (ch >= '0' && ch <= '9') { - state = sw_chunk_size; - ctx->size = ch - '0'; - break; - } - - c = (u_char) (ch | 0x20); - - if (c >= 'a' && c <= 'f') { - state = sw_chunk_size; - ctx->size = c - 'a' + 10; - break; - } - - goto invalid; - - case sw_chunk_size: - if (ch >= '0' && ch <= '9') { - ctx->size = ctx->size * 16 + (ch - '0'); - break; - } - - c = (u_char) (ch | 0x20); - - if (c >= 'a' && c <= 'f') { - ctx->size = ctx->size * 16 + (c - 'a' + 10); - break; - } - - if (ctx->size == 0) { - - switch (ch) { - case CR: - state = sw_last_chunk_extension_almost_done; - break; - case LF: - state = sw_trailer; - break; - case ';': - case ' ': - case '\t': - state = sw_last_chunk_extension; - break; - default: - goto invalid; - } - - break; - } - - switch (ch) { - case CR: - state = sw_chunk_extension_almost_done; - break; - case LF: - state = sw_chunk_data; - break; - case ';': - case ' ': - case '\t': - state = sw_chunk_extension; - break; - default: - goto invalid; - } - - break; - - case sw_chunk_extension: - switch (ch) { - case CR: - state = sw_chunk_extension_almost_done; - break; - case LF: - state = sw_chunk_data; - } - break; - - case sw_chunk_extension_almost_done: - if (ch == LF) { - state = sw_chunk_data; - break; - } - goto invalid; - - case sw_chunk_data: - rc = NGX_OK; - goto data; - - case sw_after_data: - switch (ch) { - case CR: - state = sw_after_data_almost_done; - break; - case LF: - state = sw_chunk_start; - } - break; - - case sw_after_data_almost_done: - if (ch == LF) { - state = sw_chunk_start; - break; - } - goto invalid; - - case sw_last_chunk_extension: - switch (ch) { - case CR: - state = sw_last_chunk_extension_almost_done; - break; - case LF: - state = sw_trailer; - } - break; - - case sw_last_chunk_extension_almost_done: - if (ch == LF) { - state = sw_trailer; - break; - } - goto invalid; - - case sw_trailer: - switch (ch) { - case CR: - state = sw_trailer_almost_done; - break; - case LF: - goto done; - default: - state = sw_trailer_header; - } - break; - - case sw_trailer_almost_done: - if (ch == LF) { - goto done; - } - goto invalid; - - case sw_trailer_header: - switch (ch) { - case CR: - state = sw_trailer_header_almost_done; - break; - case LF: - state = sw_trailer; - } - break; - - case sw_trailer_header_almost_done: - if (ch == LF) { - state = sw_trailer; - break; - } - goto invalid; - - } - } - -data: - - ctx->state = state; - buf->pos = pos; - - switch (state) { - - case sw_chunk_start: - ctx->length = 3 /* "0" LF LF */; - break; - case sw_chunk_size: - ctx->length = 2 /* LF LF */ - + (ctx->size ? ctx->size + 4 /* LF "0" LF LF */ : 0); - break; - case sw_chunk_extension: - case sw_chunk_extension_almost_done: - ctx->length = 1 /* LF */ + ctx->size + 4 /* LF "0" LF LF */; - break; - case sw_chunk_data: - ctx->length = ctx->size + 4 /* LF "0" LF LF */; - break; - case sw_after_data: - case sw_after_data_almost_done: - ctx->length = 4 /* LF "0" LF LF */; - break; - case sw_last_chunk_extension: - case sw_last_chunk_extension_almost_done: - ctx->length = 2 /* LF LF */; - break; - case sw_trailer: - case sw_trailer_almost_done: - ctx->length = 1 /* LF */; - break; - case sw_trailer_header: - case sw_trailer_header_almost_done: - ctx->length = 2 /* LF LF */; - break; - - } - - return rc; - -done: - - return NGX_DONE; - -invalid: - - return NGX_ERROR; -} - - static ngx_int_t ngx_http_proxy_chunked_filter(ngx_event_pipe_t *p, ngx_buf_t *buf) { @@ -1901,7 +1639,7 @@ for ( ;; ) { - rc = ngx_http_proxy_parse_chunked(r, buf); + rc = ngx_http_parse_chunked(r, buf, &ctx->chunked); if (rc == NGX_OK) { @@ -1952,16 +1690,16 @@ ngx_log_debug2(NGX_LOG_DEBUG_EVENT, p->log, 0, "input buf #%d %p", b->num, b->pos); - if (buf->last - buf->pos >= ctx->size) { + if (buf->last - buf->pos >= ctx->chunked.size) { - buf->pos += ctx->size; + buf->pos += ctx->chunked.size; b->last = buf->pos; - ctx->size = 0; + ctx->chunked.size = 0; continue; } - ctx->size -= buf->last - buf->pos; + ctx->chunked.size -= buf->last - buf->pos; buf->pos = buf->last; b->last = buf->last; @@ -1982,7 +1720,7 @@ /* set p->length, minimal amount of data we want to see */ - p->length = ctx->length; + p->length = ctx->chunked.length; break; } @@ -1997,7 +1735,7 @@ ngx_log_debug2(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "http proxy chunked state %d, length %d", - ctx->state, p->length); + ctx->chunked.state, p->length); if (b) { b->shadow = buf; @@ -2094,7 +1832,7 @@ for ( ;; ) { - rc = ngx_http_proxy_parse_chunked(r, buf); + rc = ngx_http_parse_chunked(r, buf, &ctx->chunked); if (rc == NGX_OK) { @@ -2116,13 +1854,13 @@ b->pos = buf->pos; b->tag = u->output.tag; - if (buf->last - buf->pos >= ctx->size) { - buf->pos += ctx->size; + if (buf->last - buf->pos >= ctx->chunked.size) { + buf->pos += ctx->chunked.size; b->last = buf->pos; - ctx->size = 0; + ctx->chunked.size = 0; } else { - ctx->size -= buf->last - buf->pos; + ctx->chunked.size -= buf->last - buf->pos; buf->pos = buf->last; b->last = buf->last; } Modified: trunk/src/http/ngx_http.h =================================================================== --- trunk/src/http/ngx_http.h 2012-11-21 00:57:56 UTC (rev 4925) +++ trunk/src/http/ngx_http.h 2012-11-21 00:59:16 UTC (rev 4926) @@ -18,6 +18,7 @@ typedef struct ngx_http_cache_s ngx_http_cache_t; typedef struct ngx_http_file_cache_s ngx_http_file_cache_t; typedef struct ngx_http_log_ctx_s ngx_http_log_ctx_t; +typedef struct ngx_http_chunked_s ngx_http_chunked_t; typedef ngx_int_t (*ngx_http_header_handler_pt)(ngx_http_request_t *r, ngx_table_elt_t *h, ngx_uint_t offset); @@ -52,6 +53,13 @@ }; +struct ngx_http_chunked_s { + ngx_uint_t state; + off_t size; + off_t length; +}; + + typedef struct { ngx_uint_t http_version; ngx_uint_t code; @@ -92,6 +100,8 @@ ngx_str_t *value); void ngx_http_split_args(ngx_http_request_t *r, ngx_str_t *uri, ngx_str_t *args); +ngx_int_t ngx_http_parse_chunked(ngx_http_request_t *r, ngx_buf_t *b, + ngx_http_chunked_t *ctx); ngx_int_t ngx_http_find_server_conf(ngx_http_request_t *r); Modified: trunk/src/http/ngx_http_parse.c =================================================================== --- trunk/src/http/ngx_http_parse.c 2012-11-21 00:57:56 UTC (rev 4925) +++ trunk/src/http/ngx_http_parse.c 2012-11-21 00:59:16 UTC (rev 4926) @@ -1818,3 +1818,256 @@ args->len = 0; } } + + +ngx_int_t +ngx_http_parse_chunked(ngx_http_request_t *r, ngx_buf_t *b, + ngx_http_chunked_t *ctx) +{ + u_char *pos, ch, c; + ngx_int_t rc; + enum { + sw_chunk_start = 0, + sw_chunk_size, + sw_chunk_extension, + sw_chunk_extension_almost_done, + sw_chunk_data, + sw_after_data, + sw_after_data_almost_done, + sw_last_chunk_extension, + sw_last_chunk_extension_almost_done, + sw_trailer, + sw_trailer_almost_done, + sw_trailer_header, + sw_trailer_header_almost_done + } state; + + state = ctx->state; + + if (state == sw_chunk_data && ctx->size == 0) { + state = sw_after_data; + } + + rc = NGX_AGAIN; + + for (pos = b->pos; pos < b->last; pos++) { + + ch = *pos; + + ngx_log_debug2(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "http chunked byte: %02Xd s:%d", ch, state); + + switch (state) { + + case sw_chunk_start: + if (ch >= '0' && ch <= '9') { + state = sw_chunk_size; + ctx->size = ch - '0'; + break; + } + + c = (u_char) (ch | 0x20); + + if (c >= 'a' && c <= 'f') { + state = sw_chunk_size; + ctx->size = c - 'a' + 10; + break; + } + + goto invalid; + + case sw_chunk_size: + if (ch >= '0' && ch <= '9') { + ctx->size = ctx->size * 16 + (ch - '0'); + break; + } + + c = (u_char) (ch | 0x20); + + if (c >= 'a' && c <= 'f') { + ctx->size = ctx->size * 16 + (c - 'a' + 10); + break; + } + + if (ctx->size == 0) { + + switch (ch) { + case CR: + state = sw_last_chunk_extension_almost_done; + break; + case LF: + state = sw_trailer; + break; + case ';': + case ' ': + case '\t': + state = sw_last_chunk_extension; + break; + default: + goto invalid; + } + + break; + } + + switch (ch) { + case CR: + state = sw_chunk_extension_almost_done; + break; + case LF: + state = sw_chunk_data; + break; + case ';': + case ' ': + case '\t': + state = sw_chunk_extension; + break; + default: + goto invalid; + } + + break; + + case sw_chunk_extension: + switch (ch) { + case CR: + state = sw_chunk_extension_almost_done; + break; + case LF: + state = sw_chunk_data; + } + break; + + case sw_chunk_extension_almost_done: + if (ch == LF) { + state = sw_chunk_data; + break; + } + goto invalid; + + case sw_chunk_data: + rc = NGX_OK; + goto data; + + case sw_after_data: + switch (ch) { + case CR: + state = sw_after_data_almost_done; + break; + case LF: + state = sw_chunk_start; + } + break; + + case sw_after_data_almost_done: + if (ch == LF) { + state = sw_chunk_start; + break; + } + goto invalid; + + case sw_last_chunk_extension: + switch (ch) { + case CR: + state = sw_last_chunk_extension_almost_done; + break; + case LF: + state = sw_trailer; + } + break; + + case sw_last_chunk_extension_almost_done: + if (ch == LF) { + state = sw_trailer; + break; + } + goto invalid; + + case sw_trailer: + switch (ch) { + case CR: + state = sw_trailer_almost_done; + break; + case LF: + goto done; + default: + state = sw_trailer_header; + } + break; + + case sw_trailer_almost_done: + if (ch == LF) { + goto done; + } + goto invalid; + + case sw_trailer_header: + switch (ch) { + case CR: + state = sw_trailer_header_almost_done; + break; + case LF: + state = sw_trailer; + } + break; + + case sw_trailer_header_almost_done: + if (ch == LF) { + state = sw_trailer; + break; + } + goto invalid; + + } + } + +data: + + ctx->state = state; + b->pos = pos; + + switch (state) { + + case sw_chunk_start: + ctx->length = 3 /* "0" LF LF */; + break; + case sw_chunk_size: + ctx->length = 2 /* LF LF */ + + (ctx->size ? ctx->size + 4 /* LF "0" LF LF */ : 0); + break; + case sw_chunk_extension: + case sw_chunk_extension_almost_done: + ctx->length = 1 /* LF */ + ctx->size + 4 /* LF "0" LF LF */; + break; + case sw_chunk_data: + ctx->length = ctx->size + 4 /* LF "0" LF LF */; + break; + case sw_after_data: + case sw_after_data_almost_done: + ctx->length = 4 /* LF "0" LF LF */; + break; + case sw_last_chunk_extension: + case sw_last_chunk_extension_almost_done: + ctx->length = 2 /* LF LF */; + break; + case sw_trailer: + case sw_trailer_almost_done: + ctx->length = 1 /* LF */; + break; + case sw_trailer_header: + case sw_trailer_header_almost_done: + ctx->length = 2 /* LF LF */; + break; + + } + + return rc; + +done: + + return NGX_DONE; + +invalid: + + return NGX_ERROR; +} From mdounin at mdounin.ru Wed Nov 21 01:02:56 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Wed, 21 Nov 2012 01:02:56 +0000 Subject: [nginx] svn commit: r4927 - trunk/src/http Message-ID: <20121121010256.D7E933F9F2F@mail.nginx.com> Author: mdounin Date: 2012-11-21 01:02:56 +0000 (Wed, 21 Nov 2012) New Revision: 4927 URL: http://trac.nginx.org/nginx/changeset/4927/nginx Log: Request body: adjust b->pos when chunked parsing done. This is a nop for the current code, though will allow to correctly parse pipelined requests. Modified: trunk/src/http/ngx_http_parse.c Modified: trunk/src/http/ngx_http_parse.c =================================================================== --- trunk/src/http/ngx_http_parse.c 2012-11-21 00:59:16 UTC (rev 4926) +++ trunk/src/http/ngx_http_parse.c 2012-11-21 01:02:56 UTC (rev 4927) @@ -2065,6 +2065,9 @@ done: + ctx->state = 0; + b->pos = pos + 1; + return NGX_DONE; invalid: From mdounin at mdounin.ru Wed Nov 21 01:03:48 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Wed, 21 Nov 2012 01:03:48 +0000 Subject: [nginx] svn commit: r4928 - trunk/src/http/modules Message-ID: <20121121010349.046213F9F2F@mail.nginx.com> Author: mdounin Date: 2012-11-21 01:03:48 +0000 (Wed, 21 Nov 2012) New Revision: 4928 URL: http://trac.nginx.org/nginx/changeset/4928/nginx Log: Request body: always use calculated size of a request body in proxy. This allows to handle requests with chunked body, and also simplifies handling of various request body modifications. Modified: trunk/src/http/modules/ngx_http_proxy_module.c Modified: trunk/src/http/modules/ngx_http_proxy_module.c =================================================================== --- trunk/src/http/modules/ngx_http_proxy_module.c 2012-11-21 01:02:56 UTC (rev 4927) +++ trunk/src/http/modules/ngx_http_proxy_module.c 2012-11-21 01:03:48 UTC (rev 4928) @@ -83,7 +83,7 @@ ngx_http_status_t status; ngx_http_chunked_t chunked; ngx_http_proxy_vars_t vars; - size_t internal_body_length; + off_t internal_body_length; ngx_uint_t head; /* unsigned head:1 */ } ngx_http_proxy_ctx_t; @@ -555,6 +555,8 @@ static ngx_keyval_t ngx_http_proxy_headers[] = { { ngx_string("Host"), ngx_string("$proxy_host") }, { ngx_string("Connection"), ngx_string("close") }, + { ngx_string("Content-Length"), ngx_string("$proxy_internal_body_length") }, + { ngx_string("Transfer-Encoding"), ngx_string("") }, { ngx_string("Keep-Alive"), ngx_string("") }, { ngx_string("Expect"), ngx_string("") }, { ngx_string("Upgrade"), ngx_string("") }, @@ -580,6 +582,8 @@ static ngx_keyval_t ngx_http_proxy_cache_headers[] = { { ngx_string("Host"), ngx_string("$proxy_host") }, { ngx_string("Connection"), ngx_string("close") }, + { ngx_string("Content-Length"), ngx_string("$proxy_internal_body_length") }, + { ngx_string("Transfer-Encoding"), ngx_string("") }, { ngx_string("Keep-Alive"), ngx_string("") }, { ngx_string("Expect"), ngx_string("") }, { ngx_string("Upgrade"), ngx_string("") }, @@ -1003,6 +1007,9 @@ ctx->internal_body_length = body_len; len += body_len; + + } else { + ctx->internal_body_length = r->headers_in.content_length_n; } le.ip = plcf->headers_set_len->elts; @@ -2039,7 +2046,7 @@ ctx = ngx_http_get_module_ctx(r, ngx_http_proxy_module); - if (ctx == NULL) { + if (ctx == NULL || ctx->internal_body_length < 0) { v->not_found = 1; return NGX_OK; } @@ -2048,13 +2055,13 @@ v->no_cacheable = 0; v->not_found = 0; - v->data = ngx_pnalloc(r->connection->pool, NGX_SIZE_T_LEN); + v->data = ngx_pnalloc(r->connection->pool, NGX_OFF_T_LEN); if (v->data == NULL) { return NGX_ERROR; } - v->len = ngx_sprintf(v->data, "%uz", ctx->internal_body_length) - v->data; + v->len = ngx_sprintf(v->data, "%O", ctx->internal_body_length) - v->data; return NGX_OK; } @@ -2822,8 +2829,6 @@ } if (conf->headers_set_hash.buckets - && ((conf->body_source.data == NULL) - == (prev->body_source.data == NULL)) #if (NGX_HTTP_CACHE) && ((conf->upstream.cache == NULL) == (prev->upstream.cache == NULL)) #endif @@ -2906,17 +2911,7 @@ h++; } - if (conf->body_source.data) { - s = ngx_array_push(&headers_merged); - if (s == NULL) { - return NGX_ERROR; - } - ngx_str_set(&s->key, "Content-Length"); - ngx_str_set(&s->value, "$proxy_internal_body_length"); - } - - src = headers_merged.elts; for (i = 0; i < headers_merged.nelts; i++) { From mdounin at mdounin.ru Wed Nov 21 01:05:09 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Wed, 21 Nov 2012 01:05:09 +0000 Subject: [nginx] svn commit: r4929 - trunk/src/http Message-ID: <20121121010509.9B7133F9F2F@mail.nginx.com> Author: mdounin Date: 2012-11-21 01:05:08 +0000 (Wed, 21 Nov 2012) New Revision: 4929 URL: http://trac.nginx.org/nginx/changeset/4929/nginx Log: Request body: $content_length variable to honor real body size. This allows to handle requests with chunked body by fastcgi and uwsgi modules, and also simplifies handling of various request body modifications. Modified: trunk/src/http/ngx_http_variables.c Modified: trunk/src/http/ngx_http_variables.c =================================================================== --- trunk/src/http/ngx_http_variables.c 2012-11-21 01:03:48 UTC (rev 4928) +++ trunk/src/http/ngx_http_variables.c 2012-11-21 01:05:08 UTC (rev 4929) @@ -39,6 +39,8 @@ ngx_http_variable_value_t *v, uintptr_t data); #endif +static ngx_int_t ngx_http_variable_content_length(ngx_http_request_t *r, + ngx_http_variable_value_t *v, uintptr_t data); static ngx_int_t ngx_http_variable_host(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data); static ngx_int_t ngx_http_variable_binary_remote_addr(ngx_http_request_t *r, @@ -153,8 +155,8 @@ { ngx_string("http_cookie"), NULL, ngx_http_variable_headers, offsetof(ngx_http_request_t, headers_in.cookies), 0, 0 }, - { ngx_string("content_length"), NULL, ngx_http_variable_header, - offsetof(ngx_http_request_t, headers_in.content_length), 0, 0 }, + { ngx_string("content_length"), NULL, ngx_http_variable_content_length, + 0, 0, 0 }, { ngx_string("content_type"), NULL, ngx_http_variable_header, offsetof(ngx_http_request_t, headers_in.content_type), 0, 0 }, @@ -990,6 +992,39 @@ static ngx_int_t +ngx_http_variable_content_length(ngx_http_request_t *r, + ngx_http_variable_value_t *v, uintptr_t data) +{ + u_char *p; + + if (r->headers_in.content_length) { + v->len = r->headers_in.content_length->value.len; + v->data = r->headers_in.content_length->value.data; + v->valid = 1; + v->no_cacheable = 0; + v->not_found = 0; + + } else if (r->headers_in.content_length_n >= 0) { + p = ngx_pnalloc(r->pool, NGX_OFF_T_LEN); + if (p == NULL) { + return NGX_ERROR; + } + + v->len = ngx_sprintf(p, "%O", r->headers_in.content_length_n) - p; + v->data = p; + v->valid = 1; + v->no_cacheable = 0; + v->not_found = 0; + + } else { + v->not_found = 1; + } + + return NGX_OK; +} + + +static ngx_int_t ngx_http_variable_host(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data) { From mdounin at mdounin.ru Wed Nov 21 01:06:54 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Wed, 21 Nov 2012 01:06:54 +0000 Subject: [nginx] svn commit: r4930 - trunk/src/http/modules Message-ID: <20121121010654.25DFB3F9F35@mail.nginx.com> Author: mdounin Date: 2012-11-21 01:06:53 +0000 (Wed, 21 Nov 2012) New Revision: 4930 URL: http://trac.nginx.org/nginx/changeset/4930/nginx Log: Request body: recalculate size of a request body in scgi module. This allows to handle requests with chunked body by scgi module, and also simplifies handling of various request body modifications. Modified: trunk/src/http/modules/ngx_http_scgi_module.c Modified: trunk/src/http/modules/ngx_http_scgi_module.c =================================================================== --- trunk/src/http/modules/ngx_http_scgi_module.c 2012-11-21 01:05:08 UTC (rev 4929) +++ trunk/src/http/modules/ngx_http_scgi_module.c 2012-11-21 01:06:53 UTC (rev 4930) @@ -533,10 +533,11 @@ static ngx_int_t ngx_http_scgi_create_request(ngx_http_request_t *r) { + off_t content_length_n; u_char ch, *key, *val, *lowcase_key; size_t len, key_len, val_len, allocated; ngx_buf_t *b; - ngx_str_t *content_length; + ngx_str_t content_length; ngx_uint_t i, n, hash, skip_empty, header_params; ngx_chain_t *cl, *body; ngx_list_part_t *part; @@ -545,13 +546,21 @@ ngx_http_script_engine_t e, le; ngx_http_scgi_loc_conf_t *scf; ngx_http_script_len_code_pt lcode; - static ngx_str_t zero = ngx_string("0"); + u_char buffer[NGX_OFF_T_LEN]; - content_length = r->headers_in.content_length ? - &r->headers_in.content_length->value : &zero; + content_length_n = 0; + body = r->upstream->request_bufs; - len = sizeof("CONTENT_LENGTH") + content_length->len + 1; + while (body) { + content_length_n += ngx_buf_size(body->buf); + body = body->next; + } + content_length.data = buffer; + content_length.len = ngx_sprintf(buffer, "%O", content_length_n) - buffer; + + len = sizeof("CONTENT_LENGTH") + content_length.len + 1; + header_params = 0; ignored = NULL; @@ -672,11 +681,8 @@ cl->buf = b; - b->last = ngx_snprintf(b->last, - NGX_SIZE_T_LEN + 1 + sizeof("CONTENT_LENGTH") - + NGX_OFF_T_LEN + 1, - "%ui:CONTENT_LENGTH%Z%V%Z", - len, content_length); + b->last = ngx_sprintf(b->last, "%ui:CONTENT_LENGTH%Z%V%Z", + len, &content_length); if (scf->params_len) { ngx_memzero(&e, sizeof(ngx_http_script_engine_t)); From mdounin at mdounin.ru Wed Nov 21 01:08:11 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Wed, 21 Nov 2012 01:08:11 +0000 Subject: [nginx] svn commit: r4931 - trunk/src/http Message-ID: <20121121010812.178FD3F9F35@mail.nginx.com> Author: mdounin Date: 2012-11-21 01:08:11 +0000 (Wed, 21 Nov 2012) New Revision: 4931 URL: http://trac.nginx.org/nginx/changeset/4931/nginx Log: Request body: chunked transfer encoding support. Modified: trunk/src/http/ngx_http_core_module.c trunk/src/http/ngx_http_request.c trunk/src/http/ngx_http_request.h trunk/src/http/ngx_http_request_body.c Modified: trunk/src/http/ngx_http_core_module.c =================================================================== --- trunk/src/http/ngx_http_core_module.c 2012-11-21 01:06:53 UTC (rev 4930) +++ trunk/src/http/ngx_http_core_module.c 2012-11-21 01:08:11 UTC (rev 4931) @@ -848,7 +848,8 @@ break; } - r->lingering_close = (r->headers_in.content_length_n > 0); + r->lingering_close = (r->headers_in.content_length_n > 0 + || r->headers_in.chunked); r->phase_handler = 0; } else { Modified: trunk/src/http/ngx_http_request.c =================================================================== --- trunk/src/http/ngx_http_request.c 2012-11-21 01:06:53 UTC (rev 4930) +++ trunk/src/http/ngx_http_request.c 2012-11-21 01:08:11 UTC (rev 4931) @@ -1574,19 +1574,11 @@ if (r->headers_in.content_length_n == NGX_ERROR) { ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, "client sent invalid \"Content-Length\" header"); - ngx_http_finalize_request(r, NGX_HTTP_LENGTH_REQUIRED); + ngx_http_finalize_request(r, NGX_HTTP_BAD_REQUEST); return NGX_ERROR; } } - if (r->method & NGX_HTTP_PUT && r->headers_in.content_length_n == -1) { - ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, - "client sent %V method without \"Content-Length\" header", - &r->method_name); - ngx_http_finalize_request(r, NGX_HTTP_LENGTH_REQUIRED); - return NGX_ERROR; - } - if (r->method & NGX_HTTP_TRACE) { ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, "client sent TRACE method"); @@ -1594,14 +1586,25 @@ return NGX_ERROR; } - if (r->headers_in.transfer_encoding - && ngx_strcasestrn(r->headers_in.transfer_encoding->value.data, - "chunked", 7 - 1)) - { - ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, - "client sent \"Transfer-Encoding: chunked\" header"); - ngx_http_finalize_request(r, NGX_HTTP_LENGTH_REQUIRED); - return NGX_ERROR; + if (r->headers_in.transfer_encoding) { + if (r->headers_in.transfer_encoding->value.len == 7 + && ngx_strncasecmp(r->headers_in.transfer_encoding->value.data, + (u_char *) "chunked", 7) == 0) + { + r->headers_in.content_length = NULL; + r->headers_in.content_length_n = -1; + r->headers_in.chunked = 1; + + } else if (r->headers_in.transfer_encoding->value.len != 8 + || ngx_strncasecmp(r->headers_in.transfer_encoding->value.data, + (u_char *) "identity", 8) != 0) + { + ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, + "client sent unknown \"Transfer-Encoding\": \"%V\"", + &r->headers_in.transfer_encoding->value); + ngx_http_finalize_request(r, NGX_HTTP_NOT_IMPLEMENTED); + return NGX_ERROR; + } } if (r->headers_in.connection_type == NGX_HTTP_CONNECTION_KEEP_ALIVE) { Modified: trunk/src/http/ngx_http_request.h =================================================================== --- trunk/src/http/ngx_http_request.h 2012-11-21 01:06:53 UTC (rev 4930) +++ trunk/src/http/ngx_http_request.h 2012-11-21 01:08:11 UTC (rev 4931) @@ -224,6 +224,7 @@ time_t keep_alive_n; unsigned connection_type:2; + unsigned chunked:1; unsigned msie:1; unsigned msie6:1; unsigned opera:1; @@ -276,7 +277,9 @@ ngx_chain_t *bufs; ngx_buf_t *buf; off_t rest; - ngx_chain_t *to_write; + ngx_chain_t *free; + ngx_chain_t *busy; + ngx_http_chunked_t *chunked; ngx_http_client_body_handler_pt post_handler; } ngx_http_request_body_t; Modified: trunk/src/http/ngx_http_request_body.c =================================================================== --- trunk/src/http/ngx_http_request_body.c 2012-11-21 01:06:53 UTC (rev 4930) +++ trunk/src/http/ngx_http_request_body.c 2012-11-21 01:08:11 UTC (rev 4931) @@ -12,18 +12,21 @@ static void ngx_http_read_client_request_body_handler(ngx_http_request_t *r); static ngx_int_t ngx_http_do_read_client_request_body(ngx_http_request_t *r); -static ngx_int_t ngx_http_write_request_body(ngx_http_request_t *r, - ngx_chain_t *body); +static ngx_int_t ngx_http_write_request_body(ngx_http_request_t *r); static ngx_int_t ngx_http_read_discarded_request_body(ngx_http_request_t *r); +static ngx_int_t ngx_http_discard_request_body_filter(ngx_http_request_t *r, + ngx_buf_t *b); static ngx_int_t ngx_http_test_expect(ngx_http_request_t *r); +static ngx_int_t ngx_http_request_body_filter(ngx_http_request_t *r, + ngx_chain_t *in); +static ngx_int_t ngx_http_request_body_length_filter(ngx_http_request_t *r, + ngx_chain_t *in); +static ngx_int_t ngx_http_request_body_chunked_filter(ngx_http_request_t *r, + ngx_chain_t *in); +static ngx_int_t ngx_http_request_body_save_filter(ngx_http_request_t *r, + ngx_chain_t *in); -/* - * on completion ngx_http_read_client_request_body() adds to - * r->request_body->bufs one or two bufs: - * *) one memory buf that was preread in r->header_in; - * *) one memory or file buf that contains the rest of the body - */ ngx_int_t ngx_http_read_client_request_body(ngx_http_request_t *r, @@ -32,8 +35,7 @@ size_t preread; ssize_t size; ngx_int_t rc; - ngx_buf_t *b; - ngx_chain_t *cl, **next; + ngx_chain_t out; ngx_http_request_body_t *rb; ngx_http_core_loc_conf_t *clcf; @@ -55,39 +57,26 @@ goto done; } - r->request_body = rb; + /* + * set by ngx_pcalloc(): + * + * rb->bufs = NULL; + * rb->buf = NULL; + * rb->free = NULL; + * rb->busy = NULL; + * rb->chunked = NULL; + */ - if (r->headers_in.content_length_n < 0) { - post_handler(r); - return NGX_OK; - } + rb->rest = -1; + rb->post_handler = post_handler; - clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module); + r->request_body = rb; - if (r->headers_in.content_length_n == 0) { - - if (r->request_body_in_file_only) { - if (ngx_http_write_request_body(r, NULL) != NGX_OK) { - rc = NGX_HTTP_INTERNAL_SERVER_ERROR; - goto done; - } - } - + if (r->headers_in.content_length_n < 0 && !r->headers_in.chunked) { post_handler(r); - return NGX_OK; } - rb->post_handler = post_handler; - - /* - * set by ngx_pcalloc(): - * - * rb->bufs = NULL; - * rb->buf = NULL; - * rb->rest = 0; - */ - preread = r->header_in->last - r->header_in->pos; if (preread) { @@ -97,83 +86,62 @@ ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "http client request body preread %uz", preread); - b = ngx_calloc_buf(r->pool); - if (b == NULL) { - rc = NGX_HTTP_INTERNAL_SERVER_ERROR; - goto done; - } + out.buf = r->header_in; + out.next = NULL; - b->temporary = 1; - b->start = r->header_in->pos; - b->pos = r->header_in->pos; - b->last = r->header_in->last; - b->end = r->header_in->end; + rc = ngx_http_request_body_filter(r, &out); - rb->bufs = ngx_alloc_chain_link(r->pool); - if (rb->bufs == NULL) { - rc = NGX_HTTP_INTERNAL_SERVER_ERROR; + if (rc != NGX_OK) { goto done; } - rb->bufs->buf = b; - rb->bufs->next = NULL; + r->request_length += preread - (r->header_in->last - r->header_in->pos); - rb->buf = b; + if (!r->headers_in.chunked + && rb->rest > 0 + && rb->rest <= (off_t) (r->header_in->end - r->header_in->last)) + { + /* the whole request body may be placed in r->header_in */ - if ((off_t) preread >= r->headers_in.content_length_n) { + rb->buf = r->header_in; + r->read_event_handler = ngx_http_read_client_request_body_handler; - /* the whole request body was pre-read */ + rc = ngx_http_do_read_client_request_body(r); + goto done; + } - r->header_in->pos += (size_t) r->headers_in.content_length_n; - r->request_length += r->headers_in.content_length_n; - b->last = r->header_in->pos; + } else { + /* set rb->rest */ - if (r->request_body_in_file_only) { - if (ngx_http_write_request_body(r, rb->bufs) != NGX_OK) { - rc = NGX_HTTP_INTERNAL_SERVER_ERROR; - goto done; - } - } - - post_handler(r); - - return NGX_OK; + if (ngx_http_request_body_filter(r, NULL) != NGX_OK) { + rc = NGX_HTTP_INTERNAL_SERVER_ERROR; + goto done; } + } - /* - * to not consider the body as pipelined request in - * ngx_http_set_keepalive() - */ - r->header_in->pos = r->header_in->last; + if (rb->rest == 0) { + /* the whole request body was pre-read */ - r->request_length += preread; - - rb->rest = r->headers_in.content_length_n - preread; - - if (rb->rest <= (off_t) (b->end - b->last)) { - - /* the whole request body may be placed in r->header_in */ - - rb->to_write = rb->bufs; - - r->read_event_handler = ngx_http_read_client_request_body_handler; - - rc = ngx_http_do_read_client_request_body(r); - goto done; + if (r->request_body_in_file_only) { + if (ngx_http_write_request_body(r) != NGX_OK) { + rc = NGX_HTTP_INTERNAL_SERVER_ERROR; + goto done; + } } - next = &rb->bufs->next; + post_handler(r); - } else { - b = NULL; - rb->rest = r->headers_in.content_length_n; - next = &rb->bufs; + return NGX_OK; } + clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module); + size = clcf->client_body_buffer_size; size += size >> 2; - if (rb->rest < size) { + /* TODO: honor r->request_body_in_single_buf */ + + if (!r->headers_in.chunked && rb->rest < size) { size = (ssize_t) rb->rest; if (r->request_body_in_single_buf) { @@ -182,9 +150,6 @@ } else { size = clcf->client_body_buffer_size; - - /* disable copying buffer for r->request_body_in_single_buf */ - b = NULL; } rb->buf = ngx_create_temp_buf(r->pool, size); @@ -193,32 +158,6 @@ goto done; } - cl = ngx_alloc_chain_link(r->pool); - if (cl == NULL) { - rc = NGX_HTTP_INTERNAL_SERVER_ERROR; - goto done; - } - - cl->buf = rb->buf; - cl->next = NULL; - - if (b && r->request_body_in_single_buf) { - size = b->last - b->pos; - ngx_memcpy(rb->buf->pos, b->pos, size); - rb->buf->last += size; - - next = &rb->bufs; - } - - *next = cl; - - if (r->request_body_in_file_only || r->request_body_in_single_buf) { - rb->to_write = rb->bufs; - - } else { - rb->to_write = rb->bufs->next ? rb->bufs->next : rb->bufs; - } - r->read_event_handler = ngx_http_read_client_request_body_handler; rc = ngx_http_do_read_client_request_body(r); @@ -255,9 +194,12 @@ static ngx_int_t ngx_http_do_read_client_request_body(ngx_http_request_t *r) { + off_t rest; size_t size; ssize_t n; + ngx_int_t rc; ngx_buf_t *b; + ngx_chain_t *cl, out; ngx_connection_t *c; ngx_http_request_body_t *rb; ngx_http_core_loc_conf_t *clcf; @@ -272,18 +214,44 @@ for ( ;; ) { if (rb->buf->last == rb->buf->end) { - if (ngx_http_write_request_body(r, rb->to_write) != NGX_OK) { + /* pass buffer to request body filter chain */ + + out.buf = rb->buf; + out.next = NULL; + + rc = ngx_http_request_body_filter(r, &out); + + if (rc != NGX_OK) { + return rc; + } + + /* write to file */ + + if (ngx_http_write_request_body(r) != NGX_OK) { return NGX_HTTP_INTERNAL_SERVER_ERROR; } - rb->to_write = rb->bufs->next ? rb->bufs->next : rb->bufs; + /* update chains */ + + rc = ngx_http_request_body_filter(r, NULL); + + if (rc != NGX_OK) { + return rc; + } + + if (rb->busy != NULL) { + return NGX_HTTP_INTERNAL_SERVER_ERROR; + } + + rb->buf->pos = rb->buf->start; rb->buf->last = rb->buf->start; } size = rb->buf->end - rb->buf->last; + rest = rb->rest - (rb->buf->last - rb->buf->pos); - if ((off_t) size > rb->rest) { - size = (size_t) rb->rest; + if ((off_t) size > rest) { + size = (size_t) rest; } n = c->recv(c, rb->buf->last, size); @@ -306,9 +274,21 @@ } rb->buf->last += n; - rb->rest -= n; r->request_length += n; + if (n == rest) { + /* pass buffer to request body filter chain */ + + out.buf = rb->buf; + out.next = NULL; + + rc = ngx_http_request_body_filter(r, &out); + + if (rc != NGX_OK) { + return rc; + } + } + if (rb->rest == 0) { break; } @@ -345,34 +325,26 @@ /* save the last part */ - if (ngx_http_write_request_body(r, rb->to_write) != NGX_OK) { + if (ngx_http_write_request_body(r) != NGX_OK) { return NGX_HTTP_INTERNAL_SERVER_ERROR; } - b = ngx_calloc_buf(r->pool); - if (b == NULL) { + cl = ngx_chain_get_free_buf(r->pool, &rb->free); + if (cl == NULL) { return NGX_HTTP_INTERNAL_SERVER_ERROR; } + b = cl->buf; + + ngx_memzero(b, sizeof(ngx_buf_t)); + b->in_file = 1; - b->file_pos = 0; b->file_last = rb->temp_file->file.offset; b->file = &rb->temp_file->file; - if (rb->bufs->next) { - rb->bufs->next->buf = b; - - } else { - rb->bufs->buf = b; - } + rb->bufs = cl; } - if (rb->bufs->next - && (r->request_body_in_file_only || r->request_body_in_single_buf)) - { - rb->bufs = rb->bufs->next; - } - r->read_event_handler = ngx_http_block_reading; rb->post_handler(r); @@ -382,15 +354,19 @@ static ngx_int_t -ngx_http_write_request_body(ngx_http_request_t *r, ngx_chain_t *body) +ngx_http_write_request_body(ngx_http_request_t *r) { ssize_t n; + ngx_chain_t *cl; ngx_temp_file_t *tf; ngx_http_request_body_t *rb; ngx_http_core_loc_conf_t *clcf; rb = r->request_body; + ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "http write client request body, bufs %p", rb->bufs); + if (rb->temp_file == NULL) { tf = ngx_pcalloc(r->pool, sizeof(ngx_temp_file_t)); if (tf == NULL) { @@ -414,7 +390,7 @@ rb->temp_file = tf; - if (body == NULL) { + if (rb->bufs == NULL) { /* empty body with r->request_body_in_file_only */ if (ngx_create_temp_file(&tf->file, tf->path, tf->pool, @@ -428,8 +404,12 @@ } } - n = ngx_write_chain_to_temp_file(rb->temp_file, body); + if (rb->bufs == NULL) { + return NGX_OK; + } + n = ngx_write_chain_to_temp_file(rb->temp_file, rb->bufs); + /* TODO: n == 0 or not complete and level event */ if (n == NGX_ERROR) { @@ -438,6 +418,14 @@ rb->temp_file->offset += n; + /* mark all buffers as written */ + + for (cl = rb->bufs; cl; cl = cl->next) { + cl->buf->pos = cl->buf->last; + } + + rb->bufs = NULL; + return NGX_OK; } @@ -446,9 +434,10 @@ ngx_http_discard_request_body(ngx_http_request_t *r) { ssize_t size; + ngx_int_t rc; ngx_event_t *rev; - if (r != r->main || r->discard_body) { + if (r != r->main || r->discard_body || r->request_body) { return NGX_OK; } @@ -464,20 +453,20 @@ ngx_del_timer(rev); } - if (r->headers_in.content_length_n <= 0 || r->request_body) { + if (r->headers_in.content_length_n <= 0 && !r->headers_in.chunked) { return NGX_OK; } size = r->header_in->last - r->header_in->pos; if (size) { - if (r->headers_in.content_length_n > size) { - r->header_in->pos += size; - r->headers_in.content_length_n -= size; + rc = ngx_http_discard_request_body_filter(r, r->header_in); - } else { - r->header_in->pos += (size_t) r->headers_in.content_length_n; - r->headers_in.content_length_n = 0; + if (rc != NGX_OK) { + return rc; + } + + if (r->headers_in.content_length_n == 0) { return NGX_OK; } } @@ -570,13 +559,19 @@ static ngx_int_t ngx_http_read_discarded_request_body(ngx_http_request_t *r) { - size_t size; - ssize_t n; - u_char buffer[NGX_HTTP_DISCARD_BUFFER_SIZE]; + size_t size; + ssize_t n; + ngx_int_t rc; + ngx_buf_t b; + u_char buffer[NGX_HTTP_DISCARD_BUFFER_SIZE]; ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "http read discarded body"); + ngx_memzero(&b, sizeof(ngx_buf_t)); + + b.temporary = 1; + for ( ;; ) { if (r->headers_in.content_length_n == 0) { r->read_event_handler = ngx_http_block_reading; @@ -587,9 +582,8 @@ return NGX_AGAIN; } - size = (r->headers_in.content_length_n > NGX_HTTP_DISCARD_BUFFER_SIZE) ? - NGX_HTTP_DISCARD_BUFFER_SIZE: - (size_t) r->headers_in.content_length_n; + size = (size_t) ngx_min(r->headers_in.content_length_n, + NGX_HTTP_DISCARD_BUFFER_SIZE); n = r->connection->recv(r->connection, buffer, size); @@ -606,12 +600,109 @@ return NGX_OK; } - r->headers_in.content_length_n -= n; + b.pos = buffer; + b.last = buffer + n; + + rc = ngx_http_discard_request_body_filter(r, &b); + + if (rc != NGX_OK) { + r->connection->error = 1; + return NGX_OK; + } } } static ngx_int_t +ngx_http_discard_request_body_filter(ngx_http_request_t *r, ngx_buf_t *b) +{ + size_t size; + ngx_int_t rc; + ngx_http_request_body_t *rb; + + if (r->headers_in.chunked) { + + rb = r->request_body; + + if (rb == NULL) { + + rb = ngx_pcalloc(r->pool, sizeof(ngx_http_request_body_t)); + if (rb == NULL) { + return NGX_HTTP_INTERNAL_SERVER_ERROR; + } + + rb->chunked = ngx_pcalloc(r->pool, sizeof(ngx_http_chunked_t)); + if (rb == NULL) { + return NGX_HTTP_INTERNAL_SERVER_ERROR; + } + + r->request_body = rb; + } + + for ( ;; ) { + + rc = ngx_http_parse_chunked(r, b, rb->chunked); + + if (rc == NGX_OK) { + + /* a chunk has been parsed successfully */ + + size = b->last - b->pos; + + if ((off_t) size > rb->chunked->size) { + b->pos += rb->chunked->size; + rb->chunked->size = 0; + + } else { + rb->chunked->size -= size; + b->pos = b->last; + } + + continue; + } + + if (rc == NGX_DONE) { + + /* a whole response has been parsed successfully */ + + r->headers_in.content_length_n = 0; + break; + } + + if (rc == NGX_AGAIN) { + + /* set amount of data we want to see next time */ + + r->headers_in.content_length_n = rb->chunked->length; + break; + } + + /* invalid */ + + ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, + "client sent invalid chunked body"); + + return NGX_HTTP_BAD_REQUEST; + } + + } else { + size = b->last - b->pos; + + if ((off_t) size > r->headers_in.content_length_n) { + b->pos += r->headers_in.content_length_n; + r->headers_in.content_length_n = 0; + + } else { + b->pos = b->last; + r->headers_in.content_length_n -= size; + } + } + + return NGX_OK; +} + + +static ngx_int_t ngx_http_test_expect(ngx_http_request_t *r) { ngx_int_t n; @@ -651,3 +742,274 @@ return NGX_ERROR; } + + +static ngx_int_t +ngx_http_request_body_filter(ngx_http_request_t *r, ngx_chain_t *in) +{ + if (r->headers_in.chunked) { + return ngx_http_request_body_chunked_filter(r, in); + + } else { + return ngx_http_request_body_length_filter(r, in); + } +} + + +static ngx_int_t +ngx_http_request_body_length_filter(ngx_http_request_t *r, ngx_chain_t *in) +{ + size_t size; + ngx_int_t rc; + ngx_buf_t *b; + ngx_chain_t *cl, *tl, *out, **ll; + ngx_http_request_body_t *rb; + + rb = r->request_body; + + if (rb->rest == -1) { + ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "http request body content length filter"); + + rb->rest = r->headers_in.content_length_n; + } + + out = NULL; + ll = &out; + + for (cl = in; cl; cl = cl->next) { + + tl = ngx_chain_get_free_buf(r->pool, &rb->free); + if (tl == NULL) { + return NGX_HTTP_INTERNAL_SERVER_ERROR; + } + + b = tl->buf; + + ngx_memzero(b, sizeof(ngx_buf_t)); + + b->temporary = 1; + b->tag = (ngx_buf_tag_t) &ngx_http_read_client_request_body; + b->start = cl->buf->start; + b->pos = cl->buf->pos; + b->last = cl->buf->last; + b->end = cl->buf->end; + + size = cl->buf->last - cl->buf->pos; + + if ((off_t) size < rb->rest) { + cl->buf->pos = cl->buf->last; + rb->rest -= size; + + } else { + cl->buf->pos += rb->rest; + rb->rest = 0; + b->last = cl->buf->pos; + b->last_buf = 1; + } + + *ll = tl; + ll = &tl->next; + } + + rc = ngx_http_request_body_save_filter(r, out); + + ngx_chain_update_chains(r->pool, &rb->free, &rb->busy, &out, + (ngx_buf_tag_t) &ngx_http_read_client_request_body); + + return rc; +} + + +static ngx_int_t +ngx_http_request_body_chunked_filter(ngx_http_request_t *r, ngx_chain_t *in) +{ + size_t size; + ngx_int_t rc; + ngx_buf_t *b; + ngx_chain_t *cl, *out, *tl, **ll; + ngx_http_request_body_t *rb; + ngx_http_core_loc_conf_t *clcf; + + rb = r->request_body; + + if (rb->rest == -1) { + + ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "http request body chunked filter"); + + rb->chunked = ngx_pcalloc(r->pool, sizeof(ngx_http_chunked_t)); + if (rb->chunked == NULL) { + return NGX_HTTP_INTERNAL_SERVER_ERROR; + } + + r->headers_in.content_length_n = 0; + rb->rest = 3; + } + + out = NULL; + ll = &out; + + for (cl = in; cl; cl = cl->next) { + + for ( ;; ) { + + ngx_log_debug7(NGX_LOG_DEBUG_EVENT, r->connection->log, 0, + "http body chunked buf " + "t:%d f:%d %p, pos %p, size: %z file: %O, size: %z", + cl->buf->temporary, cl->buf->in_file, + cl->buf->start, cl->buf->pos, + cl->buf->last - cl->buf->pos, + cl->buf->file_pos, + cl->buf->file_last - cl->buf->file_pos); + + rc = ngx_http_parse_chunked(r, cl->buf, rb->chunked); + + if (rc == NGX_OK) { + + /* a chunk has been parsed successfully */ + + clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module); + + if (clcf->client_max_body_size + && clcf->client_max_body_size + < r->headers_in.content_length_n + rb->chunked->size) + { + ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, + "client intended to send too large chunked " + "body: %O bytes", + r->headers_in.content_length_n + + rb->chunked->size); + + r->lingering_close = 1; + + return NGX_HTTP_REQUEST_ENTITY_TOO_LARGE; + } + + tl = ngx_chain_get_free_buf(r->pool, &rb->free); + if (tl == NULL) { + return NGX_HTTP_INTERNAL_SERVER_ERROR; + } + + b = tl->buf; + + ngx_memzero(b, sizeof(ngx_buf_t)); + + b->temporary = 1; + b->tag = (ngx_buf_tag_t) &ngx_http_read_client_request_body; + b->start = cl->buf->start; + b->pos = cl->buf->pos; + b->last = cl->buf->last; + b->end = cl->buf->end; + + *ll = tl; + ll = &tl->next; + + size = cl->buf->last - cl->buf->pos; + + if ((off_t) size > rb->chunked->size) { + cl->buf->pos += rb->chunked->size; + r->headers_in.content_length_n += rb->chunked->size; + rb->chunked->size = 0; + + } else { + rb->chunked->size -= size; + r->headers_in.content_length_n += size; + cl->buf->pos = cl->buf->last; + } + + b->last = cl->buf->pos; + + continue; + } + + if (rc == NGX_DONE) { + + /* a whole response has been parsed successfully */ + + rb->rest = 0; + + tl = ngx_chain_get_free_buf(r->pool, &rb->free); + if (tl == NULL) { + return NGX_HTTP_INTERNAL_SERVER_ERROR; + } + + b = tl->buf; + + ngx_memzero(b, sizeof(ngx_buf_t)); + + b->last_buf = 1; + + *ll = tl; + ll = &tl->next; + + break; + } + + if (rc == NGX_AGAIN) { + + /* set rb->rest, amount of data we want to see next time */ + + rb->rest = rb->chunked->length; + + break; + } + + /* invalid */ + + ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, + "client sent invalid chunked body"); + + return NGX_HTTP_BAD_REQUEST; + } + } + + rc = ngx_http_request_body_save_filter(r, out); + + ngx_chain_update_chains(r->pool, &rb->free, &rb->busy, &out, + (ngx_buf_tag_t) &ngx_http_read_client_request_body); + + return rc; +} + + +static ngx_int_t +ngx_http_request_body_save_filter(ngx_http_request_t *r, ngx_chain_t *in) +{ + ngx_chain_t *cl; + ngx_http_request_body_t *rb; + + rb = r->request_body; + +#if (NGX_DEBUG) + + for (cl = rb->bufs; cl; cl = cl->next) { + ngx_log_debug7(NGX_LOG_DEBUG_EVENT, r->connection->log, 0, + "http body old buf t:%d f:%d %p, pos %p, size: %z " + "file: %O, size: %z", + cl->buf->temporary, cl->buf->in_file, + cl->buf->start, cl->buf->pos, + cl->buf->last - cl->buf->pos, + cl->buf->file_pos, + cl->buf->file_last - cl->buf->file_pos); + } + + for (cl = in; cl; cl = cl->next) { + ngx_log_debug7(NGX_LOG_DEBUG_EVENT, r->connection->log, 0, + "http body new buf t:%d f:%d %p, pos %p, size: %z " + "file: %O, size: %z", + cl->buf->temporary, cl->buf->in_file, + cl->buf->start, cl->buf->pos, + cl->buf->last - cl->buf->pos, + cl->buf->file_pos, + cl->buf->file_last - cl->buf->file_pos); + } + +#endif + + /* TODO: coalesce neighbouring buffers */ + + ngx_chain_add_copy(r->pool, &rb->bufs, in); + + return NGX_OK; +} From mdounin at mdounin.ru Wed Nov 21 01:31:46 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 21 Nov 2012 05:31:46 +0400 Subject: =?UTF-8?B?UmU6IFNlZ21lbnRhdGlvbiBmYXVsdCDQsiBEQVYg0LzQvtC00YPQu9C1?= In-Reply-To: References: <20121120092156.GN40452@mdounin.ru> Message-ID: <20121121013146.GQ40452@mdounin.ru> Hello! On Tue, Nov 20, 2012 at 05:29:58PM +0300, Dmitry Petrov wrote: > Hello. > > Finally I was able to make a test that reproduces bug 100% of time. > Here it is (nginx sources included): http://rghost.net/41694908 > You need to: > 1. tar zxf nginx-webdav-segfault.tar.gz > 2. cd dav-bug/nginx-1.3.8 > 3. ./cf ? this will configure nginx with needed modules and /tmp as prefix > 4. cd .. > 5. In one console start nginx via 'make r' command. It will automatically > use dav.conf which will start nginx on 0.0.0.0:8080 > 6. Attach to nginx process: gdb --pid `pgrep -n nginx` > 7. Execute make ? this will launch test trying to connect to > localhost:8080, upload some files etc. Make sure that 8181 port is free > before running test case - it create dummy "HTTP server" that simply > ignores all requests. I wasn't able to get this bug without proxy_pass'es > between webdav PUT's. > 8. Monitor console where you've started gdb - you will get SIGSEGV here > The symptoms of the crash are the same as I described earlier: > r->request_body is not NULL, set from some kind of previous request. > r->request_body->temp_file is NULL. > > This test case just replaying dump recorded from Iphone HTTP session using > TCP_CORK to achive exactly the same packets send (with scrambled payload > ofc). > > Hope this helps. Thanks, I was able to reproduce the problem here using you test code. It indeed looks like a problem related to extra write events, much like one recently reported here: http://serverfault.com/questions/449195/nginx-webdav-server-with-auth-request Please try the following patch: --- a/src/http/ngx_http_request_body.c +++ b/src/http/ngx_http_request_body.c @@ -174,6 +174,7 @@ ngx_http_read_client_request_body(ngx_ht rb->to_write = rb->bufs; r->read_event_handler = ngx_http_read_client_request_body_handler; + r->write_event_handler = ngx_http_request_empty_handler; return ngx_http_do_read_client_request_body(r); } @@ -234,6 +235,7 @@ ngx_http_read_client_request_body(ngx_ht } r->read_event_handler = ngx_http_read_client_request_body_handler; + r->write_event_handler = ngx_http_request_empty_handler; return ngx_http_do_read_client_request_body(r); } > On Tue, Nov 20, 2012 at 1:21 PM, Maxim Dounin wrote: > > > Hello! > > > > On Tue, Nov 20, 2012 at 12:05:07PM +0300, Dmitry Petrov wrote: > > > > > On Mon, Nov 19, 2012 at 7:40 PM, Arnaud GRANAL > > wrote: > > > > > > > I guess it was fixed in a recent commit by Maxim: > > > > [..] > > > > diff --git a/src/http/modules/ngx_http_dav_module.c > > > > b/src/http/modules/ngx_http_dav_module.c > > > > --- a/src/http/modules/ngx_http_dav_module.c > > > > +++ b/src/http/modules/ngx_http_dav_module.c > > > > @@ -209,6 +209,11 @@ ngx_http_dav_put_handler(ngx_http_reques > > > > ngx_ext_rename_file_t ext; > > > > ngx_http_dav_loc_conf_t *dlcf; > > > > > > > > + if (r->request_body == NULL || r->request_body->temp_file == > > NULL) { > > > > + ngx_http_finalize_request(r, NGX_HTTP_INTERNAL_SERVER_ERROR); > > > > + return; > > > > + } > > > > + > > > > ngx_http_map_uri_to_path(r, &path, &root, 0); > > > > > > > > path.len--; > > > > > > > The problem is that I got this behavior on sequential valid PUT requests. > > > After handling first request, nginx even doesn't try to read body of next > > > request. It just calls ngx_http_dav_put_handler with r->response_body > > from > > > > It looks like this is a generic request body handling problem, > > which leads to incorrect body processing if an extra write event > > happens during request body reading. > > > > The patch quoted will fix resulting segfault in dav module (which > > may also happen in other conditions), but the underlying problem > > also needs to be fixed. If you have a test case to reproduce the > > problem it might be helpful. > > > > > previous request but with r->response_body->temp_file set to NULL. With > > > extra check for temp_file not being NULL nginx actually reads second > > > request body and handles it properly. > > > > The extra check you've added isn't correct as > > r->response_body->temp_file may be NULL for valid reasons, e.g. > > request body reading is done by a normal module like proxy, which > > doesn't require request body to be in file. > > > > -- > > Maxim Dounin > > http://nginx.com/support.html > > > > _______________________________________________ > > nginx-devel mailing list > > nginx-devel at nginx.org > > http://mailman.nginx.org/mailman/listinfo/nginx-devel > > > > > > -- > Regards, > Dmitry > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel -- Maxim Dounin http://nginx.com/support.html From mdounin at mdounin.ru Wed Nov 21 01:40:12 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Wed, 21 Nov 2012 01:40:12 +0000 Subject: [nginx] svn commit: r4932 - trunk/src/http Message-ID: <20121121014012.4D5883F9ED0@mail.nginx.com> Author: mdounin Date: 2012-11-21 01:40:11 +0000 (Wed, 21 Nov 2012) New Revision: 4932 URL: http://trac.nginx.org/nginx/changeset/4932/nginx Log: Request body: unbreak build without debug. Modified: trunk/src/http/ngx_http_request_body.c Modified: trunk/src/http/ngx_http_request_body.c =================================================================== --- trunk/src/http/ngx_http_request_body.c 2012-11-21 01:08:11 UTC (rev 4931) +++ trunk/src/http/ngx_http_request_body.c 2012-11-21 01:40:11 UTC (rev 4932) @@ -976,7 +976,9 @@ static ngx_int_t ngx_http_request_body_save_filter(ngx_http_request_t *r, ngx_chain_t *in) { +#if (NGX_DEBUG) ngx_chain_t *cl; +#endif ngx_http_request_body_t *rb; rb = r->request_body; From jefftk at google.com Wed Nov 21 02:45:36 2012 From: jefftk at google.com (Jeff Kaufman) Date: Tue, 20 Nov 2012 21:45:36 -0500 Subject: clearing etags when gzipping Message-ID: In ngx_http_gzip_header_filter there is a call to ngx_http_clear_etag. While this spec-compliant it loses the benefits of etags. Two alternate approaches would be: - Only clear the etag if it's strong. - Append ";gzip" to the etag, remove it from incoming "if-none-match" requests. Would a patch to implement one of these be welcome? Jeff From mdounin at mdounin.ru Wed Nov 21 08:24:12 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 21 Nov 2012 12:24:12 +0400 Subject: clearing etags when gzipping In-Reply-To: References: Message-ID: <20121121082411.GR40452@mdounin.ru> Hello! On Tue, Nov 20, 2012 at 09:45:36PM -0500, Jeff Kaufman wrote: > In ngx_http_gzip_header_filter there is a call to ngx_http_clear_etag. > While this spec-compliant it loses the benefits of etags. Two > alternate approaches would be: > > - Only clear the etag if it's strong. > - Append ";gzip" to the etag, remove it from > incoming "if-none-match" requests. > > Would a patch to implement one of these be welcome? Appending ";gzip" would not work as a result of a gzip might be different depending on gzip settings. The only aproach which will work is to change etag to a week one. I'm ok with this, though it will generally need week etag support in various places. -- Maxim Dounin http://nginx.com/support.html From crk_world at yahoo.com.cn Wed Nov 21 08:36:06 2012 From: crk_world at yahoo.com.cn (chen cw) Date: Wed, 21 Nov 2012 16:36:06 +0800 Subject: nginx proxy cache support contional request( If-Modified-Since) In-Reply-To: <26a9a8ec.69515.13b203d45f7.Coremail.flygoast@126.com> References: <26a9a8ec.69515.13b203d45f7.Coremail.flygoast@126.com> Message-ID: do proxy_cache and proxy_store meets your requirement? On Wed, Nov 21, 2012 at 7:53 AM, ?? wrote: > Hi, > > I'm using nginx as a reverse proxy with cache. My content almost never > changes. So I want to use a conditional GET request when fetch content > from backend. > However, nginx don't support it. Who have a such patch for it? Can give > me? Thanks. > > PS, I want to know someday nginx will implement it or refuse to implement > it?** > > Thank you. > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -- -- Charles Chen Software Engineer Server Platforms Team at Taobao.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmitry.petroff at gmail.com Wed Nov 21 10:23:31 2012 From: dmitry.petroff at gmail.com (Dmitry Petrov) Date: Wed, 21 Nov 2012 13:23:31 +0300 Subject: =?UTF-8?B?UmU6IFNlZ21lbnRhdGlvbiBmYXVsdCDQsiBEQVYg0LzQvtC00YPQu9C1?= In-Reply-To: <20121121013146.GQ40452@mdounin.ru> References: <20121120092156.GN40452@mdounin.ru> <20121121013146.GQ40452@mdounin.ru> Message-ID: On Wed, Nov 21, 2012 at 5:31 AM, Maxim Dounin wrote: > Hello! > [...] > Please try the following patch: > > --- a/src/http/ngx_http_request_body.c > +++ b/src/http/ngx_http_request_body.c > @@ -174,6 +174,7 @@ ngx_http_read_client_request_body(ngx_ht > rb->to_write = rb->bufs; > > r->read_event_handler = > ngx_http_read_client_request_body_handler; > + r->write_event_handler = ngx_http_request_empty_handler; > > return ngx_http_do_read_client_request_body(r); > } > @@ -234,6 +235,7 @@ ngx_http_read_client_request_body(ngx_ht > } > > r->read_event_handler = ngx_http_read_client_request_body_handler; > + r->write_event_handler = ngx_http_request_empty_handler; > > return ngx_http_do_read_client_request_body(r); > } > Hi Maxim, Your patch solved the problem for me. Now files are uploaded correctly in both synthetic tests and from real Iphone application. Thank you! -- Regards, Dmitry -------------- next part -------------- An HTML attachment was scrubbed... URL: From jefftk at google.com Wed Nov 21 12:51:02 2012 From: jefftk at google.com (Jeff Kaufman) Date: Wed, 21 Nov 2012 07:51:02 -0500 Subject: clearing etags when gzipping In-Reply-To: <20121121082411.GR40452@mdounin.ru> References: <20121121082411.GR40452@mdounin.ru> Message-ID: On Wed, Nov 21, 2012 at 3:24 AM, Maxim Dounin wrote: > Appending ";gzip" would not work as a result of a gzip might be > different depending on gzip settings. The only aproach which will > work is to change etag to a week one. I'm ok with this, though it > will generally need week etag support in various places. What would you think of simply not clearing the etag when it's already weak? Jeff From flygoast at 126.com Wed Nov 21 14:12:25 2012 From: flygoast at 126.com (=?GBK?B?t+u5yw==?=) Date: Wed, 21 Nov 2012 22:12:25 +0800 (CST) Subject: nginx proxy cache support contional request( If-Modified-Since) In-Reply-To: References: <26a9a8ec.69515.13b203d45f7.Coremail.flygoast@126.com> Message-ID: <3ea951da.14ebf.13b23500879.Coremail.flygoast@126.com> No. When the cache expires, the nginx will fetch it from backend. For saving bandwidth, I want to add 'If-Modified-Since' header to the request, so if the content is not modified, the backend will give a 304 response without body. ? 2012-11-21 16:36:06?"chen cw" ??? do proxy_cache and proxy_store meets your requirement? On Wed, Nov 21, 2012 at 7:53 AM, ?? wrote: Hi, I'm using nginx as a reverse proxy with cache. My content almost never changes. So I want to use a conditional GET request when fetch content from backend. However, nginx don't support it. Who have a such patch for it? Can give me? Thanks. PS, I want to know someday nginx will implement it or refuse to implement it? Thank you. _______________________________________________ nginx-devel mailing list nginx-devel at nginx.org http://mailman.nginx.org/mailman/listinfo/nginx-devel -- -- Charles Chen Software Engineer Server Platforms Team at Taobao.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ru at nginx.com Wed Nov 21 14:23:27 2012 From: ru at nginx.com (ru at nginx.com) Date: Wed, 21 Nov 2012 14:23:27 +0000 Subject: [nginx] svn commit: r4933 - trunk/src/core Message-ID: <20121121142327.384CB3F9FF4@mail.nginx.com> Author: ru Date: 2012-11-21 14:23:26 +0000 (Wed, 21 Nov 2012) New Revision: 4933 URL: http://trac.nginx.org/nginx/changeset/4933/nginx Log: Fixed location of debug message in ngx_shmtx_lock(). Modified: trunk/src/core/ngx_shmtx.c Modified: trunk/src/core/ngx_shmtx.c =================================================================== --- trunk/src/core/ngx_shmtx.c 2012-11-21 01:40:11 UTC (rev 4932) +++ trunk/src/core/ngx_shmtx.c 2012-11-21 14:23:26 UTC (rev 4933) @@ -117,11 +117,11 @@ "sem_wait() failed while waiting on shmtx"); break; } - - ngx_log_debug0(NGX_LOG_DEBUG_CORE, ngx_cycle->log, 0, - "shmtx awoke"); } + ngx_log_debug0(NGX_LOG_DEBUG_CORE, ngx_cycle->log, 0, + "shmtx awoke"); + continue; } From goelvivek2011 at gmail.com Wed Nov 21 16:51:51 2012 From: goelvivek2011 at gmail.com (Vivek Goel) Date: Wed, 21 Nov 2012 22:21:51 +0530 Subject: How nginx define a free worker? Message-ID: Hi, If I have n cores and I am running n nginx worker process how nginx will decide free worker for next connection? 1. Will it be doing round robin? If it is not using round robin what method it use? Is there a way I can force it to use round robin method? regards Vivek Goel -------------- next part -------------- An HTML attachment was scrubbed... URL: From goelvivek2011 at gmail.com Wed Nov 21 17:00:53 2012 From: goelvivek2011 at gmail.com (Vivek Goel) Date: Wed, 21 Nov 2012 22:30:53 +0530 Subject: Request are getting assigned to same worker Message-ID: Hi, My setup description is: 1. I have one fastcgi module 2. I have one location module. 3. I am running nginx with 20 worker process. Location module don't use upstream method so it is blocking call. It takes near about 100ms-1 sec to give a response. According to nginx logs sometime fastcgi request is taking lot of time to give response but actually fastcgi request is finishing soon according to fastcgi logs. What I am thinking problem is,Same worker is handling fastcgi and location request, due to blocking nature of location handler response of fastcgi is waiting to be served? Is there a way I can avoid this condition without replacing the module? 1. Reserving one worker to server only fastcgi request? 2. Using round robin worker assignment to avoid such condition. regards Vivek Goel -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Wed Nov 21 21:02:46 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 22 Nov 2012 01:02:46 +0400 Subject: clearing etags when gzipping In-Reply-To: References: <20121121082411.GR40452@mdounin.ru> Message-ID: <20121121210246.GW40452@mdounin.ru> Hello! On Wed, Nov 21, 2012 at 07:51:02AM -0500, Jeff Kaufman wrote: > On Wed, Nov 21, 2012 at 3:24 AM, Maxim Dounin wrote: > > Appending ";gzip" would not work as a result of a gzip might be > > different depending on gzip settings. The only aproach which will > > work is to change etag to a week one. I'm ok with this, though it > > will generally need week etag support in various places. > > What would you think of simply not clearing the etag when it's already weak? I think it would be ok. -- Maxim Dounin http://nginx.com/support.html From nginx at lukaperkov.net Thu Nov 22 14:15:38 2012 From: nginx at lukaperkov.net (Luka Perkov) Date: Thu, 22 Nov 2012 15:15:38 +0100 Subject: plans for chunked encoding In-Reply-To: <20121116131758.GA32599@w500> References: <20121019140704.GA7781@w500> <20121019141310.GM40452@mdounin.ru> <20121019142825.GB7781@w500> <20121116114855.GQ40452@mdounin.ru> <20121116131758.GA32599@w500> Message-ID: <20121122141538-18990@mutt-kz> Hi Maxim, On Fri, Nov 16, 2012 at 02:17:58PM +0100, Luka Perkov wrote: > On Fri, Nov 16, 2012 at 03:48:55PM +0400, Maxim Dounin wrote: > > > On Fri, Oct 19, 2012 at 06:13:10PM +0400, Maxim Dounin wrote: > > > > > I'm wondering if there is any work done on this already ? If so I could > > > > > test some patches... > > > > > > > > I'll post it here once there is something to test. > > > > > > Thank you. > > > > JFYI, chunked request body patches are available here: > > > > http://mailman.nginx.org/pipermail/nginx-devel/2012-November/002961.html > > > > Review and testing appreciated. > > Thank you for the patches. I'll test them this weekend (or early next > week) and I'll give you my feedback after that. I have started testing your patches. I'm working on dummy module for dealing with post requests, but in the meantime I have a question. This is the dummy post data: $ cat > /tmp/post_data << EOF test post data EOF This is what I have tested: $ curl -v -X GET http://127.0.0.1:80/ Returns 404 in headers and html body as expected. $ curl -v -X POST -d "@/tmp/post_data" http://127.0.0.1:80/ Returns 404 in headers and html body as expected. $ curl -v -X POST -d "@/tmp/post_data" --header "Transfer-Encoding: chunked" http://127.0.0.1:80/ Returns 404 in headers and html body as expected. $ curl -v -X POST -d "@/dev/null" http://127.0.0.1:80/ Returns 404 in headers and html body as expected. $ curl -v -X POST -d "@/dev/null" --header "Transfer-Encoding: chunked" http://127.0.0.1:80/ Returns 404 in headers but 500 in html body. Is this how it's supposed to be ? Luka From mdounin at mdounin.ru Thu Nov 22 16:24:29 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 22 Nov 2012 20:24:29 +0400 Subject: plans for chunked encoding In-Reply-To: <20121122141538-18990@mutt-kz> References: <20121019140704.GA7781@w500> <20121019141310.GM40452@mdounin.ru> <20121019142825.GB7781@w500> <20121116114855.GQ40452@mdounin.ru> <20121116131758.GA32599@w500> <20121122141538-18990@mutt-kz> Message-ID: <20121122162428.GZ40452@mdounin.ru> Hello! On Thu, Nov 22, 2012 at 03:15:38PM +0100, Luka Perkov wrote: > Hi Maxim, > > On Fri, Nov 16, 2012 at 02:17:58PM +0100, Luka Perkov wrote: > > On Fri, Nov 16, 2012 at 03:48:55PM +0400, Maxim Dounin wrote: > > > > On Fri, Oct 19, 2012 at 06:13:10PM +0400, Maxim Dounin wrote: > > > > > > I'm wondering if there is any work done on this already ? If so I could > > > > > > test some patches... > > > > > > > > > > I'll post it here once there is something to test. > > > > > > > > Thank you. > > > > > > JFYI, chunked request body patches are available here: > > > > > > http://mailman.nginx.org/pipermail/nginx-devel/2012-November/002961.html > > > > > > Review and testing appreciated. > > > > Thank you for the patches. I'll test them this weekend (or early next > > week) and I'll give you my feedback after that. > > I have started testing your patches. I'm working on dummy module for > dealing with post requests, but in the meantime I have a question. > > This is the dummy post data: > > $ cat > /tmp/post_data << EOF > test post data > EOF > > This is what I have tested: > > $ curl -v -X GET http://127.0.0.1:80/ > > Returns 404 in headers and html body as expected. > > $ curl -v -X POST -d "@/tmp/post_data" http://127.0.0.1:80/ > > Returns 404 in headers and html body as expected. > > $ curl -v -X POST -d "@/tmp/post_data" --header "Transfer-Encoding: chunked" http://127.0.0.1:80/ > > Returns 404 in headers and html body as expected. > > $ curl -v -X POST -d "@/dev/null" http://127.0.0.1:80/ > > Returns 404 in headers and html body as expected. > > $ curl -v -X POST -d "@/dev/null" --header "Transfer-Encoding: chunked" http://127.0.0.1:80/ > > Returns 404 in headers but 500 in html body. Is this how it's supposed to be ? I don't see 500 here, but last curl command produces incorrect http request. Here is a dump (url was slightly modified to match my sandbox): 20:17:08.787666 IP 127.0.0.1.57689 > 127.0.0.1.8080: P 1:240(239) ack 1 win 35840 0x0000: 4500 0123 465d 4000 4006 f575 7f00 0001 E..#F]@. at ..u.... 0x0010: 7f00 0001 e159 1f90 9ff4 3616 6c27 21eb .....Y....6.l'!. 0x0020: 8018 8c00 9e71 0000 0101 080a 059b 4d6f .....q........Mo 0x0030: 059b 4d6f 504f 5354 202f 666f 6f62 6172 ..MoPOST./foobar 0x0040: 2048 5454 502f 312e 310d 0a55 7365 722d .HTTP/1.1..User- 0x0050: 4167 656e 743a 2063 7572 6c2f 372e 3231 Agent:.curl/7.21 0x0060: 2e33 2028 6933 3836 2d70 6f72 7462 6c64 .3.(i386-portbld 0x0070: 2d66 7265 6562 7364 362e 3229 206c 6962 -freebsd6.2).lib 0x0080: 6375 726c 2f37 2e32 312e 3320 4f70 656e curl/7.21.3.Open 0x0090: 5353 4c2f 302e 392e 3765 207a 6c69 622f SSL/0.9.7e.zlib/ 0x00a0: 312e 322e 330d 0a48 6f73 743a 2031 3237 1.2.3..Host:.127 0x00b0: 2e30 2e30 2e31 3a38 3038 300d 0a41 6363 .0.0.1:8080..Acc 0x00c0: 6570 743a 202a 2f2a 0d0a 5472 616e 7366 ept:.*/*..Transf 0x00d0: 6572 2d45 6e63 6f64 696e 673a 2063 6875 er-Encoding:.chu 0x00e0: 6e6b 6564 0d0a 436f 6e74 656e 742d 5479 nked..Content-Ty 0x00f0: 7065 3a20 6170 706c 6963 6174 696f 6e2f pe:.application/ 0x0100: 782d 7777 772d 666f 726d 2d75 726c 656e x-www-form-urlen 0x0110: 636f 6465 640d 0a0d 0a30 0d0a 0d0a 300d coded....0....0. 0x0120: 0a0d 0a ... As you can see there are two final chunks ("0" CRLF CRLF), which is incorrect. In nginx error log it can be seen as: 2012/11/22 20:17:08 [info] 82190#0: *5 client sent invalid method while reading client pipelined request line, client: 127.0.0.1, server: , request: "0" And corresponding access log entries: 127.0.0.1 - - [22/Nov/2012:20:17:08 +0400] "POST /foobar HTTP/1.1" 404 168 "-" "curl/7.21.3 (i386-portbld-freebsd6.2) libcurl/7.21.3 OpenSSL/0.9.7e zlib/1.2.3" 127.0.0.1 - - [22/Nov/2012:20:17:08 +0400] "0" 400 172 "-" "-" Two responses are returned as per http protocol, one with 404 (in headers and body), and another one with 400 Bad Request (again, in headers and body). If you indeed see 500 in body it might be something to do with either your module or nginx config you use. -- Maxim Dounin http://nginx.com/support.html From nginx at lukaperkov.net Thu Nov 22 20:47:42 2012 From: nginx at lukaperkov.net (Luka Perkov) Date: Thu, 22 Nov 2012 21:47:42 +0100 Subject: plans for chunked encoding In-Reply-To: <20121122162428.GZ40452@mdounin.ru> References: <20121019140704.GA7781@w500> <20121019141310.GM40452@mdounin.ru> <20121019142825.GB7781@w500> <20121116114855.GQ40452@mdounin.ru> <20121116131758.GA32599@w500> <20121122141538-18990@mutt-kz> <20121122162428.GZ40452@mdounin.ru> Message-ID: <20121122204742-1844@mutt-kz> Hi Maxim, On Thu, Nov 22, 2012 at 08:24:29PM +0400, Maxim Dounin wrote: ... > > $ curl -v -X POST -d "@/dev/null" --header "Transfer-Encoding: chunked" http://127.0.0.1:80/ > > > > Returns 404 in headers but 500 in html body. Is this how it's supposed to be ? ... > If you indeed see 500 in body it might be something to do with > either your module or nginx config you use. Here is full information from my side... First the nginx logs: 2012/11/22 21:46:00 [debug] 1237#0: *1 http cl:-1 max:1048576 2012/11/22 21:46:00 [debug] 1237#0: *1 rewrite phase: 2 2012/11/22 21:46:00 [debug] 1237#0: *1 post rewrite phase: 3 2012/11/22 21:46:00 [debug] 1237#0: *1 generic phase: 4 2012/11/22 21:46:00 [debug] 1237#0: *1 generic phase: 5 2012/11/22 21:46:00 [debug] 1237#0: *1 access phase: 6 2012/11/22 21:46:00 [debug] 1237#0: *1 post access phase: 7 2012/11/22 21:46:00 [debug] 1237#0: *1 content phase: 8 2012/11/22 21:46:00 [debug] 1237#0: *1 open index "/etc/nginx/html/index.html" 2012/11/22 21:46:00 [debug] 1237#0: *1 stat() "/etc/nginx/html/index.html" failed (2: No such file or directory) 2012/11/22 21:46:00 [debug] 1237#0: *1 http index check dir: "/etc/nginx/html" 2012/11/22 21:46:00 [error] 1237#0: *1 "/etc/nginx/html/index.html" is not found (2: No such file or directory), client: 127.0.0.1, server: , request: "POST / HTTP/1.1", host: "127.0.0.1" 2012/11/22 21:46:00 [debug] 1237#0: *1 http finalize request: 404, "/?" a:1, c:1 2012/11/22 21:46:00 [debug] 1237#0: *1 http special response: 404, "/?" 2012/11/22 21:46:00 [debug] 1237#0: *1 http set discard body 2012/11/22 21:46:00 [debug] 1237#0: *1 http chunked byte: 0D s:0 2012/11/22 21:46:00 [error] 1237#0: *1 client sent invalid chunked body, client: 127.0.0.1, server: , request: "POST / HTTP/1.1", host: "127.0.0.1" 2012/11/22 21:46:00 [debug] 1237#0: *1 HTTP/1.1 404 Not Found^M Server: nginx/1.3.9^M Date: Thu, 22 Nov 2012 20:46:00 GMT^M Content-Type: text/html^M Content-Length: 192^M Connection: keep-alive^M Keep-Alive: timeout=130^M 2012/11/22 21:46:00 [debug] 1237#0: *1 write new buf t:1 f:0 0000000001731EB8, pos 0000000001731EB8, size: 179 file: 0, size: 0 2012/11/22 21:46:00 [debug] 1237#0: *1 http write filter: l:0 f:0 s:179 2012/11/22 21:46:00 [debug] 1237#0: *1 http output filter "/?" 2012/11/22 21:46:00 [debug] 1237#0: *1 http copy filter: "/?" 2012/11/22 21:46:00 [debug] 1237#0: *1 write old buf t:1 f:0 0000000001731EB8, pos 0000000001731EB8, size: 179 file: 0, size: 0 2012/11/22 21:46:00 [debug] 1237#0: *1 write new buf t:0 f:0 0000000000000000, pos 00000000006781E0, size: 140 file: 0, size: 0 2012/11/22 21:46:00 [debug] 1237#0: *1 write new buf t:0 f:0 0000000000000000, pos 00000000006770E0, size: 52 file: 0, size: 0 2012/11/22 21:46:00 [debug] 1237#0: *1 http write filter: l:1 f:0 s:371 2012/11/22 21:46:00 [debug] 1237#0: *1 http write filter limit 0 2012/11/22 21:46:00 [debug] 1237#0: *1 writev: 371 2012/11/22 21:46:00 [debug] 1237#0: *1 http write filter 0000000000000000 2012/11/22 21:46:00 [debug] 1237#0: *1 http copy filter: 0 "/?" 2012/11/22 21:46:00 [debug] 1237#0: *1 http finalize request: 0, "/?" a:1, c:1 2012/11/22 21:46:00 [debug] 1237#0: *1 set http keepalive handler 2012/11/22 21:46:00 [debug] 1237#0: *1 http close request 2012/11/22 21:46:00 [debug] 1237#0: *1 http log handler 2012/11/22 21:46:00 [debug] 1237#0: *1 free: 0000000001731490, unused: 834 2012/11/22 21:46:00 [debug] 1237#0: *1 event timer add: 3: 130000:1353617290838 2012/11/22 21:46:00 [debug] 1237#0: *1 pipelined request 2012/11/22 21:46:00 [debug] 1237#0: *1 post event 00000000017116A0 2012/11/22 21:46:00 [debug] 1237#0: *1 delete posted event 00000000017116A0 And the curl: $ curl -v -X POST -d "@/dev/null" --header "Transfer-Encoding: chunked" http://127.0.0.1:80/ * About to connect() to 127.0.0.1 port 80 (#0) * Trying 127.0.0.1... * connected * Connected to 127.0.0.1 (127.0.0.1) port 80 (#0) > POST / HTTP/1.1 > User-Agent: curl/7.28.0 > Host: 127.0.0.1 > Accept: */* > Transfer-Encoding: chunked > Content-Type: application/x-www-form-urlencoded > * upload completely sent off: 7 out of 0 bytes < HTTP/1.1 404 Not Found < Server: nginx/1.3.9 < Date: Thu, 22 Nov 2012 20:46:00 GMT < Content-Type: text/html < Content-Length: 192 < Connection: keep-alive < Keep-Alive: timeout=130 < 500 Internal Server Error

500 Internal Server Error


nginx/1.3.9
* Connection #0 to host 127.0.0.1 left intact * Closing connection #0 I have configured nginx like this (there are no custom modules): $ _cfgdir=/etc/nginx $ _tmpdir=/var/lib/nginx $ ./configure \ --with-debug \ --prefix=$_cfgdir \ --conf-path=$_cfgdir/nginx.conf \ --sbin-path=/usr/sbin/nginx \ --pid-path=/var/run/nginx.pid \ --lock-path=/var/lock/nginx.lock \ --user=http --group=http \ --http-log-path=/var/log/nginx/access.log \ --error-log-path=/var/log/nginx/error.log \ --http-client-body-temp-path=$_tmpdir/client-body \ --http-proxy-temp-path=$_tmpdir/proxy \ --with-file-aio \ \ --without-http_auth_basic_module \ --without-http_autoindex_module \ --without-http_browser_module \ --without-http_charset_module \ --without-http_empty_gif_module \ --without-http_fastcgi_module \ --without-http_geo_module \ --without-http_map_module \ --without-http_memcached_module \ --without-http_scgi_module \ --without-http_scgi_module \ --without-http_ssi_module \ --without-http_userid_module \ --without-http_uwsgi_module And the config file I'm using: user http http; worker_processes 1; # daemon off; events { worker_connections 128; } http { include mime.types; default_type application/octet-stream; keepalive_timeout 130s 130s; gzip off; log_format minimal_generic '$remote_addr [$time_local] "$request" "$status" "$http_user_agent"'; log_format minimal_proxy '$http_x_forwarded_for [$time_local] "$status" "$http_user_agent"'; log_format full_generic '$remote_addr [$time_local] "$request" "$status" "$http_user_agent" "$request_body"'; server { listen 80; location / { access_log /var/log/nginx/local_dummy.log minimal_generic; error_log /var/log/nginx/local_dummy_error.log debug; } } } This is on the last svn version. Maybe you can tell if I have configured or using something wrong ? Luka From nginx at lukaperkov.net Fri Nov 23 00:32:11 2012 From: nginx at lukaperkov.net (Luka Perkov) Date: Fri, 23 Nov 2012 01:32:11 +0100 Subject: [RFC] dummy body inspection module Message-ID: <20121123003211-30060@mutt-kz> Hi, I wanted to check with you guys what is the best way to look into request body inside nginx module. Example use case: Client sends HTTP POST message; nginx module looks at the message and if it's ok it redirects internaly HTTP request to "/ok/" and if it's not it will redirect it to "/error/". Nginx justs looks at the contents and forwards it to correct destination. I have made a dummy module for this purpose. Please take a look at the code bellow and let me know what you think. Luka /* * Copyright (C) 2012 Luka Perkov * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include #include #include #include static ngx_int_t ngx_http_post_inspect_init(ngx_conf_t *cf); static void * ngx_http_post_inspect_create_loc_conf(ngx_conf_t *cf); static char * ngx_http_post_inspect_merge_loc_conf(ngx_conf_t *cf, void *parent, void *child); typedef struct { ngx_flag_t inspect; } ngx_http_post_inspect_loc_conf_t; static ngx_command_t ngx_http_post_inspect_commands[] = { { ngx_string("post_inspect"), NGX_HTTP_LOC_CONF|NGX_CONF_FLAG, ngx_conf_set_flag_slot, NGX_HTTP_LOC_CONF_OFFSET, offsetof(ngx_http_post_inspect_loc_conf_t, inspect), NULL }, ngx_null_command }; static ngx_http_module_t ngx_http_post_inspect_module_ctx = { NULL, /* preconfiguration */ ngx_http_post_inspect_init, /* postconfiguration */ NULL, /* create main configuration */ NULL, /* init main configuration */ NULL, /* create server configuration */ NULL, /* merge server configuration */ ngx_http_post_inspect_create_loc_conf, /* create location configuration */ ngx_http_post_inspect_merge_loc_conf /* merge location configuration */ }; ngx_module_t ngx_http_post_inspect_module = { NGX_MODULE_V1, &ngx_http_post_inspect_module_ctx, /* module context */ ngx_http_post_inspect_commands, /* module directives */ NGX_HTTP_MODULE, /* module type */ NULL, /* init master */ NULL, /* init module */ NULL, /* init process */ NULL, /* init thread */ NULL, /* exit thread */ NULL, /* exit process */ NULL, /* exit master */ NGX_MODULE_V1_PADDING }; typedef struct { ngx_flag_t done:1; ngx_flag_t waiting_more_body:1; } ngx_http_post_inspect_ctx_t; static ngx_int_t ngx_http_post_inspect_deep_inspection(ngx_http_request_t *r) { ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, \ "mod_post_inspect ngx_http_post_inspect_deep_inspection()"); u_char *p; size_t len; ngx_buf_t *buf; ngx_chain_t *cl; ngx_str_t msg; if (r->request_body == NULL || r->request_body->bufs == NULL || r->request_body->temp_file) { ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, \ "mod_post_inspect ngx_http_post_inspect_deep_inspection(): body is not found"); /* * Exit because no body was found or body is in temp_file. * TODO: extend this */ return NGX_OK; } cl = r->request_body->bufs; buf = cl->buf; if (cl->next == NULL) { /* Body data is only in one buffer. */ msg.len = buf->last - buf->pos; msg.data = buf->pos; } else { len = buf->last - buf->pos; cl = cl->next; for ( /* void */ ; cl; cl = cl->next) { buf = cl->buf; len += buf->last - buf->pos; } p = ngx_pnalloc(r->pool, len); if (p == NULL) { ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, \ "mod_post_inspect ngx_http_post_inspect_deep_inspection(): ngx_pnalloc error"); return NGX_ERROR; } msg.data = p; cl = r->request_body->bufs; for ( /* void */ ; cl; cl = cl->next) { buf = cl->buf; p = ngx_cpymem(p, buf->pos, buf->last - buf->pos); } msg.len = len; } /* * We can do some magic here with POST data in msg. * After that we can redirect to some internal url. */ ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, \ "mod_post_inspect ngx_http_post_inspect_deep_inspection(): \"%V\"", &msg); ngx_str_t redirect_to = ngx_string("/redirect1/"); ngx_http_internal_redirect(r, &redirect_to , &r->args); (void) ngx_http_discard_request_body(r); return NGX_HTTP_OK; } void ngx_http_post_inspect_payload_handler(ngx_http_request_t *r) { ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, \ "mod_post_inspect ngx_http_post_inspect_payload_handler()"); ngx_http_post_inspect_ctx_t *ctx; ctx = ngx_http_get_module_ctx(r, ngx_http_post_inspect_module); r->read_event_handler = ngx_http_request_empty_handler; r->main->count--; ctx->done = 1; if (ctx->waiting_more_body) { ctx->waiting_more_body = 0; ngx_http_core_run_phases(r); } } static ngx_int_t ngx_http_post_inspect_access_handler(ngx_http_request_t *r) { ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, \ "mod_post_inspect ngx_http_post_inspect_access_handler()"); ngx_int_t rc; ngx_http_post_inspect_ctx_t *ctx; ctx = ngx_http_get_module_ctx(r, ngx_http_post_inspect_module); if (ctx != NULL) { if (ctx->done) { return NGX_DECLINED; } return NGX_DONE; } if (r->method != NGX_HTTP_POST) { return NGX_DECLINED; } ctx = ngx_pcalloc(r->pool, sizeof(ngx_http_post_inspect_ctx_t)); if (ctx == NULL) { ngx_http_finalize_request(r, NGX_ERROR); return NGX_ERROR; } ngx_http_set_ctx(r, ctx, ngx_http_post_inspect_module); r->request_body_in_single_buf = 1; r->request_body_in_persistent_file = 1; r->request_body_in_clean_file = 1; if (r->request_body_in_file_only) { r->request_body_file_log_level = 0; } rc = ngx_http_read_client_request_body(r, ngx_http_post_inspect_payload_handler); if (rc == NGX_ERROR || rc >= NGX_HTTP_SPECIAL_RESPONSE) { return rc; } if (rc == NGX_AGAIN) { ctx->waiting_more_body = 1; return NGX_DONE; } return NGX_DECLINED; } static ngx_int_t ngx_http_post_inspect_init(ngx_conf_t *cf) { ngx_http_handler_pt *h; ngx_http_core_main_conf_t *cmcf; cmcf = ngx_http_conf_get_module_main_conf(cf, ngx_http_core_module); h = ngx_array_push(&cmcf->phases[NGX_HTTP_REWRITE_PHASE].handlers); if (h == NULL) return NGX_ERROR; *h = ngx_http_post_inspect_access_handler; return NGX_OK; } static void * ngx_http_post_inspect_create_loc_conf(ngx_conf_t *cf) { ngx_http_post_inspect_loc_conf_t *conf; conf = ngx_pcalloc(cf->pool, sizeof(ngx_http_post_inspect_loc_conf_t)); if (conf == NULL) return NGX_CONF_ERROR; conf->inspect = NGX_CONF_UNSET; return conf; } static char * ngx_http_post_inspect_merge_loc_conf(ngx_conf_t *cf, void *parent, void *child) { ngx_http_core_loc_conf_t *clcf; ngx_http_post_inspect_loc_conf_t *prev = parent; ngx_http_post_inspect_loc_conf_t *conf = child; ngx_conf_merge_value(conf->inspect, prev->inspect, 0); if (conf->inspect) { clcf = ngx_http_conf_get_module_loc_conf(cf, ngx_http_core_module); clcf->handler = ngx_http_post_inspect_deep_inspection; } return NGX_CONF_OK; } From jefftk at google.com Fri Nov 23 01:05:57 2012 From: jefftk at google.com (Jeff Kaufman) Date: Thu, 22 Nov 2012 20:05:57 -0500 Subject: clearing etags when gzipping In-Reply-To: <20121121210246.GW40452@mdounin.ru> References: <20121121082411.GR40452@mdounin.ru> <20121121210246.GW40452@mdounin.ru> Message-ID: On Wed, Nov 21, 2012 at 4:02 PM, Maxim Dounin wrote: > > > > > What would you think of simply not clearing the etag when it's already weak? > > I think it would be ok. > How do you want patches submitted? Is this good? --- nginx-1.3.8/src/http/modules/ngx_http_gzip_filter_module.c 2012-07-07 17:22:27.000000000 -0400 +++ nginx-1.3.8-weak-etags-shorter/src/http/modules/ngx_http_gzip_filter_module.c2012-11-21 17:05:12.758389000 -0500 @@ -306,7 +306,15 @@ ngx_http_clear_content_length(r); ngx_http_clear_accept_ranges(r); - ngx_http_clear_etag(r); + + /* Clear etags unless they're marked as weak (prefixed with 'W/') */ + h = r->headers_out.etag; + if (h && !(h->value.len >= 3 && + h->value.data[0] == 'W' && + h->value.data[1] == '/' && + h->value.data[2] == '"')) { + ngx_http_clear_etag(r); + } return ngx_http_next_header_filter(r); } From mdounin at mdounin.ru Fri Nov 23 02:21:20 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 23 Nov 2012 06:21:20 +0400 Subject: plans for chunked encoding In-Reply-To: <20121122204742-1844@mutt-kz> References: <20121019140704.GA7781@w500> <20121019141310.GM40452@mdounin.ru> <20121019142825.GB7781@w500> <20121116114855.GQ40452@mdounin.ru> <20121116131758.GA32599@w500> <20121122141538-18990@mutt-kz> <20121122162428.GZ40452@mdounin.ru> <20121122204742-1844@mutt-kz> Message-ID: <20121123022120.GA40452@mdounin.ru> Hello! On Thu, Nov 22, 2012 at 09:47:42PM +0100, Luka Perkov wrote: > Hi Maxim, > > On Thu, Nov 22, 2012 at 08:24:29PM +0400, Maxim Dounin wrote: > ... > > > > $ curl -v -X POST -d "@/dev/null" --header "Transfer-Encoding: chunked" http://127.0.0.1:80/ > > > > > > Returns 404 in headers but 500 in html body. Is this how it's supposed to be ? > > ... > > > If you indeed see 500 in body it might be something to do with > > either your module or nginx config you use. > > Here is full information from my side... First the nginx logs: > > 2012/11/22 21:46:00 [debug] 1237#0: *1 http cl:-1 max:1048576 > 2012/11/22 21:46:00 [debug] 1237#0: *1 rewrite phase: 2 > 2012/11/22 21:46:00 [debug] 1237#0: *1 post rewrite phase: 3 > 2012/11/22 21:46:00 [debug] 1237#0: *1 generic phase: 4 > 2012/11/22 21:46:00 [debug] 1237#0: *1 generic phase: 5 > 2012/11/22 21:46:00 [debug] 1237#0: *1 access phase: 6 > 2012/11/22 21:46:00 [debug] 1237#0: *1 post access phase: 7 > 2012/11/22 21:46:00 [debug] 1237#0: *1 content phase: 8 > 2012/11/22 21:46:00 [debug] 1237#0: *1 open index "/etc/nginx/html/index.html" > 2012/11/22 21:46:00 [debug] 1237#0: *1 stat() "/etc/nginx/html/index.html" failed (2: No such file or directory) > 2012/11/22 21:46:00 [debug] 1237#0: *1 http index check dir: "/etc/nginx/html" > 2012/11/22 21:46:00 [error] 1237#0: *1 "/etc/nginx/html/index.html" is not found (2: No such file or directory), client: 127.0.0.1, server: , request: "POST / HTTP/1.1", host: "127.0.0.1" > 2012/11/22 21:46:00 [debug] 1237#0: *1 http finalize request: 404, "/?" a:1, c:1 > 2012/11/22 21:46:00 [debug] 1237#0: *1 http special response: 404, "/?" > 2012/11/22 21:46:00 [debug] 1237#0: *1 http set discard body > 2012/11/22 21:46:00 [debug] 1237#0: *1 http chunked byte: 0D s:0 > 2012/11/22 21:46:00 [error] 1237#0: *1 client sent invalid chunked body, client: 127.0.0.1, server: , request: "POST / HTTP/1.1", host: "127.0.0.1" > 2012/11/22 21:46:00 [debug] 1237#0: *1 HTTP/1.1 404 Not Found^M > Server: nginx/1.3.9^M > Date: Thu, 22 Nov 2012 20:46:00 GMT^M > Content-Type: text/html^M > Content-Length: 192^M > Connection: keep-alive^M > Keep-Alive: timeout=130^M [...] Ok, I see what goes on. In your case (likely due to different curl version) request body is corrupted from start and nginx is able to detect the corruption while discarding request body. This results in 500 response body due to the followin code in ngx_http_special_response_handler(): if (ngx_http_discard_request_body(r) != NGX_OK) { error = NGX_HTTP_INTERNAL_SERVER_ERROR; } Two patches below fixes this, as well as couple of other problems with chunked body discarding identified while testing. # HG changeset patch # User Maxim Dounin # Date 1353632132 -14400 # Node ID 7885ce53454c5970f2479763645c4885b2bd3937 # Parent c10bf2cacc89dc3a4bab1eea0cd742dae2034fd5 Request body: fixed discard of chunked request body. Even if there is no preread data, make sure to always call ngx_http_discard_request_body_filter() in case of chunked request body to initialize r->headers_in.content_length_n for later use. diff --git a/src/http/ngx_http_request_body.c b/src/http/ngx_http_request_body.c --- a/src/http/ngx_http_request_body.c +++ b/src/http/ngx_http_request_body.c @@ -459,7 +459,7 @@ ngx_http_discard_request_body(ngx_http_r size = r->header_in->last - r->header_in->pos; - if (size) { + if (size || r->headers_in.chunked) { rc = ngx_http_discard_request_body_filter(r, r->header_in); if (rc != NGX_OK) { # HG changeset patch # User Maxim Dounin # Date 1353632948 -14400 # Node ID 66e38e53aedec0618931c748ade5f83a3489e840 # Parent 7885ce53454c5970f2479763645c4885b2bd3937 Request body: improved handling of incorrect chunked request body. While discarding chunked request body in some cases after detecting request body corruption no error was returned, while it was possible to correctly return 400 Bad Request. If error is detected too late, make sure to properly close connection. Additionally, in ngx_http_special_response_handler() don't return body of 500 Internal Server Error to a client if ngx_http_discard_request_body() fails, but disable keepalive and continue. diff --git a/src/http/ngx_http_request_body.c b/src/http/ngx_http_request_body.c --- a/src/http/ngx_http_request_body.c +++ b/src/http/ngx_http_request_body.c @@ -471,12 +471,18 @@ ngx_http_discard_request_body(ngx_http_r } } - if (ngx_http_read_discarded_request_body(r) == NGX_OK) { + rc = ngx_http_read_discarded_request_body(r); + + if (rc == NGX_OK) { r->lingering_close = 0; return NGX_OK; } - /* == NGX_AGAIN */ + if (rc >= NGX_HTTP_SPECIAL_RESPONSE) { + return rc; + } + + /* rc == NGX_AGAIN */ r->read_event_handler = ngx_http_discarded_request_body_handler; @@ -533,6 +539,12 @@ ngx_http_discarded_request_body_handler( return; } + if (rc >= NGX_HTTP_SPECIAL_RESPONSE) { + c->error = 1; + ngx_http_finalize_request(r, NGX_ERROR); + return; + } + /* rc == NGX_AGAIN */ if (ngx_handle_read_event(rev, 0) != NGX_OK) { @@ -606,8 +618,7 @@ ngx_http_read_discarded_request_body(ngx rc = ngx_http_discard_request_body_filter(r, &b); if (rc != NGX_OK) { - r->connection->error = 1; - return NGX_OK; + return rc; } } } diff --git a/src/http/ngx_http_special_response.c b/src/http/ngx_http_special_response.c --- a/src/http/ngx_http_special_response.c +++ b/src/http/ngx_http_special_response.c @@ -421,7 +421,7 @@ ngx_http_special_response_handler(ngx_ht r->expect_tested = 1; if (ngx_http_discard_request_body(r) != NGX_OK) { - error = NGX_HTTP_INTERNAL_SERVER_ERROR; + r->keepalive = 0; } if (clcf->msie_refresh -- Maxim Dounin http://nginx.com/support.html From crk_world at yahoo.com.cn Fri Nov 23 03:38:06 2012 From: crk_world at yahoo.com.cn (chen cw) Date: Fri, 23 Nov 2012 11:38:06 +0800 Subject: How nginx define a free worker? In-Reply-To: References: Message-ID: 1. How nginx define a free worker "ngx_accept_disabled = ngx_cycle->connection_n / 8 - ngx_cycle->free_connection_n;" so when the number of free connections is more than the value of "worker_connections" directive, the worker is free. 2. Will it be doing round robin? No, it is not RR. In fact, nginx use the tecnology called "accept mutax", so only one worker is listening on the port, others are waiting. you can refer to "ngx_process_events_and_timers()" for more information. On Thu, Nov 22, 2012 at 12:51 AM, Vivek Goel wrote: > Hi, > If I have n cores and I am running n nginx worker process how nginx will > decide free worker for next connection? > > > 1. Will it be doing round robin? > > If it is not using round robin what method it use? Is there a way I can > force it to use round robin method? > regards > Vivek Goel > > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -- -- Charles Chen Software Engineer Server Platforms Team at Taobao.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From nginx at lukaperkov.net Fri Nov 23 09:19:32 2012 From: nginx at lukaperkov.net (Luka Perkov) Date: Fri, 23 Nov 2012 10:19:32 +0100 Subject: plans for chunked encoding In-Reply-To: <20121123022120.GA40452@mdounin.ru> References: <20121019140704.GA7781@w500> <20121019141310.GM40452@mdounin.ru> <20121019142825.GB7781@w500> <20121116114855.GQ40452@mdounin.ru> <20121116131758.GA32599@w500> <20121122141538-18990@mutt-kz> <20121122162428.GZ40452@mdounin.ru> <20121122204742-1844@mutt-kz> <20121123022120.GA40452@mdounin.ru> Message-ID: <20121123091932-28131@mutt-kz> Hi Maxim, On Fri, Nov 23, 2012 at 06:21:20AM +0400, Maxim Dounin wrote: > > > > $ curl -v -X POST -d "@/dev/null" --header "Transfer-Encoding: chunked" http://127.0.0.1:80/ > > > > > > > > Returns 404 in headers but 500 in html body. Is this how it's supposed to be ? > > > > ... > > > > > If you indeed see 500 in body it might be something to do with > > > either your module or nginx config you use. > > > > Here is full information from my side... First the nginx logs: > > > > 2012/11/22 21:46:00 [debug] 1237#0: *1 http cl:-1 max:1048576 > > 2012/11/22 21:46:00 [debug] 1237#0: *1 rewrite phase: 2 > > 2012/11/22 21:46:00 [debug] 1237#0: *1 post rewrite phase: 3 > > 2012/11/22 21:46:00 [debug] 1237#0: *1 generic phase: 4 > > 2012/11/22 21:46:00 [debug] 1237#0: *1 generic phase: 5 > > 2012/11/22 21:46:00 [debug] 1237#0: *1 access phase: 6 > > 2012/11/22 21:46:00 [debug] 1237#0: *1 post access phase: 7 > > 2012/11/22 21:46:00 [debug] 1237#0: *1 content phase: 8 > > 2012/11/22 21:46:00 [debug] 1237#0: *1 open index "/etc/nginx/html/index.html" > > 2012/11/22 21:46:00 [debug] 1237#0: *1 stat() "/etc/nginx/html/index.html" failed (2: No such file or directory) > > 2012/11/22 21:46:00 [debug] 1237#0: *1 http index check dir: "/etc/nginx/html" > > 2012/11/22 21:46:00 [error] 1237#0: *1 "/etc/nginx/html/index.html" is not found (2: No such file or directory), client: 127.0.0.1, server: , request: "POST / HTTP/1.1", host: "127.0.0.1" > > 2012/11/22 21:46:00 [debug] 1237#0: *1 http finalize request: 404, "/?" a:1, c:1 > > 2012/11/22 21:46:00 [debug] 1237#0: *1 http special response: 404, "/?" > > 2012/11/22 21:46:00 [debug] 1237#0: *1 http set discard body > > 2012/11/22 21:46:00 [debug] 1237#0: *1 http chunked byte: 0D s:0 > > 2012/11/22 21:46:00 [error] 1237#0: *1 client sent invalid chunked body, client: 127.0.0.1, server: , request: "POST / HTTP/1.1", host: "127.0.0.1" > > 2012/11/22 21:46:00 [debug] 1237#0: *1 HTTP/1.1 404 Not Found^M > > Server: nginx/1.3.9^M > > Date: Thu, 22 Nov 2012 20:46:00 GMT^M > > Content-Type: text/html^M > > Content-Length: 192^M > > Connection: keep-alive^M > > Keep-Alive: timeout=130^M > > [...] > > Ok, I see what goes on. In your case (likely due to different > curl version) request body is corrupted from start and nginx is > able to detect the corruption while discarding request body. > > This results in 500 response body due to the followin code in > ngx_http_special_response_handler(): > > if (ngx_http_discard_request_body(r) != NGX_OK) { > error = NGX_HTTP_INTERNAL_SERVER_ERROR; > } > > Two patches below fixes this, as well as couple of other problems > with chunked body discarding identified while testing. This is my curl version: $ curl -V curl 7.28.0 (x86_64-unknown-linux-gnu) libcurl/7.28.0 OpenSSL/1.0.1c zlib/1.2.7 libssh2/1.4.2 Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp Features: AsynchDNS IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP After applying your patches this is the result: $ curl -v -X POST -d "@/dev/null" --header "Transfer-Encoding: chunked" http://127.0.0.1:80/ * About to connect() to 127.0.0.1 port 80 (#0) * Trying 127.0.0.1... * connected * Connected to 127.0.0.1 (127.0.0.1) port 80 (#0) > POST / HTTP/1.1 > User-Agent: curl/7.28.0 > Host: 127.0.0.1 > Accept: */* > Transfer-Encoding: chunked > Content-Type: application/x-www-form-urlencoded > * upload completely sent off: 7 out of 0 bytes < HTTP/1.1 400 Bad Request < Server: nginx/1.3.9 < Date: Fri, 23 Nov 2012 09:14:15 GMT < Content-Type: text/html < Content-Length: 172 < Connection: close < 400 Bad Request

400 Bad Request


nginx/1.3.9
* Closing connection #0 And the nginx logs: 2012/11/23 10:14:08 [debug] 28084#0: epoll add event: fd:7 op:1 ev:00000001 2012/11/23 10:14:15 [debug] 28084#0: accept on 0.0.0.0:80, ready: 0 2012/11/23 10:14:15 [debug] 28084#0: posix_memalign: 0000000001618820:256 @16 2012/11/23 10:14:15 [debug] 28084#0: *1 accept: 127.0.0.1 fd:3 2012/11/23 10:14:15 [debug] 28084#0: *1 event timer add: 3: 60000:1353662115189 2012/11/23 10:14:15 [debug] 28084#0: *1 epoll add event: fd:3 op:1 ev:80000001 2012/11/23 10:14:15 [debug] 28084#0: *1 malloc: 000000000162B5E0:1272 2012/11/23 10:14:15 [debug] 28084#0: *1 posix_memalign: 0000000001618980:256 @16 2012/11/23 10:14:15 [debug] 28084#0: *1 malloc: 000000000162BAE0:1024 2012/11/23 10:14:15 [debug] 28084#0: *1 posix_memalign: 000000000163E490:4096 @16 2012/11/23 10:14:15 [debug] 28084#0: *1 http process request line 2012/11/23 10:14:15 [debug] 28084#0: *1 recv: fd:3 158 of 1024 2012/11/23 10:14:15 [debug] 28084#0: *1 http request line: "POST / HTTP/1.1" 2012/11/23 10:14:15 [debug] 28084#0: *1 http uri: "/" 2012/11/23 10:14:15 [debug] 28084#0: *1 http args: "" 2012/11/23 10:14:15 [debug] 28084#0: *1 http exten: "" 2012/11/23 10:14:15 [debug] 28084#0: *1 http process request header line 2012/11/23 10:14:15 [debug] 28084#0: *1 http header: "User-Agent: curl/7.28.0" 2012/11/23 10:14:15 [debug] 28084#0: *1 http header: "Host: 127.0.0.1" 2012/11/23 10:14:15 [debug] 28084#0: *1 http header: "Accept: */*" 2012/11/23 10:14:15 [debug] 28084#0: *1 http header: "Transfer-Encoding: chunked" 2012/11/23 10:14:15 [debug] 28084#0: *1 http header: "Content-Type: application/x-www-form-urlencoded" 2012/11/23 10:14:15 [debug] 28084#0: *1 http header done 2012/11/23 10:14:15 [debug] 28084#0: *1 event timer del: 3: 1353662115189 2012/11/23 10:14:15 [debug] 28084#0: *1 rewrite phase: 0 2012/11/23 10:14:15 [debug] 28084#0: *1 test location: "/" 2012/11/23 10:14:15 [debug] 28084#0: *1 using configuration "/" 2012/11/23 10:14:15 [debug] 28084#0: *1 http cl:-1 max:1048576 2012/11/23 10:14:15 [debug] 28084#0: *1 rewrite phase: 2 2012/11/23 10:14:15 [debug] 28084#0: *1 http client request body preread 7 2012/11/23 10:14:15 [debug] 28084#0: *1 http request body chunked filter 2012/11/23 10:14:15 [debug] 28084#0: *1 http body chunked buf t:1 f:0 000000000162BAE0, pos 000000000162BB77, size: 7 file: 0, size: 0 2012/11/23 10:14:15 [debug] 28084#0: *1 http chunked byte: 0D s:0 2012/11/23 10:14:15 [error] 28084#0: *1 client sent invalid chunked body, client: 127.0.0.1, server: , request: "POST / HTTP/1.1", host: "127.0.0.1" 2012/11/23 10:14:15 [debug] 28084#0: *1 http finalize request: 400, "/?" a:1, c:1 2012/11/23 10:14:15 [debug] 28084#0: *1 http special response: 400, "/?" 2012/11/23 10:14:15 [debug] 28084#0: *1 HTTP/1.1 400 Bad Request Server: nginx/1.3.9 Date: Fri, 23 Nov 2012 09:14:15 GMT Content-Type: text/html Content-Length: 172 Connection: close 2012/11/23 10:14:15 [debug] 28084#0: *1 write new buf t:1 f:0 000000000163EE90, pos 000000000163EE90, size: 151 file: 0, size: 0 2012/11/23 10:14:15 [debug] 28084#0: *1 http write filter: l:0 f:0 s:151 2012/11/23 10:14:15 [debug] 28084#0: *1 http output filter "/?" 2012/11/23 10:14:15 [debug] 28084#0: *1 http copy filter: "/?" 2012/11/23 10:14:15 [debug] 28084#0: *1 write old buf t:1 f:0 000000000163EE90, pos 000000000163EE90, size: 151 file: 0, size: 0 2012/11/23 10:14:15 [debug] 28084#0: *1 write new buf t:0 f:0 0000000000000000, pos 00000000006780C0, size: 120 file: 0, size: 0 2012/11/23 10:14:15 [debug] 28084#0: *1 write new buf t:0 f:0 0000000000000000, pos 0000000000677C60, size: 52 file: 0, size: 0 2012/11/23 10:14:15 [debug] 28084#0: *1 http write filter: l:1 f:0 s:323 2012/11/23 10:14:15 [debug] 28084#0: *1 http write filter limit 0 2012/11/23 10:14:15 [debug] 28084#0: *1 writev: 323 2012/11/23 10:14:15 [debug] 28084#0: *1 http write filter 0000000000000000 2012/11/23 10:14:15 [debug] 28084#0: *1 http copy filter: 0 "/?" 2012/11/23 10:14:15 [debug] 28084#0: *1 http finalize request: 0, "/?" a:1, c:1 2012/11/23 10:14:15 [debug] 28084#0: *1 event timer add: 3: 5000:1353662060189 2012/11/23 10:14:15 [debug] 28084#0: *1 http lingering close handler 2012/11/23 10:14:15 [debug] 28084#0: *1 recv: fd:3 -1 of 4096 2012/11/23 10:14:15 [debug] 28084#0: *1 recv() not ready (11: Resource temporarily unavailable) 2012/11/23 10:14:15 [debug] 28084#0: *1 lingering read: -2 2012/11/23 10:14:15 [debug] 28084#0: *1 event timer: 3, old: 1353662060189, new: 1353662060189 2012/11/23 10:14:15 [debug] 28084#0: *1 http lingering close handler 2012/11/23 10:14:15 [debug] 28084#0: *1 recv: fd:3 0 of 4096 2012/11/23 10:14:15 [debug] 28084#0: *1 lingering read: 0 2012/11/23 10:14:15 [debug] 28084#0: *1 http request count:1 blk:0 2012/11/23 10:14:15 [debug] 28084#0: *1 http close request 2012/11/23 10:14:15 [debug] 28084#0: *1 http log handler 2012/11/23 10:14:15 [debug] 28084#0: *1 free: 000000000163E490, unused: 922 2012/11/23 10:14:15 [debug] 28084#0: *1 close http connection: 3 2012/11/23 10:14:15 [debug] 28084#0: *1 event timer del: 3: 1353662060189 2012/11/23 10:14:15 [debug] 28084#0: *1 reusable connection: 0 2012/11/23 10:14:15 [debug] 28084#0: *1 free: 000000000162BAE0 2012/11/23 10:14:15 [debug] 28084#0: *1 free: 000000000162B5E0 2012/11/23 10:14:15 [debug] 28084#0: *1 free: 0000000001618820, unused: 8 2012/11/23 10:14:15 [debug] 28084#0: *1 free: 0000000001618980, unused: 128 Let me know if I need to test anything else. Luka From ru at nginx.com Fri Nov 23 12:43:58 2012 From: ru at nginx.com (ru at nginx.com) Date: Fri, 23 Nov 2012 12:43:58 +0000 Subject: [nginx] svn commit: r4934 - trunk/src/core Message-ID: <20121123124358.DA4483F9C14@mail.nginx.com> Author: ru Date: 2012-11-23 12:43:58 +0000 (Fri, 23 Nov 2012) New Revision: 4934 URL: http://trac.nginx.org/nginx/changeset/4934/nginx Log: Core: don't reuse shared memory zone that changed ownership (ticket #210). nginx doesn't allow the same shared memory zone to be used for different purposes, but failed to check this on reconfiguration. If a shared memory zone was used for another purpose in the new configuration, nginx attempted to reuse it and crashed. Modified: trunk/src/core/ngx_cycle.c Modified: trunk/src/core/ngx_cycle.c =================================================================== --- trunk/src/core/ngx_cycle.c 2012-11-21 14:23:26 UTC (rev 4933) +++ trunk/src/core/ngx_cycle.c 2012-11-23 12:43:58 UTC (rev 4934) @@ -447,7 +447,9 @@ continue; } - if (shm_zone[i].shm.size == oshm_zone[n].shm.size) { + if (shm_zone[i].tag == oshm_zone[n].tag + && shm_zone[i].shm.size == oshm_zone[n].shm.size) + { shm_zone[i].shm.addr = oshm_zone[n].shm.addr; if (shm_zone[i].init(&shm_zone[i], oshm_zone[n].data) From toli at webforge.bg Fri Nov 23 15:53:53 2012 From: toli at webforge.bg (Anatoli Marinov) Date: Fri, 23 Nov 2012 17:53:53 +0200 Subject: configuration reload Message-ID: <50AF9C11.2020206@webforge.bg> *Hello colleagues, We have some troubles when nginx have to reload its configuration. *There is a*control panel and every user which has account may change its own settings. For example it may add new domain name or change other option. After that new config files are created and they are sent to the servers. When there is a new config file master process receives HUP signal and it loads this new configuration. After the configuration is loaded the master process starts new workers and waits for old workers to complete its responses with the end users. Usually there are several slow connections and old workers can stay alive for a long time. If there are several new configs for small period of time the number of workers may become very high and servers' load also gets high values. In general this is not good for the service. Do you have any idea how this issue could be solved? Is there different way to set new config file without a proc*ess *of new workers creation? Thanks in advance *Anatoli Marinov* * -------------- next part -------------- An HTML attachment was scrubbed... URL: From toli at webforge.bg Fri Nov 23 16:20:31 2012 From: toli at webforge.bg (Anatoli Marinov) Date: Fri, 23 Nov 2012 18:20:31 +0200 Subject: configuration reload In-Reply-To: <50AF9C11.2020206@webforge.bg> References: <50AF9C11.2020206@webforge.bg> Message-ID: <50AFA24F.5070905@webforge.bg> Very strange alignment :) sorry for that .... On 11/23/2012 05:53 PM, Anatoli Marinov wrote: > *Hello colleagues, > We have some troubles when nginx have to reload its configuration. > *There is a*control panel and every user which has account may change > its own settings. For example it may add new domain name or change > other option. After that new config files are created and they are > sent to the servers. > When there is a new config file master process receives HUP signal and > it loads this new configuration. After the configuration is loaded the > master process starts new workers and waits for old workers to > complete its responses with the end users. Usually there are several > slow connections and old workers can stay alive for a long time. > If there are several new configs for small period of time the number > of workers may become very high and servers' load also gets high > values. In general this is not good for the service. > > Do you have any idea how this issue could be solved? Is there > different way to set new config file without a proc*ess *of new > workers creation? > > > Thanks in advance > *Anatoli Marinov* > * > > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Fri Nov 23 20:45:17 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Sat, 24 Nov 2012 00:45:17 +0400 Subject: plans for chunked encoding In-Reply-To: <20121123091932-28131@mutt-kz> References: <20121019140704.GA7781@w500> <20121019141310.GM40452@mdounin.ru> <20121019142825.GB7781@w500> <20121116114855.GQ40452@mdounin.ru> <20121116131758.GA32599@w500> <20121122141538-18990@mutt-kz> <20121122162428.GZ40452@mdounin.ru> <20121122204742-1844@mutt-kz> <20121123022120.GA40452@mdounin.ru> <20121123091932-28131@mutt-kz> Message-ID: <20121123204516.GC40452@mdounin.ru> Hello! On Fri, Nov 23, 2012 at 10:19:32AM +0100, Luka Perkov wrote: [...] > After applying your patches this is the result: [...] > 2012/11/23 10:14:15 [debug] 28084#0: *1 http client request body preread 7 > 2012/11/23 10:14:15 [debug] 28084#0: *1 http request body chunked filter > 2012/11/23 10:14:15 [debug] 28084#0: *1 http body chunked buf t:1 f:0 000000000162BAE0, pos 000000000162BB77, size: 7 file: 0, size: 0 > 2012/11/23 10:14:15 [debug] 28084#0: *1 http chunked byte: 0D s:0 > 2012/11/23 10:14:15 [error] 28084#0: *1 client sent invalid chunked body, client: 127.0.0.1, server: , request: "POST / HTTP/1.1", host: "127.0.0.1" > 2012/11/23 10:14:15 [debug] 28084#0: *1 http finalize request: 400, "/?" a:1, c:1 > 2012/11/23 10:14:15 [debug] 28084#0: *1 http special response: 400, "/?" > 2012/11/23 10:14:15 [debug] 28084#0: *1 HTTP/1.1 400 Bad Request This log is from reading request body (in contrast to discarding request body discussed previously), and 400 Bad Request is perfectly correct result as request is indeed bad as previously discussed. [...] -- Maxim Dounin http://nginx.com/support.html From mdounin at mdounin.ru Sun Nov 25 10:55:35 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Sun, 25 Nov 2012 14:55:35 +0400 Subject: configuration reload In-Reply-To: <50AF9C11.2020206@webforge.bg> References: <50AF9C11.2020206@webforge.bg> Message-ID: <20121125105534.GD40452@mdounin.ru> Hello! On Fri, Nov 23, 2012 at 05:53:53PM +0200, Anatoli Marinov wrote: > *Hello colleagues, > We have some troubles when nginx have to reload its configuration. > *There is a*control panel and every user which has account may > change its own settings. For example it may add new domain name or > change other option. After that new config files are created and > they are sent to the servers. > When there is a new config file master process receives HUP signal > and it loads this new configuration. After the configuration is > loaded the master process starts new workers and waits for old > workers to complete its responses with the end users. Usually there > are several slow connections and old workers can stay alive for a > long time. > If there are several new configs for small period of time the number > of workers may become very high and servers' load also gets high > values. In general this is not good for the service. That's why it's generally not recommended to do automated configuration reloads. If you do automated reloads - you have to implement some safeguards to prevent situation of too many worker processes shutting down by either preventing further reloads till some workers exit, or killing oldest shutting down workers if another reload is strictly required. > Do you have any idea how this issue could be solved? Is there > different way to set new config file without a proc*ess *of new > workers creation? Current nginx configuration reload code uses new worker processes to apply new configuration. Reimplementing this with some other mechanism is certainly possible, but wouldn't be trivial (and I suspect might result in other problems). I would recommend you to focus on dynamic configuration of aspects you want to be user-configurable without changing main nginx configuration. -- Maxim Dounin http://nginx.com/support.html p.s. Please do not post html here, thank you. From goelvivek2011 at gmail.com Sun Nov 25 17:51:18 2012 From: goelvivek2011 at gmail.com (Vivek Goel) Date: Sun, 25 Nov 2012 23:21:18 +0530 Subject: Embedding third party library in nginx module Message-ID: Hi, I have one third party library which I want to integrate into nginx. What I am planning to use ngx_event to wait to finish callback of third party library. But according to my knowledge ngx_event works only for file descriptor. Is there a way I can signal that event is completed? Second question is: But according to "thumbextractor-module" it is not recommended to use thread inside. My third party library uses thread for processing. Can it create some issue? source:https://github.com/wandenberg/nginx-video-thumbextractor-module *By default ImageMagick uses OpenMP to process images in threads. I recommend to configure/compile ImageMagick using the option ?disable-openmp since that is not a good idea use threads inside workers.* * * * * regards Vivek Goel -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Mon Nov 26 17:59:31 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Mon, 26 Nov 2012 17:59:31 +0000 Subject: [nginx] svn commit: r4935 - trunk/src/http Message-ID: <20121126175932.00C8C3F9E7F@mail.nginx.com> Author: mdounin Date: 2012-11-26 17:59:30 +0000 (Mon, 26 Nov 2012) New Revision: 4935 URL: http://trac.nginx.org/nginx/changeset/4935/nginx Log: Request body: fixed discard of chunked request body. Even if there is no preread data, make sure to always call ngx_http_discard_request_body_filter() in case of chunked request body to initialize r->headers_in.content_length_n for later use. Modified: trunk/src/http/ngx_http_request_body.c Modified: trunk/src/http/ngx_http_request_body.c =================================================================== --- trunk/src/http/ngx_http_request_body.c 2012-11-23 12:43:58 UTC (rev 4934) +++ trunk/src/http/ngx_http_request_body.c 2012-11-26 17:59:30 UTC (rev 4935) @@ -459,7 +459,7 @@ size = r->header_in->last - r->header_in->pos; - if (size) { + if (size || r->headers_in.chunked) { rc = ngx_http_discard_request_body_filter(r, r->header_in); if (rc != NGX_OK) { From mdounin at mdounin.ru Mon Nov 26 18:00:15 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Mon, 26 Nov 2012 18:00:15 +0000 Subject: [nginx] svn commit: r4936 - trunk/src/http Message-ID: <20121126180015.45B3D3F9E7F@mail.nginx.com> Author: mdounin Date: 2012-11-26 18:00:14 +0000 (Mon, 26 Nov 2012) New Revision: 4936 URL: http://trac.nginx.org/nginx/changeset/4936/nginx Log: Request body: improved handling of incorrect chunked request body. While discarding chunked request body in some cases after detecting request body corruption no error was returned, while it was possible to correctly return 400 Bad Request. If error is detected too late, make sure to properly close connection. Additionally, in ngx_http_special_response_handler() don't return body of 500 Internal Server Error to a client if ngx_http_discard_request_body() fails, but disable keepalive and continue. Modified: trunk/src/http/ngx_http_request_body.c trunk/src/http/ngx_http_special_response.c Modified: trunk/src/http/ngx_http_request_body.c =================================================================== --- trunk/src/http/ngx_http_request_body.c 2012-11-26 17:59:30 UTC (rev 4935) +++ trunk/src/http/ngx_http_request_body.c 2012-11-26 18:00:14 UTC (rev 4936) @@ -471,13 +471,19 @@ } } - if (ngx_http_read_discarded_request_body(r) == NGX_OK) { + rc = ngx_http_read_discarded_request_body(r); + + if (rc == NGX_OK) { r->lingering_close = 0; return NGX_OK; } - /* == NGX_AGAIN */ + if (rc >= NGX_HTTP_SPECIAL_RESPONSE) { + return rc; + } + /* rc == NGX_AGAIN */ + r->read_event_handler = ngx_http_discarded_request_body_handler; if (ngx_handle_read_event(rev, 0) != NGX_OK) { @@ -533,6 +539,12 @@ return; } + if (rc >= NGX_HTTP_SPECIAL_RESPONSE) { + c->error = 1; + ngx_http_finalize_request(r, NGX_ERROR); + return; + } + /* rc == NGX_AGAIN */ if (ngx_handle_read_event(rev, 0) != NGX_OK) { @@ -606,8 +618,7 @@ rc = ngx_http_discard_request_body_filter(r, &b); if (rc != NGX_OK) { - r->connection->error = 1; - return NGX_OK; + return rc; } } } Modified: trunk/src/http/ngx_http_special_response.c =================================================================== --- trunk/src/http/ngx_http_special_response.c 2012-11-26 17:59:30 UTC (rev 4935) +++ trunk/src/http/ngx_http_special_response.c 2012-11-26 18:00:14 UTC (rev 4936) @@ -421,7 +421,7 @@ r->expect_tested = 1; if (ngx_http_discard_request_body(r) != NGX_OK) { - error = NGX_HTTP_INTERNAL_SERVER_ERROR; + r->keepalive = 0; } if (clcf->msie_refresh From mdounin at mdounin.ru Mon Nov 26 18:01:09 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Mon, 26 Nov 2012 18:01:09 +0000 Subject: [nginx] svn commit: r4937 - trunk/src/http Message-ID: <20121126180109.5CCD23F9F96@mail.nginx.com> Author: mdounin Date: 2012-11-26 18:01:08 +0000 (Mon, 26 Nov 2012) New Revision: 4937 URL: http://trac.nginx.org/nginx/changeset/4937/nginx Log: Request body: error checking fixes, negative rb->rest handling. Negative rb->rest can't happen with current code, but it's good to have it handled anyway. Found by Coverity (CID 744846, 744847, 744848). Modified: trunk/src/http/ngx_http_request_body.c Modified: trunk/src/http/ngx_http_request_body.c =================================================================== --- trunk/src/http/ngx_http_request_body.c 2012-11-26 18:00:14 UTC (rev 4936) +++ trunk/src/http/ngx_http_request_body.c 2012-11-26 18:01:08 UTC (rev 4937) @@ -134,6 +134,13 @@ return NGX_OK; } + if (rb->rest < 0) { + ngx_log_error(NGX_LOG_ALERT, r->connection->log, 0, + "negative request body rest"); + rc = NGX_HTTP_INTERNAL_SERVER_ERROR; + goto done; + } + clcf = ngx_http_get_module_loc_conf(r, ngx_http_core_module); size = clcf->client_body_buffer_size; @@ -643,7 +650,7 @@ } rb->chunked = ngx_pcalloc(r->pool, sizeof(ngx_http_chunked_t)); - if (rb == NULL) { + if (rb->chunked == NULL) { return NGX_HTTP_INTERNAL_SERVER_ERROR; } @@ -1022,7 +1029,9 @@ /* TODO: coalesce neighbouring buffers */ - ngx_chain_add_copy(r->pool, &rb->bufs, in); + if (ngx_chain_add_copy(r->pool, &rb->bufs, in) != NGX_OK) { + return NGX_HTTP_INTERNAL_SERVER_ERROR; + } return NGX_OK; } From mdounin at mdounin.ru Mon Nov 26 18:01:49 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Mon, 26 Nov 2012 18:01:49 +0000 Subject: [nginx] svn commit: r4938 - trunk/src/http Message-ID: <20121126180149.6586E3F9F6E@mail.nginx.com> Author: mdounin Date: 2012-11-26 18:01:49 +0000 (Mon, 26 Nov 2012) New Revision: 4938 URL: http://trac.nginx.org/nginx/changeset/4938/nginx Log: Request body: block write events while reading body. If write events are not blocked, an extra write event might happen for various reasons (e.g. as a result of a http pipelining), resulting in incorrect body being passed to a post handler. The problem manifested itself with the dav module only, as this is the only module which reads the body from a content phase handler (in contrast to exclusive content handlers like proxy). Additionally, dav module used to dump core in such situations due to ticket #238. See reports here: http://mailman.nginx.org/pipermail/nginx-devel/2012-November/002981.html http://serverfault.com/questions/449195/nginx-webdav-server-with-auth-request Modified: trunk/src/http/ngx_http_request_body.c Modified: trunk/src/http/ngx_http_request_body.c =================================================================== --- trunk/src/http/ngx_http_request_body.c 2012-11-26 18:01:08 UTC (rev 4937) +++ trunk/src/http/ngx_http_request_body.c 2012-11-26 18:01:49 UTC (rev 4938) @@ -105,6 +105,7 @@ rb->buf = r->header_in; r->read_event_handler = ngx_http_read_client_request_body_handler; + r->write_event_handler = ngx_http_request_empty_handler; rc = ngx_http_do_read_client_request_body(r); goto done; @@ -166,6 +167,7 @@ } r->read_event_handler = ngx_http_read_client_request_body_handler; + r->write_event_handler = ngx_http_request_empty_handler; rc = ngx_http_do_read_client_request_body(r); From mdounin at mdounin.ru Mon Nov 26 18:54:39 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 26 Nov 2012 22:54:39 +0400 Subject: clearing etags when gzipping In-Reply-To: References: <20121121082411.GR40452@mdounin.ru> <20121121210246.GW40452@mdounin.ru> Message-ID: <20121126185439.GM40452@mdounin.ru> Hello! On Thu, Nov 22, 2012 at 08:05:57PM -0500, Jeff Kaufman wrote: > On Wed, Nov 21, 2012 at 4:02 PM, Maxim Dounin wrote: > > > > > > > > What would you think of simply not clearing the etag when it's already weak? > > > > I think it would be ok. > > > > How do you want patches submitted? Is this good? > > --- nginx-1.3.8/src/http/modules/ngx_http_gzip_filter_module.c > 2012-07-07 17:22:27.000000000 -0400 > +++ nginx-1.3.8-weak-etags-shorter/src/http/modules/ngx_http_gzip_filter_module.c2012-11-21 > 17:05:12.758389000 -0500 > @@ -306,7 +306,15 @@ > > ngx_http_clear_content_length(r); > ngx_http_clear_accept_ranges(r); > - ngx_http_clear_etag(r); > + > + /* Clear etags unless they're marked as weak (prefixed with 'W/') */ > + h = r->headers_out.etag; > + if (h && !(h->value.len >= 3 && > + h->value.data[0] == 'W' && > + h->value.data[1] == '/' && > + h->value.data[2] == '"')) { > + ngx_http_clear_etag(r); > + } > > return ngx_http_next_header_filter(r); > } Uhm, this is not something I would like to commit. It should be a macro, probably something like ngx_http_clear_strong_etag(r); Much like ngx_http_clear_etag() itself. It would be also good idea to audit other uses of ngx_http_clear_etag() to see if they also deserve changing to a new macro introduced. Rule of thumb seems to be: we want to clear strong etags only if we clear Accept-Ranges but not Last-Modified. Style also needs fixing to match one used in nginx. -- Maxim Dounin http://nginx.com/support.html From ibobrik at gmail.com Mon Nov 26 20:10:13 2012 From: ibobrik at gmail.com (ivan babrou) Date: Tue, 27 Nov 2012 00:10:13 +0400 Subject: image_filter enhancement In-Reply-To: <20121114234732.GC40452@mdounin.ru> References: <20121106181601.GS40452@mdounin.ru> <20121114234732.GC40452@mdounin.ru> Message-ID: I tried to rework patch. Now it supports configuration through variables. Variables ox and oy renamed to crop_offset_x and crop_offset_y. I'm confused about this line: + value.data[value.len] = 0; Probably I don't understand something about memory management in nginx. diff --git a/ngx_http_image_filter_module.c b/ngx_http_image_filter_module.c index c853c33..15f8d17 100644 --- a/ngx_http_image_filter_module.c +++ b/ngx_http_image_filter_module.c @@ -32,6 +32,14 @@ #define NGX_HTTP_IMAGE_GIF 2 #define NGX_HTTP_IMAGE_PNG 3 +#define NGX_HTTP_IMAGE_OFFSET_TYPE_HORIZONTAL 0 +#define NGX_HTTP_IMAGE_OFFSET_TYPE_VERTICAL 1 + +#define NGX_HTTP_IMAGE_OFFSET_CENTER 0 +#define NGX_HTTP_IMAGE_OFFSET_LEFT 1 +#define NGX_HTTP_IMAGE_OFFSET_RIGHT 2 +#define NGX_HTTP_IMAGE_OFFSET_TOP 3 +#define NGX_HTTP_IMAGE_OFFSET_BOTTOM 4 #define NGX_HTTP_IMAGE_BUFFERED 0x08 @@ -43,11 +51,15 @@ typedef struct { ngx_uint_t angle; ngx_uint_t jpeg_quality; ngx_uint_t sharpen; + ngx_uint_t crop_offset_x; + ngx_uint_t crop_offset_y; ngx_flag_t transparency; ngx_http_complex_value_t *wcv; ngx_http_complex_value_t *hcv; + ngx_http_complex_value_t *oxv; + ngx_http_complex_value_t *oyv; ngx_http_complex_value_t *acv; ngx_http_complex_value_t *jqcv; ngx_http_complex_value_t *shcv; @@ -66,6 +78,8 @@ typedef struct { ngx_uint_t height; ngx_uint_t max_width; ngx_uint_t max_height; + ngx_uint_t crop_offset_x; + ngx_uint_t crop_offset_y; ngx_uint_t angle; ngx_uint_t phase; @@ -98,6 +112,15 @@ static u_char *ngx_http_image_out(ngx_http_request_t *r, ngx_uint_t type, static void ngx_http_image_cleanup(void *data); static ngx_uint_t ngx_http_image_filter_get_value(ngx_http_request_t *r, ngx_http_complex_value_t *cv, ngx_uint_t v); +static void +ngx_http_image_filter_crop_offset(ngx_http_request_t *r, + ngx_http_complex_value_t *cv, ngx_uint_t t, ngx_uint_t *v); +static void +ngx_http_image_filter_vertical_crop_offset(ngx_str_t *value, + ngx_uint_t *v); +static void +ngx_http_image_filter_horizontal_crop_offset(ngx_str_t *value, + ngx_uint_t *v); static ngx_uint_t ngx_http_image_filter_value(ngx_str_t *value); @@ -110,6 +133,8 @@ static char *ngx_http_image_filter_jpeg_quality(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); static char *ngx_http_image_filter_sharpen(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); +static char *ngx_http_image_filter_offset(ngx_conf_t *cf, ngx_command_t *cmd, + void *conf); static ngx_int_t ngx_http_image_filter_init(ngx_conf_t *cf); @@ -150,6 +175,13 @@ static ngx_command_t ngx_http_image_filter_commands[] = { offsetof(ngx_http_image_filter_conf_t, buffer_size), NULL }, + { ngx_string("image_filter_offset"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE2, + ngx_http_image_filter_offset, + NGX_HTTP_LOC_CONF_OFFSET, + 0, + NULL }, + ngx_null_command }; @@ -529,6 +561,22 @@ ngx_http_image_process(ngx_http_request_t *r) return NULL; } + if (conf->oxv == NULL) { + ctx->crop_offset_x = conf->crop_offset_x; + } else { + ngx_http_image_filter_crop_offset(r, conf->oxv, + NGX_HTTP_IMAGE_OFFSET_TYPE_HORIZONTAL, + &ctx->crop_offset_x); + } + + if (conf->oyv == NULL) { + ctx->crop_offset_y = conf->crop_offset_y; + } else { + ngx_http_image_filter_crop_offset(r, conf->oyv, + NGX_HTTP_IMAGE_OFFSET_TYPE_VERTICAL, + &ctx->crop_offset_y); + } + if (rc == NGX_OK && ctx->width <= ctx->max_width && ctx->height <= ctx->max_height @@ -932,8 +980,17 @@ transparent: return NULL; } - ox /= 2; - oy /= 2; + if (ctx->crop_offset_x == NGX_HTTP_IMAGE_OFFSET_LEFT) { + ox = 0; + } else if (ctx->crop_offset_x == NGX_HTTP_IMAGE_OFFSET_CENTER) { + ox /= 2; + } + + if (ctx->crop_offset_y == NGX_HTTP_IMAGE_OFFSET_TOP) { + oy = 0; + } else if (ctx->crop_offset_y == NGX_HTTP_IMAGE_OFFSET_CENTER) { + oy /= 2; + } ngx_log_debug4(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "image crop: %d x %d @ %d x %d", @@ -1139,6 +1196,52 @@ ngx_http_image_filter_get_value(ngx_http_request_t *r, return ngx_http_image_filter_value(&val); } +static void +ngx_http_image_filter_crop_offset(ngx_http_request_t *r, + ngx_http_complex_value_t *cv, ngx_uint_t t, ngx_uint_t *v) +{ + ngx_str_t value; + + if (ngx_http_complex_value(r, cv, &value) != NGX_OK) { + return; + } + + value.data[value.len] = 0; + + if (t == NGX_HTTP_IMAGE_OFFSET_TYPE_HORIZONTAL) { + ngx_http_image_filter_horizontal_crop_offset(&value, v); + } else if (t == NGX_HTTP_IMAGE_OFFSET_TYPE_VERTICAL) { + ngx_http_image_filter_vertical_crop_offset(&value, v); + } +} + + +static void +ngx_http_image_filter_horizontal_crop_offset(ngx_str_t *value, + ngx_uint_t *v) +{ + if (ngx_strcmp(value->data, "center") == 0) { + *v = NGX_HTTP_IMAGE_OFFSET_CENTER; + } else if (ngx_strcmp(value->data, "left") == 0) { + *v = NGX_HTTP_IMAGE_OFFSET_LEFT; + } else if (ngx_strcmp(value->data, "right") == 0) { + *v = NGX_HTTP_IMAGE_OFFSET_RIGHT; + } +} + +static void +ngx_http_image_filter_vertical_crop_offset(ngx_str_t *value, + ngx_uint_t *v) +{ + if (ngx_strcmp(value->data, "center") == 0) { + *v = NGX_HTTP_IMAGE_OFFSET_CENTER; + } else if (ngx_strcmp(value->data, "top") == 0) { + *v = NGX_HTTP_IMAGE_OFFSET_TOP; + } else if (ngx_strcmp(value->data, "bottom") == 0) { + *v = NGX_HTTP_IMAGE_OFFSET_BOTTOM; + } +} + static ngx_uint_t ngx_http_image_filter_value(ngx_str_t *value) @@ -1175,6 +1278,8 @@ ngx_http_image_filter_create_conf(ngx_conf_t *cf) conf->angle = NGX_CONF_UNSET_UINT; conf->transparency = NGX_CONF_UNSET; conf->buffer_size = NGX_CONF_UNSET_SIZE; + conf->crop_offset_x = NGX_CONF_UNSET_UINT; + conf->crop_offset_y = NGX_CONF_UNSET_UINT; return conf; } @@ -1223,6 +1328,17 @@ ngx_http_image_filter_merge_conf(ngx_conf_t *cf, void *parent, void *child) ngx_conf_merge_size_value(conf->buffer_size, prev->buffer_size, 1 * 1024 * 1024); + ngx_conf_merge_uint_value(conf->crop_offset_x, prev->crop_offset_x, NGX_HTTP_IMAGE_OFFSET_CENTER); + ngx_conf_merge_uint_value(conf->crop_offset_y, prev->crop_offset_y, NGX_HTTP_IMAGE_OFFSET_CENTER); + + if (conf->oxv == NULL) { + conf->oxv = prev->oxv; + } + + if (conf->oyv == NULL) { + conf->oyv = prev->oyv; + } + return NGX_CONF_OK; } @@ -1474,6 +1590,64 @@ ngx_http_image_filter_sharpen(ngx_conf_t *cf, ngx_command_t *cmd, } +static char * +ngx_http_image_filter_offset(ngx_conf_t *cf, ngx_command_t *cmd, + void *conf) +{ + ngx_http_image_filter_conf_t *imcf = conf; + + ngx_str_t *value; + ngx_http_complex_value_t cv; + ngx_http_compile_complex_value_t ccv; + + value = cf->args->elts; + + ngx_memzero(&ccv, sizeof(ngx_http_compile_complex_value_t)); + + ccv.cf = cf; + ccv.value = &value[1]; + ccv.complex_value = &cv; + + if (ngx_http_compile_complex_value(&ccv) != NGX_OK) { + return NGX_CONF_ERROR; + } + + if (cv.lengths == NULL) { + ngx_http_image_filter_horizontal_crop_offset(&value[1], &imcf->crop_offset_x); + } else { + imcf->oxv = ngx_palloc(cf->pool, sizeof(ngx_http_complex_value_t)); + if (imcf->oxv == NULL) { + return NGX_CONF_ERROR; + } + + *imcf->oxv = cv; + } + + ngx_memzero(&ccv, sizeof(ngx_http_compile_complex_value_t)); + + ccv.cf = cf; + ccv.value = &value[2]; + ccv.complex_value = &cv; + + if (ngx_http_compile_complex_value(&ccv) != NGX_OK) { + return NGX_CONF_ERROR; + } + + if (cv.lengths == NULL) { + ngx_http_image_filter_vertical_crop_offset(&value[2], &imcf->crop_offset_y); + } else { + imcf->oyv = ngx_palloc(cf->pool, sizeof(ngx_http_complex_value_t)); + if (imcf->oyv == NULL) { + return NGX_CONF_ERROR; + } + + *imcf->oyv = cv; + } + + return NGX_CONF_OK; +} + + static ngx_int_t ngx_http_image_filter_init(ngx_conf_t *cf) { On 15 November 2012 03:47, Maxim Dounin wrote: > offsets via variables. On the other hand, his patch asks for -- Regards, Ian Babrou http://bobrik.name http://twitter.com/ibobrik skype:i.babrou From mdounin at mdounin.ru Mon Nov 26 21:19:57 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Mon, 26 Nov 2012 21:19:57 +0000 Subject: [nginx] svn commit: r4939 - trunk/src/os/unix Message-ID: <20121126211957.F1AF93F9C13@mail.nginx.com> Author: mdounin Date: 2012-11-26 21:19:57 +0000 (Mon, 26 Nov 2012) New Revision: 4939 URL: http://trac.nginx.org/nginx/changeset/4939/nginx Log: Core: fixed ngx_write_chain_to_file() with IOV_MAX reached. Catched by dav_chunked.t on Solaris. In released versions this might potentially result in corruption of complex protocol responses if they were written to disk and there were more distinct buffers than IOV_MAX in a single write. Modified: trunk/src/os/unix/ngx_files.c Modified: trunk/src/os/unix/ngx_files.c =================================================================== --- trunk/src/os/unix/ngx_files.c 2012-11-26 18:01:49 UTC (rev 4938) +++ trunk/src/os/unix/ngx_files.c 2012-11-26 21:19:57 UTC (rev 4939) @@ -246,6 +246,7 @@ file->sys_offset += n; file->offset += n; + offset += n; total += n; } while (cl); From mdounin at mdounin.ru Mon Nov 26 21:30:45 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Mon, 26 Nov 2012 21:30:45 +0000 Subject: [nginx] svn commit: r4940 - trunk/src/http/modules Message-ID: <20121126213045.8053C3FA11B@mail.nginx.com> Author: mdounin Date: 2012-11-26 21:30:45 +0000 (Mon, 26 Nov 2012) New Revision: 4940 URL: http://trac.nginx.org/nginx/changeset/4940/nginx Log: Gunzip: added missing ngx_http_clear_etag(). Modified: trunk/src/http/modules/ngx_http_gunzip_filter_module.c Modified: trunk/src/http/modules/ngx_http_gunzip_filter_module.c =================================================================== --- trunk/src/http/modules/ngx_http_gunzip_filter_module.c 2012-11-26 21:19:57 UTC (rev 4939) +++ trunk/src/http/modules/ngx_http_gunzip_filter_module.c 2012-11-26 21:30:45 UTC (rev 4940) @@ -165,6 +165,7 @@ ngx_http_clear_content_length(r); ngx_http_clear_accept_ranges(r); + ngx_http_clear_etag(r); return ngx_http_next_header_filter(r); } From mdounin at mdounin.ru Tue Nov 27 13:55:35 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Tue, 27 Nov 2012 13:55:35 +0000 Subject: [nginx] svn commit: r4941 - trunk/docs/xml/nginx Message-ID: <20121127135535.5F42D3F9C11@mail.nginx.com> Author: mdounin Date: 2012-11-27 13:55:34 +0000 (Tue, 27 Nov 2012) New Revision: 4941 URL: http://trac.nginx.org/nginx/changeset/4941/nginx Log: nginx-1.3.9-RELEASE Modified: trunk/docs/xml/nginx/changes.xml Modified: trunk/docs/xml/nginx/changes.xml =================================================================== --- trunk/docs/xml/nginx/changes.xml 2012-11-26 21:30:45 UTC (rev 4940) +++ trunk/docs/xml/nginx/changes.xml 2012-11-27 13:55:34 UTC (rev 4941) @@ -5,6 +5,51 @@ + + + + +????????? chunked transfer encoding ??? ????????? ???? ???????. + + +support for chunked transfer encoding while reading client request body. + + + + + +?????????? $request_time ? $msec +?????? ????? ???????????? ?? ?????? ? ????????? log_format. + + +the $request_time and $msec variables +can now be used not only in the "log_format" directive. + + + + + +cache manager ? cache loader ????? ?? ???????????, +???? ?????????????? ????? 512 listen-???????. + + +cache manager and cache loader processes might not be able to start +if more than 512 listen sockets were used. + + + + + +? ?????? ngx_http_dav_module. + + +in the ngx_http_dav_module. + + + + + + From mdounin at mdounin.ru Tue Nov 27 13:55:54 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Tue, 27 Nov 2012 13:55:54 +0000 Subject: [nginx] svn commit: r4942 - tags Message-ID: <20121127135554.704AA3FA78D@mail.nginx.com> Author: mdounin Date: 2012-11-27 13:55:54 +0000 (Tue, 27 Nov 2012) New Revision: 4942 URL: http://trac.nginx.org/nginx/changeset/4942/nginx Log: release-1.3.9 tag Added: tags/release-1.3.9/ From mdounin at mdounin.ru Tue Nov 27 16:03:28 2012 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 27 Nov 2012 20:03:28 +0400 Subject: image_filter enhancement In-Reply-To: References: <20121106181601.GS40452@mdounin.ru> <20121114234732.GC40452@mdounin.ru> Message-ID: <20121127160328.GZ40452@mdounin.ru> Hello! On Tue, Nov 27, 2012 at 12:10:13AM +0400, ivan babrou wrote: > I tried to rework patch. Now it supports configuration through > variables. Variables ox and oy renamed to crop_offset_x and > crop_offset_y. > > I'm confused about this line: > + value.data[value.len] = 0; > > Probably I don't understand something about memory management in nginx. See below. > > > diff --git a/ngx_http_image_filter_module.c b/ngx_http_image_filter_module.c > index c853c33..15f8d17 100644 > --- a/ngx_http_image_filter_module.c > +++ b/ngx_http_image_filter_module.c > @@ -32,6 +32,14 @@ > #define NGX_HTTP_IMAGE_GIF 2 > #define NGX_HTTP_IMAGE_PNG 3 > > +#define NGX_HTTP_IMAGE_OFFSET_TYPE_HORIZONTAL 0 > +#define NGX_HTTP_IMAGE_OFFSET_TYPE_VERTICAL 1 > + > +#define NGX_HTTP_IMAGE_OFFSET_CENTER 0 > +#define NGX_HTTP_IMAGE_OFFSET_LEFT 1 > +#define NGX_HTTP_IMAGE_OFFSET_RIGHT 2 > +#define NGX_HTTP_IMAGE_OFFSET_TOP 3 > +#define NGX_HTTP_IMAGE_OFFSET_BOTTOM 4 > > #define NGX_HTTP_IMAGE_BUFFERED 0x08 > > @@ -43,11 +51,15 @@ typedef struct { > ngx_uint_t angle; > ngx_uint_t jpeg_quality; > ngx_uint_t sharpen; > + ngx_uint_t crop_offset_x; > + ngx_uint_t crop_offset_y; > > ngx_flag_t transparency; > > ngx_http_complex_value_t *wcv; > ngx_http_complex_value_t *hcv; > + ngx_http_complex_value_t *oxv; > + ngx_http_complex_value_t *oyv; Probably you mean "oxcv", "oycv" - "cv" here is abbreviation of "complex value". > ngx_http_complex_value_t *acv; > ngx_http_complex_value_t *jqcv; > ngx_http_complex_value_t *shcv; > @@ -66,6 +78,8 @@ typedef struct { > ngx_uint_t height; > ngx_uint_t max_width; > ngx_uint_t max_height; > + ngx_uint_t crop_offset_x; > + ngx_uint_t crop_offset_y; > ngx_uint_t angle; > > ngx_uint_t phase; > @@ -98,6 +112,15 @@ static u_char > *ngx_http_image_out(ngx_http_request_t *r, ngx_uint_t type, > static void ngx_http_image_cleanup(void *data); > static ngx_uint_t ngx_http_image_filter_get_value(ngx_http_request_t *r, > ngx_http_complex_value_t *cv, ngx_uint_t v); > +static void > +ngx_http_image_filter_crop_offset(ngx_http_request_t *r, > + ngx_http_complex_value_t *cv, ngx_uint_t t, ngx_uint_t *v); > +static void > +ngx_http_image_filter_vertical_crop_offset(ngx_str_t *value, > + ngx_uint_t *v); > +static void > +ngx_http_image_filter_horizontal_crop_offset(ngx_str_t *value, > + ngx_uint_t *v); > static ngx_uint_t ngx_http_image_filter_value(ngx_str_t *value); > > > @@ -110,6 +133,8 @@ static char > *ngx_http_image_filter_jpeg_quality(ngx_conf_t *cf, > ngx_command_t *cmd, void *conf); > static char *ngx_http_image_filter_sharpen(ngx_conf_t *cf, ngx_command_t *cmd, > void *conf); > +static char *ngx_http_image_filter_offset(ngx_conf_t *cf, ngx_command_t *cmd, > + void *conf); > static ngx_int_t ngx_http_image_filter_init(ngx_conf_t *cf); > > > @@ -150,6 +175,13 @@ static ngx_command_t ngx_http_image_filter_commands[] = { > offsetof(ngx_http_image_filter_conf_t, buffer_size), > NULL }, > > + { ngx_string("image_filter_offset"), > + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE2, > + ngx_http_image_filter_offset, > + NGX_HTTP_LOC_CONF_OFFSET, > + 0, > + NULL }, > + > ngx_null_command > }; > > @@ -529,6 +561,22 @@ ngx_http_image_process(ngx_http_request_t *r) > return NULL; > } > > + if (conf->oxv == NULL) { > + ctx->crop_offset_x = conf->crop_offset_x; > + } else { > + ngx_http_image_filter_crop_offset(r, conf->oxv, > + > NGX_HTTP_IMAGE_OFFSET_TYPE_HORIZONTAL, > + &ctx->crop_offset_x); > + } > + > + if (conf->oyv == NULL) { > + ctx->crop_offset_y = conf->crop_offset_y; > + } else { > + ngx_http_image_filter_crop_offset(r, conf->oyv, > + NGX_HTTP_IMAGE_OFFSET_TYPE_VERTICAL, > + &ctx->crop_offset_y); > + } > + 1) You want to follow pattern in the previous code, i.e. foo = ngx_http_image_filter_get_value(r, conf->foocv, conf->foo); (With additional support of top/bottom/left/right literals.) 2) I don't think you need to calculate offsets to decide whether image is ok as is or we need to resize it. Hence it's should be ok to move the code to where it's needed. 3) With (2) fixed you probably don't need variables in ctx anymore. > if (rc == NGX_OK > && ctx->width <= ctx->max_width > && ctx->height <= ctx->max_height > @@ -932,8 +980,17 @@ transparent: > return NULL; > } > > - ox /= 2; > - oy /= 2; > + if (ctx->crop_offset_x == NGX_HTTP_IMAGE_OFFSET_LEFT) { > + ox = 0; > + } else if (ctx->crop_offset_x == NGX_HTTP_IMAGE_OFFSET_CENTER) { > + ox /= 2; > + } > + > + if (ctx->crop_offset_y == NGX_HTTP_IMAGE_OFFSET_TOP) { > + oy = 0; > + } else if (ctx->crop_offset_y == NGX_HTTP_IMAGE_OFFSET_CENTER) { > + oy /= 2; > + } Note: this needs empty lines before "} else if ..." to match style. It should be changed anyway though, due to reasons outlined above. > > ngx_log_debug4(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, > "image crop: %d x %d @ %d x %d", > @@ -1139,6 +1196,52 @@ ngx_http_image_filter_get_value(ngx_http_request_t *r, > return ngx_http_image_filter_value(&val); > } > > +static void > +ngx_http_image_filter_crop_offset(ngx_http_request_t *r, > + ngx_http_complex_value_t *cv, ngx_uint_t t, ngx_uint_t *v) > +{ > + ngx_str_t value; > + > + if (ngx_http_complex_value(r, cv, &value) != NGX_OK) { > + return; > + } > + > + value.data[value.len] = 0; This is incorrect and will result in memory corruption. If you need null-terminated string, right aproach is to set ccv.zero flag while calling ngx_http_compile_complex_value(). On the other hand, it might be good idea to don't assume null-termination instead. > + > + if (t == NGX_HTTP_IMAGE_OFFSET_TYPE_HORIZONTAL) { > + ngx_http_image_filter_horizontal_crop_offset(&value, v); > + } else if (t == NGX_HTTP_IMAGE_OFFSET_TYPE_VERTICAL) { > + ngx_http_image_filter_vertical_crop_offset(&value, v); > + } > +} > + > + > +static void > +ngx_http_image_filter_horizontal_crop_offset(ngx_str_t *value, > + ngx_uint_t *v) > +{ > + if (ngx_strcmp(value->data, "center") == 0) { > + *v = NGX_HTTP_IMAGE_OFFSET_CENTER; > + } else if (ngx_strcmp(value->data, "left") == 0) { > + *v = NGX_HTTP_IMAGE_OFFSET_LEFT; > + } else if (ngx_strcmp(value->data, "right") == 0) { > + *v = NGX_HTTP_IMAGE_OFFSET_RIGHT; > + } > +} > + > +static void > +ngx_http_image_filter_vertical_crop_offset(ngx_str_t *value, > + ngx_uint_t *v) > +{ > + if (ngx_strcmp(value->data, "center") == 0) { > + *v = NGX_HTTP_IMAGE_OFFSET_CENTER; > + } else if (ngx_strcmp(value->data, "top") == 0) { > + *v = NGX_HTTP_IMAGE_OFFSET_TOP; > + } else if (ngx_strcmp(value->data, "bottom") == 0) { > + *v = NGX_HTTP_IMAGE_OFFSET_BOTTOM; > + } > +} > + I don't think we need so many functions to do this simple task. It would be good idea simplify this. The NGX_HTTP_IMAGE_OFFSET_TYPE_HORIZONTAL and NGX_HTTP_IMAGE_OFFSET_TYPE_VERTICAL seem to be unneeded, just a boolean value would do the trick. > static ngx_uint_t > ngx_http_image_filter_value(ngx_str_t *value) > @@ -1175,6 +1278,8 @@ ngx_http_image_filter_create_conf(ngx_conf_t *cf) > conf->angle = NGX_CONF_UNSET_UINT; > conf->transparency = NGX_CONF_UNSET; > conf->buffer_size = NGX_CONF_UNSET_SIZE; > + conf->crop_offset_x = NGX_CONF_UNSET_UINT; > + conf->crop_offset_y = NGX_CONF_UNSET_UINT; > > return conf; > } > @@ -1223,6 +1328,17 @@ ngx_http_image_filter_merge_conf(ngx_conf_t > *cf, void *parent, void *child) > ngx_conf_merge_size_value(conf->buffer_size, prev->buffer_size, > 1 * 1024 * 1024); > > + ngx_conf_merge_uint_value(conf->crop_offset_x, > prev->crop_offset_x, NGX_HTTP_IMAGE_OFFSET_CENTER); > + ngx_conf_merge_uint_value(conf->crop_offset_y, > prev->crop_offset_y, NGX_HTTP_IMAGE_OFFSET_CENTER); > + > + if (conf->oxv == NULL) { > + conf->oxv = prev->oxv; > + } > + > + if (conf->oyv == NULL) { > + conf->oyv = prev->oyv; > + } > + Style: please keep lines shorter than 80 chars. Additionally, looking at the code I tend to think it's incorrect. That is, it's pattern you've followed is incorrect, not your code. E.g. the following config will result in $arg_q incorrectly used for quality in /image/, instead of "80" explicitly set: image_filter_jpeg_quality $arg_q; location /image/ { image_filter crop $arg_w $arg_h; image_filter_jpeg_quality 80; } This needs fixing. > return NGX_CONF_OK; > } > > @@ -1474,6 +1590,64 @@ ngx_http_image_filter_sharpen(ngx_conf_t *cf, > ngx_command_t *cmd, > } > > > +static char * > +ngx_http_image_filter_offset(ngx_conf_t *cf, ngx_command_t *cmd, > + void *conf) > +{ > + ngx_http_image_filter_conf_t *imcf = conf; > + > + ngx_str_t *value; > + ngx_http_complex_value_t cv; > + ngx_http_compile_complex_value_t ccv; > + > + value = cf->args->elts; > + > + ngx_memzero(&ccv, sizeof(ngx_http_compile_complex_value_t)); > + > + ccv.cf = cf; > + ccv.value = &value[1]; > + ccv.complex_value = &cv; > + > + if (ngx_http_compile_complex_value(&ccv) != NGX_OK) { > + return NGX_CONF_ERROR; > + } > + > + if (cv.lengths == NULL) { > + ngx_http_image_filter_horizontal_crop_offset(&value[1], > &imcf->crop_offset_x); > + } else { > + imcf->oxv = ngx_palloc(cf->pool, sizeof(ngx_http_complex_value_t)); > + if (imcf->oxv == NULL) { > + return NGX_CONF_ERROR; > + } > + > + *imcf->oxv = cv; > + } > + > + ngx_memzero(&ccv, sizeof(ngx_http_compile_complex_value_t)); > + > + ccv.cf = cf; > + ccv.value = &value[2]; > + ccv.complex_value = &cv; > + > + if (ngx_http_compile_complex_value(&ccv) != NGX_OK) { > + return NGX_CONF_ERROR; > + } > + > + if (cv.lengths == NULL) { > + ngx_http_image_filter_vertical_crop_offset(&value[2], > &imcf->crop_offset_y); > + } else { > + imcf->oyv = ngx_palloc(cf->pool, sizeof(ngx_http_complex_value_t)); > + if (imcf->oyv == NULL) { > + return NGX_CONF_ERROR; > + } > + > + *imcf->oyv = cv; > + } > + > + return NGX_CONF_OK; > +} > + > + > static ngx_int_t > ngx_http_image_filter_init(ngx_conf_t *cf) > { > > > On 15 November 2012 03:47, Maxim Dounin wrote: > > offsets via variables. On the other hand, his patch asks for > > > > -- > Regards, Ian Babrou > http://bobrik.name http://twitter.com/ibobrik skype:i.babrou > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel -- Maxim Dounin http://nginx.com/support.html From mdounin at mdounin.ru Thu Nov 29 23:13:18 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Thu, 29 Nov 2012 23:13:18 +0000 Subject: [nginx] svn commit: r4943 - in trunk/src: core http/modules/perl Message-ID: <20121129231319.731373FAA47@mail.nginx.com> Author: mdounin Date: 2012-11-29 23:13:18 +0000 (Thu, 29 Nov 2012) New Revision: 4943 URL: http://trac.nginx.org/nginx/changeset/4943/nginx Log: Version bump. Modified: trunk/src/core/nginx.h trunk/src/http/modules/perl/nginx.pm Modified: trunk/src/core/nginx.h =================================================================== --- trunk/src/core/nginx.h 2012-11-27 13:55:54 UTC (rev 4942) +++ trunk/src/core/nginx.h 2012-11-29 23:13:18 UTC (rev 4943) @@ -9,8 +9,8 @@ #define _NGINX_H_INCLUDED_ -#define nginx_version 1003009 -#define NGINX_VERSION "1.3.9" +#define nginx_version 1003010 +#define NGINX_VERSION "1.3.10" #define NGINX_VER "nginx/" NGINX_VERSION #define NGINX_VAR "NGINX" Modified: trunk/src/http/modules/perl/nginx.pm =================================================================== --- trunk/src/http/modules/perl/nginx.pm 2012-11-27 13:55:54 UTC (rev 4942) +++ trunk/src/http/modules/perl/nginx.pm 2012-11-29 23:13:18 UTC (rev 4943) @@ -50,7 +50,7 @@ HTTP_INSUFFICIENT_STORAGE ); -our $VERSION = '1.3.9'; +our $VERSION = '1.3.10'; require XSLoader; XSLoader::load('nginx', $VERSION); From mdounin at mdounin.ru Thu Nov 29 23:15:41 2012 From: mdounin at mdounin.ru (mdounin at mdounin.ru) Date: Thu, 29 Nov 2012 23:15:41 +0000 Subject: [nginx] svn commit: r4944 - trunk/src/os/unix Message-ID: <20121129231541.DABDE3FAA47@mail.nginx.com> Author: mdounin Date: 2012-11-29 23:15:41 +0000 (Thu, 29 Nov 2012) New Revision: 4944 URL: http://trac.nginx.org/nginx/changeset/4944/nginx Log: Core: removed GLOB_NOSORT glob option. This will result in alphabetical sorting of included files if the "include" directive with wildcards is used. Note that the behaviour is now different from that on Windows, where alphabetical sorting is not guaranteed for FindFirsFile()/FindNextFile() (used to be alphabetical on NTFS, but not on FAT). Approved by Igor Sysoev, prodded by many. Modified: trunk/src/os/unix/ngx_files.c Modified: trunk/src/os/unix/ngx_files.c =================================================================== --- trunk/src/os/unix/ngx_files.c 2012-11-29 23:13:18 UTC (rev 4943) +++ trunk/src/os/unix/ngx_files.c 2012-11-29 23:15:41 UTC (rev 4944) @@ -363,7 +363,7 @@ { int n; - n = glob((char *) gl->pattern, GLOB_NOSORT, NULL, &gl->pglob); + n = glob((char *) gl->pattern, 0, NULL, &gl->pglob); if (n == 0) { return NGX_OK; From ru at nginx.com Fri Nov 30 11:26:50 2012 From: ru at nginx.com (ru at nginx.com) Date: Fri, 30 Nov 2012 11:26:50 +0000 Subject: [nginx] svn commit: r4945 - trunk/src/core Message-ID: <20121130112650.E84123FA105@mail.nginx.com> Author: ru Date: 2012-11-30 11:26:50 +0000 (Fri, 30 Nov 2012) New Revision: 4945 URL: http://trac.nginx.org/nginx/changeset/4945/nginx Log: Fixed the NGX_SOCKADDR_STRLEN macro definition. The ngx_sock_ntop() function, when told to print both address and port, prints IPv6 address in square brackets, followed by colon and port. Modified: trunk/src/core/ngx_inet.h Modified: trunk/src/core/ngx_inet.h =================================================================== --- trunk/src/core/ngx_inet.h 2012-11-29 23:15:41 UTC (rev 4944) +++ trunk/src/core/ngx_inet.h 2012-11-30 11:26:50 UTC (rev 4945) @@ -30,7 +30,7 @@ #if (NGX_HAVE_UNIX_DOMAIN) #define NGX_SOCKADDR_STRLEN (sizeof("unix:") - 1 + NGX_UNIX_ADDRSTRLEN) #else -#define NGX_SOCKADDR_STRLEN (NGX_INET6_ADDRSTRLEN + sizeof(":65535") - 1) +#define NGX_SOCKADDR_STRLEN (NGX_INET6_ADDRSTRLEN + sizeof("[]:65535") - 1) #endif #if (NGX_HAVE_UNIX_DOMAIN)