From info at kliemeck.de Thu May 1 11:42:51 2014 From: info at kliemeck.de (info at kliemeck.de) Date: Thu, 01 May 2014 13:42:51 +0200 Subject: [PATCH] Added nonlocal to the listen directive In-Reply-To: <20140427221012.Horde.pUFTrYPMwbaBfz1QpJpkjA1@webmail.your-server.de> References: <16eacd8609c8362e9dd7.1395999953@miitool> <20140328122528.GL34696@mdounin.ru> <20140427221012.Horde.pUFTrYPMwbaBfz1QpJpkjA1@webmail.your-server.de> Message-ID: <20140501134251.Horde.ANn5O3mbOALn1gKPafFulg1@webmail.your-server.de> Hey, i thought that this is important but I have received no response. Any update on this? greets Hans-Joachim Quoting info at kliemeck.de: > Hey, > > but it is still not possible to work with IPv6, if you want to bind > to a specific address (not [::]) that is not a local address. The > "ip_nonlocal_bind-sysctl" use-case is not fulfilled with this and i > think it is a common use-case that nginx is used within a high > availability environment with a shared ip address. It is possible > that this important feature is integrated within 1.6, since it may > be a reason not to use IPv6? > > greets > Hans-Joachim Kliemeck > > Quoting mdounin at mdounin.ru: > >> Hello! >> >> On Fri, Mar 28, 2014 at 10:45:53AM +0100, Trygve Vea wrote: >> >>> # HG changeset patch >>> # User Trygve Vea >>> # Date 1395999940 -3600 >>> # Fri Mar 28 10:45:40 2014 +0100 >>> # Node ID 16eacd8609c8362e9dd729c743ed7a869c2993fe >>> # Parent 2411d4b5be2ca690a5a00a1d8ad96ff69a00317f >>> Added nonlocal to the listen directive >>> >>> The nonlocal option is used to set the needed socket options to be >>> able to bind >>> to an address not necessarily owned by the host. >>> >>> This patch currently implements this for Linux >= 2.4 IPv4/IPv6. >>> >>> The problem we solve by doing this, is in an environment where the >>> following >>> conditions are met: >>> >>> * HTTPS with multiple certificates, and a client base that are >>> unable to use >>> SNI - thus having the need to tie specific certificates to >>> specific ip/ports. >>> * Setting the ip_nonlocal_bind-sysctl is not an option (for >>> example for Linux >>> IPv6) >>> * Used in a failover-setup, where the service IP-addresses are >>> moved around by >>> a daemon like linux-ha or keepalived. >> >> As already explained, the patch is not needed for the use case >> claimed. Just a bind on INADDR_ANY/IN6ADDR_ANY will do the trick. >> >> -- >> Maxim Dounin >> http://nginx.org/ > > > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel From mp+nginx at hezmatt.org Sun May 4 05:44:47 2014 From: mp+nginx at hezmatt.org (Matt Palmer) Date: Sun, 4 May 2014 15:44:47 +1000 Subject: [patch] Properly terminate line-endings in $ssl_client_cert Message-ID: <20140504054442.GC24230@hezmatt.org> The below patch is a small one, to make the common use-case for $ssl_client_cert (including it in an HTTP request header) protocol-compliant. Some receiving webservers don't like a plain '\n' in the requests they receive. I considered digging deeper to find a more "natural" place to ensure protocol compliance, but then I figured that since we're *already* mangling the "native" look of the PEM data (adding leading tabs), adding some '\r' wasn't a huge further leap. --- nginx-1.6.0.orig/src/event/ngx_event_openssl.c +++ nginx-1.6.0/src/event/ngx_event_openssl.c @@ -2615,7 +2615,7 @@ ngx_ssl_get_certificate(ngx_connection_t for (i = 0; i < cert.len - 1; i++) { if (cert.data[i] == LF) { - len++; + len += 2; } } @@ -2628,9 +2628,12 @@ ngx_ssl_get_certificate(ngx_connection_t p = s->data; for (i = 0; i < cert.len - 1; i++) { - *p++ = cert.data[i]; if (cert.data[i] == LF) { + *p++ = '\r'; + *p++ = '\n'; *p++ = '\t'; + } else { + *p++ = cert.data[i]; } } - Matt -- Judging by this particular thread, many people in this group spent their school years taking illogical, pointless orders from morons and having their will to live systematically crushed. And people say school doesn't prepare kids for the real world. -- Rayner, in the Monastery From s.martin49 at gmail.com Sun May 4 06:08:29 2014 From: s.martin49 at gmail.com (Samuel Martin) Date: Sun, 04 May 2014 08:08:29 +0200 Subject: [PATCH 1 of 2] auto/type/sizeof: rework autotest to be cross-compilation friendly In-Reply-To: References: Message-ID: # HG changeset patch # User Samuel Martin # Date 1398374852 -7200 # Thu Apr 24 23:27:32 2014 +0200 # Node ID f0f3f5208b12c6afbbdfb55b32c968ebd7206ad5 # Parent b9553b4b8e670a0231afc0484f23953c0d8b5f22 auto/type/sizeof: rework autotest to be cross-compilation friendly Rework the sizeof test to do the checks at compile time instead of at runtime. This way, it does not break when cross-compiling for a different CPU architecture. diff -r b9553b4b8e67 -r f0f3f5208b12 auto/types/sizeof --- a/auto/types/sizeof Tue Apr 29 22:22:38 2014 +0200 +++ b/auto/types/sizeof Thu Apr 24 23:27:32 2014 +0200 @@ -14,7 +14,7 @@ ngx_size= -cat << END > $NGX_AUTOTEST.c +cat << _EOF > $NGX_AUTOTEST.c #include #include @@ -25,25 +25,52 @@ $NGX_INCLUDE_INTTYPES_H $NGX_INCLUDE_AUTO_CONFIG_H -int main() { - printf("%d", (int) sizeof($ngx_type)); +#if !defined( PASTE) +#define PASTE2( x, y) x##y +#define PASTE( x, y) PASTE2( x, y) +#endif /* PASTE */ + +#define SAY_IF_SIZEOF( typename, type, size) \\ + static char PASTE( PASTE( PASTE( sizeof_, typename), _is_), size) \\ + [(sizeof(type) == (size)) ? -1 : 1] +#define SAY_SIZEOF_END( typename) \\ + static char PASTE( end_search_for_sizeof_, typename)[-1] + +#define SAY_SIZEOF( typename, type) \\ + SAY_IF_SIZEOF( typename, type, 1); \\ + SAY_IF_SIZEOF( typename, type, 2); \\ + SAY_IF_SIZEOF( typename, type, 3); \\ + SAY_IF_SIZEOF( typename, type, 4); \\ + SAY_IF_SIZEOF( typename, type, 5); \\ + SAY_IF_SIZEOF( typename, type, 6); \\ + SAY_IF_SIZEOF( typename, type, 7); \\ + SAY_IF_SIZEOF( typename, type, 8); \\ + SAY_IF_SIZEOF( typename, type, 9); \\ + SAY_IF_SIZEOF( typename, type, 10); \\ + SAY_IF_SIZEOF( typename, type, 11); \\ + SAY_IF_SIZEOF( typename, type, 12); \\ + SAY_IF_SIZEOF( typename, type, 13); \\ + SAY_IF_SIZEOF( typename, type, 14); \\ + SAY_IF_SIZEOF( typename, type, 15); \\ + SAY_IF_SIZEOF( typename, type, 16); \\ + SAY_SIZEOF_END( typename) + +SAY_SIZEOF(TEST_TYPENAME, TEST_TYPE) + +int main(void) +{ return 0; } -END +_EOF - -ngx_test="$CC $CC_TEST_FLAGS $CC_AUX_FLAGS \ - -o $NGX_AUTOTEST $NGX_AUTOTEST.c $NGX_LD_OPT $ngx_feature_libs" - -eval "$ngx_test >> $NGX_AUTOCONF_ERR 2>&1" - - -if [ -x $NGX_AUTOTEST ]; then - ngx_size=`$NGX_AUTOTEST` - echo " $ngx_size bytes" -fi - +_ngx_typename=`echo "$ngx_type" | sed 's/ /_/g;s/\*/p/'` +ngx_size=`$CC $CC_TEST_FLAGS $CC_AUX_FLAGS \ + -DTEST_TYPENAME="$_ngx_typename" -DTEST_TYPE="$ngx_type" \ + $NGX_AUTOTEST.c \ + $NGX_LD_OPT $ngx_feature_libs 2>&1 | \ + sed -nr "/.*sizeof_${_ngx_typename}_is_([0-9]+).*/ s//\1/p" || \ + true` rm -rf $NGX_AUTOTEST* From s.martin49 at gmail.com Sun May 4 06:08:30 2014 From: s.martin49 at gmail.com (Samuel Martin) Date: Sun, 04 May 2014 08:08:30 +0200 Subject: [PATCH 2 of 2] auto/feature: disable feature autotest when cross-compiling In-Reply-To: References: Message-ID: <03617a96c528d1f02abf.1399183710@bobook.home> # HG changeset patch # User Samuel Martin # Date 1399156849 -7200 # Sun May 04 00:40:49 2014 +0200 # Node ID 03617a96c528d1f02abf829df964fcb29b06c942 # Parent f0f3f5208b12c6afbbdfb55b32c968ebd7206ad5 auto/feature: disable feature autotest when cross-compiling This patch introduce a new NGX_CROSSBUILD configure variable that records whether --crossbuild=... option was passed on the command line. Then this variable is used to disable the feature runtest because they cannot usually be executed when cross-compiling. diff -r f0f3f5208b12 -r 03617a96c528 auto/configure --- a/auto/configure Thu Apr 24 23:27:32 2014 +0200 +++ b/auto/configure Sun May 04 00:40:49 2014 +0200 @@ -44,6 +44,7 @@ else echo "building for $NGX_PLATFORM" NGX_SYSTEM=$NGX_PLATFORM + NGX_CROSSBUILD=crossbuild fi . auto/cc/conf diff -r f0f3f5208b12 -r 03617a96c528 auto/feature --- a/auto/feature Thu Apr 24 23:27:32 2014 +0200 +++ b/auto/feature Sun May 04 00:40:49 2014 +0200 @@ -49,7 +49,7 @@ if [ -x $NGX_AUTOTEST ]; then - case "$ngx_feature_run" in + case "$NGX_CROSSBUILD$ngx_feature_run" in yes) # /bin/sh is used to intercept "Killed" or "Abort trap" messages diff -r f0f3f5208b12 -r 03617a96c528 auto/options --- a/auto/options Thu Apr 24 23:27:32 2014 +0200 +++ b/auto/options Sun May 04 00:40:49 2014 +0200 @@ -32,6 +32,7 @@ NGX_TEST_BUILD_RTSIG=NO NGX_TEST_BUILD_SOLARIS_SENDFILEV=NO +NGX_CROSSBUILD= NGX_PLATFORM= NGX_WINE= From s.martin49 at gmail.com Sun May 4 06:08:28 2014 From: s.martin49 at gmail.com (Samuel Martin) Date: Sun, 04 May 2014 08:08:28 +0200 Subject: [PATCH 0 of 2] Cross-compilation support improvement Message-ID: Hi, Here is a short series improving nginx build-system support for cross-compilation. This series tries to be as less intrusive as possible, and intends to make easier integration in cross-compilataion frameworks such as Buildroot. Yours, Samuel From s.martin49 at gmail.com Sun May 4 19:33:55 2014 From: s.martin49 at gmail.com (Samuel Martin) Date: Sun, 04 May 2014 21:33:55 +0200 Subject: [PATCH 3 of 3] auto/lib/*: prepend include location with sysroot path Message-ID: # HG changeset patch # User Samuel Martin # Date 1399231473 -7200 # Sun May 04 21:24:33 2014 +0200 # Node ID b9787fdc8bac3559fd9da679dc04e8fc50cb74f3 # Parent 03617a96c528d1f02abf829df964fcb29b06c942 auto/lib/*: prepend include location with sysroot path This patch introduce a new NGX_SYSROOT variable used when cross-compiling with gcc. When cross-compiling nginx, the gcc-based cross-compiler will automatically set the NGX_SYSROOT value to the right value. This variable remains empty if not cross-compiling or no gcc compiler is detected. This NGX_SYSROOT is prepended to the ngx_feature_path content when additional location should be added to the include directory list. This is necessary to prevent gcc from complaining about unsafe include location for cross-compilation (-Wpoision-system-directories), currently only triggered by libxslt. diff -r 03617a96c528 -r b9787fdc8bac auto/cc/gcc --- a/auto/cc/gcc Sun May 04 00:40:49 2014 +0200 +++ b/auto/cc/gcc Sun May 04 21:24:33 2014 +0200 @@ -15,6 +15,10 @@ have=NGX_COMPILER value="\"gcc $NGX_GCC_VER\"" . auto/define +# set sysroot in case of cross-compilation +if [ x$NGX_CROSSBUILD = xcrossbuild ]; then + NGX_SYSROOT=`$CC -print-sysroot` +fi # Solaris 7's /usr/ccs/bin/as does not support "-pipe" diff -r 03617a96c528 -r b9787fdc8bac auto/lib/libxslt/conf --- a/auto/lib/libxslt/conf Sun May 04 00:40:49 2014 +0200 +++ b/auto/lib/libxslt/conf Sun May 04 21:24:33 2014 +0200 @@ -12,7 +12,7 @@ #include #include #include " - ngx_feature_path="/usr/include/libxml2" + ngx_feature_path="$NGX_SYSROOT/usr/include/libxml2" ngx_feature_libs="-lxml2 -lxslt" ngx_feature_test="xmlParserCtxtPtr ctxt = NULL; xsltStylesheetPtr sheet = NULL; diff -r 03617a96c528 -r b9787fdc8bac auto/options --- a/auto/options Sun May 04 00:40:49 2014 +0200 +++ b/auto/options Sun May 04 21:24:33 2014 +0200 @@ -33,6 +33,7 @@ NGX_TEST_BUILD_SOLARIS_SENDFILEV=NO NGX_CROSSBUILD= +NGX_SYSROOT= NGX_PLATFORM= NGX_WINE= From s.martin49 at gmail.com Sun May 4 19:35:54 2014 From: s.martin49 at gmail.com (Samuel Martin) Date: Sun, 4 May 2014 21:35:54 +0200 Subject: [PATCH 0 of 2] Cross-compilation support improvement In-Reply-To: References: Message-ID: Hi all, On Sun, May 4, 2014 at 8:08 AM, Samuel Martin wrote: > Hi, > > > Here is a short series improving nginx build-system support for > cross-compilation. > > This series tries to be as less intrusive as possible, and intends to > make easier integration in cross-compilataion frameworks such as > Buildroot. Just sent the 3rd and last patch of the series. Regards, -- Samuel From ru at nginx.com Mon May 5 09:56:37 2014 From: ru at nginx.com (Ruslan Ermilov) Date: Mon, 5 May 2014 13:56:37 +0400 Subject: [patch] Properly terminate line-endings in $ssl_client_cert In-Reply-To: <20140504054442.GC24230@hezmatt.org> References: <20140504054442.GC24230@hezmatt.org> Message-ID: <20140505095637.GB83088@lo0.su> On Sun, May 04, 2014 at 03:44:47PM +1000, Matt Palmer wrote: > The below patch is a small one, to make the common use-case for > $ssl_client_cert (including it in an HTTP request header) > protocol-compliant. Some receiving webservers don't like a plain '\n' in > the requests they receive. > > I considered digging deeper to find a more "natural" place to ensure > protocol compliance, but then I figured that since we're *already* mangling > the "native" look of the PEM data (adding leading tabs), adding some '\r' > wasn't a huge further leap. > > --- nginx-1.6.0.orig/src/event/ngx_event_openssl.c > +++ nginx-1.6.0/src/event/ngx_event_openssl.c > @@ -2615,7 +2615,7 @@ ngx_ssl_get_certificate(ngx_connection_t > > for (i = 0; i < cert.len - 1; i++) { > if (cert.data[i] == LF) { > - len++; > + len += 2; > } > } > > @@ -2628,9 +2628,12 @@ ngx_ssl_get_certificate(ngx_connection_t > p = s->data; > > for (i = 0; i < cert.len - 1; i++) { > - *p++ = cert.data[i]; > if (cert.data[i] == LF) { > + *p++ = '\r'; > + *p++ = '\n'; > *p++ = '\t'; > + } else { > + *p++ = cert.data[i]; > } > } > Better use macros CR and LF instead of '\r' and '\n'. Otherwise, your patch looks good to me. Please also see: http://nginx.org/en/docs/contributing_changes.html From gooogle.liest.mit at googlemail.com Mon May 5 12:21:15 2014 From: gooogle.liest.mit at googlemail.com (Frederik Schwan) Date: Mon, 05 May 2014 14:21:15 +0200 Subject: [PATCH] Add RFC 2560 "non-delegated" model where issuer is OCSP signee Message-ID: <5367823B.6010202@googlemail.com> # HG changeset patch # User Frederik Schwan # Date 1399288827 -7200 # Mon May 05 13:20:27 2014 +0200 # Node ID a1f71e35b4a851e1a2c8756f885683518665c528 # Parent 48c97d83ab7f0a3f641987fb32ace8af7720aefc Add RFC 2560 "non-delegated" model where issuer is OCSP signee diff -r 48c97d83ab7f -r a1f71e35b4a8 src/event/ngx_event_openssl_stapling.c --- a/src/event/ngx_event_openssl_stapling.c Tue Apr 29 22:22:38 2014 +0200 +++ b/src/event/ngx_event_openssl_stapling.c Mon May 05 13:20:27 2014 +0200 @@ -529,11 +529,11 @@ const #endif u_char *p; - int n; + int n, i; size_t len; ngx_str_t response; X509_STORE *store; - STACK_OF(X509) *chain; + STACK_OF(X509) *chain, *issuer_st; OCSP_CERTID *id; OCSP_RESPONSE *ocsp; OCSP_BASICRESP *basic; @@ -589,13 +589,19 @@ #else chain = staple->ssl_ctx->extra_certs; #endif + issuer_st = sk_X509_new_null(); + sk_X509_push(issuer_st, staple->issuer); + i = OCSP_basic_verify(basic, chain, store, + staple->verify ? OCSP_TRUSTOTHER : OCSP_NOVERIFY); + + if (i <= 0 && issuer_st){ + i = OCSP_basic_verify(basic, issuer_st, store,OCSP_TRUSTOTHER); + if (i > 0) ERR_clear_error(); + } - if (OCSP_basic_verify(basic, chain, store, - staple->verify ? OCSP_TRUSTOTHER : OCSP_NOVERIFY) - != 1) - { + if(i <= 0){ ngx_ssl_error(NGX_LOG_ERR, ctx->log, 0, - "OCSP_basic_verify() failed"); + "OCSP_basic_verify() failed"); goto error; } -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 341 bytes Desc: OpenPGP digital signature URL: From mp+nginx at hezmatt.org Mon May 5 23:35:26 2014 From: mp+nginx at hezmatt.org (Matt Palmer) Date: Tue, 6 May 2014 09:35:26 +1000 Subject: [patch, take 2] Properly terminate line-endings in $ssl_client_cert In-Reply-To: <20140505095637.GB83088@lo0.su> References: <20140504054442.GC24230@hezmatt.org> <20140505095637.GB83088@lo0.su> Message-ID: <20140505233521.GL24230@hezmatt.org> On Mon, May 05, 2014 at 01:56:37PM +0400, Ruslan Ermilov wrote: > On Sun, May 04, 2014 at 03:44:47PM +1000, Matt Palmer wrote: > > The below patch is a small one, to make the common use-case for > > $ssl_client_cert (including it in an HTTP request header) > > protocol-compliant. Some receiving webservers don't like a plain '\n' in > > the requests they receive. > > > > I considered digging deeper to find a more "natural" place to ensure > > protocol compliance, but then I figured that since we're *already* mangling > > the "native" look of the PEM data (adding leading tabs), adding some '\r' > > wasn't a huge further leap. > > > > Better use macros CR and LF instead of '\r' and '\n'. > Otherwise, your patch looks good to me. OK then, take 2: -----8<----- Properly escape $ssl_client_cert for HTTP protocol compliance The common use-case for $ssl_client_cert (including it in an HTTP request header) means that each line should be terminated with CRLF, rather than just a plain LF. Some receiving webservers really don't like a plain LF to terminate lines. --- a/src/event/ngx_event_openssl.c +++ b/src/event/ngx_event_openssl.c @@ -2615,7 +2615,7 @@ ngx_ssl_get_certificate(ngx_connection_t for (i = 0; i < cert.len - 1; i++) { if (cert.data[i] == LF) { - len++; + len += 2; } } @@ -2628,9 +2628,12 @@ ngx_ssl_get_certificate(ngx_connection_t p = s->data; for (i = 0; i < cert.len - 1; i++) { - *p++ = cert.data[i]; if (cert.data[i] == LF) { + *p++ = CR; + *p++ = LF; *p++ = '\t'; + } else { + *p++ = cert.data[i]; } } - Matt -- Sure, it's possible to write C in an object-oriented way. But, in practice, getting an entire team to do that is like telling them to walk along a straight line painted on the floor, with the lights off. -- Tess Snider, slug-chat at slug.org.au From thierry.magnien at sfr.com Wed May 7 09:16:55 2014 From: thierry.magnien at sfr.com (MAGNIEN, Thierry) Date: Wed, 7 May 2014 09:16:55 +0000 Subject: Does ngx_pool_t survives reload ? Message-ID: <5D103CE839D50E4CBC62C9FD7B83287C8FB781AE@EXCN015.encara.local.ads> Hi, I wrote a Nginx module and am facing a problem: I need to keep some data across reloads, so I'm using a shared memory segment and everything works fine except one thing. In the data structure I keep, there are ngx_regex_compile_t data, which were allocated on a ngx_pool_t. And what I see is that all the data are kept, except regex code, which was allocated on this ngx_pool_t. Are pools resetted on a reload and is there a way: - either to allocate regex codes on something else than a pool - or use a pool that would not be resetted ? Thanks for your help, Thierry From wandenberg at gmail.com Wed May 7 12:52:47 2014 From: wandenberg at gmail.com (Wandenberg Peixoto) Date: Wed, 7 May 2014 09:52:47 -0300 Subject: Does ngx_pool_t survives reload ? In-Reply-To: <5D103CE839D50E4CBC62C9FD7B83287C8FB781AE@EXCN015.encara.local.ads> References: <5D103CE839D50E4CBC62C9FD7B83287C8FB781AE@EXCN015.encara.local.ads> Message-ID: As far as I know, ngx_pool_t are created on process memory area, so, after a reload the new process has a new memory area and do not have access to the old. You can recompile the regex on your module startup, at init worker hook. On Wed, May 7, 2014 at 6:16 AM, MAGNIEN, Thierry wrote: > Hi, > > I wrote a Nginx module and am facing a problem: I need to keep some data > across reloads, so I'm using a shared memory segment and everything works > fine except one thing. > > In the data structure I keep, there are ngx_regex_compile_t data, which > were allocated on a ngx_pool_t. And what I see is that all the data are > kept, except regex code, which was allocated on this ngx_pool_t. > > Are pools resetted on a reload and is there a way: > - either to allocate regex codes on something else than a pool > - or use a pool that would not be resetted ? > > Thanks for your help, > Thierry > > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shuxinyang.oss at gmail.com Wed May 7 19:42:39 2014 From: shuxinyang.oss at gmail.com (Shuxin Yang) Date: Wed, 07 May 2014 12:42:39 -0700 Subject: Fwd: Question about slab allocator In-Reply-To: <536A8B98.5030500@gmail.com> References: <536A8B98.5030500@gmail.com> Message-ID: <536A8CAF.3070806@gmail.com> Sorry for sending my previous mail to wrong list. My co-worker told me this list is the right place to be. Again, profuse thanks in advance! Shuxin -------- Original Message -------- Subject: Question about slab allocator Date: Wed, 07 May 2014 12:38:00 -0700 From: Shuxin Yang To: nginx at nginx.org Hi, I'm nginx newbie. I'm reading src/core/ngx_slab.c, and am confused as to the purpose of NGX_SLAB_PAGE_START. As far as I can understand, when allocating a block, the most significant bit (MSB) of first page's corresponding ngx_slab_page_s::slab is set "1". as we can see from : cat -n ngx_slab.c at pristine-1.7-release 644 page->slab = pages | NGX_SLAB_PAGE_START; However, the MSB of is cleared later on: 362 } else if (shift == ngx_slab_exact_shift) { 363 364 page->slab = 1; So, what is the purpose of NGX_SLAB_PAGE_START (i.e the MSB of the field slab)? If we really meant to keep the MSB, I guess there is another bug over here: the condition at line 514 is always true, and hence we never get chance to free the empty page. 512 page->slab &= ~m; 513 514 if (page->slab) { 515 goto done; 516 } 517 518 ngx_slab_free_pages(pool, page, 1); Thanks Shuxin -------------- next part -------------- An HTML attachment was scrubbed... URL: From renenglish at gmail.com Thu May 8 02:47:18 2014 From: renenglish at gmail.com (Shafreeck Sea) Date: Thu, 8 May 2014 10:47:18 +0800 Subject: How to submit a third pary module ? Message-ID: Hi all: I developed a nginx module and I am glad to share with others . How to submit this module to wiki.nginx.org ? I tried to create an account at wiki.nginx.org but failed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From codeeply at gmail.com Thu May 8 07:52:24 2014 From: codeeply at gmail.com (Jianjun Zheng) Date: Thu, 8 May 2014 15:52:24 +0800 Subject: [PATCH] Upstream: add consistent hash module Message-ID: usage example: cons_hash $request_uri; Not only consistent hash is introduced as a load balancing, but also increase the hits of cache in upstream servers. This modules is written according to round_robin, ip_hash and least_conn modules for upstream. So it inherits all the essential logic from them. This implementation is simple but has a high performance, The time comlexity remains O(log(T)) with a low constant coefficient, even when half of servers get down, regardless of the weight of servers, as analysis below. Suppose there are N upstream servers, with D down servers, and the weights are W1,W2,...,W(N-1). V is the amount of virtual nodes per unit. P is the probability to get an alive server from all nodes at random. Let T = (W1+W2+...+W(N-1))*V. then the space complexity is O(T), and the time comlexity is O(log(T) + (1-P)*N/(N-D)), 'N/(N-D)' of which is independent of W and V. # HG changeset patch # User Jianjun Zheng # Date 1399531525 -28800 # Thu May 08 14:45:25 2014 +0800 # Node ID 063f37f1654ef6cc03bd311a7fe6189b299ce2f2 # Parent 48c97d83ab7f0a3f641987fb32ace8af7720aefc Upstream: add consistent hash module diff -r 48c97d83ab7f -r 063f37f1654e auto/modules --- a/auto/modules Tue Apr 29 22:22:38 2014 +0200 +++ b/auto/modules Thu May 08 14:45:25 2014 +0800 @@ -381,6 +381,11 @@ HTTP_SRCS="$HTTP_SRCS $HTTP_UPSTREAM_LEAST_CONN_SRCS" fi +if [ $HTTP_UPSTREAM_CONS_HASH = YES ]; then + HTTP_MODULES="$HTTP_MODULES $HTTP_UPSTREAM_CONS_HASH_MODULE" + HTTP_SRCS="$HTTP_SRCS $HTTP_UPSTREAM_CONS_HASH_SRCS" +fi + if [ $HTTP_UPSTREAM_KEEPALIVE = YES ]; then HTTP_MODULES="$HTTP_MODULES $HTTP_UPSTREAM_KEEPALIVE_MODULE" HTTP_SRCS="$HTTP_SRCS $HTTP_UPSTREAM_KEEPALIVE_SRCS" diff -r 48c97d83ab7f -r 063f37f1654e auto/options --- a/auto/options Tue Apr 29 22:22:38 2014 +0200 +++ b/auto/options Thu May 08 14:45:25 2014 +0800 @@ -100,6 +100,7 @@ HTTP_GZIP_STATIC=NO HTTP_UPSTREAM_IP_HASH=YES HTTP_UPSTREAM_LEAST_CONN=YES +HTTP_UPSTREAM_CONS_HASH=YES HTTP_UPSTREAM_KEEPALIVE=YES # STUB diff -r 48c97d83ab7f -r 063f37f1654e auto/sources --- a/auto/sources Tue Apr 29 22:22:38 2014 +0200 +++ b/auto/sources Thu May 08 14:45:25 2014 +0800 @@ -504,6 +504,11 @@ src/http/modules/ngx_http_upstream_least_conn_module.c" +HTTP_UPSTREAM_CONS_HASH_MODULE=ngx_http_upstream_cons_hash_module +HTTP_UPSTREAM_CONS_HASH_SRCS=" \ + src/http/modules/ngx_http_upstream_cons_hash_module.c" + + HTTP_UPSTREAM_KEEPALIVE_MODULE=ngx_http_upstream_keepalive_module HTTP_UPSTREAM_KEEPALIVE_SRCS=" \ src/http/modules/ngx_http_upstream_keepalive_module.c" diff -r 48c97d83ab7f -r 063f37f1654e src/http/modules/ngx_http_upstream_cons_hash_module.c --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/http/modules/ngx_http_upstream_cons_hash_module.c Thu May 08 14:45:25 2014 +0800 @@ -0,0 +1,591 @@ +#include +#include +#include + + +#define NGX_HTTP_UPSTREAM_CH_VNODE_NUM 141 + + +typedef struct { + uint32_t hash; + + ngx_uint_t index; +} ngx_http_upstream_cons_hash_node_t; + + +typedef struct { + ngx_array_t *values; + ngx_array_t *lengths; + + ngx_uint_t node_number; + ngx_http_upstream_cons_hash_node_t *nodes; + + ngx_uint_t *nearest; + + ngx_http_upstream_rr_peers_t *peers; + + ngx_log_t *log; + + ngx_pool_t *pool; +} ngx_http_upstream_cons_hash_conf_t; + + +typedef struct { + /* the round robin data must be first */ + ngx_http_upstream_rr_peer_data_t rrp; + + ngx_uint_t found; + + ngx_http_upstream_cons_hash_conf_t *chcf; + + ngx_event_get_peer_pt get_rr_peer; +} ngx_http_upstream_ch_peer_data_t; + + +static void *ngx_http_upstream_cons_hash_create_conf(ngx_conf_t *cf); +static char *ngx_http_upstream_cons_hash(ngx_conf_t *cf, ngx_command_t *cmd, + void *conf); +static ngx_int_t ngx_http_upstream_init_cons_hash(ngx_conf_t *cf, + ngx_http_upstream_srv_conf_t *us); + +static ngx_int_t ngx_http_upstream_init_cons_hash_peer(ngx_http_request_t *r, + ngx_http_upstream_srv_conf_t *us); +static ngx_int_t ngx_http_upstream_get_cons_hash_peer( + ngx_peer_connection_t *pc, void *data); + +static ngx_uint_t ngx_http_upstream_find_cons_hash_peer( + ngx_http_upstream_cons_hash_conf_t *chcf, uint32_t hash); +static int ngx_http_upstream_cons_hash_cmp_dist(const void *one, + const void *two); +static int ngx_http_upstream_cons_hash_cmp_node(const void *one, + const void *two); +static ngx_int_t ngx_http_upstream_cons_hash_random( + ngx_http_upstream_cons_hash_conf_t *chcf, ngx_str_t value, ngx_uint_t id, + uint32_t *ret); +static ngx_int_t ngx_http_upstream_cons_hash_init_nearest( + ngx_http_upstream_cons_hash_conf_t *chcf); + +inline static ngx_int_t ngx_http_upstream_get_cons_hash_try_peer( + ngx_peer_connection_t *pc, void *data, ngx_uint_t index); + + +static ngx_command_t ngx_http_upstream_cons_hash_commands[] = { + + { ngx_string("cons_hash"), + NGX_HTTP_UPS_CONF | NGX_CONF_TAKE1, + ngx_http_upstream_cons_hash, + 0, + 0, + NULL }, + + ngx_null_command +}; + + +static ngx_http_module_t ngx_http_upstream_cons_hash_module_ctx = { + NULL, /* preconfiguration */ + NULL, /* postconfiguration */ + + NULL, /* create main configuration */ + NULL, /* init main configuration */ + + ngx_http_upstream_cons_hash_create_conf, /* create server configuration*/ + NULL, /* merge server configuration */ + + NULL, /* create location configuration */ + NULL /* merge location configuration */ +}; + + +ngx_module_t ngx_http_upstream_cons_hash_module = { + NGX_MODULE_V1, + &ngx_http_upstream_cons_hash_module_ctx, /* module context */ + ngx_http_upstream_cons_hash_commands, /* module directives */ + NGX_HTTP_MODULE, /* module type */ + NULL, /* init master */ + NULL, /* init module */ + NULL, /* init process */ + NULL, /* init thread */ + NULL, /* exit thread */ + NULL, /* exit process */ + NULL, /* exit master */ + NGX_MODULE_V1_PADDING +}; + + +static void * +ngx_http_upstream_cons_hash_create_conf(ngx_conf_t *cf) +{ + ngx_http_upstream_cons_hash_conf_t *conf; + + conf = ngx_pcalloc(cf->pool, sizeof(ngx_http_upstream_cons_hash_conf_t)); + if (conf == NULL) { + return NULL; + } + + return conf; +} + + +static ngx_int_t +ngx_http_upstream_init_cons_hash(ngx_conf_t *cf, + ngx_http_upstream_srv_conf_t *us) +{ + uint32_t hash; + ngx_int_t rc; + ngx_str_t name; + ngx_uint_t i, j, k, n, nn, m; + ngx_http_upstream_rr_peers_t *peers; + ngx_http_upstream_cons_hash_conf_t *chcf; + + if (ngx_http_upstream_init_round_robin(cf, us) == NGX_ERROR) { + return NGX_ERROR; + } + + peers = us->peer.data; + + n = peers->number; + nn = 0; + + for (i = 0; i < n; ++i) { + nn += peers->peer[i].weight; + } + + nn *= (NGX_HTTP_UPSTREAM_CH_VNODE_NUM + 1); + + chcf = ngx_http_conf_upstream_srv_conf(us, + ngx_http_upstream_cons_hash_module); + + /* + * to guarantee nn % n == 0, there's no side effect, + * but much more convenient to construct the 'nearest' + */ + + nn = (nn + n - 1) / n * n; + chcf->node_number = nn; + + chcf->log = cf->log; + chcf->pool = cf->pool; + chcf->peers = peers; + + chcf->nodes = ngx_pcalloc(cf->pool, nn * + sizeof(ngx_http_upstream_cons_hash_node_t)); + if (chcf->nodes == NULL) { + return NGX_ERROR; + } + + for (i = 0, k = 0; i < n; ++i) { + + name = peers->peer[i].name; + m = peers->peer[i].weight * (1 + NGX_HTTP_UPSTREAM_CH_VNODE_NUM); + + for (j = 0; j < m; ++j, ++k) { + + chcf->nodes[k].index = i; + + rc = ngx_http_upstream_cons_hash_random(chcf, name, j, &hash); + if (rc == NGX_ERROR) { + return NGX_ERROR; + } + + chcf->nodes[k].hash = hash; + } + } + + for (i = 0; i < nn - k; ++i) { + chcf->nodes[k + i].index = chcf->nodes[0].index; + chcf->nodes[k + i].hash = chcf->nodes[0].hash; + } + + ngx_qsort(chcf->nodes, nn, sizeof(ngx_http_upstream_cons_hash_node_t), + ngx_http_upstream_cons_hash_cmp_node); + + rc = ngx_http_upstream_cons_hash_init_nearest(chcf); + if (rc == NGX_ERROR) { + return NGX_ERROR; + } + + us->peer.init = ngx_http_upstream_init_cons_hash_peer; + + return NGX_OK; +} + + +static ngx_int_t +ngx_http_upstream_init_cons_hash_peer(ngx_http_request_t *r, + ngx_http_upstream_srv_conf_t *us) +{ + uint32_t hash; + ngx_str_t raw_value; + ngx_http_upstream_cons_hash_conf_t *chcf; + ngx_http_upstream_ch_peer_data_t *chp; + + chcf = ngx_http_conf_upstream_srv_conf(us, + ngx_http_upstream_cons_hash_module); + if (chcf == NULL) { + return NGX_ERROR; + } + + chp = ngx_pcalloc(r->pool, sizeof(ngx_http_upstream_ch_peer_data_t)); + if (chp == NULL) { + return NGX_ERROR; + } + + r->upstream->peer.data = &chp->rrp; + + if (ngx_http_upstream_init_round_robin_peer(r, us) != NGX_OK) { + return NGX_ERROR; + } + + if (!chp->rrp.peers->single) { + + /* raw_value.data is allocated in ngx_http_script_run from r->pool */ + + if (ngx_http_script_run(r, &raw_value, + chcf->lengths->elts, 0, chcf->values->elts) == NULL) { + return NGX_ERROR; + } + + hash = ngx_murmur_hash2(raw_value.data, raw_value.len); + + ngx_pfree(r->pool, raw_value.data); + + chp->found = ngx_http_upstream_find_cons_hash_peer(chcf, hash); + } + + r->upstream->peer.get = ngx_http_upstream_get_cons_hash_peer; + + chp->chcf = chcf; + chp->get_rr_peer = ngx_http_upstream_get_round_robin_peer; + + return NGX_OK; +} + + +inline static ngx_int_t +ngx_http_upstream_get_cons_hash_try_peer(ngx_peer_connection_t *pc, void *data, + ngx_uint_t index) +{ + ngx_http_upstream_ch_peer_data_t *chp = data; + + time_t now; + ngx_int_t rc; + ngx_uint_t n, m; + ngx_http_upstream_rr_peer_t *peer; + + n = index / (8 * sizeof(uintptr_t)); + m = (uintptr_t) 1 << index % (8 * sizeof(uintptr_t)); + + if (chp->rrp.tried[n] & m) { + return NGX_AGAIN; + } + + rc = NGX_AGAIN; + + now = ngx_time(); + + peer = &chp->chcf->peers->peer[index]; + + /* ngx_lock_mutex(chp->rrp.peers->mutex); */ + + if (!peer->down) { + if (peer->max_fails == 0 || peer->fails < peer->max_fails) { + rc = NGX_OK; + } + + if (now - peer->checked > peer->fail_timeout) { + peer->checked = now; + rc = NGX_OK; + } + } + + if (rc == NGX_OK) { + chp->rrp.current = index; + + pc->sockaddr = peer->sockaddr; + pc->socklen = peer->socklen; + pc->name = &peer->name; + } + + /* ngx_unlock_mutex(chp->rrp.peers->mutex); */ + + chp->rrp.tried[n] |= m; + + return rc; +} + + +static ngx_int_t +ngx_http_upstream_get_cons_hash_peer(ngx_peer_connection_t *pc, void *data) +{ + ngx_http_upstream_ch_peer_data_t *chp = data; + + ngx_int_t rc; + ngx_uint_t i, j, n, nn; + ngx_uint_t *nearest; + ngx_http_upstream_rr_peers_t *peers; + ngx_http_upstream_cons_hash_node_t *nodes; + + if (chp->rrp.peers->single) { + return chp->get_rr_peer(pc, &chp->rrp); + } + + pc->cached = 0; + pc->connection = NULL; + + peers = chp->chcf->peers; + nodes = chp->chcf->nodes; + nearest = chp->chcf->nearest; + + n = peers->number; + nn = chp->chcf->node_number; + + for (i = chp->found; i % n != 0; i = (i + 1) % nn) { + + rc = ngx_http_upstream_get_cons_hash_try_peer(pc, data, + nodes[i].index); + if (rc == NGX_OK) { + return NGX_OK; + } + } + + for (j = (i + n) % nn; i != j; i = (i + 1) % nn) { + + rc = ngx_http_upstream_get_cons_hash_try_peer(pc, data, + nearest[i]); + if (rc == NGX_OK) { + return NGX_OK; + } + } + + /* all peers failed, mark them as live for quick recovery */ + + for (i = 0; i < peers->number; i++) { + peers->peer[i].fails = 0; + } + + pc->name = peers->name; + + return NGX_BUSY; +} + + +static ngx_uint_t +ngx_http_upstream_find_cons_hash_peer(ngx_http_upstream_cons_hash_conf_t *chcf, + uint32_t hash) +{ + uint32_t mid_hash; + ngx_int_t l, r, mid; + + l = 0; + r = chcf->node_number - 1; + + while (l <= r) { + mid = (l + r) >> 1; + mid_hash = chcf->nodes[mid].hash; + + if (mid_hash < hash) { + l = mid + 1; + } else { + r = mid - 1; + } + } + + if (l == (ngx_int_t)chcf->node_number) { + l = 0; + } + + return l; +} + + +static char * +ngx_http_upstream_cons_hash(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) +{ + ngx_str_t *value; + ngx_http_script_compile_t sc; + ngx_http_upstream_srv_conf_t *uscf; + ngx_http_upstream_cons_hash_conf_t *chcf; + + uscf = ngx_http_conf_get_module_srv_conf(cf, ngx_http_upstream_module); + + chcf = ngx_http_conf_upstream_srv_conf(uscf, + ngx_http_upstream_cons_hash_module); + if (uscf->peer.init_upstream) { + ngx_conf_log_error(NGX_LOG_WARN, cf, 0, + "load balancing method redefined"); + } + uscf->peer.init_upstream = ngx_http_upstream_init_cons_hash; + + value = cf->args->elts; + ngx_memzero(&sc, sizeof(ngx_http_script_compile_t)); + + sc.cf = cf; + sc.source = &value[1]; + sc.lengths = &chcf->lengths;; + sc.values = &chcf->values; + sc.complete_lengths = 1; + sc.complete_values = 1; + + if (ngx_http_script_compile(&sc) != NGX_OK) { + return NGX_CONF_ERROR; + } + + uscf->flags = NGX_HTTP_UPSTREAM_CREATE + |NGX_HTTP_UPSTREAM_WEIGHT + |NGX_HTTP_UPSTREAM_MAX_FAILS + |NGX_HTTP_UPSTREAM_FAIL_TIMEOUT + |NGX_HTTP_UPSTREAM_DOWN; + + return NGX_CONF_OK; +} + + +static ngx_int_t +ngx_http_upstream_cons_hash_random(ngx_http_upstream_cons_hash_conf_t *chcf, + ngx_str_t value, ngx_uint_t id, uint32_t *ret) +{ + /* repeatable random with same (val, id) */ + + u_char *buf, *pos; + ngx_uint_t total; + + total = NGX_INT_T_LEN + value.len; + + buf = ngx_calloc(total, chcf->log); + if (buf == NULL) { + return NGX_ERROR; + } + + pos = ngx_snprintf(buf, total, "%i-%*s", id, value.len, value.data); + + *ret = ngx_murmur_hash2(buf, pos - buf); + + ngx_free(buf); + + return NGX_OK; +} + + +static ngx_int_t +ngx_http_upstream_cons_hash_init_nearest( + ngx_http_upstream_cons_hash_conf_t *chcf) +{ + ngx_int_t k; + ngx_uint_t i, j, n, nn; + ngx_uint_t *nearest, *temp; + ngx_http_upstream_cons_hash_node_t *nodes; + + n = chcf->peers->number; + nn = chcf->node_number; + + nodes = chcf->nodes; + + nearest = ngx_pcalloc(chcf->pool, nn * sizeof(ngx_uint_t)); + if (nearest == NULL) { + return NGX_ERROR; + } + + chcf->nearest = nearest; + + temp = ngx_pcalloc(chcf->pool, n * sizeof(ngx_uint_t)); + if (temp == NULL) { + return NGX_ERROR; + } + + for (i = 0; i < nn; ++i) { + nearest[i] = nn; + } + + for (i = 0; i < nn; i += n) { + for (j = 0; j < n; ++j) { + temp[j] = nn; + } + for (k = n - 1; k >= 0; --k) { + temp[nodes[i + k].index] = i + k; + } + for (j = 0; j < n; ++j) { + nearest[i + j] = temp[j]; + } + } + + ngx_pfree(chcf->pool, temp); + + /* update the 'nearest' twice */ + + for (i = 0; i < 2; ++i) { + for (k = nn - n; k >= 0; k -= n) { + for (j = 0; j < n; ++j) { + if (nearest[k + j] == nn) { + nearest[k + j] = nearest[(k + j + n) % nn]; + } + } + } + } + + for (i = 0; i < nn; i += n) { + + /* there is no elt equals to nn in the 'nearest' now */ + + for (j = 0; j < n; ++j) { + if (nearest[i + j] < i) { + nearest[i + j] += nn; + } + } + + ngx_qsort(nearest + i, n, sizeof(ngx_uint_t), + ngx_http_upstream_cons_hash_cmp_dist); + + for (j = 0; j < n; ++j) { + if (nearest[i + j] >= nn) { + nearest[i + j] -= nn; + } + } + } + + for (i = 0; i < nn; ++i) { + nearest[i] = nodes[nearest[i]].index; + } + + return NGX_OK; +} + + +static int +ngx_http_upstream_cons_hash_cmp_dist(const void *one, const void *two) +{ + ngx_uint_t first, second; + + first = *(ngx_uint_t *)one; + second = *(ngx_uint_t *)two; + + if (first < second) { + return -1; + } + + if (first > second) { + return 1; + } + + return 0; +} + + +static int +ngx_http_upstream_cons_hash_cmp_node(const void *one, const void *two) +{ + ngx_http_upstream_cons_hash_node_t *first, *second; + + first = (ngx_http_upstream_cons_hash_node_t *) one; + second = (ngx_http_upstream_cons_hash_node_t *) two; + + if (first->hash < second->hash) { + return -1; + } + + if (first->hash > second->hash) { + return 1; + } + + return 0; +} -------------- next part -------------- An HTML attachment was scrubbed... URL: From sarah at nginx.com Thu May 8 14:13:50 2014 From: sarah at nginx.com (Sarah Novotny) Date: Thu, 08 May 2014 07:13:50 -0700 Subject: How to submit a third pary module ? In-Reply-To: References: Message-ID: <0C2BA140-8662-4888-926B-B7C5EB6777B9@nginx.com> On 7 May 2014, at 19:47, Shafreeck Sea wrote: > Hi all: > I developed a nginx module and I am glad to share with others . How to > submit this module to wiki.nginx.org ? I tried to create an account at > wiki.nginx.org but failed. please shoot me a note directly, with your preferred username and i'll create an account for you. we've closed down public account creation because of the sea of spam we were getting. sarah From shin at sterneck.asia Thu May 8 14:19:26 2014 From: shin at sterneck.asia (Shin Sterneck) Date: Thu, 8 May 2014 23:19:26 +0900 Subject: new autoindex_show_dot_files option patch submitted, please review Message-ID: Dear Developers, this is concerning ticket/patch #557 ( http://trac.nginx.org/nginx/ticket/557 ), proposing the introduction of a new option to the autoindex module. It allows one to control whether the autoindex module should show "dot-files" or not (files starting with a 'dot', commonly used for hidden files in linux/unix). The patch introduces the following option: syntax: autoindex_show_dot_files on | off; default: autoindex_show_dot_files off; context: http, server, location Patch: http://trac.nginx.org/nginx/attachment/ticket/557/ngx_autoindex_show_dot_files.patch Forum: http://forum.nginx.org/read.php?2,216751,249931#msg-249931 Hope this is usefully and will be merged into the next release of nginx best Regards, Shin -------------- next part -------------- An HTML attachment was scrubbed... URL: From steven.hartland at multiplay.co.uk Thu May 8 15:23:05 2014 From: steven.hartland at multiplay.co.uk (Steven Hartland) Date: Thu, 8 May 2014 16:23:05 +0100 Subject: [PATCH] Upstream: add consistent hash module References: Message-ID: Be good to see this in core, currently we use the 3rd party module to achieve this: http://wiki.nginx.org/HttpUpstreamRequestHashModule One question on the patch, you appear to have some commented out locking is this an oversight? Regards Steve ----- Original Message ----- From: "Jianjun Zheng" To: Sent: Thursday, May 08, 2014 8:52 AM Subject: [PATCH] Upstream: add consistent hash module > usage example: cons_hash $request_uri; > > Not only consistent hash is introduced as a load balancing, but also > increase the hits of cache in upstream servers. > > This modules is written according to round_robin, ip_hash > and least_conn modules for upstream. So it inherits all the > essential logic from them. > > This implementation is simple but has a high performance, > The time comlexity remains O(log(T)) with a low constant coefficient, > even when half of servers get down, > regardless of the weight of servers, as analysis below. > > Suppose there are N upstream servers, with D down servers, > and the weights are W1,W2,...,W(N-1). > V is the amount of virtual nodes per unit. > P is the probability to get an alive server from all nodes at random. > > Let T = (W1+W2+...+W(N-1))*V. > then the space complexity is O(T), > and the time comlexity is O(log(T) + (1-P)*N/(N-D)), > 'N/(N-D)' of which is independent of W and V. > > > # HG changeset patch > # User Jianjun Zheng > # Date 1399531525 -28800 > # Thu May 08 14:45:25 2014 +0800 > # Node ID 063f37f1654ef6cc03bd311a7fe6189b299ce2f2 > # Parent 48c97d83ab7f0a3f641987fb32ace8af7720aefc > Upstream: add consistent hash module > > diff -r 48c97d83ab7f -r 063f37f1654e auto/modules > --- a/auto/modules Tue Apr 29 22:22:38 2014 +0200 > +++ b/auto/modules Thu May 08 14:45:25 2014 +0800 > @@ -381,6 +381,11 @@ > HTTP_SRCS="$HTTP_SRCS $HTTP_UPSTREAM_LEAST_CONN_SRCS" > fi > > +if [ $HTTP_UPSTREAM_CONS_HASH = YES ]; then > + HTTP_MODULES="$HTTP_MODULES $HTTP_UPSTREAM_CONS_HASH_MODULE" > + HTTP_SRCS="$HTTP_SRCS $HTTP_UPSTREAM_CONS_HASH_SRCS" > +fi > + > if [ $HTTP_UPSTREAM_KEEPALIVE = YES ]; then > HTTP_MODULES="$HTTP_MODULES $HTTP_UPSTREAM_KEEPALIVE_MODULE" > HTTP_SRCS="$HTTP_SRCS $HTTP_UPSTREAM_KEEPALIVE_SRCS" > diff -r 48c97d83ab7f -r 063f37f1654e auto/options > --- a/auto/options Tue Apr 29 22:22:38 2014 +0200 > +++ b/auto/options Thu May 08 14:45:25 2014 +0800 > @@ -100,6 +100,7 @@ > HTTP_GZIP_STATIC=NO > HTTP_UPSTREAM_IP_HASH=YES > HTTP_UPSTREAM_LEAST_CONN=YES > +HTTP_UPSTREAM_CONS_HASH=YES > HTTP_UPSTREAM_KEEPALIVE=YES > > # STUB > diff -r 48c97d83ab7f -r 063f37f1654e auto/sources > --- a/auto/sources Tue Apr 29 22:22:38 2014 +0200 > +++ b/auto/sources Thu May 08 14:45:25 2014 +0800 > @@ -504,6 +504,11 @@ > src/http/modules/ngx_http_upstream_least_conn_module.c" > > > +HTTP_UPSTREAM_CONS_HASH_MODULE=ngx_http_upstream_cons_hash_module > +HTTP_UPSTREAM_CONS_HASH_SRCS=" \ > + src/http/modules/ngx_http_upstream_cons_hash_module.c" > + > + > HTTP_UPSTREAM_KEEPALIVE_MODULE=ngx_http_upstream_keepalive_module > HTTP_UPSTREAM_KEEPALIVE_SRCS=" \ > src/http/modules/ngx_http_upstream_keepalive_module.c" > diff -r 48c97d83ab7f -r 063f37f1654e > src/http/modules/ngx_http_upstream_cons_hash_module.c > --- /dev/null Thu Jan 01 00:00:00 1970 +0000 > +++ b/src/http/modules/ngx_http_upstream_cons_hash_module.c Thu May 08 > 14:45:25 2014 +0800 > @@ -0,0 +1,591 @@ > +#include > +#include > +#include > + > + > +#define NGX_HTTP_UPSTREAM_CH_VNODE_NUM 141 > + > + > +typedef struct { > + uint32_t hash; > + > + ngx_uint_t index; > +} ngx_http_upstream_cons_hash_node_t; > + > + > +typedef struct { > + ngx_array_t *values; > + ngx_array_t *lengths; > + > + ngx_uint_t node_number; > + ngx_http_upstream_cons_hash_node_t *nodes; > + > + ngx_uint_t *nearest; > + > + ngx_http_upstream_rr_peers_t *peers; > + > + ngx_log_t *log; > + > + ngx_pool_t *pool; > +} ngx_http_upstream_cons_hash_conf_t; > + > + > +typedef struct { > + /* the round robin data must be first */ > + ngx_http_upstream_rr_peer_data_t rrp; > + > + ngx_uint_t found; > + > + ngx_http_upstream_cons_hash_conf_t *chcf; > + > + ngx_event_get_peer_pt get_rr_peer; > +} ngx_http_upstream_ch_peer_data_t; > + > + > +static void *ngx_http_upstream_cons_hash_create_conf(ngx_conf_t *cf); > +static char *ngx_http_upstream_cons_hash(ngx_conf_t *cf, ngx_command_t > *cmd, > + void *conf); > +static ngx_int_t ngx_http_upstream_init_cons_hash(ngx_conf_t *cf, > + ngx_http_upstream_srv_conf_t *us); > + > +static ngx_int_t ngx_http_upstream_init_cons_hash_peer(ngx_http_request_t > *r, > + ngx_http_upstream_srv_conf_t *us); > +static ngx_int_t ngx_http_upstream_get_cons_hash_peer( > + ngx_peer_connection_t *pc, void *data); > + > +static ngx_uint_t ngx_http_upstream_find_cons_hash_peer( > + ngx_http_upstream_cons_hash_conf_t *chcf, uint32_t hash); > +static int ngx_http_upstream_cons_hash_cmp_dist(const void *one, > + const void *two); > +static int ngx_http_upstream_cons_hash_cmp_node(const void *one, > + const void *two); > +static ngx_int_t ngx_http_upstream_cons_hash_random( > + ngx_http_upstream_cons_hash_conf_t *chcf, ngx_str_t value, ngx_uint_t > id, > + uint32_t *ret); > +static ngx_int_t ngx_http_upstream_cons_hash_init_nearest( > + ngx_http_upstream_cons_hash_conf_t *chcf); > + > +inline static ngx_int_t ngx_http_upstream_get_cons_hash_try_peer( > + ngx_peer_connection_t *pc, void *data, ngx_uint_t index); > + > + > +static ngx_command_t ngx_http_upstream_cons_hash_commands[] = { > + > + { ngx_string("cons_hash"), > + NGX_HTTP_UPS_CONF | NGX_CONF_TAKE1, > + ngx_http_upstream_cons_hash, > + 0, > + 0, > + NULL }, > + > + ngx_null_command > +}; > + > + > +static ngx_http_module_t ngx_http_upstream_cons_hash_module_ctx = { > + NULL, /* preconfiguration */ > + NULL, /* postconfiguration */ > + > + NULL, /* create main configuration */ > + NULL, /* init main configuration */ > + > + ngx_http_upstream_cons_hash_create_conf, /* create server > configuration*/ > + NULL, /* merge server configuration */ > + > + NULL, /* create location configuration > */ > + NULL /* merge location configuration > */ > +}; > + > + > +ngx_module_t ngx_http_upstream_cons_hash_module = { > + NGX_MODULE_V1, > + &ngx_http_upstream_cons_hash_module_ctx, /* module context */ > + ngx_http_upstream_cons_hash_commands, /* module directives */ > + NGX_HTTP_MODULE, /* module type */ > + NULL, /* init master */ > + NULL, /* init module */ > + NULL, /* init process */ > + NULL, /* init thread */ > + NULL, /* exit thread */ > + NULL, /* exit process */ > + NULL, /* exit master */ > + NGX_MODULE_V1_PADDING > +}; > + > + > +static void * > +ngx_http_upstream_cons_hash_create_conf(ngx_conf_t *cf) > +{ > + ngx_http_upstream_cons_hash_conf_t *conf; > + > + conf = ngx_pcalloc(cf->pool, > sizeof(ngx_http_upstream_cons_hash_conf_t)); > + if (conf == NULL) { > + return NULL; > + } > + > + return conf; > +} > + > + > +static ngx_int_t > +ngx_http_upstream_init_cons_hash(ngx_conf_t *cf, > + ngx_http_upstream_srv_conf_t *us) > +{ > + uint32_t hash; > + ngx_int_t rc; > + ngx_str_t name; > + ngx_uint_t i, j, k, n, nn, m; > + ngx_http_upstream_rr_peers_t *peers; > + ngx_http_upstream_cons_hash_conf_t *chcf; > + > + if (ngx_http_upstream_init_round_robin(cf, us) == NGX_ERROR) { > + return NGX_ERROR; > + } > + > + peers = us->peer.data; > + > + n = peers->number; > + nn = 0; > + > + for (i = 0; i < n; ++i) { > + nn += peers->peer[i].weight; > + } > + > + nn *= (NGX_HTTP_UPSTREAM_CH_VNODE_NUM + 1); > + > + chcf = ngx_http_conf_upstream_srv_conf(us, > + > ngx_http_upstream_cons_hash_module); > + > + /* > + * to guarantee nn % n == 0, there's no side effect, > + * but much more convenient to construct the 'nearest' > + */ > + > + nn = (nn + n - 1) / n * n; > + chcf->node_number = nn; > + > + chcf->log = cf->log; > + chcf->pool = cf->pool; > + chcf->peers = peers; > + > + chcf->nodes = ngx_pcalloc(cf->pool, nn * > + sizeof(ngx_http_upstream_cons_hash_node_t)); > + if (chcf->nodes == NULL) { > + return NGX_ERROR; > + } > + > + for (i = 0, k = 0; i < n; ++i) { > + > + name = peers->peer[i].name; > + m = peers->peer[i].weight * (1 + NGX_HTTP_UPSTREAM_CH_VNODE_NUM); > + > + for (j = 0; j < m; ++j, ++k) { > + > + chcf->nodes[k].index = i; > + > + rc = ngx_http_upstream_cons_hash_random(chcf, name, j, &hash); > + if (rc == NGX_ERROR) { > + return NGX_ERROR; > + } > + > + chcf->nodes[k].hash = hash; > + } > + } > + > + for (i = 0; i < nn - k; ++i) { > + chcf->nodes[k + i].index = chcf->nodes[0].index; > + chcf->nodes[k + i].hash = chcf->nodes[0].hash; > + } > + > + ngx_qsort(chcf->nodes, nn, sizeof(ngx_http_upstream_cons_hash_node_t), > + ngx_http_upstream_cons_hash_cmp_node); > + > + rc = ngx_http_upstream_cons_hash_init_nearest(chcf); > + if (rc == NGX_ERROR) { > + return NGX_ERROR; > + } > + > + us->peer.init = ngx_http_upstream_init_cons_hash_peer; > + > + return NGX_OK; > +} > + > + > +static ngx_int_t > +ngx_http_upstream_init_cons_hash_peer(ngx_http_request_t *r, > + ngx_http_upstream_srv_conf_t *us) > +{ > + uint32_t hash; > + ngx_str_t raw_value; > + ngx_http_upstream_cons_hash_conf_t *chcf; > + ngx_http_upstream_ch_peer_data_t *chp; > + > + chcf = ngx_http_conf_upstream_srv_conf(us, > + > ngx_http_upstream_cons_hash_module); > + if (chcf == NULL) { > + return NGX_ERROR; > + } > + > + chp = ngx_pcalloc(r->pool, sizeof(ngx_http_upstream_ch_peer_data_t)); > + if (chp == NULL) { > + return NGX_ERROR; > + } > + > + r->upstream->peer.data = &chp->rrp; > + > + if (ngx_http_upstream_init_round_robin_peer(r, us) != NGX_OK) { > + return NGX_ERROR; > + } > + > + if (!chp->rrp.peers->single) { > + > + /* raw_value.data is allocated in ngx_http_script_run from r->pool > */ > + > + if (ngx_http_script_run(r, &raw_value, > + chcf->lengths->elts, 0, chcf->values->elts) == > NULL) { > + return NGX_ERROR; > + } > + > + hash = ngx_murmur_hash2(raw_value.data, raw_value.len); > + > + ngx_pfree(r->pool, raw_value.data); > + > + chp->found = ngx_http_upstream_find_cons_hash_peer(chcf, hash); > + } > + > + r->upstream->peer.get = ngx_http_upstream_get_cons_hash_peer; > + > + chp->chcf = chcf; > + chp->get_rr_peer = ngx_http_upstream_get_round_robin_peer; > + > + return NGX_OK; > +} > + > + > +inline static ngx_int_t > +ngx_http_upstream_get_cons_hash_try_peer(ngx_peer_connection_t *pc, void > *data, > + ngx_uint_t index) > +{ > + ngx_http_upstream_ch_peer_data_t *chp = data; > + > + time_t now; > + ngx_int_t rc; > + ngx_uint_t n, m; > + ngx_http_upstream_rr_peer_t *peer; > + > + n = index / (8 * sizeof(uintptr_t)); > + m = (uintptr_t) 1 << index % (8 * sizeof(uintptr_t)); > + > + if (chp->rrp.tried[n] & m) { > + return NGX_AGAIN; > + } > + > + rc = NGX_AGAIN; > + > + now = ngx_time(); > + > + peer = &chp->chcf->peers->peer[index]; > + > + /* ngx_lock_mutex(chp->rrp.peers->mutex); */ > + > + if (!peer->down) { > + if (peer->max_fails == 0 || peer->fails < peer->max_fails) { > + rc = NGX_OK; > + } > + > + if (now - peer->checked > peer->fail_timeout) { > + peer->checked = now; > + rc = NGX_OK; > + } > + } > + > + if (rc == NGX_OK) { > + chp->rrp.current = index; > + > + pc->sockaddr = peer->sockaddr; > + pc->socklen = peer->socklen; > + pc->name = &peer->name; > + } > + > + /* ngx_unlock_mutex(chp->rrp.peers->mutex); */ > + > + chp->rrp.tried[n] |= m; > + > + return rc; > +} > + > + > +static ngx_int_t > +ngx_http_upstream_get_cons_hash_peer(ngx_peer_connection_t *pc, void *data) > +{ > + ngx_http_upstream_ch_peer_data_t *chp = data; > + > + ngx_int_t rc; > + ngx_uint_t i, j, n, nn; > + ngx_uint_t *nearest; > + ngx_http_upstream_rr_peers_t *peers; > + ngx_http_upstream_cons_hash_node_t *nodes; > + > + if (chp->rrp.peers->single) { > + return chp->get_rr_peer(pc, &chp->rrp); > + } > + > + pc->cached = 0; > + pc->connection = NULL; > + > + peers = chp->chcf->peers; > + nodes = chp->chcf->nodes; > + nearest = chp->chcf->nearest; > + > + n = peers->number; > + nn = chp->chcf->node_number; > + > + for (i = chp->found; i % n != 0; i = (i + 1) % nn) { > + > + rc = ngx_http_upstream_get_cons_hash_try_peer(pc, data, > + nodes[i].index); > + if (rc == NGX_OK) { > + return NGX_OK; > + } > + } > + > + for (j = (i + n) % nn; i != j; i = (i + 1) % nn) { > + > + rc = ngx_http_upstream_get_cons_hash_try_peer(pc, data, > + nearest[i]); > + if (rc == NGX_OK) { > + return NGX_OK; > + } > + } > + > + /* all peers failed, mark them as live for quick recovery */ > + > + for (i = 0; i < peers->number; i++) { > + peers->peer[i].fails = 0; > + } > + > + pc->name = peers->name; > + > + return NGX_BUSY; > +} > + > + > +static ngx_uint_t > +ngx_http_upstream_find_cons_hash_peer(ngx_http_upstream_cons_hash_conf_t > *chcf, > + uint32_t hash) > +{ > + uint32_t mid_hash; > + ngx_int_t l, r, mid; > + > + l = 0; > + r = chcf->node_number - 1; > + > + while (l <= r) { > + mid = (l + r) >> 1; > + mid_hash = chcf->nodes[mid].hash; > + > + if (mid_hash < hash) { > + l = mid + 1; > + } else { > + r = mid - 1; > + } > + } > + > + if (l == (ngx_int_t)chcf->node_number) { > + l = 0; > + } > + > + return l; > +} > + > + > +static char * > +ngx_http_upstream_cons_hash(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) > +{ > + ngx_str_t *value; > + ngx_http_script_compile_t sc; > + ngx_http_upstream_srv_conf_t *uscf; > + ngx_http_upstream_cons_hash_conf_t *chcf; > + > + uscf = ngx_http_conf_get_module_srv_conf(cf, ngx_http_upstream_module); > + > + chcf = ngx_http_conf_upstream_srv_conf(uscf, > + > ngx_http_upstream_cons_hash_module); > + if (uscf->peer.init_upstream) { > + ngx_conf_log_error(NGX_LOG_WARN, cf, 0, > + "load balancing method redefined"); > + } > + uscf->peer.init_upstream = ngx_http_upstream_init_cons_hash; > + > + value = cf->args->elts; > + ngx_memzero(&sc, sizeof(ngx_http_script_compile_t)); > + > + sc.cf = cf; > + sc.source = &value[1]; > + sc.lengths = &chcf->lengths;; > + sc.values = &chcf->values; > + sc.complete_lengths = 1; > + sc.complete_values = 1; > + > + if (ngx_http_script_compile(&sc) != NGX_OK) { > + return NGX_CONF_ERROR; > + } > + > + uscf->flags = NGX_HTTP_UPSTREAM_CREATE > + |NGX_HTTP_UPSTREAM_WEIGHT > + |NGX_HTTP_UPSTREAM_MAX_FAILS > + |NGX_HTTP_UPSTREAM_FAIL_TIMEOUT > + |NGX_HTTP_UPSTREAM_DOWN; > + > + return NGX_CONF_OK; > +} > + > + > +static ngx_int_t > +ngx_http_upstream_cons_hash_random(ngx_http_upstream_cons_hash_conf_t > *chcf, > + ngx_str_t value, ngx_uint_t id, uint32_t *ret) > +{ > + /* repeatable random with same (val, id) */ > + > + u_char *buf, *pos; > + ngx_uint_t total; > + > + total = NGX_INT_T_LEN + value.len; > + > + buf = ngx_calloc(total, chcf->log); > + if (buf == NULL) { > + return NGX_ERROR; > + } > + > + pos = ngx_snprintf(buf, total, "%i-%*s", id, value.len, value.data); > + > + *ret = ngx_murmur_hash2(buf, pos - buf); > + > + ngx_free(buf); > + > + return NGX_OK; > +} > + > + > +static ngx_int_t > +ngx_http_upstream_cons_hash_init_nearest( > + ngx_http_upstream_cons_hash_conf_t *chcf) > +{ > + ngx_int_t k; > + ngx_uint_t i, j, n, nn; > + ngx_uint_t *nearest, *temp; > + ngx_http_upstream_cons_hash_node_t *nodes; > + > + n = chcf->peers->number; > + nn = chcf->node_number; > + > + nodes = chcf->nodes; > + > + nearest = ngx_pcalloc(chcf->pool, nn * sizeof(ngx_uint_t)); > + if (nearest == NULL) { > + return NGX_ERROR; > + } > + > + chcf->nearest = nearest; > + > + temp = ngx_pcalloc(chcf->pool, n * sizeof(ngx_uint_t)); > + if (temp == NULL) { > + return NGX_ERROR; > + } > + > + for (i = 0; i < nn; ++i) { > + nearest[i] = nn; > + } > + > + for (i = 0; i < nn; i += n) { > + for (j = 0; j < n; ++j) { > + temp[j] = nn; > + } > + for (k = n - 1; k >= 0; --k) { > + temp[nodes[i + k].index] = i + k; > + } > + for (j = 0; j < n; ++j) { > + nearest[i + j] = temp[j]; > + } > + } > + > + ngx_pfree(chcf->pool, temp); > + > + /* update the 'nearest' twice */ > + > + for (i = 0; i < 2; ++i) { > + for (k = nn - n; k >= 0; k -= n) { > + for (j = 0; j < n; ++j) { > + if (nearest[k + j] == nn) { > + nearest[k + j] = nearest[(k + j + n) % nn]; > + } > + } > + } > + } > + > + for (i = 0; i < nn; i += n) { > + > + /* there is no elt equals to nn in the 'nearest' now */ > + > + for (j = 0; j < n; ++j) { > + if (nearest[i + j] < i) { > + nearest[i + j] += nn; > + } > + } > + > + ngx_qsort(nearest + i, n, sizeof(ngx_uint_t), > + ngx_http_upstream_cons_hash_cmp_dist); > + > + for (j = 0; j < n; ++j) { > + if (nearest[i + j] >= nn) { > + nearest[i + j] -= nn; > + } > + } > + } > + > + for (i = 0; i < nn; ++i) { > + nearest[i] = nodes[nearest[i]].index; > + } > + > + return NGX_OK; > +} > + > + > +static int > +ngx_http_upstream_cons_hash_cmp_dist(const void *one, const void *two) > +{ > + ngx_uint_t first, second; > + > + first = *(ngx_uint_t *)one; > + second = *(ngx_uint_t *)two; > + > + if (first < second) { > + return -1; > + } > + > + if (first > second) { > + return 1; > + } > + > + return 0; > +} > + > + > +static int > +ngx_http_upstream_cons_hash_cmp_node(const void *one, const void *two) > +{ > + ngx_http_upstream_cons_hash_node_t *first, *second; > + > + first = (ngx_http_upstream_cons_hash_node_t *) one; > + second = (ngx_http_upstream_cons_hash_node_t *) two; > + > + if (first->hash < second->hash) { > + return -1; > + } > + > + if (first->hash > second->hash) { > + return 1; > + } > + > + return 0; > +} > -------------------------------------------------------------------------------- > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel From mdounin at mdounin.ru Thu May 8 23:52:44 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 9 May 2014 03:52:44 +0400 Subject: nginx/backend keepalive question In-Reply-To: References: Message-ID: <20140508235244.GB1849@mdounin.ru> Hello! On Wed, Apr 30, 2014 at 02:08:57PM -0700, Yu Zhao wrote: > Greetings! > > I couldn't find my answer of following questions in the manual or somewhere > else. Your help is much appreciated! See src/http/modules/ngx_http_upstream_keepalive_module.c. > 1) does nginx manage frontend/nginx and nginx/backend connections > separately? In another word, will a nginx/backend connection be bond to a > certain frontend/nginx connection for the lifetime of the frontend/nginx > connection that is also kept-alive? My guess is it's not, nginx uses M:N > model which M is the number of frontend/nginx connections that may or may > not be kept-alive and N is the number of nginx/backend kept-alive > connections, and M connections share N connections. Cache of connections to upstream servers is completely unrelated to client connections. > 2) If my guess is correct, how does nginx make sure the request/response > pair is not broken/mismatched? Nginx wil NOT send new request to a backend > until the response of the last one is received, right? Yes. Moreover, even documentation specifically talks about "idle keepalive connections to upstream servers that are preserved in the cache", see http://nginx.org/r/keepalive. > 3) If I'm still on the right track: because nginx has to wait the response > from a backend connection, it cannot reuse it for next request. That's why > nginx still needs to open new connections to backend even there are > existing connections. But It will only do so when all the existing > connection are non-idle (i.e. responses from them haven't been fully > received). Yes, see above. -- Maxim Dounin http://nginx.org/ From codeeply at gmail.com Fri May 9 02:17:24 2014 From: codeeply at gmail.com (Jianjun Zheng) Date: Fri, 9 May 2014 10:17:24 +0800 Subject: [PATCH] Upstream: add consistent hash module In-Reply-To: References: Message-ID: The commented out locking is to protect rrp.peers as other load balancing modules. If one would like to move rrp.peers to share memory, than the comment should be removed. 2014-05-08 23:23 GMT+08:00 Steven Hartland : > Be good to see this in core, currently we use the 3rd party module to > achieve this: > http://wiki.nginx.org/HttpUpstreamRequestHashModule > > One question on the patch, you appear to have some commented out locking is > this an oversight? > > Regards > Steve > > ----- Original Message ----- From: "Jianjun Zheng" > To: > Sent: Thursday, May 08, 2014 8:52 AM > Subject: [PATCH] Upstream: add consistent hash module > > > > usage example: cons_hash $request_uri; >> >> Not only consistent hash is introduced as a load balancing, but also >> increase the hits of cache in upstream servers. >> >> This modules is written according to round_robin, ip_hash >> and least_conn modules for upstream. So it inherits all the >> essential logic from them. >> >> This implementation is simple but has a high performance, >> The time comlexity remains O(log(T)) with a low constant coefficient, >> even when half of servers get down, >> regardless of the weight of servers, as analysis below. >> >> Suppose there are N upstream servers, with D down servers, >> and the weights are W1,W2,...,W(N-1). >> V is the amount of virtual nodes per unit. >> P is the probability to get an alive server from all nodes at random. >> >> Let T = (W1+W2+...+W(N-1))*V. >> then the space complexity is O(T), >> and the time comlexity is O(log(T) + (1-P)*N/(N-D)), >> 'N/(N-D)' of which is independent of W and V. >> >> >> # HG changeset patch >> # User Jianjun Zheng >> # Date 1399531525 -28800 >> # Thu May 08 14:45:25 2014 +0800 >> # Node ID 063f37f1654ef6cc03bd311a7fe6189b299ce2f2 >> # Parent 48c97d83ab7f0a3f641987fb32ace8af7720aefc >> Upstream: add consistent hash module >> >> diff -r 48c97d83ab7f -r 063f37f1654e auto/modules >> --- a/auto/modules Tue Apr 29 22:22:38 2014 +0200 >> +++ b/auto/modules Thu May 08 14:45:25 2014 +0800 >> @@ -381,6 +381,11 @@ >> HTTP_SRCS="$HTTP_SRCS $HTTP_UPSTREAM_LEAST_CONN_SRCS" >> fi >> >> +if [ $HTTP_UPSTREAM_CONS_HASH = YES ]; then >> + HTTP_MODULES="$HTTP_MODULES $HTTP_UPSTREAM_CONS_HASH_MODULE" >> + HTTP_SRCS="$HTTP_SRCS $HTTP_UPSTREAM_CONS_HASH_SRCS" >> +fi >> + >> if [ $HTTP_UPSTREAM_KEEPALIVE = YES ]; then >> HTTP_MODULES="$HTTP_MODULES $HTTP_UPSTREAM_KEEPALIVE_MODULE" >> HTTP_SRCS="$HTTP_SRCS $HTTP_UPSTREAM_KEEPALIVE_SRCS" >> diff -r 48c97d83ab7f -r 063f37f1654e auto/options >> --- a/auto/options Tue Apr 29 22:22:38 2014 +0200 >> +++ b/auto/options Thu May 08 14:45:25 2014 +0800 >> @@ -100,6 +100,7 @@ >> HTTP_GZIP_STATIC=NO >> HTTP_UPSTREAM_IP_HASH=YES >> HTTP_UPSTREAM_LEAST_CONN=YES >> +HTTP_UPSTREAM_CONS_HASH=YES >> HTTP_UPSTREAM_KEEPALIVE=YES >> >> # STUB >> diff -r 48c97d83ab7f -r 063f37f1654e auto/sources >> --- a/auto/sources Tue Apr 29 22:22:38 2014 +0200 >> +++ b/auto/sources Thu May 08 14:45:25 2014 +0800 >> @@ -504,6 +504,11 @@ >> src/http/modules/ngx_http_upstream_least_conn_module.c" >> >> >> +HTTP_UPSTREAM_CONS_HASH_MODULE=ngx_http_upstream_cons_hash_module >> +HTTP_UPSTREAM_CONS_HASH_SRCS=" \ >> + src/http/modules/ngx_http_upstream_cons_hash_module.c" >> + >> + >> HTTP_UPSTREAM_KEEPALIVE_MODULE=ngx_http_upstream_keepalive_module >> HTTP_UPSTREAM_KEEPALIVE_SRCS=" \ >> src/http/modules/ngx_http_upstream_keepalive_module.c" >> diff -r 48c97d83ab7f -r 063f37f1654e >> src/http/modules/ngx_http_upstream_cons_hash_module.c >> --- /dev/null Thu Jan 01 00:00:00 1970 +0000 >> +++ b/src/http/modules/ngx_http_upstream_cons_hash_module.c Thu May 08 >> 14:45:25 2014 +0800 >> @@ -0,0 +1,591 @@ >> +#include >> +#include >> +#include >> + >> + >> +#define NGX_HTTP_UPSTREAM_CH_VNODE_NUM 141 >> + >> + >> +typedef struct { >> + uint32_t hash; >> + >> + ngx_uint_t index; >> +} ngx_http_upstream_cons_hash_node_t; >> + >> + >> +typedef struct { >> + ngx_array_t *values; >> + ngx_array_t *lengths; >> + >> + ngx_uint_t node_number; >> + ngx_http_upstream_cons_hash_node_t *nodes; >> + >> + ngx_uint_t *nearest; >> + >> + ngx_http_upstream_rr_peers_t *peers; >> + >> + ngx_log_t *log; >> + >> + ngx_pool_t *pool; >> +} ngx_http_upstream_cons_hash_conf_t; >> + >> + >> +typedef struct { >> + /* the round robin data must be first */ >> + ngx_http_upstream_rr_peer_data_t rrp; >> + >> + ngx_uint_t found; >> + >> + ngx_http_upstream_cons_hash_conf_t *chcf; >> + >> + ngx_event_get_peer_pt get_rr_peer; >> +} ngx_http_upstream_ch_peer_data_t; >> + >> + >> +static void *ngx_http_upstream_cons_hash_create_conf(ngx_conf_t *cf); >> +static char *ngx_http_upstream_cons_hash(ngx_conf_t *cf, ngx_command_t >> *cmd, >> + void *conf); >> +static ngx_int_t ngx_http_upstream_init_cons_hash(ngx_conf_t *cf, >> + ngx_http_upstream_srv_conf_t *us); >> + >> +static ngx_int_t ngx_http_upstream_init_cons_ >> hash_peer(ngx_http_request_t >> *r, >> + ngx_http_upstream_srv_conf_t *us); >> +static ngx_int_t ngx_http_upstream_get_cons_hash_peer( >> + ngx_peer_connection_t *pc, void *data); >> + >> +static ngx_uint_t ngx_http_upstream_find_cons_hash_peer( >> + ngx_http_upstream_cons_hash_conf_t *chcf, uint32_t hash); >> +static int ngx_http_upstream_cons_hash_cmp_dist(const void *one, >> + const void *two); >> +static int ngx_http_upstream_cons_hash_cmp_node(const void *one, >> + const void *two); >> +static ngx_int_t ngx_http_upstream_cons_hash_random( >> + ngx_http_upstream_cons_hash_conf_t *chcf, ngx_str_t value, >> ngx_uint_t >> id, >> + uint32_t *ret); >> +static ngx_int_t ngx_http_upstream_cons_hash_init_nearest( >> + ngx_http_upstream_cons_hash_conf_t *chcf); >> + >> +inline static ngx_int_t ngx_http_upstream_get_cons_hash_try_peer( >> + ngx_peer_connection_t *pc, void *data, ngx_uint_t index); >> + >> + >> +static ngx_command_t ngx_http_upstream_cons_hash_commands[] = { >> + >> + { ngx_string("cons_hash"), >> + NGX_HTTP_UPS_CONF | NGX_CONF_TAKE1, >> + ngx_http_upstream_cons_hash, >> + 0, >> + 0, >> + NULL }, >> + >> + ngx_null_command >> +}; >> + >> + >> +static ngx_http_module_t ngx_http_upstream_cons_hash_module_ctx = { >> + NULL, /* preconfiguration */ >> + NULL, /* postconfiguration */ >> + >> + NULL, /* create main configuration */ >> + NULL, /* init main configuration */ >> + >> + ngx_http_upstream_cons_hash_create_conf, /* create server >> configuration*/ >> + NULL, /* merge server configuration >> */ >> + >> + NULL, /* create location >> configuration >> */ >> + NULL /* merge location configuration >> */ >> +}; >> + >> + >> +ngx_module_t ngx_http_upstream_cons_hash_module = { >> + NGX_MODULE_V1, >> + &ngx_http_upstream_cons_hash_module_ctx, /* module context */ >> + ngx_http_upstream_cons_hash_commands, /* module directives */ >> + NGX_HTTP_MODULE, /* module type */ >> + NULL, /* init master */ >> + NULL, /* init module */ >> + NULL, /* init process */ >> + NULL, /* init thread */ >> + NULL, /* exit thread */ >> + NULL, /* exit process */ >> + NULL, /* exit master */ >> + NGX_MODULE_V1_PADDING >> +}; >> + >> + >> +static void * >> +ngx_http_upstream_cons_hash_create_conf(ngx_conf_t *cf) >> +{ >> + ngx_http_upstream_cons_hash_conf_t *conf; >> + >> + conf = ngx_pcalloc(cf->pool, >> sizeof(ngx_http_upstream_cons_hash_conf_t)); >> + if (conf == NULL) { >> + return NULL; >> + } >> + >> + return conf; >> +} >> + >> + >> +static ngx_int_t >> +ngx_http_upstream_init_cons_hash(ngx_conf_t *cf, >> + ngx_http_upstream_srv_conf_t *us) >> +{ >> + uint32_t hash; >> + ngx_int_t rc; >> + ngx_str_t name; >> + ngx_uint_t i, j, k, n, nn, m; >> + ngx_http_upstream_rr_peers_t *peers; >> + ngx_http_upstream_cons_hash_conf_t *chcf; >> + >> + if (ngx_http_upstream_init_round_robin(cf, us) == NGX_ERROR) { >> + return NGX_ERROR; >> + } >> + >> + peers = us->peer.data; >> + >> + n = peers->number; >> + nn = 0; >> + >> + for (i = 0; i < n; ++i) { >> + nn += peers->peer[i].weight; >> + } >> + >> + nn *= (NGX_HTTP_UPSTREAM_CH_VNODE_NUM + 1); >> + >> + chcf = ngx_http_conf_upstream_srv_conf(us, >> + >> ngx_http_upstream_cons_hash_module); >> + >> + /* >> + * to guarantee nn % n == 0, there's no side effect, >> + * but much more convenient to construct the 'nearest' >> + */ >> + >> + nn = (nn + n - 1) / n * n; >> + chcf->node_number = nn; >> + >> + chcf->log = cf->log; >> + chcf->pool = cf->pool; >> + chcf->peers = peers; >> + >> + chcf->nodes = ngx_pcalloc(cf->pool, nn * >> + sizeof(ngx_http_upstream_cons_ >> hash_node_t)); >> + if (chcf->nodes == NULL) { >> + return NGX_ERROR; >> + } >> + >> + for (i = 0, k = 0; i < n; ++i) { >> + >> + name = peers->peer[i].name; >> + m = peers->peer[i].weight * (1 + NGX_HTTP_UPSTREAM_CH_VNODE_ >> NUM); >> + >> + for (j = 0; j < m; ++j, ++k) { >> + >> + chcf->nodes[k].index = i; >> + >> + rc = ngx_http_upstream_cons_hash_random(chcf, name, j, >> &hash); >> + if (rc == NGX_ERROR) { >> + return NGX_ERROR; >> + } >> + >> + chcf->nodes[k].hash = hash; >> + } >> + } >> + >> + for (i = 0; i < nn - k; ++i) { >> + chcf->nodes[k + i].index = chcf->nodes[0].index; >> + chcf->nodes[k + i].hash = chcf->nodes[0].hash; >> + } >> + >> + ngx_qsort(chcf->nodes, nn, sizeof(ngx_http_upstream_cons_ >> hash_node_t), >> + ngx_http_upstream_cons_hash_cmp_node); >> + >> + rc = ngx_http_upstream_cons_hash_init_nearest(chcf); >> + if (rc == NGX_ERROR) { >> + return NGX_ERROR; >> + } >> + >> + us->peer.init = ngx_http_upstream_init_cons_hash_peer; >> + >> + return NGX_OK; >> +} >> + >> + >> +static ngx_int_t >> +ngx_http_upstream_init_cons_hash_peer(ngx_http_request_t *r, >> + ngx_http_upstream_srv_conf_t *us) >> +{ >> + uint32_t hash; >> + ngx_str_t raw_value; >> + ngx_http_upstream_cons_hash_conf_t *chcf; >> + ngx_http_upstream_ch_peer_data_t *chp; >> + >> + chcf = ngx_http_conf_upstream_srv_conf(us, >> + >> ngx_http_upstream_cons_hash_module); >> + if (chcf == NULL) { >> + return NGX_ERROR; >> + } >> + >> + chp = ngx_pcalloc(r->pool, sizeof(ngx_http_upstream_ch_ >> peer_data_t)); >> + if (chp == NULL) { >> + return NGX_ERROR; >> + } >> + >> + r->upstream->peer.data = &chp->rrp; >> + >> + if (ngx_http_upstream_init_round_robin_peer(r, us) != NGX_OK) { >> + return NGX_ERROR; >> + } >> + >> + if (!chp->rrp.peers->single) { >> + >> + /* raw_value.data is allocated in ngx_http_script_run from >> r->pool >> */ >> + >> + if (ngx_http_script_run(r, &raw_value, >> + chcf->lengths->elts, 0, chcf->values->elts) == >> NULL) { >> + return NGX_ERROR; >> + } >> + >> + hash = ngx_murmur_hash2(raw_value.data, raw_value.len); >> + >> + ngx_pfree(r->pool, raw_value.data); >> + >> + chp->found = ngx_http_upstream_find_cons_hash_peer(chcf, hash); >> + } >> + >> + r->upstream->peer.get = ngx_http_upstream_get_cons_hash_peer; >> + >> + chp->chcf = chcf; >> + chp->get_rr_peer = ngx_http_upstream_get_round_robin_peer; >> + >> + return NGX_OK; >> +} >> + >> + >> +inline static ngx_int_t >> +ngx_http_upstream_get_cons_hash_try_peer(ngx_peer_connection_t *pc, void >> *data, >> + ngx_uint_t index) >> +{ >> + ngx_http_upstream_ch_peer_data_t *chp = data; >> + >> + time_t now; >> + ngx_int_t rc; >> + ngx_uint_t n, m; >> + ngx_http_upstream_rr_peer_t *peer; >> + >> + n = index / (8 * sizeof(uintptr_t)); >> + m = (uintptr_t) 1 << index % (8 * sizeof(uintptr_t)); >> + >> + if (chp->rrp.tried[n] & m) { >> + return NGX_AGAIN; >> + } >> + >> + rc = NGX_AGAIN; >> + >> + now = ngx_time(); >> + >> + peer = &chp->chcf->peers->peer[index]; >> + >> + /* ngx_lock_mutex(chp->rrp.peers->mutex); */ >> + >> + if (!peer->down) { >> + if (peer->max_fails == 0 || peer->fails < peer->max_fails) { >> + rc = NGX_OK; >> + } >> + >> + if (now - peer->checked > peer->fail_timeout) { >> + peer->checked = now; >> + rc = NGX_OK; >> + } >> + } >> + >> + if (rc == NGX_OK) { >> + chp->rrp.current = index; >> + >> + pc->sockaddr = peer->sockaddr; >> + pc->socklen = peer->socklen; >> + pc->name = &peer->name; >> + } >> + >> + /* ngx_unlock_mutex(chp->rrp.peers->mutex); */ >> + >> + chp->rrp.tried[n] |= m; >> + >> + return rc; >> +} >> + >> + >> +static ngx_int_t >> +ngx_http_upstream_get_cons_hash_peer(ngx_peer_connection_t *pc, void >> *data) >> +{ >> + ngx_http_upstream_ch_peer_data_t *chp = data; >> + >> + ngx_int_t rc; >> + ngx_uint_t i, j, n, nn; >> + ngx_uint_t *nearest; >> + ngx_http_upstream_rr_peers_t *peers; >> + ngx_http_upstream_cons_hash_node_t *nodes; >> + >> + if (chp->rrp.peers->single) { >> + return chp->get_rr_peer(pc, &chp->rrp); >> + } >> + >> + pc->cached = 0; >> + pc->connection = NULL; >> + >> + peers = chp->chcf->peers; >> + nodes = chp->chcf->nodes; >> + nearest = chp->chcf->nearest; >> + >> + n = peers->number; >> + nn = chp->chcf->node_number; >> + >> + for (i = chp->found; i % n != 0; i = (i + 1) % nn) { >> + >> + rc = ngx_http_upstream_get_cons_hash_try_peer(pc, data, >> + nodes[i].index); >> + if (rc == NGX_OK) { >> + return NGX_OK; >> + } >> + } >> + >> + for (j = (i + n) % nn; i != j; i = (i + 1) % nn) { >> + >> + rc = ngx_http_upstream_get_cons_hash_try_peer(pc, data, >> + nearest[i]); >> + if (rc == NGX_OK) { >> + return NGX_OK; >> + } >> + } >> + >> + /* all peers failed, mark them as live for quick recovery */ >> + >> + for (i = 0; i < peers->number; i++) { >> + peers->peer[i].fails = 0; >> + } >> + >> + pc->name = peers->name; >> + >> + return NGX_BUSY; >> +} >> + >> + >> +static ngx_uint_t >> +ngx_http_upstream_find_cons_hash_peer(ngx_http_upstream_cons_hash_conf_t >> *chcf, >> + uint32_t hash) >> +{ >> + uint32_t mid_hash; >> + ngx_int_t l, r, mid; >> + >> + l = 0; >> + r = chcf->node_number - 1; >> + >> + while (l <= r) { >> + mid = (l + r) >> 1; >> + mid_hash = chcf->nodes[mid].hash; >> + >> + if (mid_hash < hash) { >> + l = mid + 1; >> + } else { >> + r = mid - 1; >> + } >> + } >> + >> + if (l == (ngx_int_t)chcf->node_number) { >> + l = 0; >> + } >> + >> + return l; >> +} >> + >> + >> +static char * >> +ngx_http_upstream_cons_hash(ngx_conf_t *cf, ngx_command_t *cmd, void >> *conf) >> +{ >> + ngx_str_t *value; >> + ngx_http_script_compile_t sc; >> + ngx_http_upstream_srv_conf_t *uscf; >> + ngx_http_upstream_cons_hash_conf_t *chcf; >> + >> + uscf = ngx_http_conf_get_module_srv_conf(cf, >> ngx_http_upstream_module); >> + >> + chcf = ngx_http_conf_upstream_srv_conf(uscf, >> + >> ngx_http_upstream_cons_hash_module); >> + if (uscf->peer.init_upstream) { >> + ngx_conf_log_error(NGX_LOG_WARN, cf, 0, >> + "load balancing method redefined"); >> + } >> + uscf->peer.init_upstream = ngx_http_upstream_init_cons_hash; >> + >> + value = cf->args->elts; >> + ngx_memzero(&sc, sizeof(ngx_http_script_compile_t)); >> + >> + sc.cf = cf; >> + sc.source = &value[1]; >> + sc.lengths = &chcf->lengths;; >> + sc.values = &chcf->values; >> + sc.complete_lengths = 1; >> + sc.complete_values = 1; >> + >> + if (ngx_http_script_compile(&sc) != NGX_OK) { >> + return NGX_CONF_ERROR; >> + } >> + >> + uscf->flags = NGX_HTTP_UPSTREAM_CREATE >> + |NGX_HTTP_UPSTREAM_WEIGHT >> + |NGX_HTTP_UPSTREAM_MAX_FAILS >> + |NGX_HTTP_UPSTREAM_FAIL_TIMEOUT >> + |NGX_HTTP_UPSTREAM_DOWN; >> + >> + return NGX_CONF_OK; >> +} >> + >> + >> +static ngx_int_t >> +ngx_http_upstream_cons_hash_random(ngx_http_upstream_cons_hash_conf_t >> *chcf, >> + ngx_str_t value, ngx_uint_t id, uint32_t *ret) >> +{ >> + /* repeatable random with same (val, id) */ >> + >> + u_char *buf, *pos; >> + ngx_uint_t total; >> + >> + total = NGX_INT_T_LEN + value.len; >> + >> + buf = ngx_calloc(total, chcf->log); >> + if (buf == NULL) { >> + return NGX_ERROR; >> + } >> + >> + pos = ngx_snprintf(buf, total, "%i-%*s", id, value.len, value.data); >> + >> + *ret = ngx_murmur_hash2(buf, pos - buf); >> + >> + ngx_free(buf); >> + >> + return NGX_OK; >> +} >> + >> + >> +static ngx_int_t >> +ngx_http_upstream_cons_hash_init_nearest( >> + ngx_http_upstream_cons_hash_conf_t *chcf) >> +{ >> + ngx_int_t k; >> + ngx_uint_t i, j, n, nn; >> + ngx_uint_t *nearest, *temp; >> + ngx_http_upstream_cons_hash_node_t *nodes; >> + >> + n = chcf->peers->number; >> + nn = chcf->node_number; >> + >> + nodes = chcf->nodes; >> + >> + nearest = ngx_pcalloc(chcf->pool, nn * sizeof(ngx_uint_t)); >> + if (nearest == NULL) { >> + return NGX_ERROR; >> + } >> + >> + chcf->nearest = nearest; >> + >> + temp = ngx_pcalloc(chcf->pool, n * sizeof(ngx_uint_t)); >> + if (temp == NULL) { >> + return NGX_ERROR; >> + } >> + >> + for (i = 0; i < nn; ++i) { >> + nearest[i] = nn; >> + } >> + >> + for (i = 0; i < nn; i += n) { >> + for (j = 0; j < n; ++j) { >> + temp[j] = nn; >> + } >> + for (k = n - 1; k >= 0; --k) { >> + temp[nodes[i + k].index] = i + k; >> + } >> + for (j = 0; j < n; ++j) { >> + nearest[i + j] = temp[j]; >> + } >> + } >> + >> + ngx_pfree(chcf->pool, temp); >> + >> + /* update the 'nearest' twice */ >> + >> + for (i = 0; i < 2; ++i) { >> + for (k = nn - n; k >= 0; k -= n) { >> + for (j = 0; j < n; ++j) { >> + if (nearest[k + j] == nn) { >> + nearest[k + j] = nearest[(k + j + n) % nn]; >> + } >> + } >> + } >> + } >> + >> + for (i = 0; i < nn; i += n) { >> + >> + /* there is no elt equals to nn in the 'nearest' now */ >> + >> + for (j = 0; j < n; ++j) { >> + if (nearest[i + j] < i) { >> + nearest[i + j] += nn; >> + } >> + } >> + >> + ngx_qsort(nearest + i, n, sizeof(ngx_uint_t), >> + ngx_http_upstream_cons_hash_cmp_dist); >> + >> + for (j = 0; j < n; ++j) { >> + if (nearest[i + j] >= nn) { >> + nearest[i + j] -= nn; >> + } >> + } >> + } >> + >> + for (i = 0; i < nn; ++i) { >> + nearest[i] = nodes[nearest[i]].index; >> + } >> + >> + return NGX_OK; >> +} >> + >> + >> +static int >> +ngx_http_upstream_cons_hash_cmp_dist(const void *one, const void *two) >> +{ >> + ngx_uint_t first, second; >> + >> + first = *(ngx_uint_t *)one; >> + second = *(ngx_uint_t *)two; >> + >> + if (first < second) { >> + return -1; >> + } >> + >> + if (first > second) { >> + return 1; >> + } >> + >> + return 0; >> +} >> + >> + >> +static int >> +ngx_http_upstream_cons_hash_cmp_node(const void *one, const void *two) >> +{ >> + ngx_http_upstream_cons_hash_node_t *first, *second; >> + >> + first = (ngx_http_upstream_cons_hash_node_t *) one; >> + second = (ngx_http_upstream_cons_hash_node_t *) two; >> + >> + if (first->hash < second->hash) { >> + return -1; >> + } >> + >> + if (first->hash > second->hash) { >> + return 1; >> + } >> + >> + return 0; >> +} >> >> > > ------------------------------------------------------------ > -------------------- > > > _______________________________________________ >> nginx-devel mailing list >> nginx-devel at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx-devel >> > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Fri May 9 03:12:39 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 9 May 2014 07:12:39 +0400 Subject: Fwd: Question about slab allocator In-Reply-To: <536A8CAF.3070806@gmail.com> References: <536A8B98.5030500@gmail.com> <536A8CAF.3070806@gmail.com> Message-ID: <20140509031239.GM1849@mdounin.ru> Hello! On Wed, May 07, 2014 at 12:42:39PM -0700, Shuxin Yang wrote: > I'm nginx newbie. I'm reading src/core/ngx_slab.c, and am confused > as to > the purpose of NGX_SLAB_PAGE_START. > > As far as I can understand, when allocating a block, the most > significant > bit (MSB) of first page's corresponding ngx_slab_page_s::slab is set "1". > as we can see from : > > cat -n ngx_slab.c at pristine-1.7-release > 644 page->slab = pages | NGX_SLAB_PAGE_START; > > However, the MSB of is cleared later on: > 362 } else if (shift == ngx_slab_exact_shift) { > 363 > 364 page->slab = 1; > > So, what is the purpose of NGX_SLAB_PAGE_START (i.e the MSB of the > field slab)? The NGX_SLAB_PAGE_START is only used for multi-page allocations (to mark a starting page of a multi-page allocation). In other cases it's overwritten, and that's fine. -- Maxim Dounin http://nginx.org/ From mdounin at mdounin.ru Fri May 9 03:49:19 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 9 May 2014 07:49:19 +0400 Subject: [PATCH 1 of 2] auto/type/sizeof: rework autotest to be cross-compilation friendly In-Reply-To: References: Message-ID: <20140509034919.GN1849@mdounin.ru> Hello! On Sun, May 04, 2014 at 08:08:29AM +0200, Samuel Martin wrote: > # HG changeset patch > # User Samuel Martin > # Date 1398374852 -7200 > # Thu Apr 24 23:27:32 2014 +0200 > # Node ID f0f3f5208b12c6afbbdfb55b32c968ebd7206ad5 > # Parent b9553b4b8e670a0231afc0484f23953c0d8b5f22 > auto/type/sizeof: rework autotest to be cross-compilation friendly > > Rework the sizeof test to do the checks at compile time instead of at > runtime. This way, it does not break when cross-compiling for a > different CPU architecture. No. Apart from various problems with the patch like breaking ./configure output, this approach (parsing error messages of a compiler) is not portable. E.g., it won't work with MSVC. -- Maxim Dounin http://nginx.org/ From mdounin at mdounin.ru Fri May 9 03:55:07 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 9 May 2014 07:55:07 +0400 Subject: [PATCH 3 of 3] auto/lib/*: prepend include location with sysroot path In-Reply-To: References: Message-ID: <20140509035506.GO1849@mdounin.ru> Hello! On Sun, May 04, 2014 at 09:33:55PM +0200, Samuel Martin wrote: > # HG changeset patch > # User Samuel Martin > # Date 1399231473 -7200 > # Sun May 04 21:24:33 2014 +0200 > # Node ID b9787fdc8bac3559fd9da679dc04e8fc50cb74f3 > # Parent 03617a96c528d1f02abf829df964fcb29b06c942 > auto/lib/*: prepend include location with sysroot path > > This patch introduce a new NGX_SYSROOT variable used when > cross-compiling with gcc. > > When cross-compiling nginx, the gcc-based cross-compiler will > automatically set the NGX_SYSROOT value to the right value. This > variable remains empty if not cross-compiling or no gcc compiler is > detected. > This NGX_SYSROOT is prepended to the ngx_feature_path content when > additional location should be added to the include directory list. > > This is necessary to prevent gcc from complaining about unsafe include > location for cross-compilation (-Wpoision-system-directories), currently > only triggered by libxslt. > > diff -r 03617a96c528 -r b9787fdc8bac auto/cc/gcc > --- a/auto/cc/gcc Sun May 04 00:40:49 2014 +0200 > +++ b/auto/cc/gcc Sun May 04 21:24:33 2014 +0200 > @@ -15,6 +15,10 @@ > > have=NGX_COMPILER value="\"gcc $NGX_GCC_VER\"" . auto/define > > +# set sysroot in case of cross-compilation > +if [ x$NGX_CROSSBUILD = xcrossbuild ]; then > + NGX_SYSROOT=`$CC -print-sysroot` > +fi $ gcc -print-sysroot gcc: unrecognized option '-print-sysroot' gcc: No input files specified -- Maxim Dounin http://nginx.org/ From mdounin at mdounin.ru Fri May 9 04:04:43 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 9 May 2014 08:04:43 +0400 Subject: [PATCH 2 of 2] auto/feature: disable feature autotest when cross-compiling In-Reply-To: <03617a96c528d1f02abf.1399183710@bobook.home> References: <03617a96c528d1f02abf.1399183710@bobook.home> Message-ID: <20140509040443.GP1849@mdounin.ru> Hello! On Sun, May 04, 2014 at 08:08:30AM +0200, Samuel Martin wrote: > # HG changeset patch > # User Samuel Martin > # Date 1399156849 -7200 > # Sun May 04 00:40:49 2014 +0200 > # Node ID 03617a96c528d1f02abf829df964fcb29b06c942 > # Parent f0f3f5208b12c6afbbdfb55b32c968ebd7206ad5 > auto/feature: disable feature autotest when cross-compiling > > This patch introduce a new NGX_CROSSBUILD configure variable that records > whether --crossbuild=... option was passed on the command line. > > Then this variable is used to disable the feature runtest because they > cannot usually be executed when cross-compiling. No. This is very bad approach, as it silently assumes that every test which can be compiled succeeds. Not to mention that is't just wrong for "value" and "bug" tests. -- Maxim Dounin http://nginx.org/ From mdounin at mdounin.ru Fri May 9 04:17:56 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 9 May 2014 08:17:56 +0400 Subject: [PATCH] Added nonlocal to the listen directive In-Reply-To: <20140501134251.Horde.ANn5O3mbOALn1gKPafFulg1@webmail.your-server.de> References: <16eacd8609c8362e9dd7.1395999953@miitool> <20140328122528.GL34696@mdounin.ru> <20140427221012.Horde.pUFTrYPMwbaBfz1QpJpkjA1@webmail.your-server.de> <20140501134251.Horde.ANn5O3mbOALn1gKPafFulg1@webmail.your-server.de> Message-ID: <20140509041756.GR1849@mdounin.ru> Hello! On Thu, May 01, 2014 at 01:42:51PM +0200, info at kliemeck.de wrote: > Hey, > > i thought that this is important but I have received no response. Any update > on this? Much like with ipv4, just bind on the ipv6 address you want _and_ [::]. > > greets > Hans-Joachim > > Quoting info at kliemeck.de: > > >Hey, > > > >but it is still not possible to work with IPv6, if you want to bind to a > >specific address (not [::]) that is not a local address. The > >"ip_nonlocal_bind-sysctl" use-case is not fulfilled with this and i think > >it is a common use-case that nginx is used within a high availability > >environment with a shared ip address. It is possible that this important > >feature is integrated within 1.6, since it may be a reason not to use > >IPv6? > > > >greets > >Hans-Joachim Kliemeck > > > >Quoting mdounin at mdounin.ru: > > > >>Hello! > >> > >>On Fri, Mar 28, 2014 at 10:45:53AM +0100, Trygve Vea wrote: > >> > >>># HG changeset patch > >>># User Trygve Vea > >>># Date 1395999940 -3600 > >>># Fri Mar 28 10:45:40 2014 +0100 > >>># Node ID 16eacd8609c8362e9dd729c743ed7a869c2993fe > >>># Parent 2411d4b5be2ca690a5a00a1d8ad96ff69a00317f > >>>Added nonlocal to the listen directive > >>> > >>>The nonlocal option is used to set the needed socket options to be > >>>able to bind > >>>to an address not necessarily owned by the host. > >>> > >>>This patch currently implements this for Linux >= 2.4 IPv4/IPv6. > >>> > >>>The problem we solve by doing this, is in an environment where the > >>>following > >>>conditions are met: > >>> > >>>* HTTPS with multiple certificates, and a client base that are unable > >>>to use > >>> SNI - thus having the need to tie specific certificates to specific > >>>ip/ports. > >>>* Setting the ip_nonlocal_bind-sysctl is not an option (for example > >>>for Linux > >>> IPv6) > >>>* Used in a failover-setup, where the service IP-addresses are moved > >>>around by > >>> a daemon like linux-ha or keepalived. > >> > >>As already explained, the patch is not needed for the use case > >>claimed. Just a bind on INADDR_ANY/IN6ADDR_ANY will do the trick. > >> > >>-- > >>Maxim Dounin > >>http://nginx.org/ > > > > > > > >_______________________________________________ > >nginx-devel mailing list > >nginx-devel at nginx.org > >http://mailman.nginx.org/mailman/listinfo/nginx-devel > > > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel -- Maxim Dounin http://nginx.org/ From s.martin49 at gmail.com Fri May 9 12:13:00 2014 From: s.martin49 at gmail.com (Samuel Martin) Date: Fri, 9 May 2014 14:13:00 +0200 Subject: [PATCH 1 of 2] auto/type/sizeof: rework autotest to be cross-compilation friendly In-Reply-To: <20140509034919.GN1849@mdounin.ru> References: <20140509034919.GN1849@mdounin.ru> Message-ID: Hi Maxim, all, Overall, thanks a lot for your feedback. I'm doing this in order to get nginx integrated in Buildroot [1], so it would be automatically tested by the Buildroot buildfarms [2]. I'd like to know if there is an interest from nginx team to get a proper cross-compilation support. - if yes, I'll do my best to help on it (Buildroot developers try to push upstream as much as possible the fixes they do); - if no, do the config files touch by this series often change? BTW, I have to admit the series is mostly (only) for gcc, maybe only for linux target because this is my main concerns, and I don't have access to all the hardware/platforms/compilers nginx intends to support. Patch comments inlined. On Fri, May 9, 2014 at 5:49 AM, Maxim Dounin wrote: > Hello! > > On Sun, May 04, 2014 at 08:08:29AM +0200, Samuel Martin wrote: > >> # HG changeset patch >> # User Samuel Martin >> # Date 1398374852 -7200 >> # Thu Apr 24 23:27:32 2014 +0200 >> # Node ID f0f3f5208b12c6afbbdfb55b32c968ebd7206ad5 >> # Parent b9553b4b8e670a0231afc0484f23953c0d8b5f22 >> auto/type/sizeof: rework autotest to be cross-compilation friendly >> >> Rework the sizeof test to do the checks at compile time instead of at >> runtime. This way, it does not break when cross-compiling for a >> different CPU architecture. > > No. > > Apart from various problems with the patch like breaking > ./configure output, this approach (parsing error messages of a > compiler) is not portable. E.g., it won't work with MSVC. Right, instead of parsing the compiler outputs of one file, I think I can come up with a solution based on testing the compilation status of several files. Regards, -- Samuel From s.martin49 at gmail.com Fri May 9 12:13:12 2014 From: s.martin49 at gmail.com (Samuel Martin) Date: Fri, 9 May 2014 14:13:12 +0200 Subject: [PATCH 2 of 2] auto/feature: disable feature autotest when cross-compiling In-Reply-To: <20140509040443.GP1849@mdounin.ru> References: <03617a96c528d1f02abf.1399183710@bobook.home> <20140509040443.GP1849@mdounin.ru> Message-ID: Hi Maxim, all, On Fri, May 9, 2014 at 6:04 AM, Maxim Dounin wrote: > Hello! > > On Sun, May 04, 2014 at 08:08:30AM +0200, Samuel Martin wrote: > >> # HG changeset patch >> # User Samuel Martin >> # Date 1399156849 -7200 >> # Sun May 04 00:40:49 2014 +0200 >> # Node ID 03617a96c528d1f02abf829df964fcb29b06c942 >> # Parent f0f3f5208b12c6afbbdfb55b32c968ebd7206ad5 >> auto/feature: disable feature autotest when cross-compiling >> >> This patch introduce a new NGX_CROSSBUILD configure variable that records >> whether --crossbuild=... option was passed on the command line. >> >> Then this variable is used to disable the feature runtest because they >> cannot usually be executed when cross-compiling. > > No. > > This is very bad approach, as it silently assumes that every test > which can be compiled succeeds. Note that, when doing cross-compilation, one knows (or should better know) exactly what he/she is compiling. So, it's usual (at least ok) to edit or generate the config.h file (or equivalent files) used at build time. > > Not to mention that is't just wrong for "value" and "bug" tests. To be honest, I did not really check what this is. AFAICS, there is: - only 1 feature check using "bug", which is for darwin, so not really relevant to me (though I can try to find some solution for this, but I have no way to test it); - only 1 feature (but up-to 3 checks) using "value", this is about the sys_nerr, sys_errlist support in the C-library. However AFAICS, these checks are useless because nginx does implement its own ngx_sys_errlist, and never uses the one from the C-library. Regards, -- Samuel From s.martin49 at gmail.com Fri May 9 12:18:19 2014 From: s.martin49 at gmail.com (Samuel Martin) Date: Fri, 9 May 2014 14:18:19 +0200 Subject: [PATCH 3 of 3] auto/lib/*: prepend include location with sysroot path In-Reply-To: <20140509035506.GO1849@mdounin.ru> References: <20140509035506.GO1849@mdounin.ru> Message-ID: Hi Maxim, all, On Fri, May 9, 2014 at 5:55 AM, Maxim Dounin wrote: > Hello! > > On Sun, May 04, 2014 at 09:33:55PM +0200, Samuel Martin wrote: > >> # HG changeset patch >> # User Samuel Martin >> # Date 1399231473 -7200 >> # Sun May 04 21:24:33 2014 +0200 >> # Node ID b9787fdc8bac3559fd9da679dc04e8fc50cb74f3 >> # Parent 03617a96c528d1f02abf829df964fcb29b06c942 >> auto/lib/*: prepend include location with sysroot path >> >> This patch introduce a new NGX_SYSROOT variable used when >> cross-compiling with gcc. >> >> When cross-compiling nginx, the gcc-based cross-compiler will >> automatically set the NGX_SYSROOT value to the right value. This >> variable remains empty if not cross-compiling or no gcc compiler is >> detected. >> This NGX_SYSROOT is prepended to the ngx_feature_path content when >> additional location should be added to the include directory list. >> >> This is necessary to prevent gcc from complaining about unsafe include >> location for cross-compilation (-Wpoision-system-directories), currently >> only triggered by libxslt. >> >> diff -r 03617a96c528 -r b9787fdc8bac auto/cc/gcc >> --- a/auto/cc/gcc Sun May 04 00:40:49 2014 +0200 >> +++ b/auto/cc/gcc Sun May 04 21:24:33 2014 +0200 >> @@ -15,6 +15,10 @@ >> >> have=NGX_COMPILER value="\"gcc $NGX_GCC_VER\"" . auto/define >> >> +# set sysroot in case of cross-compilation >> +if [ x$NGX_CROSSBUILD = xcrossbuild ]; then >> + NGX_SYSROOT=`$CC -print-sysroot` >> +fi > > $ gcc -print-sysroot > gcc: unrecognized option '-print-sysroot' > gcc: No input files specified Is this printed in stderr or stdout? What gcc version are you using? on what distro? Even on a old debian squeeze, gcc 4.4.5 supports this option and answers nothing in stdout: --- $ gcc --version gcc (Debian 4.4.5-8) 4.4.5 Copyright (C) 2010 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. $ gcc -print-sysroot $ echo $? 0 $ --- Regards, -- Samuel From agentzh at gmail.com Mon May 12 05:13:52 2014 From: agentzh at gmail.com (Yichun Zhang (agentzh)) Date: Sun, 11 May 2014 22:13:52 -0700 Subject: [PATCH] Core: merge adjacent free slab pages to ameliorate fragmentation from multi-page blocks (Was Re: Help with shared memory usage) Message-ID: Hello! On Mon, Jul 29, 2013 at 10:11 AM, Maxim Dounin wrote: > Additionally, doing a full merge of all free blocks on a free > operation looks too much. It might be something we want to do on > allocation failure, but not on a normal path in > ngx_slab_free_pages(). And/or something lightweight may be done > in ngx_slab_free_pages(), e.g., checking if pages following pages > we are freeing are free too, and merging them in this case. > I'd propose an alternative patch taking the second approach, that is, merging adjacent free pages (for both the previous and next blocks) in ngx_slab_free_pages(). This approach has the following advantages: 1. It can effectively distribute the merging computations across all the page free operations, which can prevent potential frequent and long stalls when actually running out of large enough free blocks along the "free" list that is already very long for large zones (which usually consists of tons of one-page blocks upon allocation failures). 2. it can also make multi-page allocations generally faster because we're merging pages immediately when we can and thus it's more likely to find large enough free blocks along the (relatively short) free list for ngx_slab_alloc_pages(). The only downside is that I have to introduce an extra field "prev_slab" (8-byte for x86_64) in ngx_slab_page_t in my patch, which makes the slab page metadata a bit larger. Feedback welcome! Thanks! -agentzh # HG changeset patch # User Yichun Zhang # Date 1399870567 25200 # Sun May 11 21:56:07 2014 -0700 # Node ID 93614769dd4b6df8844c3c43c6a0b3f83bfa6746 # Parent 48c97d83ab7f0a3f641987fb32ace8af7720aefc Core: merge adjacent free slab pages to ameliorate fragmentation from multi-page blocks. diff -r 48c97d83ab7f -r 93614769dd4b src/core/ngx_slab.c --- a/src/core/ngx_slab.c Tue Apr 29 22:22:38 2014 +0200 +++ b/src/core/ngx_slab.c Sun May 11 21:56:07 2014 -0700 @@ -111,6 +111,7 @@ ngx_memzero(p, pages * sizeof(ngx_slab_page_t)); pool->pages = (ngx_slab_page_t *) p; + pool->npages = pages; pool->free.prev = 0; pool->free.next = (ngx_slab_page_t *) p; @@ -118,6 +119,7 @@ pool->pages->slab = pages; pool->pages->next = &pool->free; pool->pages->prev = (uintptr_t) &pool->free; + pool->pages->prev_slab = 0; pool->start = (u_char *) ngx_align_ptr((uintptr_t) p + pages * sizeof(ngx_slab_page_t), @@ -626,9 +628,16 @@ if (page->slab >= pages) { if (page->slab > pages) { + /* adjust the next adjacent block's "prev_slab" field */ + p = &page[page->slab]; + if (p < pool->pages + pool->npages) { + p->prev_slab = page->slab - pages; + } + page[pages].slab = page->slab - pages; page[pages].next = page->next; page[pages].prev = page->prev; + page[pages].prev_slab = pages; p = (ngx_slab_page_t *) page->prev; p->next = &page[pages]; @@ -652,6 +661,7 @@ p->slab = NGX_SLAB_PAGE_BUSY; p->next = NULL; p->prev = NGX_SLAB_PAGE; + p->prev_slab = 0; p++; } @@ -672,7 +682,7 @@ ngx_slab_free_pages(ngx_slab_pool_t *pool, ngx_slab_page_t *page, ngx_uint_t pages) { - ngx_slab_page_t *prev; + ngx_slab_page_t *prev, *p; page->slab = pages--; @@ -686,6 +696,53 @@ page->next->prev = page->prev; } + /* merge the next adjacent free block if it is free */ + + p = &page[page->slab]; + if (p < pool->pages + pool->npages + && !(p->slab & NGX_SLAB_PAGE_START) + && p->next != NULL + && (p->prev & NGX_SLAB_PAGE_MASK) == NGX_SLAB_PAGE) + { + page->slab += p->slab; + + /* remove the next adjacent block from the free list */ + + prev = (ngx_slab_page_t *) p->prev; + prev->next = p->next; + p->next->prev = p->prev; + + /* adjust the "prev_slab" field in the next next adjacent block */ + if (p + p->slab < pool->pages + pool->npages) { + p[p->slab].prev_slab = page->slab; + } + + ngx_memzero(p, sizeof(ngx_slab_page_t)); + } + + if (page->prev_slab) { + /* merge the previous adjacent block if it is free */ + + p = page - page->prev_slab; + if (!(p->slab & NGX_SLAB_PAGE_START) + && p->next != NULL + && (p->prev & NGX_SLAB_PAGE_MASK) == NGX_SLAB_PAGE) + { + /* p->slab == page->prev_slab */ + + p->slab += page->slab; + ngx_memzero(page, sizeof(ngx_slab_page_t)); + + /* adjust the "prev_slab" field in the next adjacent block */ + if (p + p->slab < pool->pages + pool->npages) { + p[p->slab].prev_slab = p->slab; + } + + /* skip adding "page" to the free list */ + return; + } + } + page->prev = (uintptr_t) &pool->free; page->next = pool->free.next; diff -r 48c97d83ab7f -r 93614769dd4b src/core/ngx_slab.h --- a/src/core/ngx_slab.h Tue Apr 29 22:22:38 2014 +0200 +++ b/src/core/ngx_slab.h Sun May 11 21:56:07 2014 -0700 @@ -19,6 +19,8 @@ uintptr_t slab; ngx_slab_page_t *next; uintptr_t prev; + uintptr_t prev_slab; + /* number of pages for the previous adjacent block */ }; @@ -31,6 +33,8 @@ ngx_slab_page_t *pages; ngx_slab_page_t free; + ngx_uint_t npages; + u_char *start; u_char *end; -------------- next part -------------- A non-text attachment was scrubbed... Name: nginx-slab-defrag.patch Type: text/x-patch Size: 4236 bytes Desc: not available URL: From mdounin at mdounin.ru Tue May 13 13:02:40 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 13 May 2014 17:02:40 +0400 Subject: [PATCH] Upstream: add consistent hash module In-Reply-To: References: Message-ID: <20140513130240.GO1849@mdounin.ru> Hello! On Thu, May 08, 2014 at 03:52:24PM +0800, Jianjun Zheng wrote: > usage example: cons_hash $request_uri; > > Not only consistent hash is introduced as a load balancing, but also > increase the hits of cache in upstream servers. > > This modules is written according to round_robin, ip_hash > and least_conn modules for upstream. So it inherits all the > essential logic from them. We are quite conservative at adding 3rd party modules into nginx base, sorry. Though it may be good idea to add your module to the 3rd party modules list at wiki, see here: http://wiki.nginx.org/3rdPartyModules I believe there are at least couple of consistent hash modules there already (and memcached_hash[1] seems to be not listed there, while it's oldest consistent hash module for nginx as far as I know). [1] http://openhack.ru/nginx-patched/wiki/MemcachedHash -- Maxim Dounin http://nginx.org/ From mdounin at mdounin.ru Tue May 13 18:11:07 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 13 May 2014 22:11:07 +0400 Subject: [PATCH 1 of 2] auto/type/sizeof: rework autotest to be cross-compilation friendly In-Reply-To: References: <20140509034919.GN1849@mdounin.ru> Message-ID: <20140513181107.GS1849@mdounin.ru> Hello! On Fri, May 09, 2014 at 02:13:00PM +0200, Samuel Martin wrote: > Hi Maxim, all, > > Overall, thanks a lot for your feedback. > > I'm doing this in order to get nginx integrated in Buildroot [1], so > it would be automatically tested by the Buildroot buildfarms [2]. > > I'd like to know if there is an interest from nginx team to get a > proper cross-compilation support. > - if yes, I'll do my best to help on it (Buildroot developers try to > push upstream as much as possible the fixes they do); While cross-compilation is something good to have, we don't consider it to be a "must-have" feature. And if added, we would like it to be done a) with minimal intrusion, and b) properly. I don't think that some-gcc-only non-portable changes are welcomed, especially if result requires manual editing. (Cross-compilation was actually discussed several times here, and I think it should be done by adding some ability to supply pre-computed results for tests which require running.) > - if no, do the config files touch by this series often change? Not really. In last 365 days, only 25 changesets touched auto/: $ hg log --date -365 auto/ | grep 'changeset:' | wc -l 25 -- Maxim Dounin http://nginx.org/ From mdounin at mdounin.ru Tue May 13 18:12:21 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 13 May 2014 22:12:21 +0400 Subject: [PATCH 3 of 3] auto/lib/*: prepend include location with sysroot path In-Reply-To: References: <20140509035506.GO1849@mdounin.ru> Message-ID: <20140513181221.GT1849@mdounin.ru> Hello! On Fri, May 09, 2014 at 02:18:19PM +0200, Samuel Martin wrote: > Hi Maxim, all, > > On Fri, May 9, 2014 at 5:55 AM, Maxim Dounin wrote: > > Hello! > > > > On Sun, May 04, 2014 at 09:33:55PM +0200, Samuel Martin wrote: > > > >> # HG changeset patch > >> # User Samuel Martin > >> # Date 1399231473 -7200 > >> # Sun May 04 21:24:33 2014 +0200 > >> # Node ID b9787fdc8bac3559fd9da679dc04e8fc50cb74f3 > >> # Parent 03617a96c528d1f02abf829df964fcb29b06c942 > >> auto/lib/*: prepend include location with sysroot path > >> > >> This patch introduce a new NGX_SYSROOT variable used when > >> cross-compiling with gcc. > >> > >> When cross-compiling nginx, the gcc-based cross-compiler will > >> automatically set the NGX_SYSROOT value to the right value. This > >> variable remains empty if not cross-compiling or no gcc compiler is > >> detected. > >> This NGX_SYSROOT is prepended to the ngx_feature_path content when > >> additional location should be added to the include directory list. > >> > >> This is necessary to prevent gcc from complaining about unsafe include > >> location for cross-compilation (-Wpoision-system-directories), currently > >> only triggered by libxslt. > >> > >> diff -r 03617a96c528 -r b9787fdc8bac auto/cc/gcc > >> --- a/auto/cc/gcc Sun May 04 00:40:49 2014 +0200 > >> +++ b/auto/cc/gcc Sun May 04 21:24:33 2014 +0200 > >> @@ -15,6 +15,10 @@ > >> > >> have=NGX_COMPILER value="\"gcc $NGX_GCC_VER\"" . auto/define > >> > >> +# set sysroot in case of cross-compilation > >> +if [ x$NGX_CROSSBUILD = xcrossbuild ]; then > >> + NGX_SYSROOT=`$CC -print-sysroot` > >> +fi > > > > $ gcc -print-sysroot > > gcc: unrecognized option '-print-sysroot' > > gcc: No input files specified > > Is this printed in stderr or stdout? > > What gcc version are you using? on what distro? > Even on a old debian squeeze, gcc 4.4.5 supports this option and > answers nothing in stdout: This comment is not about gcc version I'm using, this is about the fact that "-print-sysroot" doesn't work with all gcc versions. And nginx supports gcc starting with at least 2.95 (and you can even see code about gcc 2.7 in cc/gcc). -- Maxim Dounin http://nginx.org/ From phoem at phoempc.com Wed May 14 01:20:07 2014 From: phoem at phoempc.com (Jordan Newman) Date: Tue, 13 May 2014 21:20:07 -0400 Subject: Worker ID Message-ID: <83AB555D-725A-479A-AFB6-0C707E020435@phoempc.com> Is there a way from an nginx module to see which # worker it is? Thank you. -- Jordan Newman From wandenberg at gmail.com Wed May 14 02:51:39 2014 From: wandenberg at gmail.com (Wandenberg Peixoto) Date: Tue, 13 May 2014 23:51:39 -0300 Subject: Worker ID In-Reply-To: <83AB555D-725A-479A-AFB6-0C707E020435@phoempc.com> References: <83AB555D-725A-479A-AFB6-0C707E020435@phoempc.com> Message-ID: Check the nginx_process_slot or nginx_process_pid On May 13, 2014 10:20 PM, "Jordan Newman" wrote: > Is there a way from an nginx module to see which # worker it is? Thank you. > > -- > Jordan Newman > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yurnerola at gmail.com Wed May 14 02:59:59 2014 From: yurnerola at gmail.com (liubin) Date: Wed, 14 May 2014 10:59:59 +0800 Subject: Maybe Comment Bug Message-ID: <0A2CE991-0861-42E7-80EE-F6B346066EAF@gmail.com> Hi: As i new to mp4 file format,i confused about the comment. "send original file if moov atom resides before mdat atom and client requests integral file? Should the ?before? should be changed with ?after?? Sorry about my poor English. ngx_http_mp4_module.c static ngx_int_t ngx_http_mp4_read_moov_atom(ngx_http_mp4_file_t *mp4, uint64_t atom_data_size) { ngx_int_t rc; ngx_uint_t no_mdat; ngx_buf_t *atom; ngx_http_mp4_conf_t *conf; ngx_log_debug0(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0, "mp4 moov atom"); no_mdat = (mp4->mdat_atom.buf == NULL); if (no_mdat && mp4->start == 0 && mp4->length == 0) { /* * send original file if moov atom resides before * mdat atom and client requests integral file */ return NGX_DECLINED; } Best regards, -yurnero -------------- next part -------------- An HTML attachment was scrubbed... URL: From arut at nginx.com Wed May 14 05:53:09 2014 From: arut at nginx.com (Roman Arutyunyan) Date: Wed, 14 May 2014 09:53:09 +0400 Subject: Maybe Comment Bug In-Reply-To: <0A2CE991-0861-42E7-80EE-F6B346066EAF@gmail.com> References: <0A2CE991-0861-42E7-80EE-F6B346066EAF@gmail.com> Message-ID: Hello, Obviously, "no_mdat" is true if moov resides before mdat. On May 14, 2014, at 6:59 AM, liubin wrote: > Hi: > As i new to mp4 file format,i confused about the comment. > "send original file if moov atom resides before mdat atom and client requests integral file? > Should the ?before? should be changed with ?after?? > Sorry about my poor English. > > > ngx_http_mp4_module.c > > static ngx_int_t > ngx_http_mp4_read_moov_atom(ngx_http_mp4_file_t *mp4, uint64_t atom_data_size) > { > ngx_int_t rc; > ngx_uint_t no_mdat; > ngx_buf_t *atom; > ngx_http_mp4_conf_t *conf; > > ngx_log_debug0(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0, "mp4 moov atom"); > > no_mdat = (mp4->mdat_atom.buf == NULL); > > if (no_mdat && mp4->start == 0 && mp4->length == 0) { > /* > * send original file if moov atom resides before > * mdat atom and client requests integral file > */ > return NGX_DECLINED; > } > > > Best regards, > -yurnero > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel From citrin at citrin.ru Wed May 14 11:20:54 2014 From: citrin at citrin.ru (Anton Yuzhaninov) Date: Wed, 14 May 2014 15:20:54 +0400 Subject: [PATCH 3 of 3] auto/lib/*: prepend include location with sysroot path In-Reply-To: References: <20140509035506.GO1849@mdounin.ru> Message-ID: <53735196.6090009@citrin.ru> On 05/09/14 16:18, Samuel Martin wrote: > Even on a old debian squeeze, gcc 4.4.5 supports this option and > answers nothing in stdout: > --- > $ gcc --version > gcc (Debian 4.4.5-8) 4.4.5 > Copyright (C) 2010 Free Software Foundation, Inc. > This is free software; see the source for copying conditions. There is NO > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > $ gcc -print-sysroot > $ echo $? > 0 > $ > gcc -print-sysroot gcc: unrecognized option '-print-sysroot' gcc: No input files specified > gcc --version gcc (GCC) 4.2.1 20070831 patched [FreeBSD] Copyright (C) 2007 Free Software Foundation, Inc. ... gcc 4.2.1 is default complier in FreeBSD 9.x From mdounin at mdounin.ru Wed May 14 14:55:32 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 14 May 2014 18:55:32 +0400 Subject: Worker ID In-Reply-To: <83AB555D-725A-479A-AFB6-0C707E020435@phoempc.com> References: <83AB555D-725A-479A-AFB6-0C707E020435@phoempc.com> Message-ID: <20140514145532.GW1849@mdounin.ru> Hello! On Tue, May 13, 2014 at 09:20:07PM -0400, Jordan Newman wrote: > Is there a way from an nginx module to see which # worker it is? Thank you. If you need to know worker's number, you are likely doing something wrong. Note that there may be more than one set of workers running concurrently - e.g., after a configuration reload a new set of workers is spawned, and only after this old workes are gracefully shut down. -- Maxim Dounin http://nginx.org/ From ru at nginx.com Wed May 14 18:28:04 2014 From: ru at nginx.com (Ruslan Ermilov) Date: Wed, 14 May 2014 18:28:04 +0000 Subject: [nginx] Core: use '\r' for CR and '\n' for LF definitions. Message-ID: details: http://hg.nginx.org/nginx/rev/bc98b0f11bdd branches: changeset: 5684:bc98b0f11bdd user: Ruslan Ermilov date: Wed May 14 22:26:05 2014 +0400 description: Core: use '\r' for CR and '\n' for LF definitions. diffstat: src/core/ngx_core.h | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diffs (16 lines): diff -r 48c97d83ab7f -r bc98b0f11bdd src/core/ngx_core.h --- a/src/core/ngx_core.h Tue Apr 29 22:22:38 2014 +0200 +++ b/src/core/ngx_core.h Wed May 14 22:26:05 2014 +0400 @@ -80,9 +80,9 @@ typedef void (*ngx_connection_handler_pt #include -#define LF (u_char) 10 -#define CR (u_char) 13 -#define CRLF "\x0d\x0a" +#define LF (u_char) '\n' +#define CR (u_char) '\r' +#define CRLF "\r\n" #define ngx_abs(value) (((value) >= 0) ? (value) : - (value)) From ru at nginx.com Wed May 14 18:28:05 2014 From: ru at nginx.com (Ruslan Ermilov) Date: Wed, 14 May 2014 18:28:05 +0000 Subject: [nginx] Style: use %N instead of '\n' where appropriate. Message-ID: details: http://hg.nginx.org/nginx/rev/0cbefdcf82a6 branches: changeset: 5685:0cbefdcf82a6 user: Ruslan Ermilov date: Wed May 14 22:26:30 2014 +0400 description: Style: use %N instead of '\n' where appropriate. diffstat: src/http/modules/ngx_http_proxy_module.c | 2 +- src/mail/ngx_mail_auth_http_module.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diffs (24 lines): diff -r bc98b0f11bdd -r 0cbefdcf82a6 src/http/modules/ngx_http_proxy_module.c --- a/src/http/modules/ngx_http_proxy_module.c Wed May 14 22:26:05 2014 +0400 +++ b/src/http/modules/ngx_http_proxy_module.c Wed May 14 22:26:30 2014 +0400 @@ -1294,7 +1294,7 @@ ngx_http_proxy_create_request(ngx_http_r } ngx_log_debug2(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, - "http proxy header:\n\"%*s\"", + "http proxy header:%N\"%*s\"", (size_t) (b->last - b->pos), b->pos); if (plcf->body_set == NULL && plcf->upstream.pass_request_body) { diff -r bc98b0f11bdd -r 0cbefdcf82a6 src/mail/ngx_mail_auth_http_module.c --- a/src/mail/ngx_mail_auth_http_module.c Wed May 14 22:26:05 2014 +0400 +++ b/src/mail/ngx_mail_auth_http_module.c Wed May 14 22:26:30 2014 +0400 @@ -1269,7 +1269,7 @@ ngx_mail_auth_http_create_request(ngx_ma l.len = b->last - b->pos; l.data = b->pos; ngx_log_debug1(NGX_LOG_DEBUG_MAIL, s->connection->log, 0, - "mail auth http header:\n\"%V\"", &l); + "mail auth http header:%N\"%V\"", &l); } #endif From phoem at phoempc.com Thu May 15 01:10:04 2014 From: phoem at phoempc.com (Jordan Newman) Date: Wed, 14 May 2014 21:10:04 -0400 Subject: Worker ID In-Reply-To: <20140514145532.GW1849@mdounin.ru> References: <83AB555D-725A-479A-AFB6-0C707E020435@phoempc.com> <20140514145532.GW1849@mdounin.ru> Message-ID: <2F230A56-EC9A-433B-A3C7-58D96E4544A4@phoempc.com> Thanks, ngx_process_slot was what I needed. The reason for knowing the worker # is for stat tracking, its important that the backend where the stats are being collected on knows to group them together by worker and then by server. -- Jordan Newman > On May 14, 2014, at 10:55 AM, Maxim Dounin wrote: > > Hello! > >> On Tue, May 13, 2014 at 09:20:07PM -0400, Jordan Newman wrote: >> >> Is there a way from an nginx module to see which # worker it is? Thank you. > > If you need to know worker's number, you are likely doing > something wrong. > > Note that there may be more than one set of workers running > concurrently - e.g., after a configuration reload a new set of > workers is spawned, and only after this old workes are gracefully > shut down. > > -- > Maxim Dounin > http://nginx.org/ > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel From mdounin at mdounin.ru Thu May 15 12:12:12 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 15 May 2014 16:12:12 +0400 Subject: Worker ID In-Reply-To: <2F230A56-EC9A-433B-A3C7-58D96E4544A4@phoempc.com> References: <83AB555D-725A-479A-AFB6-0C707E020435@phoempc.com> <20140514145532.GW1849@mdounin.ru> <2F230A56-EC9A-433B-A3C7-58D96E4544A4@phoempc.com> Message-ID: <20140515121212.GM1849@mdounin.ru> Hello! On Wed, May 14, 2014 at 09:10:04PM -0400, Jordan Newman wrote: > Thanks, ngx_process_slot was what I needed. The reason for > knowing the worker # is for stat tracking, its important that > the backend where the stats are being collected on knows to > group them together by worker and then by server. For this use case it should be fine to use just a process id of a worker. It's as available in the ngx_pid global variable (and as the $pid nginx variable). -- Maxim Dounin http://nginx.org/ From vbart at nginx.com Thu May 15 15:25:42 2014 From: vbart at nginx.com (Valentin Bartenev) Date: Thu, 15 May 2014 15:25:42 +0000 Subject: [nginx] SPDY: fixed one case of improper memory allocation error... Message-ID: details: http://hg.nginx.org/nginx/rev/5cf537731218 branches: changeset: 5686:5cf537731218 user: Valentin Bartenev date: Wed Apr 30 02:16:21 2014 +0400 description: SPDY: fixed one case of improper memory allocation error handling. Now ngx_http_spdy_construct_request_line() doesn't try to finalize request in case of failed memory allocation. diffstat: src/http/ngx_http_spdy.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diffs (13 lines): diff -r 0cbefdcf82a6 -r 5cf537731218 src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Wed May 14 22:26:30 2014 +0400 +++ b/src/http/ngx_http_spdy.c Wed Apr 30 02:16:21 2014 +0400 @@ -2900,7 +2900,8 @@ ngx_http_spdy_construct_request_line(ngx p = ngx_pnalloc(r->pool, r->request_line.len + 1); if (p == NULL) { - ngx_http_finalize_request(r, NGX_HTTP_INTERNAL_SERVER_ERROR); + ngx_http_spdy_close_stream(r->spdy_stream, + NGX_HTTP_INTERNAL_SERVER_ERROR); return NGX_ERROR; } From vbart at nginx.com Thu May 15 15:25:44 2014 From: vbart at nginx.com (Valentin Bartenev) Date: Thu, 15 May 2014 15:25:44 +0000 Subject: [nginx] SPDY: improved ngx_http_spdy_state_protocol_error(). Message-ID: details: http://hg.nginx.org/nginx/rev/6642690698f1 branches: changeset: 5687:6642690698f1 user: Valentin Bartenev date: Wed Apr 30 20:33:58 2014 +0400 description: SPDY: improved ngx_http_spdy_state_protocol_error(). Now ngx_http_spdy_state_protocol_error() is able to close stream, so there is no need in a separate call for this. Also fixed zero status code in logs for some cases. diffstat: src/http/ngx_http_spdy.c | 24 +++++++++++++++--------- 1 files changed, 15 insertions(+), 9 deletions(-) diffs (98 lines): diff -r 5cf537731218 -r 6642690698f1 src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Wed Apr 30 02:16:21 2014 +0400 +++ b/src/http/ngx_http_spdy.c Wed Apr 30 20:33:58 2014 +0400 @@ -1041,7 +1041,7 @@ ngx_http_spdy_state_headers(ngx_http_spd if (z != Z_OK) { ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, "spdy inflateSetDictionary() failed: %d", z); - ngx_http_spdy_close_stream(sc->stream, 0); + return ngx_http_spdy_state_protocol_error(sc); } @@ -1055,7 +1055,7 @@ ngx_http_spdy_state_headers(ngx_http_spd if (z != Z_OK) { ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, "spdy inflate() failed: %d", z); - ngx_http_spdy_close_stream(sc->stream, 0); + return ngx_http_spdy_state_protocol_error(sc); } @@ -1078,7 +1078,7 @@ ngx_http_spdy_state_headers(ngx_http_spd ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, "client sent SYN_STREAM frame " "with invalid HEADERS block"); - ngx_http_spdy_close_stream(sc->stream, NGX_HTTP_BAD_REQUEST); + return ngx_http_spdy_state_protocol_error(sc); } @@ -1159,7 +1159,7 @@ ngx_http_spdy_state_headers(ngx_http_spd if (z != Z_OK) { ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, "spdy inflate() failed: %d", z); - ngx_http_spdy_close_stream(sc->stream, 0); + return ngx_http_spdy_state_protocol_error(sc); } @@ -1175,7 +1175,7 @@ ngx_http_spdy_state_headers(ngx_http_spd /* TODO: improve error message */ ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "spdy again while last chunk"); - ngx_http_spdy_close_stream(sc->stream, 0); + return ngx_http_spdy_state_protocol_error(sc); } @@ -1196,7 +1196,7 @@ ngx_http_spdy_state_headers(ngx_http_spd ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, "client sent invalid HEADERS spdy frame"); - ngx_http_spdy_close_stream(sc->stream, NGX_HTTP_BAD_REQUEST); + return ngx_http_spdy_state_protocol_error(sc); } @@ -1208,7 +1208,7 @@ ngx_http_spdy_state_headers(ngx_http_spd if (rc == NGX_HTTP_PARSE_INVALID_HEADER) { ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, "client sent invalid HEADERS spdy frame"); - ngx_http_spdy_close_stream(sc->stream, NGX_HTTP_BAD_REQUEST); + return ngx_http_spdy_state_protocol_error(sc); } @@ -1224,7 +1224,7 @@ ngx_http_spdy_state_headers(ngx_http_spd ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, "client sent SYN_STREAM frame " "with invalid HEADERS block"); - ngx_http_spdy_close_stream(sc->stream, NGX_HTTP_BAD_REQUEST); + return ngx_http_spdy_state_protocol_error(sc); } @@ -1894,6 +1894,8 @@ ngx_http_spdy_state_complete(ngx_http_sp u_char *end) { sc->handler = ngx_http_spdy_state_head; + sc->stream = NULL; + return pos; } @@ -1929,8 +1931,12 @@ ngx_http_spdy_state_protocol_error(ngx_h ngx_log_debug0(NGX_LOG_DEBUG_HTTP, sc->connection->log, 0, "spdy state protocol error"); - /* TODO */ + if (sc->stream) { + ngx_http_spdy_close_stream(sc->stream, NGX_HTTP_BAD_REQUEST); + } + ngx_http_spdy_finalize_connection(sc, NGX_HTTP_CLIENT_CLOSED_REQUEST); + return NULL; } From vbart at nginx.com Thu May 15 15:25:45 2014 From: vbart at nginx.com (Valentin Bartenev) Date: Thu, 15 May 2014 15:25:45 +0000 Subject: [nginx] SPDY: prevented creation of RST_STREAM in protocol error... Message-ID: details: http://hg.nginx.org/nginx/rev/86232c95623d branches: changeset: 5688:86232c95623d user: Valentin Bartenev date: Thu May 15 19:18:26 2014 +0400 description: SPDY: prevented creation of RST_STREAM in protocol error state. Previously, the frame wasn't sent anyway (and had a wrong status code). diffstat: src/http/ngx_http_spdy.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (11 lines): diff -r 6642690698f1 -r 86232c95623d src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Wed Apr 30 20:33:58 2014 +0400 +++ b/src/http/ngx_http_spdy.c Thu May 15 19:18:26 2014 +0400 @@ -1932,6 +1932,7 @@ ngx_http_spdy_state_protocol_error(ngx_h "spdy state protocol error"); if (sc->stream) { + sc->stream->out_closed = 1; ngx_http_spdy_close_stream(sc->stream, NGX_HTTP_BAD_REQUEST); } From vbart at nginx.com Thu May 15 15:25:46 2014 From: vbart at nginx.com (Valentin Bartenev) Date: Thu, 15 May 2014 15:25:46 +0000 Subject: [nginx] SPDY: removed ngx_http_spdy_state_headers_error(). Message-ID: details: http://hg.nginx.org/nginx/rev/5d55f03b1e12 branches: changeset: 5689:5d55f03b1e12 user: Valentin Bartenev date: Wed Apr 30 20:34:20 2014 +0400 description: SPDY: removed ngx_http_spdy_state_headers_error(). The function just calls ngx_http_spdy_state_headers_skip() most of the time. There was also an attempt of optimization to stop parsing if the client already closed connection, but it looks strange and unfinished anyway. diffstat: src/http/ngx_http_spdy.c | 26 ++++++-------------------- 1 files changed, 6 insertions(+), 20 deletions(-) diffs (83 lines): diff -r 86232c95623d -r 5d55f03b1e12 src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Thu May 15 19:18:26 2014 +0400 +++ b/src/http/ngx_http_spdy.c Wed Apr 30 20:34:20 2014 +0400 @@ -103,8 +103,6 @@ static u_char *ngx_http_spdy_state_syn_s u_char *pos, u_char *end); static u_char *ngx_http_spdy_state_headers(ngx_http_spdy_connection_t *sc, u_char *pos, u_char *end); -static u_char *ngx_http_spdy_state_headers_error(ngx_http_spdy_connection_t *sc, - u_char *pos, u_char *end); static u_char *ngx_http_spdy_state_headers_skip(ngx_http_spdy_connection_t *sc, u_char *pos, u_char *end); static u_char *ngx_http_spdy_state_window_update(ngx_http_spdy_connection_t *sc, @@ -1100,7 +1098,7 @@ ngx_http_spdy_state_headers(ngx_http_spd { ngx_http_spdy_close_stream(sc->stream, NGX_HTTP_INTERNAL_SERVER_ERROR); - return ngx_http_spdy_state_headers_error(sc, pos, end); + return ngx_http_spdy_state_headers_skip(sc, pos, end); } if (ngx_array_init(&r->headers_in.cookies, r->pool, 2, @@ -1109,7 +1107,7 @@ ngx_http_spdy_state_headers(ngx_http_spd { ngx_http_spdy_close_stream(sc->stream, NGX_HTTP_INTERNAL_SERVER_ERROR); - return ngx_http_spdy_state_headers_error(sc, pos, end); + return ngx_http_spdy_state_headers_skip(sc, pos, end); } } @@ -1135,13 +1133,13 @@ ngx_http_spdy_state_headers(ngx_http_spd /* TODO logging */ ngx_http_finalize_request(r, NGX_HTTP_REQUEST_HEADER_TOO_LARGE); - return ngx_http_spdy_state_headers_error(sc, pos, end); + return ngx_http_spdy_state_headers_skip(sc, pos, end); } if (rc != NGX_OK) { ngx_http_spdy_close_stream(sc->stream, NGX_HTTP_INTERNAL_SERVER_ERROR); - return ngx_http_spdy_state_headers_error(sc, pos, end); + return ngx_http_spdy_state_headers_skip(sc, pos, end); } /* null-terminate the last processed header name or value */ @@ -1190,7 +1188,7 @@ ngx_http_spdy_state_headers(ngx_http_spd ngx_http_finalize_request(r, NGX_HTTP_BAD_REQUEST); - return ngx_http_spdy_state_headers_error(sc, pos, end); + return ngx_http_spdy_state_headers_skip(sc, pos, end); default: /* NGX_HTTP_PARSE_INVALID_HEADER */ @@ -1216,7 +1214,7 @@ ngx_http_spdy_state_headers(ngx_http_spd ngx_http_finalize_request(r, NGX_HTTP_BAD_REQUEST); } - return ngx_http_spdy_state_headers_error(sc, pos, end); + return ngx_http_spdy_state_headers_skip(sc, pos, end); } } @@ -1243,18 +1241,6 @@ ngx_http_spdy_state_headers(ngx_http_spd static u_char * -ngx_http_spdy_state_headers_error(ngx_http_spdy_connection_t *sc, u_char *pos, - u_char *end) -{ - if (sc->connection->error) { - return ngx_http_spdy_state_internal_error(sc); - } - - return ngx_http_spdy_state_headers_skip(sc, pos, end); -} - - -static u_char * ngx_http_spdy_state_headers_skip(ngx_http_spdy_connection_t *sc, u_char *pos, u_char *end) { From vbart at nginx.com Thu May 15 15:25:48 2014 From: vbart at nginx.com (Valentin Bartenev) Date: Thu, 15 May 2014 15:25:48 +0000 Subject: [nginx] SPDY: improved error handling of header block decompress... Message-ID: details: http://hg.nginx.org/nginx/rev/fb56f5d612a0 branches: changeset: 5690:fb56f5d612a0 user: Valentin Bartenev date: Wed Apr 30 20:34:20 2014 +0400 description: SPDY: improved error handling of header block decompression. Now cases when decompression failed due to internal error and when a client sent corrupted data are distinguished. diffstat: src/http/ngx_http_spdy.c | 62 ++++++++++++++++++++++++++++++++++------------- 1 files changed, 44 insertions(+), 18 deletions(-) diffs (117 lines): diff -r 5d55f03b1e12 -r fb56f5d612a0 src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Wed Apr 30 20:34:20 2014 +0400 +++ b/src/http/ngx_http_spdy.c Wed Apr 30 20:34:20 2014 +0400 @@ -123,6 +123,9 @@ static u_char *ngx_http_spdy_state_compl u_char *pos, u_char *end); static u_char *ngx_http_spdy_state_save(ngx_http_spdy_connection_t *sc, u_char *pos, u_char *end, ngx_http_spdy_handler_pt handler); + +static u_char *ngx_http_spdy_state_inflate_error( + ngx_http_spdy_connection_t *sc, int rc); static u_char *ngx_http_spdy_state_protocol_error( ngx_http_spdy_connection_t *sc); static u_char *ngx_http_spdy_state_internal_error( @@ -1036,11 +1039,21 @@ ngx_http_spdy_state_headers(ngx_http_spd if (z == Z_NEED_DICT) { z = inflateSetDictionary(&sc->zstream_in, ngx_http_spdy_dict, sizeof(ngx_http_spdy_dict)); + if (z != Z_OK) { - ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, - "spdy inflateSetDictionary() failed: %d", z); - - return ngx_http_spdy_state_protocol_error(sc); + if (z == Z_DATA_ERROR) { + ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, + "client sent SYN_STREAM frame with header " + "block encoded using wrong dictionary: %ul", + (u_long) sc->zstream_in.adler); + + return ngx_http_spdy_state_protocol_error(sc); + } + + ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, + "inflateSetDictionary() failed: %d", z); + + return ngx_http_spdy_state_internal_error(sc); } ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, @@ -1051,10 +1064,7 @@ ngx_http_spdy_state_headers(ngx_http_spd } if (z != Z_OK) { - ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, - "spdy inflate() failed: %d", z); - - return ngx_http_spdy_state_protocol_error(sc); + return ngx_http_spdy_state_inflate_error(sc, z); } ngx_log_debug5(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, @@ -1155,10 +1165,7 @@ ngx_http_spdy_state_headers(ngx_http_spd z = inflate(&sc->zstream_in, Z_NO_FLUSH); if (z != Z_OK) { - ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, - "spdy inflate() failed: %d", z); - - return ngx_http_spdy_state_protocol_error(sc); + return ngx_http_spdy_state_inflate_error(sc, z); } sc->length -= sc->zstream_in.next_in - pos; @@ -1268,12 +1275,8 @@ ngx_http_spdy_state_headers_skip(ngx_htt n = inflate(&sc->zstream_in, Z_NO_FLUSH); - ngx_log_debug1(NGX_LOG_DEBUG_HTTP, sc->connection->log, 0, - "spdy inflate(): %d", n); - if (n != Z_OK) { - /* TODO: logging */ - return ngx_http_spdy_state_protocol_error(sc); + return ngx_http_spdy_state_inflate_error(sc, n); } } @@ -1912,6 +1915,24 @@ ngx_http_spdy_state_save(ngx_http_spdy_c static u_char * +ngx_http_spdy_state_inflate_error(ngx_http_spdy_connection_t *sc, int rc) +{ + if (rc == Z_DATA_ERROR || rc == Z_STREAM_END) { + ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, + "client sent SYN_STREAM frame with " + "corrupted header block, inflate() failed: %d", rc); + + return ngx_http_spdy_state_protocol_error(sc); + } + + ngx_log_error(NGX_LOG_ERR, sc->connection->log, 0, + "inflate() failed: %d", rc); + + return ngx_http_spdy_state_internal_error(sc); +} + + +static u_char * ngx_http_spdy_state_protocol_error(ngx_http_spdy_connection_t *sc) { ngx_log_debug0(NGX_LOG_DEBUG_HTTP, sc->connection->log, 0, @@ -1934,8 +1955,13 @@ ngx_http_spdy_state_internal_error(ngx_h ngx_log_debug0(NGX_LOG_DEBUG_HTTP, sc->connection->log, 0, "spdy state internal error"); - /* TODO */ + if (sc->stream) { + sc->stream->out_closed = 1; + ngx_http_spdy_close_stream(sc->stream, NGX_HTTP_INTERNAL_SERVER_ERROR); + } + ngx_http_spdy_finalize_connection(sc, NGX_HTTP_INTERNAL_SERVER_ERROR); + return NULL; } From vbart at nginx.com Thu May 15 15:25:49 2014 From: vbart at nginx.com (Valentin Bartenev) Date: Thu, 15 May 2014 15:25:49 +0000 Subject: [nginx] SPDY: ngx_http_spdy_state_headers() error handling cleanup. Message-ID: details: http://hg.nginx.org/nginx/rev/31349361558e branches: changeset: 5691:31349361558e user: Valentin Bartenev date: Wed Apr 30 20:34:20 2014 +0400 description: SPDY: ngx_http_spdy_state_headers() error handling cleanup. - Specification-friendly handling of invalid header block or special headers. Such errors are not fatal for session and shouldn't lead to connection close; - Avoid mix of NGX_HTTP_PARSE_INVALID_REQUEST/NGX_HTTP_PARSE_INVALID_HEADER. diffstat: src/http/ngx_http_spdy.c | 104 ++++++++++++++++++++++++++++------------------ 1 files changed, 63 insertions(+), 41 deletions(-) diffs (252 lines): diff -r fb56f5d612a0 -r 31349361558e src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Wed Apr 30 20:34:20 2014 +0400 +++ b/src/http/ngx_http_spdy.c Wed Apr 30 20:34:20 2014 +0400 @@ -105,6 +105,8 @@ static u_char *ngx_http_spdy_state_heade u_char *pos, u_char *end); static u_char *ngx_http_spdy_state_headers_skip(ngx_http_spdy_connection_t *sc, u_char *pos, u_char *end); +static u_char *ngx_http_spdy_state_headers_error(ngx_http_spdy_connection_t *sc, + u_char *pos, u_char *end); static u_char *ngx_http_spdy_state_window_update(ngx_http_spdy_connection_t *sc, u_char *pos, u_char *end); static u_char *ngx_http_spdy_state_data(ngx_http_spdy_connection_t *sc, @@ -1083,11 +1085,10 @@ ngx_http_spdy_state_headers(ngx_http_spd if (buf->last - buf->pos < NGX_SPDY_NV_NUM_SIZE) { if (complete) { - ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, - "client sent SYN_STREAM frame " - "with invalid HEADERS block"); - - return ngx_http_spdy_state_protocol_error(sc); + ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "premature end of spdy header block"); + + return ngx_http_spdy_state_headers_error(sc, pos, end); } return ngx_http_spdy_state_save(sc, pos, end, @@ -1181,13 +1182,13 @@ ngx_http_spdy_state_headers(ngx_http_spd ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "spdy again while last chunk"); - return ngx_http_spdy_state_protocol_error(sc); + return ngx_http_spdy_state_headers_error(sc, pos, end); } return ngx_http_spdy_state_save(sc, pos, end, ngx_http_spdy_state_headers); - case NGX_HTTP_PARSE_INVALID_REQUEST: + case NGX_HTTP_PARSE_INVALID_HEADER: /* TODO: improve error message */ ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, @@ -1197,12 +1198,8 @@ ngx_http_spdy_state_headers(ngx_http_spd return ngx_http_spdy_state_headers_skip(sc, pos, end); - default: /* NGX_HTTP_PARSE_INVALID_HEADER */ - - ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, - "client sent invalid HEADERS spdy frame"); - - return ngx_http_spdy_state_protocol_error(sc); + default: /* NGX_ERROR */ + return ngx_http_spdy_state_headers_error(sc, pos, end); } /* a header line has been parsed successfully */ @@ -1211,14 +1208,13 @@ ngx_http_spdy_state_headers(ngx_http_spd if (rc != NGX_OK) { if (rc == NGX_HTTP_PARSE_INVALID_HEADER) { - ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, - "client sent invalid HEADERS spdy frame"); - - return ngx_http_spdy_state_protocol_error(sc); + ngx_http_finalize_request(r, NGX_HTTP_BAD_REQUEST); + return ngx_http_spdy_state_headers_skip(sc, pos, end); } - if (rc == NGX_HTTP_PARSE_INVALID_REQUEST) { - ngx_http_finalize_request(r, NGX_HTTP_BAD_REQUEST); + if (rc != NGX_ABORT) { + ngx_http_spdy_close_stream(sc->stream, + NGX_HTTP_INTERNAL_SERVER_ERROR); } return ngx_http_spdy_state_headers_skip(sc, pos, end); @@ -1226,11 +1222,10 @@ ngx_http_spdy_state_headers(ngx_http_spd } if (buf->pos != buf->last || sc->zstream_in.avail_in) { - ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, - "client sent SYN_STREAM frame " - "with invalid HEADERS block"); - - return ngx_http_spdy_state_protocol_error(sc); + ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "incorrect number of spdy header block entries"); + + return ngx_http_spdy_state_headers_error(sc, pos, end); } if (!complete) { @@ -1293,6 +1288,33 @@ ngx_http_spdy_state_headers_skip(ngx_htt static u_char * +ngx_http_spdy_state_headers_error(ngx_http_spdy_connection_t *sc, u_char *pos, + u_char *end) +{ + ngx_http_spdy_stream_t *stream; + + stream = sc->stream; + + ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, + "client sent SYN_STREAM frame for stream %ui " + "with invalid header block", stream->id); + + if (ngx_http_spdy_send_rst_stream(sc, stream->id, NGX_SPDY_PROTOCOL_ERROR, + stream->priority) + != NGX_OK) + { + return ngx_http_spdy_state_internal_error(sc); + } + + stream->out_closed = 1; + + ngx_http_spdy_close_stream(stream, NGX_HTTP_BAD_REQUEST); + + return ngx_http_spdy_state_headers_skip(sc, pos, end); +} + + +static u_char * ngx_http_spdy_state_window_update(ngx_http_spdy_connection_t *sc, u_char *pos, u_char *end) { @@ -2425,7 +2447,7 @@ ngx_http_spdy_parse_header(ngx_http_requ r->lowcase_index = ngx_spdy_frame_parse_uint32(p); if (r->lowcase_index == 0) { - return NGX_HTTP_PARSE_INVALID_HEADER; + return NGX_ERROR; } /* null-terminate the previous header value */ @@ -2475,11 +2497,11 @@ ngx_http_spdy_parse_header(ngx_http_requ case LF: case CR: case ':': - return NGX_HTTP_PARSE_INVALID_REQUEST; + return NGX_HTTP_PARSE_INVALID_HEADER; } if (ch >= 'A' && ch <= 'Z') { - return NGX_HTTP_PARSE_INVALID_HEADER; + return NGX_ERROR; } r->invalid_header = 1; @@ -2642,13 +2664,11 @@ ngx_http_spdy_handle_request_header(ngx_ return sh->handler(r); } - return NGX_HTTP_PARSE_INVALID_REQUEST; + return NGX_HTTP_PARSE_INVALID_HEADER; } h = ngx_list_push(&r->headers_in.headers); if (h == NULL) { - ngx_http_spdy_close_stream(r->spdy_stream, - NGX_HTTP_INTERNAL_SERVER_ERROR); return NGX_ERROR; } @@ -2752,7 +2772,7 @@ ngx_http_spdy_parse_method(ngx_http_requ if ((*p < 'A' || *p > 'Z') && *p != '_') { ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, "client sent invalid method"); - return NGX_HTTP_PARSE_INVALID_REQUEST; + return NGX_HTTP_PARSE_INVALID_HEADER; } p++; @@ -2788,8 +2808,6 @@ ngx_http_spdy_parse_host(ngx_http_reques h = ngx_list_push(&r->headers_in.headers); if (h == NULL) { - ngx_http_spdy_close_stream(r->spdy_stream, - NGX_HTTP_INTERNAL_SERVER_ERROR); return NGX_ERROR; } @@ -2820,11 +2838,15 @@ ngx_http_spdy_parse_path(ngx_http_reques r->uri_end = r->header_end; if (ngx_http_parse_uri(r) != NGX_OK) { - return NGX_HTTP_PARSE_INVALID_REQUEST; + return NGX_HTTP_PARSE_INVALID_HEADER; } if (ngx_http_process_request_uri(r) != NGX_OK) { - return NGX_ERROR; + /* + * request has been finalized already + * in ngx_http_process_request_uri() + */ + return NGX_ABORT; } return NGX_OK; @@ -2843,13 +2865,13 @@ ngx_http_spdy_parse_version(ngx_http_req p = r->header_start; if (r->header_end - p < 8 || !(ngx_str5cmp(p, 'H', 'T', 'T', 'P', '/'))) { - return NGX_HTTP_PARSE_INVALID_REQUEST; + return NGX_HTTP_PARSE_INVALID_HEADER; } ch = *(p + 5); if (ch < '1' || ch > '9') { - return NGX_HTTP_PARSE_INVALID_REQUEST; + return NGX_HTTP_PARSE_INVALID_HEADER; } r->http_major = ch - '0'; @@ -2863,20 +2885,20 @@ ngx_http_spdy_parse_version(ngx_http_req } if (ch < '0' || ch > '9') { - return NGX_HTTP_PARSE_INVALID_REQUEST; + return NGX_HTTP_PARSE_INVALID_HEADER; } r->http_major = r->http_major * 10 + ch - '0'; } if (*p != '.') { - return NGX_HTTP_PARSE_INVALID_REQUEST; + return NGX_HTTP_PARSE_INVALID_HEADER; } ch = *(p + 1); if (ch < '0' || ch > '9') { - return NGX_HTTP_PARSE_INVALID_REQUEST; + return NGX_HTTP_PARSE_INVALID_HEADER; } r->http_minor = ch - '0'; @@ -2886,7 +2908,7 @@ ngx_http_spdy_parse_version(ngx_http_req ch = *p; if (ch < '0' || ch > '9') { - return NGX_HTTP_PARSE_INVALID_REQUEST; + return NGX_HTTP_PARSE_INVALID_HEADER; } r->http_minor = r->http_minor * 10 + ch - '0'; From vbart at nginx.com Thu May 15 15:25:50 2014 From: vbart at nginx.com (Valentin Bartenev) Date: Thu, 15 May 2014 15:25:50 +0000 Subject: [nginx] SPDY: set log action for PROXY protocol only while parsi... Message-ID: details: http://hg.nginx.org/nginx/rev/ed5fb7d22d46 branches: changeset: 5692:ed5fb7d22d46 user: Valentin Bartenev date: Thu May 15 19:22:06 2014 +0400 description: SPDY: set log action for PROXY protocol only while parsing it. Handling of PROXY protocol for SPDY connection is currently implemented as a SPDY state. And while nginx waiting for PROXY protocol data it continues to process SPDY connection: initializes zlib context, sends control frames. diffstat: src/http/ngx_http_spdy.c | 17 +++++++++-------- 1 files changed, 9 insertions(+), 8 deletions(-) diffs (40 lines): diff -r 31349361558e -r ed5fb7d22d46 src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Wed Apr 30 20:34:20 2014 +0400 +++ b/src/http/ngx_http_spdy.c Thu May 15 19:22:06 2014 +0400 @@ -424,12 +424,8 @@ ngx_http_spdy_init(ngx_event_t *rev) sc->init_window = NGX_SPDY_INIT_STREAM_WINDOW; - sc->handler = ngx_http_spdy_state_head; - - if (hc->proxy_protocol) { - c->log->action = "reading PROXY protocol"; - sc->handler = ngx_http_spdy_proxy_protocol; - } + sc->handler = hc->proxy_protocol ? ngx_http_spdy_proxy_protocol + : ngx_http_spdy_state_head; sc->zstream_in.zalloc = ngx_http_spdy_zalloc; sc->zstream_in.zfree = ngx_http_spdy_zfree; @@ -823,14 +819,19 @@ static u_char * ngx_http_spdy_proxy_protocol(ngx_http_spdy_connection_t *sc, u_char *pos, u_char *end) { + ngx_log_t *log; + + log = sc->connection->log; + log->action = "reading PROXY protocol"; + pos = ngx_proxy_protocol_parse(sc->connection, pos, end); + log->action = "processing SPDY"; + if (pos == NULL) { return ngx_http_spdy_state_protocol_error(sc); } - sc->connection->log->action = "processing SPDY"; - return ngx_http_spdy_state_complete(sc, pos, end); } From vbart at nginx.com Thu May 15 15:25:52 2014 From: vbart at nginx.com (Valentin Bartenev) Date: Thu, 15 May 2014 15:25:52 +0000 Subject: [nginx] SPDY: improved logging. Message-ID: details: http://hg.nginx.org/nginx/rev/701d6e17e42c branches: changeset: 5693:701d6e17e42c user: Valentin Bartenev date: Wed Apr 30 20:34:20 2014 +0400 description: SPDY: improved logging. diffstat: src/http/ngx_http_spdy.c | 210 +++++++++++++++++++++++--------- src/http/ngx_http_spdy_filter_module.c | 3 +- 2 files changed, 151 insertions(+), 62 deletions(-) diffs (truncated from 656 to 300 lines): diff -r ed5fb7d22d46 -r 701d6e17e42c src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Thu May 15 19:22:06 2014 +0400 +++ b/src/http/ngx_http_spdy.c Wed Apr 30 20:34:20 2014 +0400 @@ -395,8 +395,7 @@ ngx_http_spdy_init(ngx_event_t *rev) c = rev->data; hc = c->data; - ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, - "init spdy request"); + ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "init spdy request"); c->log->action = "processing SPDY"; @@ -556,7 +555,7 @@ ngx_http_spdy_read_handler(ngx_event_t * if (n == 0 && (sc->incomplete || sc->processing)) { ngx_log_error(NGX_LOG_INFO, c->log, 0, - "client closed prematurely connection"); + "client prematurely closed connection"); } if (n == 0 || n == NGX_ERROR) { @@ -644,7 +643,7 @@ ngx_http_spdy_write_handler(ngx_event_t stream->handled = 0; ngx_log_debug1(NGX_LOG_DEBUG_HTTP, c->log, 0, - "spdy run stream %ui", stream->id); + "run spdy stream %ui", stream->id); wev = stream->request->connection->write; wev->handler(wev); @@ -860,7 +859,7 @@ ngx_http_spdy_state_head(ngx_http_spdy_c pos += sizeof(uint32_t); ngx_log_debug3(NGX_LOG_DEBUG_HTTP, sc->connection->log, 0, - "spdy process frame head:%08XD f:%Xd l:%uz", + "process spdy frame head:%08XD f:%Xd l:%uz", head, sc->flags, sc->length); if (ngx_spdy_ctl_frame_check(head)) { @@ -872,6 +871,8 @@ ngx_http_spdy_state_head(ngx_http_spdy_c return ngx_http_spdy_state_syn_stream(sc, pos, end); case NGX_SPDY_SYN_REPLY: + ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, + "client sent unexpected SYN_REPLY frame"); return ngx_http_spdy_state_protocol_error(sc); case NGX_SPDY_RST_STREAM: @@ -887,6 +888,8 @@ ngx_http_spdy_state_head(ngx_http_spdy_c return ngx_http_spdy_state_skip(sc, pos, end); /* TODO */ case NGX_SPDY_HEADERS: + ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, + "client sent unexpected HEADERS frame"); return ngx_http_spdy_state_protocol_error(sc); case NGX_SPDY_WINDOW_UPDATE: @@ -904,10 +907,8 @@ ngx_http_spdy_state_head(ngx_http_spdy_c return ngx_http_spdy_state_data(sc, pos, end); } - - /* TODO version & type check */ - ngx_log_debug0(NGX_LOG_DEBUG_HTTP, sc->connection->log, 0, - "spdy unknown frame"); + ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, + "client sent invalid frame"); return ngx_http_spdy_state_protocol_error(sc); } @@ -927,7 +928,10 @@ ngx_http_spdy_state_syn_stream(ngx_http_ } if (sc->length <= NGX_SPDY_SYN_STREAM_SIZE) { - /* TODO logging */ + ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, + "client sent SYN_STREAM frame with incorrect length %uz", + sc->length); + return ngx_http_spdy_state_protocol_error(sc); } @@ -968,7 +972,7 @@ ngx_http_spdy_state_syn_stream(ngx_http_ if (sc->processing >= sscf->concurrent_streams) { ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, - "spdy concurrent streams exceeded %ui", sc->processing); + "concurrent streams exceeded %ui", sc->processing); if (ngx_http_spdy_send_rst_stream(sc, sid, NGX_SPDY_REFUSED_STREAM, prio) @@ -1026,7 +1030,7 @@ ngx_http_spdy_state_headers(ngx_http_spd r = sc->stream->request; ngx_log_debug2(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, - "spdy process HEADERS %uz of %uz", size, sc->length); + "process spdy header block %uz of %uz", size, sc->length); buf = r->header_in; @@ -1101,7 +1105,7 @@ ngx_http_spdy_state_headers(ngx_http_spd buf->pos += NGX_SPDY_NV_NUM_SIZE; ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, - "spdy HEADERS block consists of %ui entries", + "spdy header block has %ui entries", sc->entries); if (ngx_list_init(&r->headers_in.headers, r->pool, 20, @@ -1142,7 +1146,6 @@ ngx_http_spdy_state_headers(ngx_http_spd rc = ngx_http_spdy_alloc_large_header_buffer(r); if (rc == NGX_DECLINED) { - /* TODO logging */ ngx_http_finalize_request(r, NGX_HTTP_REQUEST_HEADER_TOO_LARGE); return ngx_http_spdy_state_headers_skip(sc, pos, end); @@ -1179,9 +1182,8 @@ ngx_http_spdy_state_headers(ngx_http_spd } if (complete) { - /* TODO: improve error message */ ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, - "spdy again while last chunk"); + "premature end of spdy header block"); return ngx_http_spdy_state_headers_error(sc, pos, end); } @@ -1190,13 +1192,7 @@ ngx_http_spdy_state_headers(ngx_http_spd ngx_http_spdy_state_headers); case NGX_HTTP_PARSE_INVALID_HEADER: - - /* TODO: improve error message */ - ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, - "client sent invalid header line"); - ngx_http_finalize_request(r, NGX_HTTP_BAD_REQUEST); - return ngx_http_spdy_state_headers_skip(sc, pos, end); default: /* NGX_ERROR */ @@ -1262,6 +1258,9 @@ ngx_http_spdy_state_headers_skip(ngx_htt ngx_http_spdy_state_headers_skip); } + ngx_log_debug2(NGX_LOG_DEBUG_HTTP, sc->connection->log, 0, + "spdy header block skip %uz of %uz", size, sc->length); + sc->zstream_in.next_in = pos; sc->zstream_in.avail_in = (size < sc->length) ? size : sc->length; @@ -1364,7 +1363,7 @@ ngx_http_spdy_state_window_update(ngx_ht ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, "client violated flow control for stream %ui: " "received WINDOW_UPDATE frame with delta %uz " - "that is not allowed for window %z", + "not allowed for window %z", sid, delta, stream->send_window); if (ngx_http_spdy_terminate_stream(sc, stream, @@ -1397,7 +1396,7 @@ ngx_http_spdy_state_window_update(ngx_ht ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, "client violated connection flow control: " "received WINDOW_UPDATE frame with delta %uz " - "that is not allowed for window %uz", + "not allowed for window %uz", delta, sc->send_window); return ngx_http_spdy_state_protocol_error(sc); @@ -1440,8 +1439,8 @@ ngx_http_spdy_state_data(ngx_http_spdy_c if (sc->length > sc->recv_window) { ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, - "client violated connection flow control: length of " - "received DATA frame %uz, while available window %uz", + "client violated connection flow control: " + "received DATA frame length %uz, available window %uz", sc->length, sc->recv_window); return ngx_http_spdy_state_protocol_error(sc); @@ -1465,13 +1464,16 @@ ngx_http_spdy_state_data(ngx_http_spdy_c stream = sc->stream; if (stream == NULL) { + ngx_log_debug0(NGX_LOG_DEBUG_HTTP, sc->connection->log, 0, + "unknown spdy stream"); + return ngx_http_spdy_state_skip(sc, pos, end); } if (sc->length > stream->recv_window) { ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, - "client violated flow control for stream %ui: length of " - "received DATA frame %uz, while available window %uz", + "client violated flow control for stream %ui: " + "received DATA frame length %uz, available window %uz", stream->id, sc->length, stream->recv_window); if (ngx_http_spdy_terminate_stream(sc, stream, @@ -1501,7 +1503,7 @@ ngx_http_spdy_state_data(ngx_http_spdy_c if (stream->in_closed) { ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, - "client sent DATA frame for half closed stream %ui", + "client sent DATA frame for half-closed stream %ui", stream->id); if (ngx_http_spdy_terminate_stream(sc, stream, @@ -1544,7 +1546,10 @@ ngx_http_spdy_state_read_data(ngx_http_s stream->in_closed = 1; } - /* TODO log and accounting */ + ngx_log_debug1(NGX_LOG_DEBUG_HTTP, sc->connection->log, 0, + "skipping spdy DATA frame, reason: %d", + stream->skip_data); + return ngx_http_spdy_state_skip(sc, pos, end); } @@ -1573,7 +1578,10 @@ ngx_http_spdy_state_read_data(ngx_http_s if (r->headers_in.content_length_n != -1 && r->headers_in.content_length_n < rb->rest) { - /* TODO logging */ + ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, + "client intended to send body data " + "larger than declared"); + stream->skip_data = NGX_SPDY_DATA_ERROR; goto error; @@ -1584,9 +1592,8 @@ ngx_http_spdy_state_read_data(ngx_http_s && clcf->client_max_body_size < rb->rest) { ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, - "client intended to send too large chunked " - "body: %O bytes", - rb->rest); + "client intended to send " + "too large chunked body: %O bytes", rb->rest); stream->skip_data = NGX_SPDY_DATA_ERROR; goto error; @@ -1638,7 +1645,7 @@ ngx_http_spdy_state_read_data(ngx_http_s } else if (r->headers_in.content_length_n != rb->rest) { ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, "client prematurely closed stream: " - "%O of %O bytes of request body received", + "only %O out of %O bytes of request body received", rb->rest, r->headers_in.content_length_n); stream->skip_data = NGX_SPDY_DATA_ERROR; @@ -1717,9 +1724,11 @@ ngx_http_spdy_state_rst_stream(ngx_http_ "spdy RST_STREAM sid:%ui st:%ui", sid, status); stream = ngx_http_spdy_get_stream_by_id(sc, sid); + if (stream == NULL) { ngx_log_debug0(NGX_LOG_DEBUG_HTTP, sc->connection->log, 0, - "unknown stream, probably it has been closed already"); + "unknown spdy stream"); + return ngx_http_spdy_state_complete(sc, pos, end); } @@ -1738,7 +1747,7 @@ ngx_http_spdy_state_rst_stream(ngx_http_ case NGX_SPDY_INTERNAL_ERROR: ngx_log_error(NGX_LOG_INFO, fc->log, 0, - "client terminated stream %ui because of internal error", + "client terminated stream %ui due to internal error", sid); break; @@ -1770,7 +1779,10 @@ ngx_http_spdy_state_ping(ngx_http_spdy_c } if (sc->length != NGX_SPDY_PING_SIZE) { - /* TODO logging */ + ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, + "client sent PING frame with incorrect length %uz", + sc->length); + return ngx_http_spdy_state_protocol_error(sc); } @@ -1810,6 +1822,9 @@ ngx_http_spdy_state_skip(ngx_http_spdy_c size = end - pos; + ngx_log_debug2(NGX_LOG_DEBUG_HTTP, sc->connection->log, 0, + "spdy frame skip %uz of %uz", size, sc->length); + if (size < sc->length) { sc->length -= size; return ngx_http_spdy_state_save(sc, end, end, @@ -1839,13 +1854,16 @@ ngx_http_spdy_state_settings(ngx_http_sp From vbart at nginx.com Thu May 15 15:25:53 2014 From: vbart at nginx.com (Valentin Bartenev) Date: Thu, 15 May 2014 15:25:53 +0000 Subject: [nginx] SPDY: refactored ngx_http_spdy_state_headers(). Message-ID: details: http://hg.nginx.org/nginx/rev/9a95eb9f448b branches: changeset: 5694:9a95eb9f448b user: Valentin Bartenev date: Wed Apr 30 20:34:20 2014 +0400 description: SPDY: refactored ngx_http_spdy_state_headers(). This change is similar to d2ac5cf4056d. Special flag of completeness looks surplus when there is also a counter of frame bytes left. diffstat: src/http/ngx_http_spdy.c | 13 ++++--------- 1 files changed, 4 insertions(+), 9 deletions(-) diffs (52 lines): diff -r 701d6e17e42c -r 9a95eb9f448b src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Wed Apr 30 20:34:20 2014 +0400 +++ b/src/http/ngx_http_spdy.c Wed Apr 30 20:34:20 2014 +0400 @@ -1009,7 +1009,6 @@ ngx_http_spdy_state_headers(ngx_http_spd size_t size; ngx_buf_t *buf; ngx_int_t rc; - ngx_uint_t complete; ngx_http_request_t *r; size = end - pos; @@ -1019,12 +1018,8 @@ ngx_http_spdy_state_headers(ngx_http_spd ngx_http_spdy_state_headers); } - if (size >= sc->length) { + if (size > sc->length) { size = sc->length; - complete = 1; - - } else { - complete = 0; } r = sc->stream->request; @@ -1089,7 +1084,7 @@ ngx_http_spdy_state_headers(ngx_http_spd if (buf->last - buf->pos < NGX_SPDY_NV_NUM_SIZE) { - if (complete) { + if (sc->length == 0) { ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "premature end of spdy header block"); @@ -1181,7 +1176,7 @@ ngx_http_spdy_state_headers(ngx_http_spd continue; } - if (complete) { + if (sc->length == 0) { ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "premature end of spdy header block"); @@ -1225,7 +1220,7 @@ ngx_http_spdy_state_headers(ngx_http_spd return ngx_http_spdy_state_headers_error(sc, pos, end); } - if (!complete) { + if (sc->length) { return ngx_http_spdy_state_save(sc, pos, end, ngx_http_spdy_state_headers); } From vbart at nginx.com Thu May 15 15:25:55 2014 From: vbart at nginx.com (Valentin Bartenev) Date: Thu, 15 May 2014 15:25:55 +0000 Subject: [nginx] SPDY: added a debug point to the state buffer overflow p... Message-ID: details: http://hg.nginx.org/nginx/rev/231588611230 branches: changeset: 5695:231588611230 user: Valentin Bartenev date: Wed Apr 30 20:34:20 2014 +0400 description: SPDY: added a debug point to the state buffer overflow protection. diffstat: src/http/ngx_http_spdy.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (11 lines): diff -r 9a95eb9f448b -r 231588611230 src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Wed Apr 30 20:34:20 2014 +0400 +++ b/src/http/ngx_http_spdy.c Wed Apr 30 20:34:20 2014 +0400 @@ -1944,6 +1944,7 @@ ngx_http_spdy_state_save(ngx_http_spdy_c ngx_log_error(NGX_LOG_ALERT, sc->connection->log, 0, "state buffer overflow: %uz bytes required", size); + ngx_debug_point(); return ngx_http_spdy_state_internal_error(sc); } From vbart at nginx.com Thu May 15 15:25:57 2014 From: vbart at nginx.com (Valentin Bartenev) Date: Thu, 15 May 2014 15:25:57 +0000 Subject: [nginx] SPDY: added protection from overrun of the receive buffer. Message-ID: details: http://hg.nginx.org/nginx/rev/19a14a484707 branches: changeset: 5696:19a14a484707 user: Valentin Bartenev date: Wed Apr 30 20:34:20 2014 +0400 description: SPDY: added protection from overrun of the receive buffer. diffstat: src/http/ngx_http_spdy.c | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-) diffs (18 lines): diff -r 231588611230 -r 19a14a484707 src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Wed Apr 30 20:34:20 2014 +0400 +++ b/src/http/ngx_http_spdy.c Wed Apr 30 20:34:20 2014 +0400 @@ -1921,6 +1921,14 @@ ngx_http_spdy_state_complete(ngx_http_sp ngx_log_debug2(NGX_LOG_DEBUG_HTTP, sc->connection->log, 0, "spdy frame complete pos:%p end:%p", pos, end); + if (pos > end) { + ngx_log_error(NGX_LOG_ALERT, sc->connection->log, 0, + "receive buffer overrun"); + + ngx_debug_point(); + return ngx_http_spdy_state_internal_error(sc); + } + sc->handler = ngx_http_spdy_state_head; sc->stream = NULL; From mdounin at mdounin.ru Fri May 16 14:21:06 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 16 May 2014 14:21:06 +0000 Subject: [nginx] Fixed alias in regex locations with limit_except/if. Message-ID: details: http://hg.nginx.org/nginx/rev/c985d90a8d1f branches: changeset: 5697:c985d90a8d1f user: Maxim Dounin date: Fri May 16 17:42:24 2014 +0400 description: Fixed alias in regex locations with limit_except/if. The ngx_http_map_uri_to_path() function used clcf->regex to detect if it's working within a location given by a regular expression and have to replace full URI with alias (instead of a part matching the location prefix). This is incorrect due to clcf->regex being false in implicit locations created by if and limit_except. Fix is to preserve relevant information in clcf->alias instead, by setting it to NGX_MAX_SIZE_T_VALUE if an alias was specified in a regex location. diffstat: src/http/ngx_http_core_module.c | 30 +++++++++++------------------- 1 files changed, 11 insertions(+), 19 deletions(-) diffs (80 lines): diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c +++ b/src/http/ngx_http_core_module.c @@ -1245,10 +1245,8 @@ ngx_http_core_try_files_phase(ngx_http_r if (!alias) { reserve = len > r->uri.len ? len - r->uri.len : 0; -#if (NGX_PCRE) - } else if (clcf->regex) { + } else if (alias == NGX_MAX_SIZE_T_VALUE) { reserve = len; -#endif } else { reserve = len > r->uri.len - alias ? len - (r->uri.len - alias) : 0; @@ -1365,13 +1363,12 @@ ngx_http_core_try_files_phase(ngx_http_r if (!alias) { r->uri = path; -#if (NGX_PCRE) - } else if (clcf->regex) { + } else if (alias == NGX_MAX_SIZE_T_VALUE) { if (!test_dir) { r->uri = path; r->add_uri_to_alias = 1; } -#endif + } else { r->uri.len = alias + path.len; r->uri.data = ngx_pnalloc(r->pool, r->uri.len); @@ -2006,16 +2003,12 @@ ngx_http_map_uri_to_path(ngx_http_reques } else { -#if (NGX_PCRE) - ngx_uint_t captures; - - captures = alias && clcf->regex; - - reserved += captures ? r->add_uri_to_alias ? r->uri.len + 1 : 1 - : r->uri.len - alias + 1; -#else - reserved += r->uri.len - alias + 1; -#endif + if (alias == NGX_MAX_SIZE_T_VALUE) { + reserved += r->add_uri_to_alias ? r->uri.len + 1 : 1; + + } else { + reserved += r->uri.len - alias + 1; + } if (ngx_http_script_run(r, path, clcf->root_lengths->elts, reserved, clcf->root_values->elts) @@ -2033,8 +2026,7 @@ ngx_http_map_uri_to_path(ngx_http_reques *root_length = path->len - reserved; last = path->data + *root_length; -#if (NGX_PCRE) - if (captures) { + if (alias == NGX_MAX_SIZE_T_VALUE) { if (!r->add_uri_to_alias) { *last = '\0'; return last; @@ -2042,7 +2034,6 @@ ngx_http_map_uri_to_path(ngx_http_reques alias = 0; } -#endif } last = ngx_cpystrn(last, r->uri.data + alias, r->uri.len - alias + 1); @@ -4476,6 +4467,7 @@ ngx_http_core_root(ngx_conf_t *cf, ngx_c #if (NGX_PCRE) if (alias && clcf->regex) { + clcf->alias = NGX_MAX_SIZE_T_VALUE; n = 1; } #endif From idinasabari at gmail.com Fri May 16 18:26:34 2014 From: idinasabari at gmail.com (idina sabari) Date: Fri, 16 May 2014 11:26:34 -0700 Subject: Fwd: nginx-devel Digest, Vol 55, Issue 15 In-Reply-To: References: Message-ID: ---------- Forwarded message ---------- From: Date: 2014-05-16 5:00 GMT-07:00 Subject: nginx-devel Digest, Vol 55, Issue 15 To: nginx-devel at nginx.org Send nginx-devel mailing list submissions to nginx-devel at nginx.org To subscribe or unsubscribe via the World Wide Web, visit http://mailman.nginx.org/mailman/listinfo/nginx-devel or, via email, send a message with subject or body 'help' to nginx-devel-request at nginx.org You can reach the person managing the list at nginx-devel-owner at nginx.org When replying, please edit your Subject line so it is more specific than "Re: Contents of nginx-devel digest..." Today's Topics: 1. [nginx] SPDY: set log action for PROXY protocol only while parsi... (Valentin Bartenev) 2. [nginx] SPDY: improved logging. (Valentin Bartenev) 3. [nginx] SPDY: refactored ngx_http_spdy_state_headers(). (Valentin Bartenev) 4. [nginx] SPDY: added a debug point to the state buffer overflow p... (Valentin Bartenev) 5. [nginx] SPDY: added protection from overrun of the receive buffer. (Valentin Bartenev) ---------------------------------------------------------------------- Message: 1 Date: Thu, 15 May 2014 15:25:50 +0000 From: Valentin Bartenev To: nginx-devel at nginx.org Subject: [nginx] SPDY: set log action for PROXY protocol only while parsi... Message-ID: Content-Type: text/plain; charset="us-ascii" details: http://hg.nginx.org/nginx/rev/ed5fb7d22d46 branches: changeset: 5692:ed5fb7d22d46 user: Valentin Bartenev date: Thu May 15 19:22:06 2014 +0400 description: SPDY: set log action for PROXY protocol only while parsing it. Handling of PROXY protocol for SPDY connection is currently implemented as a SPDY state. And while nginx waiting for PROXY protocol data it continues to process SPDY connection: initializes zlib context, sends control frames. diffstat: src/http/ngx_http_spdy.c | 17 +++++++++-------- 1 files changed, 9 insertions(+), 8 deletions(-) diffs (40 lines): diff -r 31349361558e -r ed5fb7d22d46 src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Wed Apr 30 20:34:20 2014 +0400 +++ b/src/http/ngx_http_spdy.c Thu May 15 19:22:06 2014 +0400 @@ -424,12 +424,8 @@ ngx_http_spdy_init(ngx_event_t *rev) sc->init_window = NGX_SPDY_INIT_STREAM_WINDOW; - sc->handler = ngx_http_spdy_state_head; - - if (hc->proxy_protocol) { - c->log->action = "reading PROXY protocol"; - sc->handler = ngx_http_spdy_proxy_protocol; - } + sc->handler = hc->proxy_protocol ? ngx_http_spdy_proxy_protocol + : ngx_http_spdy_state_head; sc->zstream_in.zalloc = ngx_http_spdy_zalloc; sc->zstream_in.zfree = ngx_http_spdy_zfree; @@ -823,14 +819,19 @@ static u_char * ngx_http_spdy_proxy_protocol(ngx_http_spdy_connection_t *sc, u_char *pos, u_char *end) { + ngx_log_t *log; + + log = sc->connection->log; + log->action = "reading PROXY protocol"; + pos = ngx_proxy_protocol_parse(sc->connection, pos, end); + log->action = "processing SPDY"; + if (pos == NULL) { return ngx_http_spdy_state_protocol_error(sc); } - sc->connection->log->action = "processing SPDY"; - return ngx_http_spdy_state_complete(sc, pos, end); } ------------------------------ Message: 2 Date: Thu, 15 May 2014 15:25:52 +0000 From: Valentin Bartenev To: nginx-devel at nginx.org Subject: [nginx] SPDY: improved logging. Message-ID: Content-Type: text/plain; charset="us-ascii" details: http://hg.nginx.org/nginx/rev/701d6e17e42c branches: changeset: 5693:701d6e17e42c user: Valentin Bartenev date: Wed Apr 30 20:34:20 2014 +0400 description: SPDY: improved logging. diffstat: src/http/ngx_http_spdy.c | 210 +++++++++++++++++++++++--------- src/http/ngx_http_spdy_filter_module.c | 3 +- 2 files changed, 151 insertions(+), 62 deletions(-) diffs (truncated from 656 to 300 lines): diff -r ed5fb7d22d46 -r 701d6e17e42c src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Thu May 15 19:22:06 2014 +0400 +++ b/src/http/ngx_http_spdy.c Wed Apr 30 20:34:20 2014 +0400 @@ -395,8 +395,7 @@ ngx_http_spdy_init(ngx_event_t *rev) c = rev->data; hc = c->data; - ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, - "init spdy request"); + ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "init spdy request"); c->log->action = "processing SPDY"; @@ -556,7 +555,7 @@ ngx_http_spdy_read_handler(ngx_event_t * if (n == 0 && (sc->incomplete || sc->processing)) { ngx_log_error(NGX_LOG_INFO, c->log, 0, - "client closed prematurely connection"); + "client prematurely closed connection"); } if (n == 0 || n == NGX_ERROR) { @@ -644,7 +643,7 @@ ngx_http_spdy_write_handler(ngx_event_t stream->handled = 0; ngx_log_debug1(NGX_LOG_DEBUG_HTTP, c->log, 0, - "spdy run stream %ui", stream->id); + "run spdy stream %ui", stream->id); wev = stream->request->connection->write; wev->handler(wev); @@ -860,7 +859,7 @@ ngx_http_spdy_state_head(ngx_http_spdy_c pos += sizeof(uint32_t); ngx_log_debug3(NGX_LOG_DEBUG_HTTP, sc->connection->log, 0, - "spdy process frame head:%08XD f:%Xd l:%uz", + "process spdy frame head:%08XD f:%Xd l:%uz", head, sc->flags, sc->length); if (ngx_spdy_ctl_frame_check(head)) { @@ -872,6 +871,8 @@ ngx_http_spdy_state_head(ngx_http_spdy_c return ngx_http_spdy_state_syn_stream(sc, pos, end); case NGX_SPDY_SYN_REPLY: + ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, + "client sent unexpected SYN_REPLY frame"); return ngx_http_spdy_state_protocol_error(sc); case NGX_SPDY_RST_STREAM: @@ -887,6 +888,8 @@ ngx_http_spdy_state_head(ngx_http_spdy_c return ngx_http_spdy_state_skip(sc, pos, end); /* TODO */ case NGX_SPDY_HEADERS: + ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, + "client sent unexpected HEADERS frame"); return ngx_http_spdy_state_protocol_error(sc); case NGX_SPDY_WINDOW_UPDATE: @@ -904,10 +907,8 @@ ngx_http_spdy_state_head(ngx_http_spdy_c return ngx_http_spdy_state_data(sc, pos, end); } - - /* TODO version & type check */ - ngx_log_debug0(NGX_LOG_DEBUG_HTTP, sc->connection->log, 0, - "spdy unknown frame"); + ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, + "client sent invalid frame"); return ngx_http_spdy_state_protocol_error(sc); } @@ -927,7 +928,10 @@ ngx_http_spdy_state_syn_stream(ngx_http_ } if (sc->length <= NGX_SPDY_SYN_STREAM_SIZE) { - /* TODO logging */ + ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, + "client sent SYN_STREAM frame with incorrect length %uz", + sc->length); + return ngx_http_spdy_state_protocol_error(sc); } @@ -968,7 +972,7 @@ ngx_http_spdy_state_syn_stream(ngx_http_ if (sc->processing >= sscf->concurrent_streams) { ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, - "spdy concurrent streams exceeded %ui", sc->processing); + "concurrent streams exceeded %ui", sc->processing); if (ngx_http_spdy_send_rst_stream(sc, sid, NGX_SPDY_REFUSED_STREAM, prio) @@ -1026,7 +1030,7 @@ ngx_http_spdy_state_headers(ngx_http_spd r = sc->stream->request; ngx_log_debug2(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, - "spdy process HEADERS %uz of %uz", size, sc->length); + "process spdy header block %uz of %uz", size, sc->length); buf = r->header_in; @@ -1101,7 +1105,7 @@ ngx_http_spdy_state_headers(ngx_http_spd buf->pos += NGX_SPDY_NV_NUM_SIZE; ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, - "spdy HEADERS block consists of %ui entries", + "spdy header block has %ui entries", sc->entries); if (ngx_list_init(&r->headers_in.headers, r->pool, 20, @@ -1142,7 +1146,6 @@ ngx_http_spdy_state_headers(ngx_http_spd rc = ngx_http_spdy_alloc_large_header_buffer(r); if (rc == NGX_DECLINED) { - /* TODO logging */ ngx_http_finalize_request(r, NGX_HTTP_REQUEST_HEADER_TOO_LARGE); return ngx_http_spdy_state_headers_skip(sc, pos, end); @@ -1179,9 +1182,8 @@ ngx_http_spdy_state_headers(ngx_http_spd } if (complete) { - /* TODO: improve error message */ ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, - "spdy again while last chunk"); + "premature end of spdy header block"); return ngx_http_spdy_state_headers_error(sc, pos, end); } @@ -1190,13 +1192,7 @@ ngx_http_spdy_state_headers(ngx_http_spd ngx_http_spdy_state_headers); case NGX_HTTP_PARSE_INVALID_HEADER: - - /* TODO: improve error message */ - ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, - "client sent invalid header line"); - ngx_http_finalize_request(r, NGX_HTTP_BAD_REQUEST); - return ngx_http_spdy_state_headers_skip(sc, pos, end); default: /* NGX_ERROR */ @@ -1262,6 +1258,9 @@ ngx_http_spdy_state_headers_skip(ngx_htt ngx_http_spdy_state_headers_skip); } + ngx_log_debug2(NGX_LOG_DEBUG_HTTP, sc->connection->log, 0, + "spdy header block skip %uz of %uz", size, sc->length); + sc->zstream_in.next_in = pos; sc->zstream_in.avail_in = (size < sc->length) ? size : sc->length; @@ -1364,7 +1363,7 @@ ngx_http_spdy_state_window_update(ngx_ht ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, "client violated flow control for stream %ui: " "received WINDOW_UPDATE frame with delta %uz " - "that is not allowed for window %z", + "not allowed for window %z", sid, delta, stream->send_window); if (ngx_http_spdy_terminate_stream(sc, stream, @@ -1397,7 +1396,7 @@ ngx_http_spdy_state_window_update(ngx_ht ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, "client violated connection flow control: " "received WINDOW_UPDATE frame with delta %uz " - "that is not allowed for window %uz", + "not allowed for window %uz", delta, sc->send_window); return ngx_http_spdy_state_protocol_error(sc); @@ -1440,8 +1439,8 @@ ngx_http_spdy_state_data(ngx_http_spdy_c if (sc->length > sc->recv_window) { ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, - "client violated connection flow control: length of " - "received DATA frame %uz, while available window %uz", + "client violated connection flow control: " + "received DATA frame length %uz, available window %uz", sc->length, sc->recv_window); return ngx_http_spdy_state_protocol_error(sc); @@ -1465,13 +1464,16 @@ ngx_http_spdy_state_data(ngx_http_spdy_c stream = sc->stream; if (stream == NULL) { + ngx_log_debug0(NGX_LOG_DEBUG_HTTP, sc->connection->log, 0, + "unknown spdy stream"); + return ngx_http_spdy_state_skip(sc, pos, end); } if (sc->length > stream->recv_window) { ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, - "client violated flow control for stream %ui: length of " - "received DATA frame %uz, while available window %uz", + "client violated flow control for stream %ui: " + "received DATA frame length %uz, available window %uz", stream->id, sc->length, stream->recv_window); if (ngx_http_spdy_terminate_stream(sc, stream, @@ -1501,7 +1503,7 @@ ngx_http_spdy_state_data(ngx_http_spdy_c if (stream->in_closed) { ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, - "client sent DATA frame for half closed stream %ui", + "client sent DATA frame for half-closed stream %ui", stream->id); if (ngx_http_spdy_terminate_stream(sc, stream, @@ -1544,7 +1546,10 @@ ngx_http_spdy_state_read_data(ngx_http_s stream->in_closed = 1; } - /* TODO log and accounting */ + ngx_log_debug1(NGX_LOG_DEBUG_HTTP, sc->connection->log, 0, + "skipping spdy DATA frame, reason: %d", + stream->skip_data); + return ngx_http_spdy_state_skip(sc, pos, end); } @@ -1573,7 +1578,10 @@ ngx_http_spdy_state_read_data(ngx_http_s if (r->headers_in.content_length_n != -1 && r->headers_in.content_length_n < rb->rest) { - /* TODO logging */ + ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, + "client intended to send body data " + "larger than declared"); + stream->skip_data = NGX_SPDY_DATA_ERROR; goto error; @@ -1584,9 +1592,8 @@ ngx_http_spdy_state_read_data(ngx_http_s && clcf->client_max_body_size < rb->rest) { ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, - "client intended to send too large chunked " - "body: %O bytes", - rb->rest); + "client intended to send " + "too large chunked body: %O bytes", rb->rest); stream->skip_data = NGX_SPDY_DATA_ERROR; goto error; @@ -1638,7 +1645,7 @@ ngx_http_spdy_state_read_data(ngx_http_s } else if (r->headers_in.content_length_n != rb->rest) { ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, "client prematurely closed stream: " - "%O of %O bytes of request body received", + "only %O out of %O bytes of request body received", rb->rest, r->headers_in.content_length_n); stream->skip_data = NGX_SPDY_DATA_ERROR; @@ -1717,9 +1724,11 @@ ngx_http_spdy_state_rst_stream(ngx_http_ "spdy RST_STREAM sid:%ui st:%ui", sid, status); stream = ngx_http_spdy_get_stream_by_id(sc, sid); + if (stream == NULL) { ngx_log_debug0(NGX_LOG_DEBUG_HTTP, sc->connection->log, 0, - "unknown stream, probably it has been closed already"); + "unknown spdy stream"); + return ngx_http_spdy_state_complete(sc, pos, end); } @@ -1738,7 +1747,7 @@ ngx_http_spdy_state_rst_stream(ngx_http_ case NGX_SPDY_INTERNAL_ERROR: ngx_log_error(NGX_LOG_INFO, fc->log, 0, - "client terminated stream %ui because of internal error", + "client terminated stream %ui due to internal error", sid); break; @@ -1770,7 +1779,10 @@ ngx_http_spdy_state_ping(ngx_http_spdy_c } if (sc->length != NGX_SPDY_PING_SIZE) { - /* TODO logging */ + ngx_log_error(NGX_LOG_INFO, sc->connection->log, 0, + "client sent PING frame with incorrect length %uz", + sc->length); + return ngx_http_spdy_state_protocol_error(sc); } @@ -1810,6 +1822,9 @@ ngx_http_spdy_state_skip(ngx_http_spdy_c size = end - pos; + ngx_log_debug2(NGX_LOG_DEBUG_HTTP, sc->connection->log, 0, + "spdy frame skip %uz of %uz", size, sc->length); + if (size < sc->length) { sc->length -= size; return ngx_http_spdy_state_save(sc, end, end, @@ -1839,13 +1854,16 @@ ngx_http_spdy_state_settings(ngx_http_sp ------------------------------ Message: 3 Date: Thu, 15 May 2014 15:25:53 +0000 From: Valentin Bartenev To: nginx-devel at nginx.org Subject: [nginx] SPDY: refactored ngx_http_spdy_state_headers(). Message-ID: Content-Type: text/plain; charset="us-ascii" details: http://hg.nginx.org/nginx/rev/9a95eb9f448b branches: changeset: 5694:9a95eb9f448b user: Valentin Bartenev date: Wed Apr 30 20:34:20 2014 +0400 description: SPDY: refactored ngx_http_spdy_state_headers(). This change is similar to d2ac5cf4056d. Special flag of completeness looks surplus when there is also a counter of frame bytes left. diffstat: src/http/ngx_http_spdy.c | 13 ++++--------- 1 files changed, 4 insertions(+), 9 deletions(-) diffs (52 lines): diff -r 701d6e17e42c -r 9a95eb9f448b src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Wed Apr 30 20:34:20 2014 +0400 +++ b/src/http/ngx_http_spdy.c Wed Apr 30 20:34:20 2014 +0400 @@ -1009,7 +1009,6 @@ ngx_http_spdy_state_headers(ngx_http_spd size_t size; ngx_buf_t *buf; ngx_int_t rc; - ngx_uint_t complete; ngx_http_request_t *r; size = end - pos; @@ -1019,12 +1018,8 @@ ngx_http_spdy_state_headers(ngx_http_spd ngx_http_spdy_state_headers); } - if (size >= sc->length) { + if (size > sc->length) { size = sc->length; - complete = 1; - - } else { - complete = 0; } r = sc->stream->request; @@ -1089,7 +1084,7 @@ ngx_http_spdy_state_headers(ngx_http_spd if (buf->last - buf->pos < NGX_SPDY_NV_NUM_SIZE) { - if (complete) { + if (sc->length == 0) { ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "premature end of spdy header block"); @@ -1181,7 +1176,7 @@ ngx_http_spdy_state_headers(ngx_http_spd continue; } - if (complete) { + if (sc->length == 0) { ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "premature end of spdy header block"); @@ -1225,7 +1220,7 @@ ngx_http_spdy_state_headers(ngx_http_spd return ngx_http_spdy_state_headers_error(sc, pos, end); } - if (!complete) { + if (sc->length) { return ngx_http_spdy_state_save(sc, pos, end, ngx_http_spdy_state_headers); } ------------------------------ Message: 4 Date: Thu, 15 May 2014 15:25:55 +0000 From: Valentin Bartenev To: nginx-devel at nginx.org Subject: [nginx] SPDY: added a debug point to the state buffer overflow p... Message-ID: Content-Type: text/plain; charset="us-ascii" details: http://hg.nginx.org/nginx/rev/231588611230 branches: changeset: 5695:231588611230 user: Valentin Bartenev date: Wed Apr 30 20:34:20 2014 +0400 description: SPDY: added a debug point to the state buffer overflow protection. diffstat: src/http/ngx_http_spdy.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (11 lines): diff -r 9a95eb9f448b -r 231588611230 src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Wed Apr 30 20:34:20 2014 +0400 +++ b/src/http/ngx_http_spdy.c Wed Apr 30 20:34:20 2014 +0400 @@ -1944,6 +1944,7 @@ ngx_http_spdy_state_save(ngx_http_spdy_c ngx_log_error(NGX_LOG_ALERT, sc->connection->log, 0, "state buffer overflow: %uz bytes required", size); + ngx_debug_point(); return ngx_http_spdy_state_internal_error(sc); } ------------------------------ Message: 5 Date: Thu, 15 May 2014 15:25:57 +0000 From: Valentin Bartenev To: nginx-devel at nginx.org Subject: [nginx] SPDY: added protection from overrun of the receive buffer. Message-ID: Content-Type: text/plain; charset="us-ascii" details: http://hg.nginx.org/nginx/rev/19a14a484707 branches: changeset: 5696:19a14a484707 user: Valentin Bartenev date: Wed Apr 30 20:34:20 2014 +0400 description: SPDY: added protection from overrun of the receive buffer. diffstat: src/http/ngx_http_spdy.c | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-) diffs (18 lines): diff -r 231588611230 -r 19a14a484707 src/http/ngx_http_spdy.c --- a/src/http/ngx_http_spdy.c Wed Apr 30 20:34:20 2014 +0400 +++ b/src/http/ngx_http_spdy.c Wed Apr 30 20:34:20 2014 +0400 @@ -1921,6 +1921,14 @@ ngx_http_spdy_state_complete(ngx_http_sp ngx_log_debug2(NGX_LOG_DEBUG_HTTP, sc->connection->log, 0, "spdy frame complete pos:%p end:%p", pos, end); + if (pos > end) { + ngx_log_error(NGX_LOG_ALERT, sc->connection->log, 0, + "receive buffer overrun"); + + ngx_debug_point(); + return ngx_http_spdy_state_internal_error(sc); + } + sc->handler = ngx_http_spdy_state_head; sc->stream = NULL; ------------------------------ _______________________________________________ nginx-devel mailing list nginx-devel at nginx.org http://mailman.nginx.org/mailman/listinfo/nginx-devel End of nginx-devel Digest, Vol 55, Issue 15 ******************************************* -------------- next part -------------- An HTML attachment was scrubbed... URL: From yurnerola at gmail.com Sat May 17 13:55:08 2014 From: yurnerola at gmail.com (liubin) Date: Sat, 17 May 2014 21:55:08 +0800 Subject: mp4_module doubt Message-ID: Hello? I have a question with ngx_http_mp4_module. As i know,ngx_http_mp4_module compute start_module using stts. For example?we get start_sample=3 from the stts. However,the stss is 1,5,9. Then,why don?t make start_sample=5? Can 3 display nomally while there is no 1 as sync sample? Best regards, -yurnero -------------- next part -------------- An HTML attachment was scrubbed... URL: From arut at nginx.com Sat May 17 16:26:36 2014 From: arut at nginx.com (Roman Arutyunyan) Date: Sat, 17 May 2014 20:26:36 +0400 Subject: mp4_module doubt In-Reply-To: References: Message-ID: <1983AA0D-88E3-4583-A29D-89655F6DCA02@nginx.com> Hello, The mp4 module does not align start time to key frames. If you specify unaligned start time you'll get video stream starting with a non-key frame. Most players just ignore video data until the first key frame. On May 17, 2014, at 5:55 PM, liubin wrote: > Hello? > I have a question with ngx_http_mp4_module. > As i know,ngx_http_mp4_module compute start_module using stts. > For example?we get start_sample=3 from the stts. > However,the stss is 1,5,9. > Then,why don?t make start_sample=5? > Can 3 display nomally while there is no 1 as sync sample? > Best regards, > -yurnero > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel From yurnerola at gmail.com Sun May 18 12:22:49 2014 From: yurnerola at gmail.com (liubin) Date: Sun, 18 May 2014 20:22:49 +0800 Subject: mp4_module doubt In-Reply-To: References: Message-ID: <0BA9B631-70BE-4020-B476-C24B93994025@gmail.com> Hello, Thanks for your reply.And is it necessary to optimize it? > Date: Sat, 17 May 2014 20:26:36 +0400 > From: Roman Arutyunyan > To: nginx-devel at nginx.org > Subject: Re: mp4_module doubt > Message-ID: <1983AA0D-88E3-4583-A29D-89655F6DCA02 at nginx.com> > Content-Type: text/plain; charset=utf-8 > > Hello, > > The mp4 module does not align start time to key frames. > If you specify unaligned start time you'll get video stream starting > with a non-key frame. Most players just ignore video data until > the first key frame. > > On May 17, 2014, at 5:55 PM, liubin wrote: > >> Hello? >> I have a question with ngx_http_mp4_module. >> As i know,ngx_http_mp4_module compute start_module using stts. >> For example?we get start_sample=3 from the stts. >> However,the stss is 1,5,9. >> Then,why don?t make start_sample=5? >> Can 3 display nomally while there is no 1 as sync sample? >> Best regards, >> -yurnero >> >> _______________________________________________ >> nginx-devel mailing list >> nginx-devel at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From arut at nginx.com Sun May 18 16:00:32 2014 From: arut at nginx.com (Roman Arutyunyan) Date: Sun, 18 May 2014 20:00:32 +0400 Subject: mp4_module doubt In-Reply-To: <0BA9B631-70BE-4020-B476-C24B93994025@gmail.com> References: <0BA9B631-70BE-4020-B476-C24B93994025@gmail.com> Message-ID: Hello, No plans for this feature yet. What is your exact use case? What is the software you have problems with? On May 18, 2014, at 4:22 PM, liubin wrote: > Hello, > Thanks for your reply.And is it necessary to optimize it? >> Date: Sat, 17 May 2014 20:26:36 +0400 >> From: Roman Arutyunyan >> To: nginx-devel at nginx.org >> Subject: Re: mp4_module doubt >> Message-ID: <1983AA0D-88E3-4583-A29D-89655F6DCA02 at nginx.com> >> Content-Type: text/plain; charset=utf-8 >> >> Hello, >> >> The mp4 module does not align start time to key frames. >> If you specify unaligned start time you'll get video stream starting >> with a non-key frame. Most players just ignore video data until >> the first key frame. >> >> On May 17, 2014, at 5:55 PM, liubin wrote: >> >>> Hello? >>> I have a question with ngx_http_mp4_module. >>> As i know,ngx_http_mp4_module compute start_module using stts. >>> For example?we get start_sample=3 from the stts. >>> However,the stss is 1,5,9. >>> Then,why don?t make start_sample=5? >>> Can 3 display nomally while there is no 1 as sync sample? >>> Best regards, >>> -yurnero >>> >>> _______________________________________________ >>> nginx-devel mailing list >>> nginx-devel at nginx.org >>> http://mailman.nginx.org/mailman/listinfo/nginx-devel >> >> > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel From yurnerola at gmail.com Mon May 19 00:56:56 2014 From: yurnerola at gmail.com (=?UTF-8?B?5YiY5paM?=) Date: Mon, 19 May 2014 08:56:56 +0800 Subject: mp4_module doubt In-Reply-To: References: <0BA9B631-70BE-4020-B476-C24B93994025@gmail.com> Message-ID: Hello, I'm just reading this module and wandering if we can reduce flow from nginx by align start time to key frames.Our company is a video website from China.And i want to know does it make sense to do this. 2014-05-19 0:00 GMT+08:00, Roman Arutyunyan : > Hello, > > No plans for this feature yet. > What is your exact use case? What is the software you have problems with? > > On May 18, 2014, at 4:22 PM, liubin wrote: > >> Hello, >> Thanks for your reply.And is it necessary to optimize it? >>> Date: Sat, 17 May 2014 20:26:36 +0400 >>> From: Roman Arutyunyan >>> To: nginx-devel at nginx.org >>> Subject: Re: mp4_module doubt >>> Message-ID: <1983AA0D-88E3-4583-A29D-89655F6DCA02 at nginx.com> >>> Content-Type: text/plain; charset=utf-8 >>> >>> Hello, >>> >>> The mp4 module does not align start time to key frames. >>> If you specify unaligned start time you'll get video stream starting >>> with a non-key frame. Most players just ignore video data until >>> the first key frame. >>> >>> On May 17, 2014, at 5:55 PM, liubin wrote: >>> >>>> Hello? >>>> I have a question with ngx_http_mp4_module. >>>> As i know,ngx_http_mp4_module compute start_module using stts. >>>> For example?we get start_sample=3 from the stts. >>>> However,the stss is 1,5,9. >>>> Then,why don?t make start_sample=5? >>>> Can 3 display nomally while there is no 1 as sync sample? >>>> Best regards, >>>> -yurnero >>>> >>>> _______________________________________________ >>>> nginx-devel mailing list >>>> nginx-devel at nginx.org >>>> http://mailman.nginx.org/mailman/listinfo/nginx-devel >>> >>> >> >> _______________________________________________ >> nginx-devel mailing list >> nginx-devel at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx-devel > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -- ------------------------------ Best regards, -yurnerola From mdounin at mdounin.ru Mon May 19 11:17:47 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 19 May 2014 15:17:47 +0400 Subject: mp4_module doubt In-Reply-To: References: <0BA9B631-70BE-4020-B476-C24B93994025@gmail.com> Message-ID: <20140519111747.GB1849@mdounin.ru> Hello! On Mon, May 19, 2014 at 08:56:56AM +0800, ?? wrote: > Hello, > I'm just reading this module and wandering if we can reduce flow > from nginx by align start time to key frames.Our company is a video > website from China.And i want to know does it make sense to do this. It doesn't. It's player responsibility to do requests they actually need. > 2014-05-19 0:00 GMT+08:00, Roman Arutyunyan : > > Hello, > > > > No plans for this feature yet. > > What is your exact use case? What is the software you have problems with? > > > > On May 18, 2014, at 4:22 PM, liubin wrote: > > > >> Hello, > >> Thanks for your reply.And is it necessary to optimize it? > >>> Date: Sat, 17 May 2014 20:26:36 +0400 > >>> From: Roman Arutyunyan > >>> To: nginx-devel at nginx.org > >>> Subject: Re: mp4_module doubt > >>> Message-ID: <1983AA0D-88E3-4583-A29D-89655F6DCA02 at nginx.com> > >>> Content-Type: text/plain; charset=utf-8 > >>> > >>> Hello, > >>> > >>> The mp4 module does not align start time to key frames. > >>> If you specify unaligned start time you'll get video stream starting > >>> with a non-key frame. Most players just ignore video data until > >>> the first key frame. > >>> > >>> On May 17, 2014, at 5:55 PM, liubin wrote: > >>> > >>>> Hello? > >>>> I have a question with ngx_http_mp4_module. > >>>> As i know,ngx_http_mp4_module compute start_module using stts. > >>>> For example?we get start_sample=3 from the stts. > >>>> However,the stss is 1,5,9. > >>>> Then,why don?t make start_sample=5? > >>>> Can 3 display nomally while there is no 1 as sync sample? > >>>> Best regards, > >>>> -yurnero > >>>> > >>>> _______________________________________________ > >>>> nginx-devel mailing list > >>>> nginx-devel at nginx.org > >>>> http://mailman.nginx.org/mailman/listinfo/nginx-devel > >>> > >>> > >> > >> _______________________________________________ > >> nginx-devel mailing list > >> nginx-devel at nginx.org > >> http://mailman.nginx.org/mailman/listinfo/nginx-devel > > > > _______________________________________________ > > nginx-devel mailing list > > nginx-devel at nginx.org > > http://mailman.nginx.org/mailman/listinfo/nginx-devel > > > > > -- > > ------------------------------ > Best regards, > -yurnerola > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel -- Maxim Dounin http://nginx.org/ From arut at nginx.com Mon May 19 12:02:43 2014 From: arut at nginx.com (Roman Arutyunyan) Date: Mon, 19 May 2014 16:02:43 +0400 Subject: mp4_module doubt In-Reply-To: References: <0BA9B631-70BE-4020-B476-C24B93994025@gmail.com> Message-ID: On 19 May 2014, at 04:56, ?? wrote: > Hello, > I'm just reading this module and wandering if we can reduce flow > from nginx by align start time to key frames.Our company is a video > website from China.And i want to know does it make sense to do this. Here?s the patch solving your problem. Currently there are no plans to add this to upstream but that can change. It would be nice to hear your feedback. Here?s an example of ?mp4_align_start? directive added by the patch. location /mp4 { mp4; mp4_align_start on; root /tmp/; } # HG changeset patch # User Roman Arutyunyan # Date 1400496445 -14400 # Mon May 19 14:47:25 2014 +0400 # Node ID fccafb910a3361e3e2d6ca8afd6edfb576444997 # Parent 3a48775f1535fe37cd9c034d92c5a5e9ae208c1e Mp4: align start to video key frame. The new directive mp4_align_start adjusts start time to make the result mp4 start with a key frame. This makes sense for some players displaying garbage until the first key frame. diff -r 3a48775f1535 -r fccafb910a33 src/http/modules/ngx_http_mp4_module.c --- a/src/http/modules/ngx_http_mp4_module.c Tue Apr 29 12:28:41 2014 +0400 +++ b/src/http/modules/ngx_http_mp4_module.c Mon May 19 14:47:25 2014 +0400 @@ -43,6 +43,7 @@ typedef struct { size_t buffer_size; size_t max_buffer_size; + ngx_flag_t align_start; } ngx_http_mp4_conf_t; @@ -303,6 +304,8 @@ static ngx_int_t ngx_http_mp4_update_co6 ngx_http_mp4_trak_t *trak); static void ngx_http_mp4_adjust_co64_atom(ngx_http_mp4_file_t *mp4, ngx_http_mp4_trak_t *trak, off_t adjustment); +static ngx_int_t ngx_http_mp4_align_start(ngx_http_mp4_file_t *mp4, + ngx_http_mp4_trak_t *trak); static char *ngx_http_mp4(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); static void *ngx_http_mp4_create_conf(ngx_conf_t *cf); @@ -332,6 +335,13 @@ static ngx_command_t ngx_http_mp4_comma offsetof(ngx_http_mp4_conf_t, max_buffer_size), NULL }, + { ngx_string("mp4_align_start"), + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, + ngx_conf_set_flag_slot, + NGX_HTTP_LOC_CONF_OFFSET, + offsetof(ngx_http_mp4_conf_t, align_start), + NULL }, + ngx_null_command }; @@ -742,6 +752,14 @@ ngx_http_mp4_process(ngx_http_mp4_file_t end_offset = 0; trak = mp4->trak.elts; + if (conf->align_start) { + for (i = 0; i < mp4->trak.nelts; i++) { + if (ngx_http_mp4_align_start(mp4, &trak[i]) == NGX_OK) { + break; + } + } + } + for (i = 0; i < mp4->trak.nelts; i++) { if (ngx_http_mp4_update_stts_atom(mp4, &trak[i]) != NGX_OK) { @@ -3457,6 +3475,110 @@ ngx_http_mp4_adjust_co64_atom(ngx_http_m } +static ngx_int_t +ngx_http_mp4_align_start(ngx_http_mp4_file_t *mp4, ngx_http_mp4_trak_t *trak) +{ + uint32_t count, duration, sample, start_sample, + *stss_entry, *stss_end; + uint64_t start_time; + ngx_buf_t *stts_data, *stss_data; + ngx_mp4_stts_entry_t *stts_entry, *stts_end; + + stss_data = trak->out[NGX_HTTP_MP4_STSS_DATA].buf; + if (stss_data == NULL) { + return NGX_AGAIN; + } + + stts_data = trak->out[NGX_HTTP_MP4_STTS_DATA].buf; + if (stts_data == NULL) { + return NGX_AGAIN; + } + + ngx_log_debug0(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0, "align start"); + + start_time = (uint64_t) mp4->start * trak->timescale / 1000; + start_sample = 0; + + stts_entry = (ngx_mp4_stts_entry_t *) stts_data->pos; + stts_end = (ngx_mp4_stts_entry_t *) stts_data->last; + + while (stts_entry < stts_end) { + count = ngx_mp4_get_32value(stts_entry->count); + duration = ngx_mp4_get_32value(stts_entry->duration); + + ngx_log_debug3(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0, + "time:%uL, count:%uD, duration:%uD", + start_time, count, duration); + + if (start_time < (uint64_t) count * duration) { + start_sample += (ngx_uint_t) (start_time / duration); + goto found_stts; + } + + start_sample += count; + start_time -= count * duration; + stts_entry++; + } + + ngx_log_debug0(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0, "out of stts samples"); + + return NGX_AGAIN; + +found_stts: + + stss_entry = (uint32_t *) stss_data->pos; + stss_end = (uint32_t *) stss_data->last; + + while (stss_entry < stss_end) { + sample = ngx_mp4_get_32value(stss_entry); + + ngx_log_debug1(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0, + "sync:%uD", sample); + + if (sample >= start_sample + 1) { + sample--; + goto found_stss; + } + + stss_entry++; + } + + ngx_log_debug0(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0, "out of stss samples"); + + return NGX_AGAIN; + +found_stss: + + while (stts_entry < stts_end) { + count = ngx_mp4_get_32value(stts_entry->count); + duration = ngx_mp4_get_32value(stts_entry->duration); + + ngx_log_debug3(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0, + "time:%uL, count:%uD, duration:%uD", + start_time, count, duration); + + if (start_sample + count >= sample) { + start_time += (sample - start_sample) * duration; + mp4->start = (uint64_t) start_time * 1000 / trak->timescale; + + ngx_log_debug2(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0, + "aligned start:%ui, samples:%uD", + mp4->start, sample - start_sample); + + return NGX_OK; + } + + start_sample += count; + start_time -= count * duration; + stts_entry++; + } + + ngx_log_debug0(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0, "out of stts samples"); + + return NGX_AGAIN; +} + + static char * ngx_http_mp4(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) { @@ -3481,6 +3603,7 @@ ngx_http_mp4_create_conf(ngx_conf_t *cf) conf->buffer_size = NGX_CONF_UNSET_SIZE; conf->max_buffer_size = NGX_CONF_UNSET_SIZE; + conf->align_start = NGX_CONF_UNSET; return conf; } @@ -3495,6 +3618,7 @@ ngx_http_mp4_merge_conf(ngx_conf_t *cf, ngx_conf_merge_size_value(conf->buffer_size, prev->buffer_size, 512 * 1024); ngx_conf_merge_size_value(conf->max_buffer_size, prev->max_buffer_size, 10 * 1024 * 1024); + ngx_conf_merge_value(conf->align_start, prev->align_start, 0); return NGX_CONF_OK; } From mdounin at mdounin.ru Mon May 19 18:47:16 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 19 May 2014 18:47:16 +0000 Subject: [nginx] Charset filter: fixed charset setting on encoded replies. Message-ID: details: http://hg.nginx.org/nginx/rev/fcb0420f90de branches: changeset: 5698:fcb0420f90de user: Maxim Dounin date: Mon May 19 22:45:34 2014 +0400 description: Charset filter: fixed charset setting on encoded replies. If response is gzipped we can't recode response, but in case it's not needed we still can add charset to Content-Type. The r->ignore_content_encoding is dropped accordingly, charset with gzip_static now properly works without any special flags. diffstat: src/http/modules/ngx_http_charset_filter_module.c | 34 ++++++++++++---------- src/http/modules/ngx_http_gzip_static_module.c | 2 - src/http/ngx_http_request.h | 1 - 3 files changed, 19 insertions(+), 18 deletions(-) diffs (84 lines): diff --git a/src/http/modules/ngx_http_charset_filter_module.c b/src/http/modules/ngx_http_charset_filter_module.c --- a/src/http/modules/ngx_http_charset_filter_module.c +++ b/src/http/modules/ngx_http_charset_filter_module.c @@ -272,12 +272,27 @@ ngx_http_charset_header_filter(ngx_http_ return ngx_http_next_header_filter(r); } + if (source_charset == charset) { + r->headers_out.content_type.len = r->headers_out.content_type_len; + + ngx_http_set_charset(r, &dst); + + return ngx_http_next_header_filter(r); + } + + /* source_charset != charset */ + + if (r->headers_out.content_encoding + && r->headers_out.content_encoding->value.len) + { + return ngx_http_next_header_filter(r); + } + mcf = ngx_http_get_module_main_conf(r, ngx_http_charset_filter_module); charsets = mcf->charsets.elts; - if (source_charset != charset - && (charsets[source_charset].tables == NULL - || charsets[source_charset].tables[charset] == NULL)) + if (charsets[source_charset].tables == NULL + || charsets[source_charset].tables[charset] == NULL) { goto no_charset_map; } @@ -286,11 +301,7 @@ ngx_http_charset_header_filter(ngx_http_ ngx_http_set_charset(r, &dst); - if (source_charset != charset) { - return ngx_http_charset_ctx(r, charsets, charset, source_charset); - } - - return ngx_http_next_header_filter(r); + return ngx_http_charset_ctx(r, charsets, charset, source_charset); no_charset_map: @@ -311,13 +322,6 @@ ngx_http_destination_charset(ngx_http_re ngx_http_charset_loc_conf_t *mlcf; ngx_http_charset_main_conf_t *mcf; - if (!r->ignore_content_encoding - && r->headers_out.content_encoding - && r->headers_out.content_encoding->value.len) - { - return NGX_DECLINED; - } - if (r->headers_out.content_type.len == 0) { return NGX_DECLINED; } diff --git a/src/http/modules/ngx_http_gzip_static_module.c b/src/http/modules/ngx_http_gzip_static_module.c --- a/src/http/modules/ngx_http_gzip_static_module.c +++ b/src/http/modules/ngx_http_gzip_static_module.c @@ -246,8 +246,6 @@ ngx_http_gzip_static_handler(ngx_http_re ngx_str_set(&h->value, "gzip"); r->headers_out.content_encoding = h; - r->ignore_content_encoding = 1; - /* we need to allocate all before the header would be sent */ b = ngx_pcalloc(r->pool, sizeof(ngx_buf_t)); diff --git a/src/http/ngx_http_request.h b/src/http/ngx_http_request.h --- a/src/http/ngx_http_request.h +++ b/src/http/ngx_http_request.h @@ -511,7 +511,6 @@ struct ngx_http_request_s { unsigned discard_body:1; unsigned internal:1; unsigned error_page:1; - unsigned ignore_content_encoding:1; unsigned filter_finalize:1; unsigned post_action:1; unsigned request_complete:1; From mdounin at mdounin.ru Mon May 19 18:47:18 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 19 May 2014 18:47:18 +0000 Subject: [nginx] Setting $args now invalidates unparsed uri. Message-ID: details: http://hg.nginx.org/nginx/rev/e9dbffc18a77 branches: changeset: 5699:e9dbffc18a77 user: Maxim Dounin date: Mon May 19 22:45:35 2014 +0400 description: Setting $args now invalidates unparsed uri. Prodded by Yichun Zhang. diffstat: src/http/ngx_http_variables.c | 20 +++++++++++++++++++- 1 files changed, 19 insertions(+), 1 deletions(-) diffs (67 lines): diff --git a/src/http/ngx_http_variables.c b/src/http/ngx_http_variables.c --- a/src/http/ngx_http_variables.c +++ b/src/http/ngx_http_variables.c @@ -13,8 +13,10 @@ static ngx_int_t ngx_http_variable_request(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data); +#if 0 static void ngx_http_variable_request_set(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data); +#endif static ngx_int_t ngx_http_variable_request_get_size(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data); static void ngx_http_variable_request_set_size(ngx_http_request_t *r, @@ -64,6 +66,8 @@ static ngx_int_t ngx_http_variable_schem ngx_http_variable_value_t *v, uintptr_t data); static ngx_int_t ngx_http_variable_https(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data); +static void ngx_http_variable_set_args(ngx_http_request_t *r, + ngx_http_variable_value_t *v, uintptr_t data); static ngx_int_t ngx_http_variable_is_args(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data); static ngx_int_t ngx_http_variable_document_root(ngx_http_request_t *r, @@ -223,7 +227,7 @@ static ngx_http_variable_t ngx_http_cor NGX_HTTP_VAR_NOCACHEABLE, 0 }, { ngx_string("args"), - ngx_http_variable_request_set, + ngx_http_variable_set_args, ngx_http_variable_request, offsetof(ngx_http_request_t, args), NGX_HTTP_VAR_CHANGEABLE|NGX_HTTP_VAR_NOCACHEABLE, 0 }, @@ -662,6 +666,8 @@ ngx_http_variable_request(ngx_http_reque } +#if 0 + static void ngx_http_variable_request_set(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data) @@ -674,6 +680,8 @@ ngx_http_variable_request_set(ngx_http_r s->data = v->data; } +#endif + static ngx_int_t ngx_http_variable_request_get_size(ngx_http_request_t *r, @@ -1371,6 +1379,16 @@ ngx_http_variable_https(ngx_http_request } +static void +ngx_http_variable_set_args(ngx_http_request_t *r, + ngx_http_variable_value_t *v, uintptr_t data) +{ + r->args.len = v->len; + r->args.data = v->data; + r->valid_unparsed_uri = 0; +} + + static ngx_int_t ngx_http_variable_is_args(ngx_http_request_t *r, ngx_http_variable_value_t *v, uintptr_t data) From mdounin at mdounin.ru Mon May 19 18:50:28 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Mon, 19 May 2014 22:50:28 +0400 Subject: [PATH] Variable: setting $args should invalidate unparsed uri. In-Reply-To: References: Message-ID: <20140519185028.GR1849@mdounin.ru> Hello! On Thu, Jan 23, 2014 at 12:16:48PM -0800, Yichun Zhang (agentzh) wrote: > Hello! > > A user of mine, rvsw, reported an issue in the $args builtin variable. > Setting $args does not change the r->valid_unparsed_uri flag so > modules like ngx_proxy might still use the unparsed URI even when the > $args has been assigned to a new value. Sorry for late reply. I've committed slightly different patch, thanks. http://hg.nginx.org/nginx/rev/e9dbffc18a77 -- Maxim Dounin http://nginx.org/ From yurnerola at gmail.com Tue May 20 07:45:31 2014 From: yurnerola at gmail.com (liubin) Date: Tue, 20 May 2014 15:45:31 +0800 Subject: mp4_module doubt In-Reply-To: References: <0BA9B631-70BE-4020-B476-C24B93994025@gmail.com> Message-ID: Hi: It?s really so nice of u guys to help solve this problem. Actually ,i tried the patch. Find two little bug: 1.start_time is badly used ,may anther var should be declared. 2. > > start_time += (sample - start_sample) * duration; This entry ,maybe we forget the first part. start_sample is in this entry already. Thanks again! ? 2014?5?19????8:02?Roman Arutyunyan ??? > > On 19 May 2014, at 04:56, ?? wrote: > >> Hello, >> I'm just reading this module and wandering if we can reduce flow >> from nginx by align start time to key frames.Our company is a video >> website from China.And i want to know does it make sense to do this. > > Here?s the patch solving your problem. Currently there are no plans to > add this to upstream but that can change. It would be nice to hear your > feedback. Here?s an example of ?mp4_align_start? directive added by > the patch. > > location /mp4 { > mp4; > mp4_align_start on; > root /tmp/; > } > > # HG changeset patch > # User Roman Arutyunyan > # Date 1400496445 -14400 > # Mon May 19 14:47:25 2014 +0400 > # Node ID fccafb910a3361e3e2d6ca8afd6edfb576444997 > # Parent 3a48775f1535fe37cd9c034d92c5a5e9ae208c1e > Mp4: align start to video key frame. > > The new directive mp4_align_start adjusts start time to make > the result mp4 start with a key frame. This makes sense for > some players displaying garbage until the first key frame. > > diff -r 3a48775f1535 -r fccafb910a33 src/http/modules/ngx_http_mp4_module.c > --- a/src/http/modules/ngx_http_mp4_module.c Tue Apr 29 12:28:41 2014 +0400 > +++ b/src/http/modules/ngx_http_mp4_module.c Mon May 19 14:47:25 2014 +0400 > @@ -43,6 +43,7 @@ > typedef struct { > size_t buffer_size; > size_t max_buffer_size; > + ngx_flag_t align_start; > } ngx_http_mp4_conf_t; > > > @@ -303,6 +304,8 @@ static ngx_int_t ngx_http_mp4_update_co6 > ngx_http_mp4_trak_t *trak); > static void ngx_http_mp4_adjust_co64_atom(ngx_http_mp4_file_t *mp4, > ngx_http_mp4_trak_t *trak, off_t adjustment); > +static ngx_int_t ngx_http_mp4_align_start(ngx_http_mp4_file_t *mp4, > + ngx_http_mp4_trak_t *trak); > > static char *ngx_http_mp4(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); > static void *ngx_http_mp4_create_conf(ngx_conf_t *cf); > @@ -332,6 +335,13 @@ static ngx_command_t ngx_http_mp4_comma > offsetof(ngx_http_mp4_conf_t, max_buffer_size), > NULL }, > > + { ngx_string("mp4_align_start"), > + NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1, > + ngx_conf_set_flag_slot, > + NGX_HTTP_LOC_CONF_OFFSET, > + offsetof(ngx_http_mp4_conf_t, align_start), > + NULL }, > + > ngx_null_command > }; > > @@ -742,6 +752,14 @@ ngx_http_mp4_process(ngx_http_mp4_file_t > end_offset = 0; > trak = mp4->trak.elts; > > + if (conf->align_start) { > + for (i = 0; i < mp4->trak.nelts; i++) { > + if (ngx_http_mp4_align_start(mp4, &trak[i]) == NGX_OK) { > + break; > + } > + } > + } > + > for (i = 0; i < mp4->trak.nelts; i++) { > > if (ngx_http_mp4_update_stts_atom(mp4, &trak[i]) != NGX_OK) { > @@ -3457,6 +3475,110 @@ ngx_http_mp4_adjust_co64_atom(ngx_http_m > } > > > +static ngx_int_t > +ngx_http_mp4_align_start(ngx_http_mp4_file_t *mp4, ngx_http_mp4_trak_t *trak) > +{ > + uint32_t count, duration, sample, start_sample, > + *stss_entry, *stss_end; > + uint64_t start_time; > + ngx_buf_t *stts_data, *stss_data; > + ngx_mp4_stts_entry_t *stts_entry, *stts_end; > + > + stss_data = trak->out[NGX_HTTP_MP4_STSS_DATA].buf; > + if (stss_data == NULL) { > + return NGX_AGAIN; > + } > + > + stts_data = trak->out[NGX_HTTP_MP4_STTS_DATA].buf; > + if (stts_data == NULL) { > + return NGX_AGAIN; > + } > + > + ngx_log_debug0(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0, "align start"); > + > + start_time = (uint64_t) mp4->start * trak->timescale / 1000; > + start_sample = 0; > + > + stts_entry = (ngx_mp4_stts_entry_t *) stts_data->pos; > + stts_end = (ngx_mp4_stts_entry_t *) stts_data->last; > + > + while (stts_entry < stts_end) { > + count = ngx_mp4_get_32value(stts_entry->count); > + duration = ngx_mp4_get_32value(stts_entry->duration); > + > + ngx_log_debug3(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0, > + "time:%uL, count:%uD, duration:%uD", > + start_time, count, duration); > + > + if (start_time < (uint64_t) count * duration) { > + start_sample += (ngx_uint_t) (start_time / duration); > + goto found_stts; > + } > + > + start_sample += count; > + start_time -= count * duration; > + stts_entry++; > + } > + > + ngx_log_debug0(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0, "out of stts samples"); > + > + return NGX_AGAIN; > + > +found_stts: > + > + stss_entry = (uint32_t *) stss_data->pos; > + stss_end = (uint32_t *) stss_data->last; > + > + while (stss_entry < stss_end) { > + sample = ngx_mp4_get_32value(stss_entry); > + > + ngx_log_debug1(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0, > + "sync:%uD", sample); > + > + if (sample >= start_sample + 1) { > + sample--; > + goto found_stss; > + } > + > + stss_entry++; > + } > + > + ngx_log_debug0(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0, "out of stss samples"); > + > + return NGX_AGAIN; > + > +found_stss: > + > + while (stts_entry < stts_end) { > + count = ngx_mp4_get_32value(stts_entry->count); > + duration = ngx_mp4_get_32value(stts_entry->duration); > + > + ngx_log_debug3(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0, > + "time:%uL, count:%uD, duration:%uD", > + start_time, count, duration); > + > + if (start_sample + count >= sample) { > + start_time += (sample - start_sample) * duration; > + mp4->start = (uint64_t) start_time * 1000 / trak->timescale; > + > + ngx_log_debug2(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0, > + "aligned start:%ui, samples:%uD", > + mp4->start, sample - start_sample); > + > + return NGX_OK; > + } > + > + start_sample += count; > + start_time -= count * duration; > + stts_entry++; > + } > + > + ngx_log_debug0(NGX_LOG_DEBUG_HTTP, mp4->file.log, 0, "out of stts samples"); > + > + return NGX_AGAIN; > +} > + > + > static char * > ngx_http_mp4(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) > { > @@ -3481,6 +3603,7 @@ ngx_http_mp4_create_conf(ngx_conf_t *cf) > > conf->buffer_size = NGX_CONF_UNSET_SIZE; > conf->max_buffer_size = NGX_CONF_UNSET_SIZE; > + conf->align_start = NGX_CONF_UNSET; > > return conf; > } > @@ -3495,6 +3618,7 @@ ngx_http_mp4_merge_conf(ngx_conf_t *cf, > ngx_conf_merge_size_value(conf->buffer_size, prev->buffer_size, 512 * 1024); > ngx_conf_merge_size_value(conf->max_buffer_size, prev->max_buffer_size, > 10 * 1024 * 1024); > + ngx_conf_merge_value(conf->align_start, prev->align_start, 0); > > return NGX_CONF_OK; > } > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel From sb at waeme.net Tue May 20 10:03:48 2014 From: sb at waeme.net (Sergey Budnevitch) Date: Tue, 20 May 2014 10:03:48 +0000 Subject: [nginx] SSL: $ssl_client_fingerprint variable. Message-ID: details: http://hg.nginx.org/nginx/rev/5e892d40e5cc branches: changeset: 5700:5e892d40e5cc user: Sergey Budnevitch date: Tue May 20 14:03:03 2014 +0400 description: SSL: $ssl_client_fingerprint variable. diffstat: src/event/ngx_event_openssl.c | 34 ++++++++++++++++++++++++++++++++++ src/event/ngx_event_openssl.h | 2 ++ src/http/modules/ngx_http_ssl_module.c | 3 +++ 3 files changed, 39 insertions(+), 0 deletions(-) diffs (69 lines): diff -r e9dbffc18a77 -r 5e892d40e5cc src/event/ngx_event_openssl.c --- a/src/event/ngx_event_openssl.c Mon May 19 22:45:35 2014 +0400 +++ b/src/event/ngx_event_openssl.c Tue May 20 14:03:03 2014 +0400 @@ -2957,6 +2957,40 @@ ngx_ssl_get_serial_number(ngx_connection ngx_int_t +ngx_ssl_get_fingerprint(ngx_connection_t *c, ngx_pool_t *pool, ngx_str_t *s) +{ + X509 *cert; + unsigned int len; + u_char buf[EVP_MAX_MD_SIZE]; + + s->len = 0; + + cert = SSL_get_peer_certificate(c->ssl->connection); + if (cert == NULL) { + return NGX_OK; + } + + if (!X509_digest(cert, EVP_sha1(), buf, &len)) { + X509_free(cert); + return NGX_ERROR; + } + + s->len = 2 * len; + s->data = ngx_pnalloc(pool, 2 * len); + if (s->data == NULL) { + X509_free(cert); + return NGX_ERROR; + } + + ngx_hex_dump(s->data, buf, len); + + X509_free(cert); + + return NGX_OK; +} + + +ngx_int_t ngx_ssl_get_client_verify(ngx_connection_t *c, ngx_pool_t *pool, ngx_str_t *s) { X509 *cert; diff -r e9dbffc18a77 -r 5e892d40e5cc src/event/ngx_event_openssl.h --- a/src/event/ngx_event_openssl.h Mon May 19 22:45:35 2014 +0400 +++ b/src/event/ngx_event_openssl.h Tue May 20 14:03:03 2014 +0400 @@ -173,6 +173,8 @@ ngx_int_t ngx_ssl_get_issuer_dn(ngx_conn ngx_str_t *s); ngx_int_t ngx_ssl_get_serial_number(ngx_connection_t *c, ngx_pool_t *pool, ngx_str_t *s); +ngx_int_t ngx_ssl_get_fingerprint(ngx_connection_t *c, ngx_pool_t *pool, + ngx_str_t *s); ngx_int_t ngx_ssl_get_client_verify(ngx_connection_t *c, ngx_pool_t *pool, ngx_str_t *s); diff -r e9dbffc18a77 -r 5e892d40e5cc src/http/modules/ngx_http_ssl_module.c --- a/src/http/modules/ngx_http_ssl_module.c Mon May 19 22:45:35 2014 +0400 +++ b/src/http/modules/ngx_http_ssl_module.c Tue May 20 14:03:03 2014 +0400 @@ -292,6 +292,9 @@ static ngx_http_variable_t ngx_http_ssl { ngx_string("ssl_client_serial"), NULL, ngx_http_ssl_variable, (uintptr_t) ngx_ssl_get_serial_number, NGX_HTTP_VAR_CHANGEABLE, 0 }, + { ngx_string("ssl_client_fingerprint"), NULL, ngx_http_ssl_variable, + (uintptr_t) ngx_ssl_get_fingerprint, NGX_HTTP_VAR_CHANGEABLE, 0 }, + { ngx_string("ssl_client_verify"), NULL, ngx_http_ssl_variable, (uintptr_t) ngx_ssl_get_client_verify, NGX_HTTP_VAR_CHANGEABLE, 0 }, From ru at nginx.com Tue May 20 12:13:13 2014 From: ru at nginx.com (Ruslan Ermilov) Date: Tue, 20 May 2014 12:13:13 +0000 Subject: [nginx] Configure: the --build= option. Message-ID: details: http://hg.nginx.org/nginx/rev/1209b8a7b077 branches: changeset: 5701:1209b8a7b077 user: Ruslan Ermilov date: Tue May 20 16:10:07 2014 +0400 description: Configure: the --build= option. If set, its value is output in "nginx -v" and in the error log. diffstat: auto/configure | 4 ++++ auto/options | 3 +++ src/core/nginx.c | 2 +- src/core/nginx.h | 6 ++++++ src/os/unix/ngx_posix_init.c | 2 +- src/os/win32/ngx_win32_init.c | 2 +- 6 files changed, 16 insertions(+), 3 deletions(-) diffs (91 lines): diff -r 5e892d40e5cc -r 1209b8a7b077 auto/configure --- a/auto/configure Tue May 20 14:03:03 2014 +0400 +++ b/auto/configure Tue May 20 16:10:07 2014 +0400 @@ -108,4 +108,8 @@ have=NGX_HTTP_SCGI_TEMP_PATH value="\"$N have=NGX_USER value="\"$NGX_USER\"" . auto/define have=NGX_GROUP value="\"$NGX_GROUP\"" . auto/define +if [ ".$NGX_BUILD" != "." ]; then + have=NGX_BUILD value="\"$NGX_BUILD\"" . auto/define +fi + . auto/summary diff -r 5e892d40e5cc -r 1209b8a7b077 auto/options --- a/auto/options Tue May 20 14:03:03 2014 +0400 +++ b/auto/options Tue May 20 16:10:07 2014 +0400 @@ -14,6 +14,7 @@ NGX_PID_PATH= NGX_LOCK_PATH= NGX_USER= NGX_GROUP= +NGX_BUILD= CC=${CC:-cc} CPP= @@ -178,6 +179,7 @@ do --crossbuild=*) NGX_PLATFORM="$value" ;; + --build=*) NGX_BUILD="$value" ;; --builddir=*) NGX_OBJS="$value" ;; --with-rtsig_module) EVENT_RTSIG=YES ;; @@ -341,6 +343,7 @@ cat << END --group=GROUP set non-privileged group for worker processes + --build=NAME set build name --builddir=DIR set build directory --with-rtsig_module enable rtsig module diff -r 5e892d40e5cc -r 1209b8a7b077 src/core/nginx.c --- a/src/core/nginx.c Tue May 20 14:03:03 2014 +0400 +++ b/src/core/nginx.c Tue May 20 16:10:07 2014 +0400 @@ -217,7 +217,7 @@ main(int argc, char *const *argv) } if (ngx_show_version) { - ngx_write_stderr("nginx version: " NGINX_VER NGX_LINEFEED); + ngx_write_stderr("nginx version: " NGINX_VER_BUILD NGX_LINEFEED); if (ngx_show_help) { ngx_write_stderr( diff -r 5e892d40e5cc -r 1209b8a7b077 src/core/nginx.h --- a/src/core/nginx.h Tue May 20 14:03:03 2014 +0400 +++ b/src/core/nginx.h Tue May 20 16:10:07 2014 +0400 @@ -13,6 +13,12 @@ #define NGINX_VERSION "1.7.1" #define NGINX_VER "nginx/" NGINX_VERSION +#ifdef NGX_BUILD +#define NGINX_VER_BUILD NGINX_VER " (" NGX_BUILD ")" +#else +#define NGINX_VER_BUILD NGINX_VER +#endif + #define NGINX_VAR "NGINX" #define NGX_OLDPID_EXT ".oldbin" diff -r 5e892d40e5cc -r 1209b8a7b077 src/os/unix/ngx_posix_init.c --- a/src/os/unix/ngx_posix_init.c Tue May 20 14:03:03 2014 +0400 +++ b/src/os/unix/ngx_posix_init.c Tue May 20 16:10:07 2014 +0400 @@ -82,7 +82,7 @@ ngx_os_init(ngx_log_t *log) void ngx_os_status(ngx_log_t *log) { - ngx_log_error(NGX_LOG_NOTICE, log, 0, NGINX_VER); + ngx_log_error(NGX_LOG_NOTICE, log, 0, NGINX_VER_BUILD); #ifdef NGX_COMPILER ngx_log_error(NGX_LOG_NOTICE, log, 0, "built by " NGX_COMPILER); diff -r 5e892d40e5cc -r 1209b8a7b077 src/os/win32/ngx_win32_init.c --- a/src/os/win32/ngx_win32_init.c Tue May 20 14:03:03 2014 +0400 +++ b/src/os/win32/ngx_win32_init.c Tue May 20 16:10:07 2014 +0400 @@ -247,7 +247,7 @@ ngx_os_status(ngx_log_t *log) { ngx_osviex_stub_t *osviex_stub; - ngx_log_error(NGX_LOG_NOTICE, log, 0, NGINX_VER); + ngx_log_error(NGX_LOG_NOTICE, log, 0, NGINX_VER_BUILD); if (osviex) { From vl at nginx.com Wed May 21 13:26:29 2014 From: vl at nginx.com (Homutov Vladimir) Date: Wed, 21 May 2014 13:26:29 +0000 Subject: [nginx] Added syslog support for error_log and access_log direct... Message-ID: details: http://hg.nginx.org/nginx/rev/777202558122 branches: changeset: 5702:777202558122 user: Vladimir Homutov date: Mon May 12 16:34:15 2014 +0400 description: Added syslog support for error_log and access_log directives. diffstat: auto/sources | 6 +- src/core/ngx_core.h | 1 + src/core/ngx_log.c | 86 +++++++- src/core/ngx_log.h | 6 + src/core/ngx_syslog.c | 332 +++++++++++++++++++++++++++++++++ src/core/ngx_syslog.h | 30 ++ src/core/ngx_times.c | 21 +- src/core/ngx_times.h | 1 + src/http/modules/ngx_http_log_module.c | 71 ++++++- src/http/ngx_http_request.h | 2 + src/os/unix/ngx_process_cycle.c | 11 +- 11 files changed, 544 insertions(+), 23 deletions(-) diffs (truncated from 858 to 300 lines): diff -r 1209b8a7b077 -r 777202558122 auto/sources --- a/auto/sources Tue May 20 16:10:07 2014 +0400 +++ b/auto/sources Mon May 12 16:34:15 2014 +0400 @@ -37,7 +37,8 @@ CORE_DEPS="src/core/nginx.h \ src/core/ngx_resolver.h \ src/core/ngx_open_file_cache.h \ src/core/ngx_crypt.h \ - src/core/ngx_proxy_protocol.h" + src/core/ngx_proxy_protocol.h \ + src/core/ngx_syslog.h" CORE_SRCS="src/core/nginx.c \ @@ -69,7 +70,8 @@ CORE_SRCS="src/core/nginx.c \ src/core/ngx_resolver.c \ src/core/ngx_open_file_cache.c \ src/core/ngx_crypt.c \ - src/core/ngx_proxy_protocol.c" + src/core/ngx_proxy_protocol.c \ + src/core/ngx_syslog.c" REGEX_MODULE=ngx_regex_module diff -r 1209b8a7b077 -r 777202558122 src/core/ngx_core.h --- a/src/core/ngx_core.h Tue May 20 16:10:07 2014 +0400 +++ b/src/core/ngx_core.h Mon May 12 16:34:15 2014 +0400 @@ -77,6 +77,7 @@ typedef void (*ngx_connection_handler_pt #include #include #include +#include #include diff -r 1209b8a7b077 -r 777202558122 src/core/ngx_log.c --- a/src/core/ngx_log.c Tue May 20 16:10:07 2014 +0400 +++ b/src/core/ngx_log.c Mon May 12 16:34:15 2014 +0400 @@ -148,6 +148,12 @@ ngx_log_error_core(ngx_uint_t level, ngx break; } + if (log->writer) { + log->writer(log, level, errstr, p - errstr); + log = log->next; + continue; + } + (void) ngx_write_fd(log->file->fd, errstr, p - errstr); if (log->file->fd == ngx_stderr) { @@ -366,15 +372,33 @@ ngx_log_init(u_char *prefix) ngx_int_t ngx_log_open_default(ngx_cycle_t *cycle) { - static ngx_str_t error_log = ngx_string(NGX_ERROR_LOG_PATH); + ngx_log_t *log; + static ngx_str_t error_log = ngx_string(NGX_ERROR_LOG_PATH); - if (cycle->new_log.file == NULL) { - cycle->new_log.file = ngx_conf_open_file(cycle, &error_log); - if (cycle->new_log.file == NULL) { + if (ngx_log_get_file_log(&cycle->new_log) != NULL) { + return NGX_OK; + } + + if (cycle->new_log.log_level != 0) { + /* there are some error logs, but no files */ + + log = ngx_pcalloc(cycle->pool, sizeof(ngx_log_t)); + if (log == NULL) { return NGX_ERROR; } - cycle->new_log.log_level = NGX_LOG_ERR; + log->log_level = NGX_LOG_ERR; + ngx_log_insert(&cycle->new_log, log); + + } else { + /* no error logs at all */ + log = &cycle->new_log; + log->log_level = NGX_LOG_ERR; + } + + log->file = ngx_conf_open_file(cycle, &error_log); + if (log->file == NULL) { + return NGX_ERROR; } return NGX_OK; @@ -390,7 +414,8 @@ ngx_log_redirect_stderr(ngx_cycle_t *cyc return NGX_OK; } - fd = cycle->log->file->fd; + /* file log always exists when we are called */ + fd = ngx_log_get_file_log(cycle->log)->file->fd; if (fd != ngx_stderr) { if (ngx_set_stderr(fd) == NGX_FILE_ERROR) { @@ -405,6 +430,21 @@ ngx_log_redirect_stderr(ngx_cycle_t *cyc } +ngx_log_t * +ngx_log_get_file_log(ngx_log_t *head) +{ + ngx_log_t *log; + + for (log = head; log; log = log->next) { + if (log->file != NULL) { + return log; + } + } + + return NULL; +} + + static char * ngx_log_set_levels(ngx_conf_t *cf, ngx_log_t *log) { @@ -482,8 +522,9 @@ ngx_error_log(ngx_conf_t *cf, ngx_comman char * ngx_log_set_log(ngx_conf_t *cf, ngx_log_t **head) { - ngx_log_t *new_log; - ngx_str_t *value, name; + ngx_log_t *new_log; + ngx_str_t *value, name; + ngx_syslog_peer_t *peer; if (*head != NULL && (*head)->log_level == 0) { new_log = *head; @@ -506,13 +547,30 @@ ngx_log_set_log(ngx_conf_t *cf, ngx_log_ ngx_str_null(&name); cf->cycle->log_use_stderr = 1; + new_log->file = ngx_conf_open_file(cf->cycle, &name); + if (new_log->file == NULL) { + return NGX_CONF_ERROR; + } + + + } else if (ngx_strncmp(value[1].data, "syslog:", 7) == 0) { + peer = ngx_pcalloc(cf->pool, sizeof(ngx_syslog_peer_t)); + if (peer == NULL) { + return NGX_CONF_ERROR; + } + + if (ngx_syslog_process_conf(cf, peer) != NGX_CONF_OK) { + return NGX_CONF_ERROR; + } + + new_log->writer = ngx_syslog_writer; + new_log->wdata = peer; + } else { - name = value[1]; - } - - new_log->file = ngx_conf_open_file(cf->cycle, &name); - if (new_log->file == NULL) { - return NGX_CONF_ERROR; + new_log->file = ngx_conf_open_file(cf->cycle, &value[1]); + if (new_log->file == NULL) { + return NGX_CONF_ERROR; + } } if (ngx_log_set_levels(cf, new_log) != NGX_CONF_OK) { diff -r 1209b8a7b077 -r 777202558122 src/core/ngx_log.h --- a/src/core/ngx_log.h Tue May 20 16:10:07 2014 +0400 +++ b/src/core/ngx_log.h Mon May 12 16:34:15 2014 +0400 @@ -43,6 +43,8 @@ typedef u_char *(*ngx_log_handler_pt) (ngx_log_t *log, u_char *buf, size_t len); +typedef void (*ngx_log_writer_pt) (ngx_log_t *log, ngx_uint_t level, + u_char *buf, size_t len); struct ngx_log_s { @@ -54,6 +56,9 @@ struct ngx_log_s { ngx_log_handler_pt handler; void *data; + ngx_log_writer_pt writer; + void *wdata; + /* * we declare "action" as "char *" because the actions are usually * the static strings and in the "u_char *" case we have to override @@ -227,6 +232,7 @@ void ngx_cdecl ngx_log_stderr(ngx_err_t u_char *ngx_log_errno(u_char *buf, u_char *last, ngx_err_t err); ngx_int_t ngx_log_open_default(ngx_cycle_t *cycle); ngx_int_t ngx_log_redirect_stderr(ngx_cycle_t *cycle); +ngx_log_t *ngx_log_get_file_log(ngx_log_t *head); char *ngx_log_set_log(ngx_conf_t *cf, ngx_log_t **head); diff -r 1209b8a7b077 -r 777202558122 src/core/ngx_syslog.c --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/core/ngx_syslog.c Mon May 12 16:34:15 2014 +0400 @@ -0,0 +1,332 @@ + +/* + * Copyright (C) Nginx, Inc. + */ + + +#include +#include +#include + + +#define NGX_SYSLOG_MAX_STR \ + NGX_MAX_ERROR_STR + sizeof("<255>Jan 01 00:00:00 ") - 1 \ + + (NGX_MAXHOSTNAMELEN - 1) + 1 /* space */ \ + + 32 /* tag */ + 2 /* colon, space */ + + +static char *ngx_syslog_parse_args(ngx_conf_t *cf, ngx_syslog_peer_t *peer); +static ngx_int_t ngx_syslog_init_peer(ngx_syslog_peer_t *peer); +static void ngx_syslog_cleanup(void *data); + + +static char *facilities[] = { + "kern", "user", "mail", "daemon", "auth", "intern", "lpr", "news", "uucp", + "clock", "authpriv", "ftp", "ntp", "audit", "alert", "cron", "local0", + "local1", "local2", "local3", "local4", "local5", "local6", "local7", + NULL +}; + +/* note 'error/warn' like in nginx.conf, not 'err/warning' */ +static char *severities[] = { + "emerg", "alert", "crit", "error", "warn", "notice", "info", "debug", NULL +}; + +static ngx_log_t ngx_syslog_dummy_log; +static ngx_event_t ngx_syslog_dummy_event; + + +char * +ngx_syslog_process_conf(ngx_conf_t *cf, ngx_syslog_peer_t *peer) +{ + peer->pool = cf->pool; + peer->facility = NGX_CONF_UNSET_UINT; + peer->severity = NGX_CONF_UNSET_UINT; + + if (ngx_syslog_parse_args(cf, peer) != NGX_CONF_OK) { + return NGX_CONF_ERROR; + } + + if (peer->server.sockaddr == NULL) { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "no syslog server specified"); + return NGX_CONF_ERROR; + } + + if (peer->facility == NGX_CONF_UNSET_UINT) { + peer->facility = 23; /* local7 */ + } + + if (peer->severity == NGX_CONF_UNSET_UINT) { + peer->severity = 6; /* info */ + } + + if (peer->tag.data == NULL) { + ngx_str_set(&peer->tag, "nginx"); + } + + peer->conn.fd = (ngx_socket_t) -1; + + return NGX_CONF_OK; +} + + +static char * +ngx_syslog_parse_args(ngx_conf_t *cf, ngx_syslog_peer_t *peer) +{ + u_char *p, *comma, c; + size_t len; + ngx_str_t *value; + ngx_url_t u; + ngx_uint_t i; + + value = cf->args->elts; + + p = value[1].data + sizeof("syslog:") - 1; + + for ( ;; ) { + comma = (u_char *) ngx_strchr(p, ','); + + if (comma != NULL) { + len = comma - p; + *comma = '\0'; + + } else { + len = value[1].data + value[1].len - p; + } From marcin.stragowski at adpilot.pl Wed May 21 14:21:17 2014 From: marcin.stragowski at adpilot.pl (Marcin =?utf-8?Q?Str=C4=85gowski?=) Date: Wed, 21 May 2014 16:21:17 +0200 (CEST) Subject: [PATCH] make nginx not swappable In-Reply-To: <755329027.934950.1400681167039.JavaMail.zimbra@adpilot.pl> Message-ID: <1652221247.939763.1400682077782.JavaMail.zimbra@adpilot.pl> Hello, I'm new here but I work with nginx on daily basis at my company Adpilot.pl and I would like to suggest a patch to nginx. Recently we had a need to provide a full security to our servers by securing our encryption keys and preventing them to be written on to the hard disk. But there still was an issue with swapping out nginx - there still was (a small) possibility that in extreme situations some portion of nginx memory where keys are stored (or information which could be used to recreate keys) will be swapped out and will be written on hard drive. Also keeping nginx out of swap has few performance benefits on heavy loaded systems ;) In earlier Linux systems process could be kept out of swap by setting a sticky bit (chmod +S) but on all modern linux distributions - this flag doesn't work anymore. Now it must be done manually in code, so I'm sending a patch which is adding a configuration parameter that can enable marking all nginx memory (also workers) as nonswappable. Feedback welcome! Thanks Marcin Str?gowski -------------- next part -------------- A non-text attachment was scrubbed... Name: mlockall.patch Type: text/x-patch Size: 2751 bytes Desc: not available URL: From mdounin at mdounin.ru Wed May 21 16:48:47 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 21 May 2014 20:48:47 +0400 Subject: [PATCH] make nginx not swappable In-Reply-To: <1652221247.939763.1400682077782.JavaMail.zimbra@adpilot.pl> References: <755329027.934950.1400681167039.JavaMail.zimbra@adpilot.pl> <1652221247.939763.1400682077782.JavaMail.zimbra@adpilot.pl> Message-ID: <20140521164847.GL1849@mdounin.ru> Hello! On Wed, May 21, 2014 at 04:21:17PM +0200, Marcin Str?gowski wrote: > > Hello, I'm new here but I work with nginx on daily basis at my > company Adpilot.pl and I would like to suggest a patch to nginx. > > Recently we had a need to provide a full security to our servers > by securing our encryption keys and preventing them to be > written on to the hard disk. > > But there still was an issue with swapping out nginx - there > still was (a small) possibility that in extreme situations some > portion of nginx memory where keys are stored (or information > which could be used to recreate keys) will be swapped out and > will be written on hard drive. > > Also keeping nginx out of swap has few performance benefits on > heavy loaded systems ;) > > In earlier Linux systems process could be kept out of swap by > setting a sticky bit (chmod +S) but on all modern linux > distributions - this flag doesn't work anymore. > > Now it must be done manually in code, so I'm sending a patch > which is adding a configuration parameter that can enable > marking all nginx memory (also workers) as nonswappable. I don't like neither the functionality nor the patch. Trivial solution to the original problem would be to just disable swap on the system (and you'll have to disable dumps and hibernation as well). Better approach would be to store keys in a special secure allocation, locked and with guard pages. Akamai recently tried to provide a patch for OpenSSL for this, see thread here: http://thread.gmane.org/gmane.comp.encryption.openssl.user/51243 -- Maxim Dounin http://nginx.org/ From mdounin at mdounin.ru Wed May 21 17:56:51 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 21 May 2014 17:56:51 +0000 Subject: [nginx] Mail: added a check for the number of arguments in MAIL/... Message-ID: details: http://hg.nginx.org/nginx/rev/7deb01451486 branches: changeset: 5703:7deb01451486 user: Maxim Dounin date: Wed May 21 21:56:20 2014 +0400 description: Mail: added a check for the number of arguments in MAIL/RCPT. Missed during introduction of the SMTP pipelining support (04e43d03e153, 1.5.6). Previously, the check wasn't needed as s->buffer was used directly and the number of arguments didn't matter. Reported by Svyatoslav Nikolsky. diffstat: src/mail/ngx_mail_smtp_handler.c | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-) diffs (27 lines): diff --git a/src/mail/ngx_mail_smtp_handler.c b/src/mail/ngx_mail_smtp_handler.c --- a/src/mail/ngx_mail_smtp_handler.c +++ b/src/mail/ngx_mail_smtp_handler.c @@ -679,6 +679,11 @@ ngx_mail_smtp_mail(ngx_mail_session_t *s return NGX_OK; } + if (s->args.nelts == 0) { + ngx_str_set(&s->out, smtp_invalid_argument); + return NGX_OK; + } + arg = s->args.elts; arg += s->args.nelts - 1; @@ -713,6 +718,11 @@ ngx_mail_smtp_rcpt(ngx_mail_session_t *s return NGX_OK; } + if (s->args.nelts == 0) { + ngx_str_set(&s->out, smtp_invalid_argument); + return NGX_OK; + } + arg = s->args.elts; arg += s->args.nelts - 1; From ru at nginx.com Wed May 21 20:18:55 2014 From: ru at nginx.com (Ruslan Ermilov) Date: Wed, 21 May 2014 20:18:55 +0000 Subject: [nginx] Core: output client port number when logging accept event. Message-ID: details: http://hg.nginx.org/nginx/rev/524741fd50ed branches: changeset: 5704:524741fd50ed user: Ruslan Ermilov date: Thu May 22 00:16:09 2014 +0400 description: Core: output client port number when logging accept event. diffstat: src/event/ngx_event_accept.c | 14 +++++++++++--- 1 files changed, 11 insertions(+), 3 deletions(-) diffs (37 lines): diff -r 7deb01451486 -r 524741fd50ed src/event/ngx_event_accept.c --- a/src/event/ngx_event_accept.c Wed May 21 21:56:20 2014 +0400 +++ b/src/event/ngx_event_accept.c Thu May 22 00:16:09 2014 +0400 @@ -288,9 +288,11 @@ ngx_event_accept(ngx_event_t *ev) #if (NGX_DEBUG) { + ngx_str_t addr; struct sockaddr_in *sin; ngx_cidr_t *cidr; ngx_uint_t i; + u_char text[NGX_SOCKADDR_STRLEN]; #if (NGX_HAVE_INET6) struct sockaddr_in6 *sin6; ngx_uint_t n; @@ -340,12 +342,18 @@ ngx_event_accept(ngx_event_t *ev) continue; } + if (log->log_level & NGX_LOG_DEBUG_EVENT) { + addr.data = text; + addr.len = ngx_sock_ntop(c->sockaddr, c->socklen, text, + NGX_SOCKADDR_STRLEN, 1); + + ngx_log_debug3(NGX_LOG_DEBUG_EVENT, log, 0, + "*%uA accept: %V fd:%d", c->number, &addr, s); + } + } #endif - ngx_log_debug3(NGX_LOG_DEBUG_EVENT, log, 0, - "*%uA accept: %V fd:%d", c->number, &c->addr_text, s); - if (ngx_add_conn && (ngx_event_flags & NGX_USE_EPOLL_EVENT) == 0) { if (ngx_add_conn(c) == NGX_ERROR) { ngx_close_accepted_connection(c); From ru at nginx.com Wed May 21 20:18:56 2014 From: ru at nginx.com (Ruslan Ermilov) Date: Wed, 21 May 2014 20:18:56 +0000 Subject: [nginx] Mail: output client port number on client connects (tick... Message-ID: details: http://hg.nginx.org/nginx/rev/d5b8ee9f2201 branches: changeset: 5705:d5b8ee9f2201 user: Ruslan Ermilov date: Thu May 22 00:16:17 2014 +0400 description: Mail: output client port number on client connects (ticket #531). diffstat: src/mail/ngx_mail_handler.c | 8 ++++++-- 1 files changed, 6 insertions(+), 2 deletions(-) diffs (32 lines): diff -r 524741fd50ed -r d5b8ee9f2201 src/mail/ngx_mail_handler.c --- a/src/mail/ngx_mail_handler.c Thu May 22 00:16:09 2014 +0400 +++ b/src/mail/ngx_mail_handler.c Thu May 22 00:16:17 2014 +0400 @@ -22,6 +22,7 @@ static void ngx_mail_ssl_handshake_handl void ngx_mail_init_connection(ngx_connection_t *c) { + size_t len; ngx_uint_t i; ngx_mail_port_t *port; struct sockaddr *sa; @@ -30,6 +31,7 @@ ngx_mail_init_connection(ngx_connection_ ngx_mail_in_addr_t *addr; ngx_mail_session_t *s; ngx_mail_addr_conf_t *addr_conf; + u_char text[NGX_SOCKADDR_STRLEN]; #if (NGX_HAVE_INET6) struct sockaddr_in6 *sin6; ngx_mail_in6_addr_t *addr6; @@ -127,8 +129,10 @@ ngx_mail_init_connection(ngx_connection_ c->data = s; s->connection = c; - ngx_log_error(NGX_LOG_INFO, c->log, 0, "*%uA client %V connected to %V", - c->number, &c->addr_text, s->addr_text); + len = ngx_sock_ntop(c->sockaddr, c->socklen, text, NGX_SOCKADDR_STRLEN, 1); + + ngx_log_error(NGX_LOG_INFO, c->log, 0, "*%uA client %*s connected to %V", + c->number, len, text, s->addr_text); ctx = ngx_palloc(c->pool, sizeof(ngx_mail_log_ctx_t)); if (ctx == NULL) { From marcin.stragowski at adpilot.pl Wed May 21 20:48:58 2014 From: marcin.stragowski at adpilot.pl (Marcin =?utf-8?Q?Str=C4=85gowski?=) Date: Wed, 21 May 2014 22:48:58 +0200 (CEST) Subject: [PATCH] make nginx not swappable In-Reply-To: <20140521164847.GL1849@mdounin.ru> References: <755329027.934950.1400681167039.JavaMail.zimbra@adpilot.pl> <1652221247.939763.1400682077782.JavaMail.zimbra@adpilot.pl> <20140521164847.GL1849@mdounin.ru> Message-ID: <2123161961.986606.1400705338760.JavaMail.zimbra@adpilot.pl> > I don't like neither the functionality nor the patch. > Trivial solution to the original problem would be to just disable > swap on the system (and you'll have to disable dumps and > hibernation as well). this is not solving our problem - we still would like to have the possibility to swap some other processes on the system (there are also other services on the same machines and this is frequent scenario in RTB systems) > Better approach would be to store keys in a special secure > allocation, locked and with guard pages. Akamai recently tried to > provide a patch for OpenSSL for this, see thread here: > http://thread.gmane.org/gmane.comp.encryption.openssl.user/51243 that's nice - thank you - will look at this :) Still, I believe there is a usecase for our patch - for example in system like ours where are few processes that could make machine swap - we're still holding nginx out of swap so it's still responsive which for our systems is crucial. if there's something wrong with patch itself - I always can fix it. -- Maxim Dounin http://nginx.org/ _______________________________________________ nginx-devel mailing list nginx-devel at nginx.org http://mailman.nginx.org/mailman/listinfo/nginx-devel From piotr at cloudflare.com Wed May 21 21:52:43 2014 From: piotr at cloudflare.com (Piotr Sikora) Date: Wed, 21 May 2014 14:52:43 -0700 Subject: [PATCH] make nginx not swappable In-Reply-To: <2123161961.986606.1400705338760.JavaMail.zimbra@adpilot.pl> References: <755329027.934950.1400681167039.JavaMail.zimbra@adpilot.pl> <1652221247.939763.1400682077782.JavaMail.zimbra@adpilot.pl> <20140521164847.GL1849@mdounin.ru> <2123161961.986606.1400705338760.JavaMail.zimbra@adpilot.pl> Message-ID: Hey, > if there's something wrong with patch itself - I always can fix it. 1. m(un)lockall is POSIX, so you need to stop guarding it with NGX_LINUX and add tests for it in configure, 2. master process doesn't set rlimit(RLIMIT_MEMLOCK), 3. there is no error handling (logging isn't error handling), 4. it could use a more user-friendly name (i.e. "worker_noswap"), 5. style doesn't match nginx, i.e. indent is 4 spaces, not tab, etc. Best regards, Piotr Sikora From gfleury at gmail.com Thu May 22 05:29:23 2014 From: gfleury at gmail.com (George Fleury) Date: Thu, 22 May 2014 02:29:23 -0300 Subject: HTTP methods with '-' char Message-ID: Hi all, i'm porting apache https mod_cluster to NGINX, however the methods used for internal control mod_cluster use the character '-' (ex: ENABLE-APP), and for NGINX these methods are invalid because of character '- ?. Now comes my doubts, gave a quick read in RFCs 2616 and 822 and found nothing saying that the '-' character can not be used in token methods of http/1.1. Is that right or there is a reason? From alexeyk45 at gmail.com Thu May 22 07:32:48 2014 From: alexeyk45 at gmail.com (Alexey K.) Date: Thu, 22 May 2014 13:32:48 +0600 Subject: [PATCH] HTTP upstream: added possibility to specify variables for the ip_hash directive Message-ID: <537DA820.60705@gmail.com> # HG changeset patch # User Alexey K. # Date 1400743228 -21600 # Thu May 22 13:20:28 2014 +0600 # Node ID cbfb9bc7dc93c10baaa80571665ca1231afdafe9 # Parent d5b8ee9f2201e1d9fa7ac41da9a4a8d939cd42b1 HTTP upstream: added possibility to specify variables for the ip_hash directive. Example: ip_hash $http_x_forwarded_for; By default, without any parameters, client's IP address is used. A use case: A group of load balancers is put behind another group. Thus client's IP address for the former is always one of the latter's. With this patch ip_hash can use any variable with a valid address, including the one in X-Forwarded-For header. diff -r d5b8ee9f2201 -r cbfb9bc7dc93 src/http/modules/ngx_http_upstream_ip_hash_module.c --- a/src/http/modules/ngx_http_upstream_ip_hash_module.c Thu May 22 00:16:17 2014 +0400 +++ b/src/http/modules/ngx_http_upstream_ip_hash_module.c Thu May 22 13:20:28 2014 +0600 @@ -36,7 +36,7 @@ static ngx_command_t ngx_http_upstream_ip_hash_commands[] = { { ngx_string("ip_hash"), - NGX_HTTP_UPS_CONF|NGX_CONF_NOARGS, + NGX_HTTP_UPS_CONF|NGX_CONF_NOARGS|NGX_CONF_TAKE1, ngx_http_upstream_ip_hash, 0, 0, @@ -116,17 +116,41 @@ r->upstream->peer.get = ngx_http_upstream_get_ip_hash_peer; - switch (r->connection->sockaddr->sa_family) { + struct sockaddr *addr=r->connection->sockaddr; + + if(us->ip_hash_var != -1) { + + ngx_http_variable_value_t *v=ngx_http_get_flushed_variable(r,us->ip_hash_var); + + if (v == NULL || v->not_found) { + ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "http upstream ip_hash variable not found"); + } else { + + ngx_addr_t ngx_addr; + + if (ngx_parse_addr(r->pool, &ngx_addr, v->data, v->len) == NGX_OK) { + addr=ngx_addr.sockaddr; + } else { + ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "http upstream ip_hash address is invalid"); + } + + } + + } + + switch (addr->sa_family) { case AF_INET: - sin = (struct sockaddr_in *) r->connection->sockaddr; + sin = (struct sockaddr_in *) addr; iphp->addr = (u_char *) &sin->sin_addr.s_addr; iphp->addrlen = 3; break; #if (NGX_HAVE_INET6) case AF_INET6: - sin6 = (struct sockaddr_in6 *) r->connection->sockaddr; + sin6 = (struct sockaddr_in6 *) addr; iphp->addr = (u_char *) &sin6->sin6_addr.s6_addr; iphp->addrlen = 16; break; @@ -267,6 +291,31 @@ "load balancing method redefined"); } + if (cf->args->nelts == 2) { + + ngx_str_t name; + + name=((ngx_str_t *)cf->args->elts)[1]; + + if (name.data[0] != '$') { + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, + "invalid variable name \"%V\"", &name); + return NGX_CONF_ERROR; + } + + name.len--; + name.data++; + + uscf->ip_hash_var = ngx_http_get_variable_index(cf, &name); + + if (uscf->ip_hash_var == NGX_ERROR) { + return NGX_CONF_ERROR; + } + + } else { + uscf->ip_hash_var = -1; + } + uscf->peer.init_upstream = ngx_http_upstream_init_ip_hash; uscf->flags = NGX_HTTP_UPSTREAM_CREATE diff -r d5b8ee9f2201 -r cbfb9bc7dc93 src/http/ngx_http_upstream.h --- a/src/http/ngx_http_upstream.h Thu May 22 00:16:17 2014 +0400 +++ b/src/http/ngx_http_upstream.h Thu May 22 13:20:28 2014 +0600 @@ -119,6 +119,7 @@ in_port_t port; in_port_t default_port; ngx_uint_t no_port; /* unsigned no_port:1 */ + ngx_int_t ip_hash_var; }; From arut at nginx.com Thu May 22 07:45:06 2014 From: arut at nginx.com (Roman Arutyunyan) Date: Thu, 22 May 2014 11:45:06 +0400 Subject: [PATCH] HTTP upstream: added possibility to specify variables for the ip_hash directive In-Reply-To: <537DA820.60705@gmail.com> References: <537DA820.60705@gmail.com> Message-ID: <02341120-46BD-474F-A7E2-B29771563785@nginx.com> On 22 May 2014, at 11:32, Alexey K. wrote: > # HG changeset patch > # User Alexey K. > # Date 1400743228 -21600 > # Thu May 22 13:20:28 2014 +0600 > # Node ID cbfb9bc7dc93c10baaa80571665ca1231afdafe9 > # Parent d5b8ee9f2201e1d9fa7ac41da9a4a8d939cd42b1 > HTTP upstream: added possibility to specify variables for the ip_hash directive. > > Example: ip_hash $http_x_forwarded_for; > By default, without any parameters, client's IP address is used. > > A use case: > A group of load balancers is put behind another group. Thus client's IP address for the former is always one of the latter's. > With this patch ip_hash can use any variable with a valid address, including the one in X-Forwarded-For header. It?s better to support a complex value (expression) than just a single variable A hash balancer with this feature is planned for future nginx versions. From anthrax at unixuser.org Thu May 22 07:53:40 2014 From: anthrax at unixuser.org (anthrax at unixuser.org) Date: Thu, 22 May 2014 16:53:40 +0900 Subject: [PATCH] sub_filter bug Message-ID: <8738g26ygr.wl%anthrax@unixuser.org> Hi, Could you take a look at the following patch? # HG changeset patch # User KAWAHARA Masashi # Date 1400671481 -32400 # Wed May 21 20:24:41 2014 +0900 # Node ID 3949e591694fff37a436a076f4d4006c802587c7 # Parent 1209b8a7b077c7bbf161e502964308736c84db6f Bugfix: When both sub_filter and SSI were used and the include file was not terminated by newline, it was transmitted incorrectly. ex. conf: ssi on sub_filter '//check-string' '//replace-string' contents: index.shtml: //check-string test.html: (!! saved no newline at end of line !!) include-text/ output: include-test//replace-string correct output: include-test///replace-string diff -r 1209b8a7b077 -r 3949e591694f src/http/modules/ngx_http_sub_filter_module.c --- a/src/http/modules/ngx_http_sub_filter_module.c Tue May 20 16:10:07 2014 +0400 +++ b/src/http/modules/ngx_http_sub_filter_module.c Wed May 21 20:24:41 2014 +0900 @@ -539,6 +539,9 @@ for ( ;; ) { if (ch == match) { + if ((p + 1) == last && state != sub_match_state) { + goto match_started; + } copy_end = p; ctx->looked.data[0] = *p; looked = 1; From mdounin at mdounin.ru Thu May 22 13:22:58 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 22 May 2014 17:22:58 +0400 Subject: HTTP methods with '-' char In-Reply-To: References: Message-ID: <20140522132258.GS1849@mdounin.ru> Hello! On Thu, May 22, 2014 at 02:29:23AM -0300, George Fleury wrote: > Hi all, > > i'm porting apache https mod_cluster to NGINX, however the methods used for internal control mod_cluster > use the character '-' (ex: ENABLE-APP), and for NGINX these methods are invalid because of character '- ?. > Now comes my doubts, gave a quick read in RFCs 2616 and 822 and found nothing saying that the '-' character > can not be used in token methods of http/1.1. Is that right or there is a reason? As of now, nginx doesn't allow any characters other than uppercase latin letters and "_" in method names. While this is stricter than what HTTP requires, though covers almost all known valid uses, thus limiting potential attack vectors. See the thread here for previous discussion on this: http://mailman.nginx.org/pipermail/nginx-devel/2013-July/003929.html -- Maxim Dounin http://nginx.org/ From pauloasilva at gmail.com Thu May 22 14:21:56 2014 From: pauloasilva at gmail.com (Paulo Silva) Date: Thu, 22 May 2014 15:21:56 +0100 Subject: Buffers Chain Message-ID: Hi, Let me introduce the Elephant in the room. My name is Paulo and I'm from Portugal. I am starting developing nginx modules and I'm having a great time with buffers chain :D I'm using nginx as a reverse proxy and my goal is to log the full response body (HTML) on the server. I did start with a filter module with a body_filter and my approach is as follow: 1. Inspect given ngx_chain_t *in linked list to understand if nginx has the full response (last_buf) 2. On each body_filter call I'm updating a linked list which glues given ngx_chain_t *in to the previous ones. 3. As soon as I detect the last_buf, I iterate my internal linked list (and each linked ngx_chain_t lists), dumping buffers content. Although I'm reaching the last_buf, I'm missing some bytes. There is another/better/correct approach to do this? Thanks in advance, Paulo A. Silva http://tech.pauloasilva.com http://linkedin.com/in/devpauloasilva/ From mdounin at mdounin.ru Thu May 22 17:12:13 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 22 May 2014 21:12:13 +0400 Subject: [PATCH] sub_filter bug In-Reply-To: <8738g26ygr.wl%anthrax@unixuser.org> References: <8738g26ygr.wl%anthrax@unixuser.org> Message-ID: <20140522171213.GW1849@mdounin.ru> Hello! On Thu, May 22, 2014 at 04:53:40PM +0900, anthrax at unixuser.org wrote: > Hi, > > Could you take a look at the following patch? > > > # HG changeset patch > # User KAWAHARA Masashi > # Date 1400671481 -32400 > # Wed May 21 20:24:41 2014 +0900 > # Node ID 3949e591694fff37a436a076f4d4006c802587c7 > # Parent 1209b8a7b077c7bbf161e502964308736c84db6f > Bugfix: When both sub_filter and SSI were used and the include file was > not terminated by newline, it was transmitted incorrectly. > ex. > conf: > ssi on > sub_filter '//check-string' '//replace-string' > > contents: > index.shtml: > //check-string > > test.html: (!! saved no newline at end of line !!) > include-text/ > > output: > include-test//replace-string > > correct output: > include-test///replace-string The problem is certainly valid, thanks. The sub filter doesn't output partial match at the end of a subrequest response (if any), and this is certainly bad. The patch looks wrong though, it breaks cross-buffer matching within a single responses, and doesn't work for more than one character partial matches. Here are suggested patch (tests can be found at http://hg.nginx.org/nginx-tests/rev/811fbc213fd8): # HG changeset patch # User Maxim Dounin # Date 1400778091 -14400 # Thu May 22 21:01:31 2014 +0400 # Node ID 2fa527ce5797db76f393c4e1862175381276e7f3 # Parent 7deb014514861b296adaa57af5c955459d7e25ff Sub filter: fixed subrequests handling. In particular, properly output partial match at the end of a subrequest response (much like we do at the end of a response), and reset/set the last_in_chain flag as appropriate. Reported by KAWAHARA Masashi. diff --git a/src/http/modules/ngx_http_sub_filter_module.c b/src/http/modules/ngx_http_sub_filter_module.c --- a/src/http/modules/ngx_http_sub_filter_module.c +++ b/src/http/modules/ngx_http_sub_filter_module.c @@ -305,6 +305,7 @@ ngx_http_sub_body_filter(ngx_http_reques b->last = ctx->copy_end; b->shadow = NULL; b->last_buf = 0; + b->last_in_chain = 0; b->recycled = 0; if (b->in_file) { @@ -374,7 +375,9 @@ ngx_http_sub_body_filter(ngx_http_reques continue; } - if (ctx->buf->last_buf && ctx->looked.len) { + if (ctx->looked.len + && (ctx->buf->last_buf || ctx->buf->last_in_chain)) + { cl = ngx_chain_get_free_buf(r->pool, &ctx->free); if (cl == NULL) { return NGX_ERROR; @@ -394,7 +397,7 @@ ngx_http_sub_body_filter(ngx_http_reques ctx->looked.len = 0; } - if (ctx->buf->last_buf || ctx->buf->flush + if (ctx->buf->last_buf || ctx->buf->flush || ctx->buf->sync || ngx_buf_in_memory(ctx->buf)) { if (b == NULL) { @@ -414,6 +417,7 @@ ngx_http_sub_body_filter(ngx_http_reques } b->last_buf = ctx->buf->last_buf; + b->last_in_chain = ctx->buf->last_in_chain; b->flush = ctx->buf->flush; b->shadow = ctx->buf; -- Maxim Dounin http://nginx.org/ From apunch at brandscreen.com Thu May 22 22:39:42 2014 From: apunch at brandscreen.com (Andrew Punch) Date: Fri, 23 May 2014 08:39:42 +1000 Subject: [PATCH] make nginx not swappable Message-ID: Encrypt your swap partition then or disable it. A very minimal area of memory where any keys are stored or in the ssl library where encryption takes place should be non-swappable but keeping everything in memory means you're doing it wrong. If you really must do it for yourself, I don't see why patching a single mlockall() into the right place is so onerous. Also you haven't provided the ability to do this on other platforms e.g. Windows. I don't think this patch is worth pursuing. If you want to do it yourself, that's fine but I don't want this unnecessary complexity on any servers I run. -- NOTICE This e-mail and any attachments are confidential and may contain copyright material of Brandscreen or third parties. If you are not the intended recipient of this email you should not read, print, re-transmit, store or act in reliance on this e-mail or any attachments, and should destroy all copies of them. Brandscreen does not guarantee the integrity of any emails or any attached files. The views or opinions expressed are the author's own and may not reflect the views or opinions of Brandscreen. -------------- next part -------------- An HTML attachment was scrubbed... URL: From apunch at brandscreen.com Thu May 22 22:59:47 2014 From: apunch at brandscreen.com (Andrew Punch) Date: Fri, 23 May 2014 08:59:47 +1000 Subject: [PATCH] make nginx not swappable In-Reply-To: References: Message-ID: Alternatively you can set the per-process swappiness using cgroups. http://unix.stackexchange.com/questions/10214/per-process-swapiness-for-linux#10227 On 23 May 2014 08:39, Andrew Punch wrote: > Encrypt your swap partition then or disable it. A very minimal area of > memory where any keys are stored or in the ssl library where encryption > takes place should be non-swappable but keeping everything in memory means > you're doing it wrong. > > If you really must do it for yourself, I don't see why patching a single > mlockall() into the right place is so onerous. Also you haven't provided > the ability to do this on other platforms e.g. Windows. > > I don't think this patch is worth pursuing. If you want to do it yourself, > that's fine but I don't want this unnecessary complexity on any servers I > run. > -- NOTICE This e-mail and any attachments are confidential and may contain copyright material of Brandscreen or third parties. If you are not the intended recipient of this email you should not read, print, re-transmit, store or act in reliance on this e-mail or any attachments, and should destroy all copies of them. Brandscreen does not guarantee the integrity of any emails or any attached files. The views or opinions expressed are the author's own and may not reflect the views or opinions of Brandscreen. -------------- next part -------------- An HTML attachment was scrubbed... URL: From anthrax at unixuser.org Fri May 23 06:58:27 2014 From: anthrax at unixuser.org (=?ISO-2022-JP?B?GyRCQG44Nj4tO0obKEI=?=) Date: Fri, 23 May 2014 15:58:27 +0900 Subject: [PATCH] sub_filter bug In-Reply-To: <20140522171213.GW1849@mdounin.ru> References: <8738g26ygr.wl%anthrax@unixuser.org> <20140522171213.GW1849@mdounin.ru> Message-ID: <87mwe9gewc.wl%anthrax@unixuser.org> It looks good to me. Thx! At Thu, 22 May 2014 21:12:13 +0400, Maxim Dounin wrote: > > Hello! > > On Thu, May 22, 2014 at 04:53:40PM +0900, anthrax at unixuser.org wrote: > > > Hi, > > > > Could you take a look at the following patch? > > > > > > # HG changeset patch > > # User KAWAHARA Masashi > > # Date 1400671481 -32400 > > # Wed May 21 20:24:41 2014 +0900 > > # Node ID 3949e591694fff37a436a076f4d4006c802587c7 > > # Parent 1209b8a7b077c7bbf161e502964308736c84db6f > > Bugfix: When both sub_filter and SSI were used and the include file was > > not terminated by newline, it was transmitted incorrectly. > > ex. > > conf: > > ssi on > > sub_filter '//check-string' '//replace-string' > > > > contents: > > index.shtml: > > //check-string > > > > test.html: (!! saved no newline at end of line !!) > > include-text/ > > > > output: > > include-test//replace-string > > > > correct output: > > include-test///replace-string > > The problem is certainly valid, thanks. The sub filter doesn't > output partial match at the end of a subrequest response (if any), > and this is certainly bad. > > The patch looks wrong though, it breaks cross-buffer matching > within a single responses, and doesn't work for more than one > character partial matches. > > Here are suggested patch (tests can be found at > http://hg.nginx.org/nginx-tests/rev/811fbc213fd8): > > # HG changeset patch > # User Maxim Dounin > # Date 1400778091 -14400 > # Thu May 22 21:01:31 2014 +0400 > # Node ID 2fa527ce5797db76f393c4e1862175381276e7f3 > # Parent 7deb014514861b296adaa57af5c955459d7e25ff > Sub filter: fixed subrequests handling. > > In particular, properly output partial match at the end of a subrequest > response (much like we do at the end of a response), and reset/set the > last_in_chain flag as appropriate. > > Reported by KAWAHARA Masashi. > > diff --git a/src/http/modules/ngx_http_sub_filter_module.c b/src/http/modules/ngx_http_sub_filter_module.c > --- a/src/http/modules/ngx_http_sub_filter_module.c > +++ b/src/http/modules/ngx_http_sub_filter_module.c > @@ -305,6 +305,7 @@ ngx_http_sub_body_filter(ngx_http_reques > b->last = ctx->copy_end; > b->shadow = NULL; > b->last_buf = 0; > + b->last_in_chain = 0; > b->recycled = 0; > > if (b->in_file) { > @@ -374,7 +375,9 @@ ngx_http_sub_body_filter(ngx_http_reques > continue; > } > > - if (ctx->buf->last_buf && ctx->looked.len) { > + if (ctx->looked.len > + && (ctx->buf->last_buf || ctx->buf->last_in_chain)) > + { > cl = ngx_chain_get_free_buf(r->pool, &ctx->free); > if (cl == NULL) { > return NGX_ERROR; > @@ -394,7 +397,7 @@ ngx_http_sub_body_filter(ngx_http_reques > ctx->looked.len = 0; > } > > - if (ctx->buf->last_buf || ctx->buf->flush > + if (ctx->buf->last_buf || ctx->buf->flush || ctx->buf->sync > || ngx_buf_in_memory(ctx->buf)) > { > if (b == NULL) { > @@ -414,6 +417,7 @@ ngx_http_sub_body_filter(ngx_http_reques > } > > b->last_buf = ctx->buf->last_buf; > + b->last_in_chain = ctx->buf->last_in_chain; > b->flush = ctx->buf->flush; > b->shadow = ctx->buf; > > > -- > Maxim Dounin > http://nginx.org/ > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel From mdounin at mdounin.ru Fri May 23 09:12:12 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 23 May 2014 13:12:12 +0400 Subject: Buffers Chain In-Reply-To: References: Message-ID: <20140523091212.GD1849@mdounin.ru> Hello! On Thu, May 22, 2014 at 03:21:56PM +0100, Paulo Silva wrote: > Hi, > Let me introduce the Elephant in the room. My name is Paulo and I'm > from Portugal. > I am starting developing nginx modules and I'm having a great time > with buffers chain :D > > I'm using nginx as a reverse proxy and my goal is to log the full > response body (HTML) on the server. > > I did start with a filter module with a body_filter and my approach is > as follow: > > 1. Inspect given ngx_chain_t *in linked list to understand if nginx > has the full response (last_buf) > 2. On each body_filter call I'm updating a linked list which glues > given ngx_chain_t *in to the previous ones. > 3. As soon as I detect the last_buf, I iterate my internal linked list > (and each linked ngx_chain_t lists), dumping buffers content. > > Although I'm reaching the last_buf, I'm missing some bytes. In no particular order: - Make sure you are allocating your own ngx_chain_t structures. Trying to modify ngx_chain_t structures got in *in chain is a common mistake. - If you are passing buffers to next filters, it's incorrect to assume that buffer contents will be available later (a buffer contents will be eventually sent to a client, and then the buffer can be reused by its owner). - If you are not passing buffers to next filters, it's incorrect to assume that you'll get full response (e.g., with "output_buffers 1 64", you'll get only first 64 bytes of a response). - If you didn't set r->filter_need_in_memory, it's incorrect to assume that buffers will be in memory. To log a response body, you'll have to copy contents of all buffers sent though your filter. Just saving links to buffers is certainly not enough. -- Maxim Dounin http://nginx.org/ From pauloasilva at gmail.com Fri May 23 09:23:38 2014 From: pauloasilva at gmail.com (Paulo Silva) Date: Fri, 23 May 2014 10:23:38 +0100 Subject: Buffers Chain In-Reply-To: <20140523091212.GD1849@mdounin.ru> References: <20140523091212.GD1849@mdounin.ru> Message-ID: @Maxim: nice hints, thanks. Let me ask you about filters order. To get the contents that will be sent to client, I want to run my body_filter right before the gziping. How to accomplish that or if it is even possible I am worried that copying the buffers content the original ones can be yet modified by other filters and than I will log an outdated copy. Thanks for your help. On Fri, May 23, 2014 at 10:12 AM, Maxim Dounin wrote: > Hello! > > On Thu, May 22, 2014 at 03:21:56PM +0100, Paulo Silva wrote: > >> Hi, >> Let me introduce the Elephant in the room. My name is Paulo and I'm >> from Portugal. >> I am starting developing nginx modules and I'm having a great time >> with buffers chain :D >> >> I'm using nginx as a reverse proxy and my goal is to log the full >> response body (HTML) on the server. >> >> I did start with a filter module with a body_filter and my approach is >> as follow: >> >> 1. Inspect given ngx_chain_t *in linked list to understand if nginx >> has the full response (last_buf) >> 2. On each body_filter call I'm updating a linked list which glues >> given ngx_chain_t *in to the previous ones. >> 3. As soon as I detect the last_buf, I iterate my internal linked list >> (and each linked ngx_chain_t lists), dumping buffers content. >> >> Although I'm reaching the last_buf, I'm missing some bytes. > > In no particular order: > > - Make sure you are allocating your own ngx_chain_t structures. > Trying to modify ngx_chain_t structures got in *in chain is a > common mistake. > > - If you are passing buffers to next filters, it's incorrect to > assume that buffer contents will be available later (a buffer > contents will be eventually sent to a client, and then the > buffer can be reused by its owner). > > - If you are not passing buffers to next filters, it's incorrect > to assume that you'll get full response (e.g., with > "output_buffers 1 64", you'll get only first 64 bytes of a > response). > > - If you didn't set r->filter_need_in_memory, it's incorrect to > assume that buffers will be in memory. > > To log a response body, you'll have to copy contents of all > buffers sent though your filter. Just saving links to buffers is > certainly not enough. > > -- > Maxim Dounin > http://nginx.org/ > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel -- Paulo A. Silva http://tech.pauloasilva.com http://linkedin.com/in/devpauloasilva/ From arut at nginx.com Fri May 23 09:49:59 2014 From: arut at nginx.com (Roman Arutyunyan) Date: Fri, 23 May 2014 09:49:59 +0000 Subject: [nginx] Upstream: fix tries check in ip_hash. Message-ID: details: http://hg.nginx.org/nginx/rev/a2bf26774cd3 branches: changeset: 5706:a2bf26774cd3 user: Roman Arutyunyan date: Fri May 23 13:47:05 2014 +0400 description: Upstream: fix tries check in ip_hash. Make two checks for maximum number of tries consistent. The other one checks '>' condition. diffstat: src/http/modules/ngx_http_upstream_ip_hash_module.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r d5b8ee9f2201 -r a2bf26774cd3 src/http/modules/ngx_http_upstream_ip_hash_module.c --- a/src/http/modules/ngx_http_upstream_ip_hash_module.c Thu May 22 00:16:17 2014 +0400 +++ b/src/http/modules/ngx_http_upstream_ip_hash_module.c Fri May 23 13:47:05 2014 +0400 @@ -231,7 +231,7 @@ ngx_http_upstream_get_ip_hash_peer(ngx_p next: - if (++iphp->tries >= 20) { + if (++iphp->tries > 20) { return iphp->get_rr_peer(pc, &iphp->rrp); } } From mdounin at mdounin.ru Fri May 23 10:20:42 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 23 May 2014 14:20:42 +0400 Subject: Buffers Chain In-Reply-To: References: <20140523091212.GD1849@mdounin.ru> Message-ID: <20140523102042.GE1849@mdounin.ru> Hello! On Fri, May 23, 2014 at 10:23:38AM +0100, Paulo Silva wrote: > @Maxim: nice hints, thanks. > > Let me ask you about filters order. To get the contents that will be > sent to client, I want to run my body_filter right before the gziping. > How to accomplish that or if it is even possible > > I am worried that copying the buffers content the original ones can be > yet modified by other filters and than I will log an outdated copy. Try looking into auto/modules. -- Maxim Dounin http://nginx.org/ From pauloasilva at gmail.com Fri May 23 10:57:20 2014 From: pauloasilva at gmail.com (Paulo Silva) Date: Fri, 23 May 2014 11:57:20 +0100 Subject: Buffers Chain In-Reply-To: <20140523102042.GE1849@mdounin.ru> References: <20140523091212.GD1849@mdounin.ru> <20140523102042.GE1849@mdounin.ru> Message-ID: there is other option than modify the auto/modules file? According to my goal (capture the full request response body) I would say that my module must run right before the postpone. Am I supposed to modify the auto/modules like follows? if [ $HTTP_POSTPONE = YES ]; then HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES $HTTP_POSTPONE_FILTER_MODULE" HTTP_SRCS="$HTTP_SRCS $HTTP_POSTPONE_FILTER_SRCS" fi # insert my module here! if [ $HTTP_SSI = YES ]; then have=NGX_HTTP_SSI . auto/have HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES $HTTP_SSI_FILTER_MODULE" HTTP_DEPS="$HTTP_DEPS $HTTP_SSI_DEPS" HTTP_SRCS="$HTTP_SRCS $HTTP_SSI_SRCS" fi I did check my modules config file and I did realize that it is "queued" as HTTP_AUX_FILTER_MODULES. There are different queues for core modules and addons? May thanks for your time. A definitive guide to nginx (modules) development would be a best-seller ;) On Fri, May 23, 2014 at 11:20 AM, Maxim Dounin wrote: > Hello! > > On Fri, May 23, 2014 at 10:23:38AM +0100, Paulo Silva wrote: > >> @Maxim: nice hints, thanks. >> >> Let me ask you about filters order. To get the contents that will be >> sent to client, I want to run my body_filter right before the gziping. >> How to accomplish that or if it is even possible >> >> I am worried that copying the buffers content the original ones can be >> yet modified by other filters and than I will log an outdated copy. > > Try looking into auto/modules. > > -- > Maxim Dounin > http://nginx.org/ > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel -- Paulo A. Silva http://tech.pauloasilva.com http://linkedin.com/in/devpauloasilva/ From mdounin at mdounin.ru Fri May 23 11:58:01 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 23 May 2014 15:58:01 +0400 Subject: Buffers Chain In-Reply-To: References: <20140523091212.GD1849@mdounin.ru> <20140523102042.GE1849@mdounin.ru> Message-ID: <20140523115801.GG1849@mdounin.ru> Hello! On Fri, May 23, 2014 at 11:57:20AM +0100, Paulo Silva wrote: > there is other option than modify the auto/modules file? > > According to my goal (capture the full request response body) I would > say that my module must run right before the postpone. Before the postpone filter you'll get subrequest bodies in your filter, which is probably not what you want (the postpone filter is to glue subrequest together, correctly ordered). > Am I supposed to modify the auto/modules like follows? > > if [ $HTTP_POSTPONE = YES ]; then > HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES $HTTP_POSTPONE_FILTER_MODULE" > HTTP_SRCS="$HTTP_SRCS $HTTP_POSTPONE_FILTER_SRCS" > fi > > # insert my module here! > > if [ $HTTP_SSI = YES ]; then > have=NGX_HTTP_SSI . auto/have > HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES $HTTP_SSI_FILTER_MODULE" > HTTP_DEPS="$HTTP_DEPS $HTTP_SSI_DEPS" > HTTP_SRCS="$HTTP_SRCS $HTTP_SSI_SRCS" > fi > > > I did check my modules config file and I did realize that it is > "queued" as HTTP_AUX_FILTER_MODULES. There are different queues for > core modules and addons? The HTTP_AUX_FILTER_MODULES is a generic queue, and it's the only one currently officially supported for 3rd party modules. If you want your filter to be called right before/after postpone filter, it should be relatively safe to put it into the HTTP_POSTPONE_FILTER_MODULE variable though (and may be with some additional checks to make sure the postpone filter is enabled, or just a code to enable it unconditionally). -- Maxim Dounin http://nginx.org/ From pauloasilva at gmail.com Fri May 23 13:17:27 2014 From: pauloasilva at gmail.com (Paulo Silva) Date: Fri, 23 May 2014 14:17:27 +0100 Subject: Buffers Chain In-Reply-To: <20140523115801.GG1849@mdounin.ru> References: <20140523091212.GD1849@mdounin.ru> <20140523102042.GE1849@mdounin.ru> <20140523115801.GG1849@mdounin.ru> Message-ID: Hi, On Fri, May 23, 2014 at 12:58 PM, Maxim Dounin wrote: > Hello! > > On Fri, May 23, 2014 at 11:57:20AM +0100, Paulo Silva wrote: > >> there is other option than modify the auto/modules file? >> >> According to my goal (capture the full request response body) I would >> say that my module must run right before the postpone. > > Before the postpone filter you'll get subrequest bodies in your > filter, which is probably not what you want (the postpone filter > is to glue subrequest together, correctly ordered). > >> Am I supposed to modify the auto/modules like follows? >> >> if [ $HTTP_POSTPONE = YES ]; then >> HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES $HTTP_POSTPONE_FILTER_MODULE" >> HTTP_SRCS="$HTTP_SRCS $HTTP_POSTPONE_FILTER_SRCS" >> fi >> >> # insert my module here! >> >> if [ $HTTP_SSI = YES ]; then >> have=NGX_HTTP_SSI . auto/have >> HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES $HTTP_SSI_FILTER_MODULE" >> HTTP_DEPS="$HTTP_DEPS $HTTP_SSI_DEPS" >> HTTP_SRCS="$HTTP_SRCS $HTTP_SSI_SRCS" >> fi >> >> >> I did check my modules config file and I did realize that it is >> "queued" as HTTP_AUX_FILTER_MODULES. There are different queues for >> core modules and addons? > > The HTTP_AUX_FILTER_MODULES is a generic queue, and it's the > only one currently officially supported for 3rd party modules. > > If you want your filter to be called right before/after postpone > filter, it should be relatively safe to put it into the > HTTP_POSTPONE_FILTER_MODULE variable though (and may be with some > additional checks to make sure the postpone filter is enabled, or > just a code to enable it unconditionally). > And this is also valid when compiling nginx with the --add-module flag? How does config file look like? My knowledge is restricted to Emiller's Guide To Nginx Module Development (http://www.evanmiller.org/nginx-modules-guide.html) and a few debugging hours. Thanks a bunch, > -- > Maxim Dounin > http://nginx.org/ > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel -- Paulo A. Silva http://tech.pauloasilva.com http://linkedin.com/in/devpauloasilva/ From ru at nginx.com Fri May 23 13:18:33 2014 From: ru at nginx.com (Ruslan Ermilov) Date: Fri, 23 May 2014 13:18:33 +0000 Subject: [nginx] Events: use eventfd() instead of syscall(SYS_eventfd) if... Message-ID: details: http://hg.nginx.org/nginx/rev/31dd63dcb9ea branches: changeset: 5707:31dd63dcb9ea user: Ruslan Ermilov date: Fri May 23 16:37:05 2014 +0400 description: Events: use eventfd() instead of syscall(SYS_eventfd) if possible. This fixes --with-file-aio support on systems that lack eventfd() syscall, notably aarch64 Linux. The syscall(SYS_eventfd) may still be necessary on systems that have eventfd() syscall in the kernel but lack it in glibc, e.g. as seen in the current CentOS 5 release. diffstat: auto/unix | 33 +++++++++++++++++++++++++++------ src/event/modules/ngx_epoll_module.c | 8 ++++---- src/os/unix/ngx_linux_config.h | 3 +++ 3 files changed, 34 insertions(+), 10 deletions(-) diffs (103 lines): diff -r a2bf26774cd3 -r 31dd63dcb9ea auto/unix --- a/auto/unix Fri May 23 13:47:05 2014 +0400 +++ b/auto/unix Fri May 23 16:37:05 2014 +0400 @@ -398,16 +398,36 @@ if [ $NGX_FILE_AIO = YES ]; then if [ $ngx_found = yes ]; then CORE_SRCS="$CORE_SRCS $FILE_AIO_SRCS" + fi - elif [ $ngx_found = no ]; then + if [ $ngx_found = no ]; then ngx_feature="Linux AIO support" ngx_feature_name="NGX_HAVE_FILE_AIO" ngx_feature_run=no ngx_feature_incs="#include - #include " + #include " ngx_feature_path= ngx_feature_libs= + ngx_feature_test="struct iocb iocb; + iocb.aio_lio_opcode = IOCB_CMD_PREAD; + iocb.aio_flags = IOCB_FLAG_RESFD; + iocb.aio_resfd = -1; + (void) eventfd(0, 0)" + . auto/feature + + if [ $ngx_found = yes ]; then + have=NGX_HAVE_EVENTFD . auto/have + have=NGX_HAVE_SYS_EVENTFD_H . auto/have + CORE_SRCS="$CORE_SRCS $LINUX_AIO_SRCS" + fi + fi + + if [ $ngx_found = no ]; then + + ngx_feature="Linux AIO support (SYS_eventfd)" + ngx_feature_incs="#include + #include " ngx_feature_test="int n = SYS_eventfd; struct iocb iocb; iocb.aio_lio_opcode = IOCB_CMD_PREAD; @@ -418,16 +438,17 @@ if [ $NGX_FILE_AIO = YES ]; then if [ $ngx_found = yes ]; then have=NGX_HAVE_EVENTFD . auto/have CORE_SRCS="$CORE_SRCS $LINUX_AIO_SRCS" + fi + fi - else - cat << END + if [ $ngx_found = no ]; then + cat << END $0: no supported file AIO was found Currently file AIO is supported on FreeBSD 4.3+ and Linux 2.6.22+ only END - exit 1 - fi + exit 1 fi fi diff -r a2bf26774cd3 -r 31dd63dcb9ea src/event/modules/ngx_epoll_module.c --- a/src/event/modules/ngx_epoll_module.c Fri May 23 13:47:05 2014 +0400 +++ b/src/event/modules/ngx_epoll_module.c Fri May 23 16:37:05 2014 +0400 @@ -193,10 +193,6 @@ ngx_module_t ngx_epoll_module = { * We call io_setup(), io_destroy() io_submit(), and io_getevents() directly * as syscalls instead of libaio usage, because the library header file * supports eventfd() since 0.3.107 version only. - * - * Also we do not use eventfd() in glibc, because glibc supports it - * since 2.8 version and glibc maps two syscalls eventfd() and eventfd2() - * into single eventfd() function with different number of parameters. */ static int @@ -227,7 +223,11 @@ ngx_epoll_aio_init(ngx_cycle_t *cycle, n int n; struct epoll_event ee; +#if (NGX_HAVE_SYS_EVENTFD_H) + ngx_eventfd = eventfd(0, 0); +#else ngx_eventfd = syscall(SYS_eventfd, 0); +#endif if (ngx_eventfd == -1) { ngx_log_error(NGX_LOG_EMERG, cycle->log, ngx_errno, diff -r a2bf26774cd3 -r 31dd63dcb9ea src/os/unix/ngx_linux_config.h --- a/src/os/unix/ngx_linux_config.h Fri May 23 13:47:05 2014 +0400 +++ b/src/os/unix/ngx_linux_config.h Fri May 23 16:37:05 2014 +0400 @@ -94,6 +94,9 @@ extern ssize_t sendfile(int s, int fd, i #if (NGX_HAVE_FILE_AIO) +#if (NGX_HAVE_SYS_EVENTFD_H) +#include +#endif #include #include typedef struct iocb ngx_aiocb_t; From mdounin at mdounin.ru Fri May 23 13:41:08 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Fri, 23 May 2014 17:41:08 +0400 Subject: Buffers Chain In-Reply-To: References: <20140523091212.GD1849@mdounin.ru> <20140523102042.GE1849@mdounin.ru> <20140523115801.GG1849@mdounin.ru> Message-ID: <20140523134108.GK1849@mdounin.ru> Hello! On Fri, May 23, 2014 at 02:17:27PM +0100, Paulo Silva wrote: > Hi, > > On Fri, May 23, 2014 at 12:58 PM, Maxim Dounin wrote: > > Hello! > > > > On Fri, May 23, 2014 at 11:57:20AM +0100, Paulo Silva wrote: > > > >> there is other option than modify the auto/modules file? > >> > >> According to my goal (capture the full request response body) I would > >> say that my module must run right before the postpone. > > > > Before the postpone filter you'll get subrequest bodies in your > > filter, which is probably not what you want (the postpone filter > > is to glue subrequest together, correctly ordered). > > > >> Am I supposed to modify the auto/modules like follows? > >> > >> if [ $HTTP_POSTPONE = YES ]; then > >> HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES $HTTP_POSTPONE_FILTER_MODULE" > >> HTTP_SRCS="$HTTP_SRCS $HTTP_POSTPONE_FILTER_SRCS" > >> fi > >> > >> # insert my module here! > >> > >> if [ $HTTP_SSI = YES ]; then > >> have=NGX_HTTP_SSI . auto/have > >> HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES $HTTP_SSI_FILTER_MODULE" > >> HTTP_DEPS="$HTTP_DEPS $HTTP_SSI_DEPS" > >> HTTP_SRCS="$HTTP_SRCS $HTTP_SSI_SRCS" > >> fi > >> > >> > >> I did check my modules config file and I did realize that it is > >> "queued" as HTTP_AUX_FILTER_MODULES. There are different queues for > >> core modules and addons? > > > > The HTTP_AUX_FILTER_MODULES is a generic queue, and it's the > > only one currently officially supported for 3rd party modules. > > > > If you want your filter to be called right before/after postpone > > filter, it should be relatively safe to put it into the > > HTTP_POSTPONE_FILTER_MODULE variable though (and may be with some > > additional checks to make sure the postpone filter is enabled, or > > just a code to enable it unconditionally). > > > > And this is also valid when compiling nginx with the --add-module flag? > How does config file look like? > > My knowledge is restricted to Emiller's Guide To Nginx Module > Development (http://www.evanmiller.org/nginx-modules-guide.html) and a > few debugging hours. Uhm, looking again into auto/modules I think I was wrong, and modifying the HTTP_POSTPONE_FILTER_MODULE variable won't work (added module config scripts are executed later on), you should modify HTTP_FILTER_MODULES variable instead, and put your module into a proper position. Note that the "config" file of a module is just a shell script, and you are free to do more or less anything there. -- Maxim Dounin http://nginx.org/ From pauloasilva at gmail.com Fri May 23 13:50:46 2014 From: pauloasilva at gmail.com (Paulo Silva) Date: Fri, 23 May 2014 14:50:46 +0100 Subject: Buffers Chain In-Reply-To: <20140523134108.GK1849@mdounin.ru> References: <20140523091212.GD1849@mdounin.ru> <20140523102042.GE1849@mdounin.ru> <20140523115801.GG1849@mdounin.ru> <20140523134108.GK1849@mdounin.ru> Message-ID: Because I don't have deep knowledge of nginx internal and I can not find a proper resource about it, the best I can do and with what I am comfortable is with body_filter. Do you think I can notice whether all other 3rd party module filters finish modifying the ngx_chain_t *in ? On Fri, May 23, 2014 at 2:41 PM, Maxim Dounin wrote: > Hello! > > On Fri, May 23, 2014 at 02:17:27PM +0100, Paulo Silva wrote: > >> Hi, >> >> On Fri, May 23, 2014 at 12:58 PM, Maxim Dounin wrote: >> > Hello! >> > >> > On Fri, May 23, 2014 at 11:57:20AM +0100, Paulo Silva wrote: >> > >> >> there is other option than modify the auto/modules file? >> >> >> >> According to my goal (capture the full request response body) I would >> >> say that my module must run right before the postpone. >> > >> > Before the postpone filter you'll get subrequest bodies in your >> > filter, which is probably not what you want (the postpone filter >> > is to glue subrequest together, correctly ordered). >> > >> >> Am I supposed to modify the auto/modules like follows? >> >> >> >> if [ $HTTP_POSTPONE = YES ]; then >> >> HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES $HTTP_POSTPONE_FILTER_MODULE" >> >> HTTP_SRCS="$HTTP_SRCS $HTTP_POSTPONE_FILTER_SRCS" >> >> fi >> >> >> >> # insert my module here! >> >> >> >> if [ $HTTP_SSI = YES ]; then >> >> have=NGX_HTTP_SSI . auto/have >> >> HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES $HTTP_SSI_FILTER_MODULE" >> >> HTTP_DEPS="$HTTP_DEPS $HTTP_SSI_DEPS" >> >> HTTP_SRCS="$HTTP_SRCS $HTTP_SSI_SRCS" >> >> fi >> >> >> >> >> >> I did check my modules config file and I did realize that it is >> >> "queued" as HTTP_AUX_FILTER_MODULES. There are different queues for >> >> core modules and addons? >> > >> > The HTTP_AUX_FILTER_MODULES is a generic queue, and it's the >> > only one currently officially supported for 3rd party modules. >> > >> > If you want your filter to be called right before/after postpone >> > filter, it should be relatively safe to put it into the >> > HTTP_POSTPONE_FILTER_MODULE variable though (and may be with some >> > additional checks to make sure the postpone filter is enabled, or >> > just a code to enable it unconditionally). >> > >> >> And this is also valid when compiling nginx with the --add-module flag? >> How does config file look like? >> >> My knowledge is restricted to Emiller's Guide To Nginx Module >> Development (http://www.evanmiller.org/nginx-modules-guide.html) and a >> few debugging hours. > > Uhm, looking again into auto/modules I think I was wrong, and > modifying the HTTP_POSTPONE_FILTER_MODULE variable won't work > (added module config scripts are executed later on), you should > modify HTTP_FILTER_MODULES variable instead, and put your module > into a proper position. > > Note that the "config" file of a module is just a shell script, > and you are free to do more or less anything there. > > -- > Maxim Dounin > http://nginx.org/ > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel -- Paulo A. Silva http://tech.pauloasilva.com http://linkedin.com/in/devpauloasilva/ From pauloasilva at gmail.com Sun May 25 23:22:43 2014 From: pauloasilva at gmail.com (Paulo Silva) Date: Mon, 26 May 2014 00:22:43 +0100 Subject: Buffers Chain In-Reply-To: References: <20140523091212.GD1849@mdounin.ru> <20140523102042.GE1849@mdounin.ru> <20140523115801.GG1849@mdounin.ru> <20140523134108.GK1849@mdounin.ru> Message-ID: Hi, I'm not sure whether I will face problems with other filters modifying the response body after mine, but for know I'm comfortable as I can rebuild the full response body just iterating buffers chains. As I said before I'm using nginx as reverse proxy and my main goal is to pass the upstream (proxy_pass) response to another local process (relative to nginx). I am benchmarking Unix sockets and Shared memory as IPC. I did it already for Unix Sockets and with my prototype the nginx "performance" dropped for half the number of requests per second. Of course I'm doing something really bad. Is it OK to use socket/connect/send from inside an nginx module? I would be glad to hear from you. Thanks, On Fri, May 23, 2014 at 2:50 PM, Paulo Silva wrote: > Because I don't have deep knowledge of nginx internal and I can not > find a proper resource about it, the best I can do and with what I am > comfortable is with body_filter. > > Do you think I can notice whether all other 3rd party module filters > finish modifying the ngx_chain_t *in ? > > > > On Fri, May 23, 2014 at 2:41 PM, Maxim Dounin wrote: >> Hello! >> >> On Fri, May 23, 2014 at 02:17:27PM +0100, Paulo Silva wrote: >> >>> Hi, >>> >>> On Fri, May 23, 2014 at 12:58 PM, Maxim Dounin wrote: >>> > Hello! >>> > >>> > On Fri, May 23, 2014 at 11:57:20AM +0100, Paulo Silva wrote: >>> > >>> >> there is other option than modify the auto/modules file? >>> >> >>> >> According to my goal (capture the full request response body) I would >>> >> say that my module must run right before the postpone. >>> > >>> > Before the postpone filter you'll get subrequest bodies in your >>> > filter, which is probably not what you want (the postpone filter >>> > is to glue subrequest together, correctly ordered). >>> > >>> >> Am I supposed to modify the auto/modules like follows? >>> >> >>> >> if [ $HTTP_POSTPONE = YES ]; then >>> >> HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES $HTTP_POSTPONE_FILTER_MODULE" >>> >> HTTP_SRCS="$HTTP_SRCS $HTTP_POSTPONE_FILTER_SRCS" >>> >> fi >>> >> >>> >> # insert my module here! >>> >> >>> >> if [ $HTTP_SSI = YES ]; then >>> >> have=NGX_HTTP_SSI . auto/have >>> >> HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES $HTTP_SSI_FILTER_MODULE" >>> >> HTTP_DEPS="$HTTP_DEPS $HTTP_SSI_DEPS" >>> >> HTTP_SRCS="$HTTP_SRCS $HTTP_SSI_SRCS" >>> >> fi >>> >> >>> >> >>> >> I did check my modules config file and I did realize that it is >>> >> "queued" as HTTP_AUX_FILTER_MODULES. There are different queues for >>> >> core modules and addons? >>> > >>> > The HTTP_AUX_FILTER_MODULES is a generic queue, and it's the >>> > only one currently officially supported for 3rd party modules. >>> > >>> > If you want your filter to be called right before/after postpone >>> > filter, it should be relatively safe to put it into the >>> > HTTP_POSTPONE_FILTER_MODULE variable though (and may be with some >>> > additional checks to make sure the postpone filter is enabled, or >>> > just a code to enable it unconditionally). >>> > >>> >>> And this is also valid when compiling nginx with the --add-module flag? >>> How does config file look like? >>> >>> My knowledge is restricted to Emiller's Guide To Nginx Module >>> Development (http://www.evanmiller.org/nginx-modules-guide.html) and a >>> few debugging hours. >> >> Uhm, looking again into auto/modules I think I was wrong, and >> modifying the HTTP_POSTPONE_FILTER_MODULE variable won't work >> (added module config scripts are executed later on), you should >> modify HTTP_FILTER_MODULES variable instead, and put your module >> into a proper position. >> >> Note that the "config" file of a module is just a shell script, >> and you are free to do more or less anything there. >> >> -- >> Maxim Dounin >> http://nginx.org/ >> >> _______________________________________________ >> nginx-devel mailing list >> nginx-devel at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx-devel > > > > -- > Paulo A. Silva > http://tech.pauloasilva.com > http://linkedin.com/in/devpauloasilva/ -- Paulo A. Silva http://tech.pauloasilva.com http://linkedin.com/in/devpauloasilva/ From xinghua_hi at 126.com Mon May 26 01:36:52 2014 From: xinghua_hi at 126.com (xinghua_hi) Date: Mon, 26 May 2014 09:36:52 +0800 (CST) Subject: connection starvation problem Message-ID: <4e8960d3.4fd3.146362c5247.Coremail.xinghua_hi@126.com> hi: In man epoll , a section refer to "epoll edge trigger starvation problem", and around to this we need to maintain a ready list instead of read socket until EAGAIN. but i didn't find any related solution code in nginx. In request body read function `ngx_http_do_read_client_request_body`, it seems that nginx will read socket buffer until EAGAIN or has read Content-length data. so if a client is fast enough and post data is large enough, is it possible to starve other connections? SO do i miss something important? nginx how to deal with this situation in read and write? thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From mat999 at gmail.com Mon May 26 08:09:46 2014 From: mat999 at gmail.com (SplitIce) Date: Mon, 26 May 2014 18:09:46 +1000 Subject: Buffers Chain In-Reply-To: References: <20140523091212.GD1849@mdounin.ru> <20140523102042.GE1849@mdounin.ru> <20140523115801.GG1849@mdounin.ru> <20140523134108.GK1849@mdounin.ru> Message-ID: As in blocking send and connect? I don't know the specifics of Unix Sockets, but don't they block when the buffer fills (I know FIFO queues do)? On Mon, May 26, 2014 at 9:22 AM, Paulo Silva wrote: > Hi, > I'm not sure whether I will face problems with other filters modifying > the response body after mine, but for know I'm comfortable as I can > rebuild the full response body just iterating buffers chains. > > As I said before I'm using nginx as reverse proxy and my main goal is > to pass the upstream (proxy_pass) response to another local process > (relative to nginx). > > I am benchmarking Unix sockets and Shared memory as IPC. > I did it already for Unix Sockets and with my prototype the nginx > "performance" dropped for half the number of requests per second. Of > course I'm doing something really bad. > > Is it OK to use socket/connect/send from inside an nginx module? > > I would be glad to hear from you. > Thanks, > > On Fri, May 23, 2014 at 2:50 PM, Paulo Silva > wrote: > > Because I don't have deep knowledge of nginx internal and I can not > > find a proper resource about it, the best I can do and with what I am > > comfortable is with body_filter. > > > > Do you think I can notice whether all other 3rd party module filters > > finish modifying the ngx_chain_t *in ? > > > > > > > > On Fri, May 23, 2014 at 2:41 PM, Maxim Dounin > wrote: > >> Hello! > >> > >> On Fri, May 23, 2014 at 02:17:27PM +0100, Paulo Silva wrote: > >> > >>> Hi, > >>> > >>> On Fri, May 23, 2014 at 12:58 PM, Maxim Dounin > wrote: > >>> > Hello! > >>> > > >>> > On Fri, May 23, 2014 at 11:57:20AM +0100, Paulo Silva wrote: > >>> > > >>> >> there is other option than modify the auto/modules file? > >>> >> > >>> >> According to my goal (capture the full request response body) I > would > >>> >> say that my module must run right before the postpone. > >>> > > >>> > Before the postpone filter you'll get subrequest bodies in your > >>> > filter, which is probably not what you want (the postpone filter > >>> > is to glue subrequest together, correctly ordered). > >>> > > >>> >> Am I supposed to modify the auto/modules like follows? > >>> >> > >>> >> if [ $HTTP_POSTPONE = YES ]; then > >>> >> HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES > $HTTP_POSTPONE_FILTER_MODULE" > >>> >> HTTP_SRCS="$HTTP_SRCS $HTTP_POSTPONE_FILTER_SRCS" > >>> >> fi > >>> >> > >>> >> # insert my module here! > >>> >> > >>> >> if [ $HTTP_SSI = YES ]; then > >>> >> have=NGX_HTTP_SSI . auto/have > >>> >> HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES > $HTTP_SSI_FILTER_MODULE" > >>> >> HTTP_DEPS="$HTTP_DEPS $HTTP_SSI_DEPS" > >>> >> HTTP_SRCS="$HTTP_SRCS $HTTP_SSI_SRCS" > >>> >> fi > >>> >> > >>> >> > >>> >> I did check my modules config file and I did realize that it is > >>> >> "queued" as HTTP_AUX_FILTER_MODULES. There are different queues for > >>> >> core modules and addons? > >>> > > >>> > The HTTP_AUX_FILTER_MODULES is a generic queue, and it's the > >>> > only one currently officially supported for 3rd party modules. > >>> > > >>> > If you want your filter to be called right before/after postpone > >>> > filter, it should be relatively safe to put it into the > >>> > HTTP_POSTPONE_FILTER_MODULE variable though (and may be with some > >>> > additional checks to make sure the postpone filter is enabled, or > >>> > just a code to enable it unconditionally). > >>> > > >>> > >>> And this is also valid when compiling nginx with the --add-module flag? > >>> How does config file look like? > >>> > >>> My knowledge is restricted to Emiller's Guide To Nginx Module > >>> Development (http://www.evanmiller.org/nginx-modules-guide.html) and a > >>> few debugging hours. > >> > >> Uhm, looking again into auto/modules I think I was wrong, and > >> modifying the HTTP_POSTPONE_FILTER_MODULE variable won't work > >> (added module config scripts are executed later on), you should > >> modify HTTP_FILTER_MODULES variable instead, and put your module > >> into a proper position. > >> > >> Note that the "config" file of a module is just a shell script, > >> and you are free to do more or less anything there. > >> > >> -- > >> Maxim Dounin > >> http://nginx.org/ > >> > >> _______________________________________________ > >> nginx-devel mailing list > >> nginx-devel at nginx.org > >> http://mailman.nginx.org/mailman/listinfo/nginx-devel > > > > > > > > -- > > Paulo A. Silva > > http://tech.pauloasilva.com > > http://linkedin.com/in/devpauloasilva/ > > > > -- > Paulo A. Silva > http://tech.pauloasilva.com > http://linkedin.com/in/devpauloasilva/ > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pauloasilva at gmail.com Mon May 26 08:18:29 2014 From: pauloasilva at gmail.com (Paulo Silva) Date: Mon, 26 May 2014 09:18:29 +0100 Subject: Buffers Chain In-Reply-To: References: <20140523091212.GD1849@mdounin.ru> <20140523102042.GE1849@mdounin.ru> <20140523115801.GG1849@mdounin.ru> <20140523134108.GK1849@mdounin.ru> Message-ID: On Mon, May 26, 2014 at 9:09 AM, SplitIce wrote: > As in blocking send and connect? I don't know the specifics of Unix Sockets, > but don't they block when the buffer fills (I know FIFO queues do)? > Sorry, I don't fully understand your question. I was expecting that with the SOCK_NONBLOCK it would not block. What would be your approach? Do you know about any nginx internal mechanism to accomplish this goal (get the upstream response body out of nginx)? > > On Mon, May 26, 2014 at 9:22 AM, Paulo Silva wrote: >> >> Hi, >> I'm not sure whether I will face problems with other filters modifying >> the response body after mine, but for know I'm comfortable as I can >> rebuild the full response body just iterating buffers chains. >> >> As I said before I'm using nginx as reverse proxy and my main goal is >> to pass the upstream (proxy_pass) response to another local process >> (relative to nginx). >> >> I am benchmarking Unix sockets and Shared memory as IPC. >> I did it already for Unix Sockets and with my prototype the nginx >> "performance" dropped for half the number of requests per second. Of >> course I'm doing something really bad. >> >> Is it OK to use socket/connect/send from inside an nginx module? >> >> I would be glad to hear from you. >> Thanks, >> >> On Fri, May 23, 2014 at 2:50 PM, Paulo Silva >> wrote: >> > Because I don't have deep knowledge of nginx internal and I can not >> > find a proper resource about it, the best I can do and with what I am >> > comfortable is with body_filter. >> > >> > Do you think I can notice whether all other 3rd party module filters >> > finish modifying the ngx_chain_t *in ? >> > >> > >> > >> > On Fri, May 23, 2014 at 2:41 PM, Maxim Dounin >> > wrote: >> >> Hello! >> >> >> >> On Fri, May 23, 2014 at 02:17:27PM +0100, Paulo Silva wrote: >> >> >> >>> Hi, >> >>> >> >>> On Fri, May 23, 2014 at 12:58 PM, Maxim Dounin >> >>> wrote: >> >>> > Hello! >> >>> > >> >>> > On Fri, May 23, 2014 at 11:57:20AM +0100, Paulo Silva wrote: >> >>> > >> >>> >> there is other option than modify the auto/modules file? >> >>> >> >> >>> >> According to my goal (capture the full request response body) I >> >>> >> would >> >>> >> say that my module must run right before the postpone. >> >>> > >> >>> > Before the postpone filter you'll get subrequest bodies in your >> >>> > filter, which is probably not what you want (the postpone filter >> >>> > is to glue subrequest together, correctly ordered). >> >>> > >> >>> >> Am I supposed to modify the auto/modules like follows? >> >>> >> >> >>> >> if [ $HTTP_POSTPONE = YES ]; then >> >>> >> HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES >> >>> >> $HTTP_POSTPONE_FILTER_MODULE" >> >>> >> HTTP_SRCS="$HTTP_SRCS $HTTP_POSTPONE_FILTER_SRCS" >> >>> >> fi >> >>> >> >> >>> >> # insert my module here! >> >>> >> >> >>> >> if [ $HTTP_SSI = YES ]; then >> >>> >> have=NGX_HTTP_SSI . auto/have >> >>> >> HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES >> >>> >> $HTTP_SSI_FILTER_MODULE" >> >>> >> HTTP_DEPS="$HTTP_DEPS $HTTP_SSI_DEPS" >> >>> >> HTTP_SRCS="$HTTP_SRCS $HTTP_SSI_SRCS" >> >>> >> fi >> >>> >> >> >>> >> >> >>> >> I did check my modules config file and I did realize that it is >> >>> >> "queued" as HTTP_AUX_FILTER_MODULES. There are different queues for >> >>> >> core modules and addons? >> >>> > >> >>> > The HTTP_AUX_FILTER_MODULES is a generic queue, and it's the >> >>> > only one currently officially supported for 3rd party modules. >> >>> > >> >>> > If you want your filter to be called right before/after postpone >> >>> > filter, it should be relatively safe to put it into the >> >>> > HTTP_POSTPONE_FILTER_MODULE variable though (and may be with some >> >>> > additional checks to make sure the postpone filter is enabled, or >> >>> > just a code to enable it unconditionally). >> >>> > >> >>> >> >>> And this is also valid when compiling nginx with the --add-module >> >>> flag? >> >>> How does config file look like? >> >>> >> >>> My knowledge is restricted to Emiller's Guide To Nginx Module >> >>> Development (http://www.evanmiller.org/nginx-modules-guide.html) and a >> >>> few debugging hours. >> >> >> >> Uhm, looking again into auto/modules I think I was wrong, and >> >> modifying the HTTP_POSTPONE_FILTER_MODULE variable won't work >> >> (added module config scripts are executed later on), you should >> >> modify HTTP_FILTER_MODULES variable instead, and put your module >> >> into a proper position. >> >> >> >> Note that the "config" file of a module is just a shell script, >> >> and you are free to do more or less anything there. >> >> >> >> -- >> >> Maxim Dounin >> >> http://nginx.org/ >> >> >> >> _______________________________________________ >> >> nginx-devel mailing list >> >> nginx-devel at nginx.org >> >> http://mailman.nginx.org/mailman/listinfo/nginx-devel >> > >> > >> > >> > -- >> > Paulo A. Silva >> > http://tech.pauloasilva.com >> > http://linkedin.com/in/devpauloasilva/ >> >> >> >> -- >> Paulo A. Silva >> http://tech.pauloasilva.com >> http://linkedin.com/in/devpauloasilva/ >> >> _______________________________________________ >> nginx-devel mailing list >> nginx-devel at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx-devel > > > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel -- Paulo A. Silva http://tech.pauloasilva.com http://linkedin.com/in/devpauloasilva/ From mat999 at gmail.com Mon May 26 08:28:50 2014 From: mat999 at gmail.com (SplitIce) Date: Mon, 26 May 2014 18:28:50 +1000 Subject: Buffers Chain In-Reply-To: References: <20140523091212.GD1849@mdounin.ru> <20140523102042.GE1849@mdounin.ru> <20140523115801.GG1849@mdounin.ru> <20140523134108.GK1849@mdounin.ru> Message-ID: Yes, connecting with SOCK_NONBLOCK shouldnt block. I don't believe this is mentioned previously. If your code blocks (e.g blocking connect or blocking send) then it would reduce nginx throughput substantially. Thats the point I was trying to make. Have you investigated srcache ( https://github.com/openresty/srcache-nginx-module/)? srcache_store could be used without srcache_fetch to store only. Your store handler could be a proxy_pass call to pass a restful server or whatever you use for storing the HTML responses. Perhaps a better solution would be to use the nginx lua extension instead (less of a hack). Regards, Mathew On Mon, May 26, 2014 at 6:18 PM, Paulo Silva wrote: > On Mon, May 26, 2014 at 9:09 AM, SplitIce wrote: > > As in blocking send and connect? I don't know the specifics of Unix > Sockets, > > but don't they block when the buffer fills (I know FIFO queues do)? > > > > Sorry, I don't fully understand your question. > I was expecting that with the SOCK_NONBLOCK it would not block. > > What would be your approach? > Do you know about any nginx internal mechanism to accomplish this goal > (get the upstream response body out of nginx)? > > > > > On Mon, May 26, 2014 at 9:22 AM, Paulo Silva > wrote: > >> > >> Hi, > >> I'm not sure whether I will face problems with other filters modifying > >> the response body after mine, but for know I'm comfortable as I can > >> rebuild the full response body just iterating buffers chains. > >> > >> As I said before I'm using nginx as reverse proxy and my main goal is > >> to pass the upstream (proxy_pass) response to another local process > >> (relative to nginx). > >> > >> I am benchmarking Unix sockets and Shared memory as IPC. > >> I did it already for Unix Sockets and with my prototype the nginx > >> "performance" dropped for half the number of requests per second. Of > >> course I'm doing something really bad. > >> > >> Is it OK to use socket/connect/send from inside an nginx module? > >> > >> I would be glad to hear from you. > >> Thanks, > >> > >> On Fri, May 23, 2014 at 2:50 PM, Paulo Silva > >> wrote: > >> > Because I don't have deep knowledge of nginx internal and I can not > >> > find a proper resource about it, the best I can do and with what I am > >> > comfortable is with body_filter. > >> > > >> > Do you think I can notice whether all other 3rd party module filters > >> > finish modifying the ngx_chain_t *in ? > >> > > >> > > >> > > >> > On Fri, May 23, 2014 at 2:41 PM, Maxim Dounin > >> > wrote: > >> >> Hello! > >> >> > >> >> On Fri, May 23, 2014 at 02:17:27PM +0100, Paulo Silva wrote: > >> >> > >> >>> Hi, > >> >>> > >> >>> On Fri, May 23, 2014 at 12:58 PM, Maxim Dounin > >> >>> wrote: > >> >>> > Hello! > >> >>> > > >> >>> > On Fri, May 23, 2014 at 11:57:20AM +0100, Paulo Silva wrote: > >> >>> > > >> >>> >> there is other option than modify the auto/modules file? > >> >>> >> > >> >>> >> According to my goal (capture the full request response body) I > >> >>> >> would > >> >>> >> say that my module must run right before the postpone. > >> >>> > > >> >>> > Before the postpone filter you'll get subrequest bodies in your > >> >>> > filter, which is probably not what you want (the postpone filter > >> >>> > is to glue subrequest together, correctly ordered). > >> >>> > > >> >>> >> Am I supposed to modify the auto/modules like follows? > >> >>> >> > >> >>> >> if [ $HTTP_POSTPONE = YES ]; then > >> >>> >> HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES > >> >>> >> $HTTP_POSTPONE_FILTER_MODULE" > >> >>> >> HTTP_SRCS="$HTTP_SRCS $HTTP_POSTPONE_FILTER_SRCS" > >> >>> >> fi > >> >>> >> > >> >>> >> # insert my module here! > >> >>> >> > >> >>> >> if [ $HTTP_SSI = YES ]; then > >> >>> >> have=NGX_HTTP_SSI . auto/have > >> >>> >> HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES > >> >>> >> $HTTP_SSI_FILTER_MODULE" > >> >>> >> HTTP_DEPS="$HTTP_DEPS $HTTP_SSI_DEPS" > >> >>> >> HTTP_SRCS="$HTTP_SRCS $HTTP_SSI_SRCS" > >> >>> >> fi > >> >>> >> > >> >>> >> > >> >>> >> I did check my modules config file and I did realize that it is > >> >>> >> "queued" as HTTP_AUX_FILTER_MODULES. There are different queues > for > >> >>> >> core modules and addons? > >> >>> > > >> >>> > The HTTP_AUX_FILTER_MODULES is a generic queue, and it's the > >> >>> > only one currently officially supported for 3rd party modules. > >> >>> > > >> >>> > If you want your filter to be called right before/after postpone > >> >>> > filter, it should be relatively safe to put it into the > >> >>> > HTTP_POSTPONE_FILTER_MODULE variable though (and may be with some > >> >>> > additional checks to make sure the postpone filter is enabled, or > >> >>> > just a code to enable it unconditionally). > >> >>> > > >> >>> > >> >>> And this is also valid when compiling nginx with the --add-module > >> >>> flag? > >> >>> How does config file look like? > >> >>> > >> >>> My knowledge is restricted to Emiller's Guide To Nginx Module > >> >>> Development (http://www.evanmiller.org/nginx-modules-guide.html) > and a > >> >>> few debugging hours. > >> >> > >> >> Uhm, looking again into auto/modules I think I was wrong, and > >> >> modifying the HTTP_POSTPONE_FILTER_MODULE variable won't work > >> >> (added module config scripts are executed later on), you should > >> >> modify HTTP_FILTER_MODULES variable instead, and put your module > >> >> into a proper position. > >> >> > >> >> Note that the "config" file of a module is just a shell script, > >> >> and you are free to do more or less anything there. > >> >> > >> >> -- > >> >> Maxim Dounin > >> >> http://nginx.org/ > >> >> > >> >> _______________________________________________ > >> >> nginx-devel mailing list > >> >> nginx-devel at nginx.org > >> >> http://mailman.nginx.org/mailman/listinfo/nginx-devel > >> > > >> > > >> > > >> > -- > >> > Paulo A. Silva > >> > http://tech.pauloasilva.com > >> > http://linkedin.com/in/devpauloasilva/ > >> > >> > >> > >> -- > >> Paulo A. Silva > >> http://tech.pauloasilva.com > >> http://linkedin.com/in/devpauloasilva/ > >> > >> _______________________________________________ > >> nginx-devel mailing list > >> nginx-devel at nginx.org > >> http://mailman.nginx.org/mailman/listinfo/nginx-devel > > > > > > > > _______________________________________________ > > nginx-devel mailing list > > nginx-devel at nginx.org > > http://mailman.nginx.org/mailman/listinfo/nginx-devel > > > > -- > Paulo A. Silva > http://tech.pauloasilva.com > http://linkedin.com/in/devpauloasilva/ > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pauloasilva at gmail.com Mon May 26 09:17:35 2014 From: pauloasilva at gmail.com (Paulo Silva) Date: Mon, 26 May 2014 10:17:35 +0100 Subject: Buffers Chain In-Reply-To: References: <20140523091212.GD1849@mdounin.ru> <20140523102042.GE1849@mdounin.ru> <20140523115801.GG1849@mdounin.ru> <20140523134108.GK1849@mdounin.ru> Message-ID: On Mon, May 26, 2014 at 9:28 AM, SplitIce wrote: > Yes, connecting with SOCK_NONBLOCK shouldnt block. I don't believe this is > mentioned previously. If your code blocks (e.g blocking connect or blocking > send) then it would reduce nginx throughput substantially. Thats the point I > was trying to make. > Yap, I didn't mention SOCK_NONBLOCK. Sorry about that. Although I'm working on a prototype to benchmark, my concern is about socket() and connect() calls on every nginx filter execution. How would you manage this? > Have you investigated srcache > (https://github.com/openresty/srcache-nginx-module/)? srcache_store could be > used without srcache_fetch to store only. Your store handler could be a > proxy_pass call to pass a restful server or whatever you use for storing the > HTML responses. > I didn't know about this module. Thanks, I will have a deep look on it. Do you think that I can pick the proxy_pass response and store it on memcache easily? > Perhaps a better solution would be to use the nginx lua extension instead > (less of a hack). > Do you think that lua extension has advantages regarding an nginx filter module. Sorry about so many questions but at the beginning, nginx is a all mystery, and you'll worried to do not break its performance. Thanks a bunch, > Regards, > Mathew > > > On Mon, May 26, 2014 at 6:18 PM, Paulo Silva wrote: >> >> On Mon, May 26, 2014 at 9:09 AM, SplitIce wrote: >> > As in blocking send and connect? I don't know the specifics of Unix >> > Sockets, >> > but don't they block when the buffer fills (I know FIFO queues do)? >> > >> >> Sorry, I don't fully understand your question. >> I was expecting that with the SOCK_NONBLOCK it would not block. >> >> What would be your approach? >> Do you know about any nginx internal mechanism to accomplish this goal >> (get the upstream response body out of nginx)? >> >> > >> > On Mon, May 26, 2014 at 9:22 AM, Paulo Silva >> > wrote: >> >> >> >> Hi, >> >> I'm not sure whether I will face problems with other filters modifying >> >> the response body after mine, but for know I'm comfortable as I can >> >> rebuild the full response body just iterating buffers chains. >> >> >> >> As I said before I'm using nginx as reverse proxy and my main goal is >> >> to pass the upstream (proxy_pass) response to another local process >> >> (relative to nginx). >> >> >> >> I am benchmarking Unix sockets and Shared memory as IPC. >> >> I did it already for Unix Sockets and with my prototype the nginx >> >> "performance" dropped for half the number of requests per second. Of >> >> course I'm doing something really bad. >> >> >> >> Is it OK to use socket/connect/send from inside an nginx module? >> >> >> >> I would be glad to hear from you. >> >> Thanks, >> >> >> >> On Fri, May 23, 2014 at 2:50 PM, Paulo Silva >> >> wrote: >> >> > Because I don't have deep knowledge of nginx internal and I can not >> >> > find a proper resource about it, the best I can do and with what I am >> >> > comfortable is with body_filter. >> >> > >> >> > Do you think I can notice whether all other 3rd party module filters >> >> > finish modifying the ngx_chain_t *in ? >> >> > >> >> > >> >> > >> >> > On Fri, May 23, 2014 at 2:41 PM, Maxim Dounin >> >> > wrote: >> >> >> Hello! >> >> >> >> >> >> On Fri, May 23, 2014 at 02:17:27PM +0100, Paulo Silva wrote: >> >> >> >> >> >>> Hi, >> >> >>> >> >> >>> On Fri, May 23, 2014 at 12:58 PM, Maxim Dounin >> >> >>> wrote: >> >> >>> > Hello! >> >> >>> > >> >> >>> > On Fri, May 23, 2014 at 11:57:20AM +0100, Paulo Silva wrote: >> >> >>> > >> >> >>> >> there is other option than modify the auto/modules file? >> >> >>> >> >> >> >>> >> According to my goal (capture the full request response body) I >> >> >>> >> would >> >> >>> >> say that my module must run right before the postpone. >> >> >>> > >> >> >>> > Before the postpone filter you'll get subrequest bodies in your >> >> >>> > filter, which is probably not what you want (the postpone filter >> >> >>> > is to glue subrequest together, correctly ordered). >> >> >>> > >> >> >>> >> Am I supposed to modify the auto/modules like follows? >> >> >>> >> >> >> >>> >> if [ $HTTP_POSTPONE = YES ]; then >> >> >>> >> HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES >> >> >>> >> $HTTP_POSTPONE_FILTER_MODULE" >> >> >>> >> HTTP_SRCS="$HTTP_SRCS $HTTP_POSTPONE_FILTER_SRCS" >> >> >>> >> fi >> >> >>> >> >> >> >>> >> # insert my module here! >> >> >>> >> >> >> >>> >> if [ $HTTP_SSI = YES ]; then >> >> >>> >> have=NGX_HTTP_SSI . auto/have >> >> >>> >> HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES >> >> >>> >> $HTTP_SSI_FILTER_MODULE" >> >> >>> >> HTTP_DEPS="$HTTP_DEPS $HTTP_SSI_DEPS" >> >> >>> >> HTTP_SRCS="$HTTP_SRCS $HTTP_SSI_SRCS" >> >> >>> >> fi >> >> >>> >> >> >> >>> >> >> >> >>> >> I did check my modules config file and I did realize that it is >> >> >>> >> "queued" as HTTP_AUX_FILTER_MODULES. There are different queues >> >> >>> >> for >> >> >>> >> core modules and addons? >> >> >>> > >> >> >>> > The HTTP_AUX_FILTER_MODULES is a generic queue, and it's the >> >> >>> > only one currently officially supported for 3rd party modules. >> >> >>> > >> >> >>> > If you want your filter to be called right before/after postpone >> >> >>> > filter, it should be relatively safe to put it into the >> >> >>> > HTTP_POSTPONE_FILTER_MODULE variable though (and may be with some >> >> >>> > additional checks to make sure the postpone filter is enabled, or >> >> >>> > just a code to enable it unconditionally). >> >> >>> > >> >> >>> >> >> >>> And this is also valid when compiling nginx with the --add-module >> >> >>> flag? >> >> >>> How does config file look like? >> >> >>> >> >> >>> My knowledge is restricted to Emiller's Guide To Nginx Module >> >> >>> Development (http://www.evanmiller.org/nginx-modules-guide.html) >> >> >>> and a >> >> >>> few debugging hours. >> >> >> >> >> >> Uhm, looking again into auto/modules I think I was wrong, and >> >> >> modifying the HTTP_POSTPONE_FILTER_MODULE variable won't work >> >> >> (added module config scripts are executed later on), you should >> >> >> modify HTTP_FILTER_MODULES variable instead, and put your module >> >> >> into a proper position. >> >> >> >> >> >> Note that the "config" file of a module is just a shell script, >> >> >> and you are free to do more or less anything there. >> >> >> >> >> >> -- >> >> >> Maxim Dounin >> >> >> http://nginx.org/ >> >> >> >> >> >> _______________________________________________ >> >> >> nginx-devel mailing list >> >> >> nginx-devel at nginx.org >> >> >> http://mailman.nginx.org/mailman/listinfo/nginx-devel >> >> > >> >> > >> >> > >> >> > -- >> >> > Paulo A. Silva >> >> > http://tech.pauloasilva.com >> >> > http://linkedin.com/in/devpauloasilva/ >> >> >> >> >> >> >> >> -- >> >> Paulo A. Silva >> >> http://tech.pauloasilva.com >> >> http://linkedin.com/in/devpauloasilva/ >> >> >> >> _______________________________________________ >> >> nginx-devel mailing list >> >> nginx-devel at nginx.org >> >> http://mailman.nginx.org/mailman/listinfo/nginx-devel >> > >> > >> > >> > _______________________________________________ >> > nginx-devel mailing list >> > nginx-devel at nginx.org >> > http://mailman.nginx.org/mailman/listinfo/nginx-devel >> >> >> >> -- >> Paulo A. Silva >> http://tech.pauloasilva.com >> http://linkedin.com/in/devpauloasilva/ >> >> _______________________________________________ >> nginx-devel mailing list >> nginx-devel at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx-devel > > > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel -- Paulo A. Silva http://tech.pauloasilva.com http://linkedin.com/in/devpauloasilva/ From vbart at nginx.com Mon May 26 09:20:01 2014 From: vbart at nginx.com (Valentin V. Bartenev) Date: Mon, 26 May 2014 13:20:01 +0400 Subject: connection starvation problem In-Reply-To: <4e8960d3.4fd3.146362c5247.Coremail.xinghua_hi@126.com> References: <4e8960d3.4fd3.146362c5247.Coremail.xinghua_hi@126.com> Message-ID: <4465028.koN6tLdML8@vbart-workstation> On Monday 26 May 2014 09:36:52 xinghua_hi wrote: > hi: > > In man epoll , a section refer to "epoll edge trigger starvation problem", > and around to this we need to maintain a ready list instead of read socket > until EAGAIN. but i didn't find any related solution code in nginx. In > request body read function `ngx_http_do_read_client_request_body`, it seems > that nginx will read socket buffer until EAGAIN or has read Content-length > data. so if a client is fast enough and post data is large enough, is it > possible to starve other connections? SO do i miss something important? > nginx how to deal with this situation in read and write? The question is: have you actually seen this problem in real life with a typical TCP connection over network? wbr, Valentin V. Bartenev From mat999 at gmail.com Mon May 26 09:29:26 2014 From: mat999 at gmail.com (SplitIce) Date: Mon, 26 May 2014 19:29:26 +1000 Subject: Buffers Chain In-Reply-To: References: <20140523091212.GD1849@mdounin.ru> <20140523102042.GE1849@mdounin.ru> <20140523115801.GG1849@mdounin.ru> <20140523134108.GK1849@mdounin.ru> Message-ID: Theres actually an example using the memcached module (and echo module to get the request body). The main advantage to the lua module is if you wanted to do something custom (e.g processing) or transmit it over a different protocol not available as an extension. On Mon, May 26, 2014 at 7:17 PM, Paulo Silva wrote: > On Mon, May 26, 2014 at 9:28 AM, SplitIce wrote: > > Yes, connecting with SOCK_NONBLOCK shouldnt block. I don't believe this > is > > mentioned previously. If your code blocks (e.g blocking connect or > blocking > > send) then it would reduce nginx throughput substantially. Thats the > point I > > was trying to make. > > > > Yap, I didn't mention SOCK_NONBLOCK. Sorry about that. > > Although I'm working on a prototype to benchmark, my concern is about > socket() and connect() calls on every nginx filter execution. How > would you manage this? > > > Have you investigated srcache > > (https://github.com/openresty/srcache-nginx-module/)? srcache_store > could be > > used without srcache_fetch to store only. Your store handler could be a > > proxy_pass call to pass a restful server or whatever you use for storing > the > > HTML responses. > > > > I didn't know about this module. Thanks, I will have a deep look on it. > Do you think that I can pick the proxy_pass response and store it on > memcache easily? > > > Perhaps a better solution would be to use the nginx lua extension instead > > (less of a hack). > > > > Do you think that lua extension has advantages regarding an nginx filter > module. > > Sorry about so many questions but at the beginning, nginx is a all > mystery, and you'll worried to do not break its performance. > > Thanks a bunch, > > > Regards, > > Mathew > > > > > > On Mon, May 26, 2014 at 6:18 PM, Paulo Silva > wrote: > >> > >> On Mon, May 26, 2014 at 9:09 AM, SplitIce wrote: > >> > As in blocking send and connect? I don't know the specifics of Unix > >> > Sockets, > >> > but don't they block when the buffer fills (I know FIFO queues do)? > >> > > >> > >> Sorry, I don't fully understand your question. > >> I was expecting that with the SOCK_NONBLOCK it would not block. > >> > >> What would be your approach? > >> Do you know about any nginx internal mechanism to accomplish this goal > >> (get the upstream response body out of nginx)? > >> > >> > > >> > On Mon, May 26, 2014 at 9:22 AM, Paulo Silva > >> > wrote: > >> >> > >> >> Hi, > >> >> I'm not sure whether I will face problems with other filters > modifying > >> >> the response body after mine, but for know I'm comfortable as I can > >> >> rebuild the full response body just iterating buffers chains. > >> >> > >> >> As I said before I'm using nginx as reverse proxy and my main goal is > >> >> to pass the upstream (proxy_pass) response to another local process > >> >> (relative to nginx). > >> >> > >> >> I am benchmarking Unix sockets and Shared memory as IPC. > >> >> I did it already for Unix Sockets and with my prototype the nginx > >> >> "performance" dropped for half the number of requests per second. Of > >> >> course I'm doing something really bad. > >> >> > >> >> Is it OK to use socket/connect/send from inside an nginx module? > >> >> > >> >> I would be glad to hear from you. > >> >> Thanks, > >> >> > >> >> On Fri, May 23, 2014 at 2:50 PM, Paulo Silva > >> >> wrote: > >> >> > Because I don't have deep knowledge of nginx internal and I can not > >> >> > find a proper resource about it, the best I can do and with what I > am > >> >> > comfortable is with body_filter. > >> >> > > >> >> > Do you think I can notice whether all other 3rd party module > filters > >> >> > finish modifying the ngx_chain_t *in ? > >> >> > > >> >> > > >> >> > > >> >> > On Fri, May 23, 2014 at 2:41 PM, Maxim Dounin > >> >> > wrote: > >> >> >> Hello! > >> >> >> > >> >> >> On Fri, May 23, 2014 at 02:17:27PM +0100, Paulo Silva wrote: > >> >> >> > >> >> >>> Hi, > >> >> >>> > >> >> >>> On Fri, May 23, 2014 at 12:58 PM, Maxim Dounin < > mdounin at mdounin.ru> > >> >> >>> wrote: > >> >> >>> > Hello! > >> >> >>> > > >> >> >>> > On Fri, May 23, 2014 at 11:57:20AM +0100, Paulo Silva wrote: > >> >> >>> > > >> >> >>> >> there is other option than modify the auto/modules file? > >> >> >>> >> > >> >> >>> >> According to my goal (capture the full request response body) > I > >> >> >>> >> would > >> >> >>> >> say that my module must run right before the postpone. > >> >> >>> > > >> >> >>> > Before the postpone filter you'll get subrequest bodies in your > >> >> >>> > filter, which is probably not what you want (the postpone > filter > >> >> >>> > is to glue subrequest together, correctly ordered). > >> >> >>> > > >> >> >>> >> Am I supposed to modify the auto/modules like follows? > >> >> >>> >> > >> >> >>> >> if [ $HTTP_POSTPONE = YES ]; then > >> >> >>> >> HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES > >> >> >>> >> $HTTP_POSTPONE_FILTER_MODULE" > >> >> >>> >> HTTP_SRCS="$HTTP_SRCS $HTTP_POSTPONE_FILTER_SRCS" > >> >> >>> >> fi > >> >> >>> >> > >> >> >>> >> # insert my module here! > >> >> >>> >> > >> >> >>> >> if [ $HTTP_SSI = YES ]; then > >> >> >>> >> have=NGX_HTTP_SSI . auto/have > >> >> >>> >> HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES > >> >> >>> >> $HTTP_SSI_FILTER_MODULE" > >> >> >>> >> HTTP_DEPS="$HTTP_DEPS $HTTP_SSI_DEPS" > >> >> >>> >> HTTP_SRCS="$HTTP_SRCS $HTTP_SSI_SRCS" > >> >> >>> >> fi > >> >> >>> >> > >> >> >>> >> > >> >> >>> >> I did check my modules config file and I did realize that it > is > >> >> >>> >> "queued" as HTTP_AUX_FILTER_MODULES. There are different > queues > >> >> >>> >> for > >> >> >>> >> core modules and addons? > >> >> >>> > > >> >> >>> > The HTTP_AUX_FILTER_MODULES is a generic queue, and it's the > >> >> >>> > only one currently officially supported for 3rd party modules. > >> >> >>> > > >> >> >>> > If you want your filter to be called right before/after > postpone > >> >> >>> > filter, it should be relatively safe to put it into the > >> >> >>> > HTTP_POSTPONE_FILTER_MODULE variable though (and may be with > some > >> >> >>> > additional checks to make sure the postpone filter is enabled, > or > >> >> >>> > just a code to enable it unconditionally). > >> >> >>> > > >> >> >>> > >> >> >>> And this is also valid when compiling nginx with the --add-module > >> >> >>> flag? > >> >> >>> How does config file look like? > >> >> >>> > >> >> >>> My knowledge is restricted to Emiller's Guide To Nginx Module > >> >> >>> Development (http://www.evanmiller.org/nginx-modules-guide.html) > >> >> >>> and a > >> >> >>> few debugging hours. > >> >> >> > >> >> >> Uhm, looking again into auto/modules I think I was wrong, and > >> >> >> modifying the HTTP_POSTPONE_FILTER_MODULE variable won't work > >> >> >> (added module config scripts are executed later on), you should > >> >> >> modify HTTP_FILTER_MODULES variable instead, and put your module > >> >> >> into a proper position. > >> >> >> > >> >> >> Note that the "config" file of a module is just a shell script, > >> >> >> and you are free to do more or less anything there. > >> >> >> > >> >> >> -- > >> >> >> Maxim Dounin > >> >> >> http://nginx.org/ > >> >> >> > >> >> >> _______________________________________________ > >> >> >> nginx-devel mailing list > >> >> >> nginx-devel at nginx.org > >> >> >> http://mailman.nginx.org/mailman/listinfo/nginx-devel > >> >> > > >> >> > > >> >> > > >> >> > -- > >> >> > Paulo A. Silva > >> >> > http://tech.pauloasilva.com > >> >> > http://linkedin.com/in/devpauloasilva/ > >> >> > >> >> > >> >> > >> >> -- > >> >> Paulo A. Silva > >> >> http://tech.pauloasilva.com > >> >> http://linkedin.com/in/devpauloasilva/ > >> >> > >> >> _______________________________________________ > >> >> nginx-devel mailing list > >> >> nginx-devel at nginx.org > >> >> http://mailman.nginx.org/mailman/listinfo/nginx-devel > >> > > >> > > >> > > >> > _______________________________________________ > >> > nginx-devel mailing list > >> > nginx-devel at nginx.org > >> > http://mailman.nginx.org/mailman/listinfo/nginx-devel > >> > >> > >> > >> -- > >> Paulo A. Silva > >> http://tech.pauloasilva.com > >> http://linkedin.com/in/devpauloasilva/ > >> > >> _______________________________________________ > >> nginx-devel mailing list > >> nginx-devel at nginx.org > >> http://mailman.nginx.org/mailman/listinfo/nginx-devel > > > > > > > > _______________________________________________ > > nginx-devel mailing list > > nginx-devel at nginx.org > > http://mailman.nginx.org/mailman/listinfo/nginx-devel > > > > -- > Paulo A. Silva > http://tech.pauloasilva.com > http://linkedin.com/in/devpauloasilva/ > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pauloasilva at gmail.com Tue May 27 08:15:28 2014 From: pauloasilva at gmail.com (Paulo Silva) Date: Tue, 27 May 2014 09:15:28 +0100 Subject: Buffers Chain In-Reply-To: References: <20140523091212.GD1849@mdounin.ru> <20140523102042.GE1849@mdounin.ru> <20140523115801.GG1849@mdounin.ru> <20140523134108.GK1849@mdounin.ru> Message-ID: Hi, One more question about buffers chains. Is it possible to a body_filter to say to the engine to hang up client streaming? Maybe say "keep it in memory" (there are a plenty undocumented ngx_chain_t members). I'm searching for "" on response body and one possibility is to get, i.e., "" at the beginning of the next chain. But on the next body_filter call I will not have access to the previous chain, right? I was thinking about telling to the engine to hang up the client streaming whenever a chain ends with a substring of "". The purpose is to inject some HTML right before the so I have to manipulate the content of chain->buf. It is not possible to use SSI, because nginx is being used as reverse proxy and the backend application doesn't know about nginx. Do you know about some nginx internals resources other than Evan Millers post? I would like to say "sorry" about my (possibly) lame questions. Thanks a bunch, On Mon, May 26, 2014 at 10:29 AM, SplitIce wrote: > Theres actually an example using the memcached module (and echo module to > get the request body). The main advantage to the lua module is if you wanted > to do something custom (e.g processing) or transmit it over a different > protocol not available as an extension. > > > On Mon, May 26, 2014 at 7:17 PM, Paulo Silva wrote: >> >> On Mon, May 26, 2014 at 9:28 AM, SplitIce wrote: >> > Yes, connecting with SOCK_NONBLOCK shouldnt block. I don't believe this >> > is >> > mentioned previously. If your code blocks (e.g blocking connect or >> > blocking >> > send) then it would reduce nginx throughput substantially. Thats the >> > point I >> > was trying to make. >> > >> >> Yap, I didn't mention SOCK_NONBLOCK. Sorry about that. >> >> Although I'm working on a prototype to benchmark, my concern is about >> socket() and connect() calls on every nginx filter execution. How >> would you manage this? >> >> > Have you investigated srcache >> > (https://github.com/openresty/srcache-nginx-module/)? srcache_store >> > could be >> > used without srcache_fetch to store only. Your store handler could be a >> > proxy_pass call to pass a restful server or whatever you use for storing >> > the >> > HTML responses. >> > >> >> I didn't know about this module. Thanks, I will have a deep look on it. >> Do you think that I can pick the proxy_pass response and store it on >> memcache easily? >> >> > Perhaps a better solution would be to use the nginx lua extension >> > instead >> > (less of a hack). >> > >> >> Do you think that lua extension has advantages regarding an nginx filter >> module. >> >> Sorry about so many questions but at the beginning, nginx is a all >> mystery, and you'll worried to do not break its performance. >> >> Thanks a bunch, >> >> > Regards, >> > Mathew >> > >> > >> > On Mon, May 26, 2014 at 6:18 PM, Paulo Silva >> > wrote: >> >> >> >> On Mon, May 26, 2014 at 9:09 AM, SplitIce wrote: >> >> > As in blocking send and connect? I don't know the specifics of Unix >> >> > Sockets, >> >> > but don't they block when the buffer fills (I know FIFO queues do)? >> >> > >> >> >> >> Sorry, I don't fully understand your question. >> >> I was expecting that with the SOCK_NONBLOCK it would not block. >> >> >> >> What would be your approach? >> >> Do you know about any nginx internal mechanism to accomplish this goal >> >> (get the upstream response body out of nginx)? >> >> >> >> > >> >> > On Mon, May 26, 2014 at 9:22 AM, Paulo Silva >> >> > wrote: >> >> >> >> >> >> Hi, >> >> >> I'm not sure whether I will face problems with other filters >> >> >> modifying >> >> >> the response body after mine, but for know I'm comfortable as I can >> >> >> rebuild the full response body just iterating buffers chains. >> >> >> >> >> >> As I said before I'm using nginx as reverse proxy and my main goal >> >> >> is >> >> >> to pass the upstream (proxy_pass) response to another local process >> >> >> (relative to nginx). >> >> >> >> >> >> I am benchmarking Unix sockets and Shared memory as IPC. >> >> >> I did it already for Unix Sockets and with my prototype the nginx >> >> >> "performance" dropped for half the number of requests per second. Of >> >> >> course I'm doing something really bad. >> >> >> >> >> >> Is it OK to use socket/connect/send from inside an nginx module? >> >> >> >> >> >> I would be glad to hear from you. >> >> >> Thanks, >> >> >> >> >> >> On Fri, May 23, 2014 at 2:50 PM, Paulo Silva >> >> >> wrote: >> >> >> > Because I don't have deep knowledge of nginx internal and I can >> >> >> > not >> >> >> > find a proper resource about it, the best I can do and with what I >> >> >> > am >> >> >> > comfortable is with body_filter. >> >> >> > >> >> >> > Do you think I can notice whether all other 3rd party module >> >> >> > filters >> >> >> > finish modifying the ngx_chain_t *in ? >> >> >> > >> >> >> > >> >> >> > >> >> >> > On Fri, May 23, 2014 at 2:41 PM, Maxim Dounin >> >> >> > wrote: >> >> >> >> Hello! >> >> >> >> >> >> >> >> On Fri, May 23, 2014 at 02:17:27PM +0100, Paulo Silva wrote: >> >> >> >> >> >> >> >>> Hi, >> >> >> >>> >> >> >> >>> On Fri, May 23, 2014 at 12:58 PM, Maxim Dounin >> >> >> >>> >> >> >> >>> wrote: >> >> >> >>> > Hello! >> >> >> >>> > >> >> >> >>> > On Fri, May 23, 2014 at 11:57:20AM +0100, Paulo Silva wrote: >> >> >> >>> > >> >> >> >>> >> there is other option than modify the auto/modules file? >> >> >> >>> >> >> >> >> >>> >> According to my goal (capture the full request response body) >> >> >> >>> >> I >> >> >> >>> >> would >> >> >> >>> >> say that my module must run right before the postpone. >> >> >> >>> > >> >> >> >>> > Before the postpone filter you'll get subrequest bodies in >> >> >> >>> > your >> >> >> >>> > filter, which is probably not what you want (the postpone >> >> >> >>> > filter >> >> >> >>> > is to glue subrequest together, correctly ordered). >> >> >> >>> > >> >> >> >>> >> Am I supposed to modify the auto/modules like follows? >> >> >> >>> >> >> >> >> >>> >> if [ $HTTP_POSTPONE = YES ]; then >> >> >> >>> >> HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES >> >> >> >>> >> $HTTP_POSTPONE_FILTER_MODULE" >> >> >> >>> >> HTTP_SRCS="$HTTP_SRCS $HTTP_POSTPONE_FILTER_SRCS" >> >> >> >>> >> fi >> >> >> >>> >> >> >> >> >>> >> # insert my module here! >> >> >> >>> >> >> >> >> >>> >> if [ $HTTP_SSI = YES ]; then >> >> >> >>> >> have=NGX_HTTP_SSI . auto/have >> >> >> >>> >> HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES >> >> >> >>> >> $HTTP_SSI_FILTER_MODULE" >> >> >> >>> >> HTTP_DEPS="$HTTP_DEPS $HTTP_SSI_DEPS" >> >> >> >>> >> HTTP_SRCS="$HTTP_SRCS $HTTP_SSI_SRCS" >> >> >> >>> >> fi >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> I did check my modules config file and I did realize that it >> >> >> >>> >> is >> >> >> >>> >> "queued" as HTTP_AUX_FILTER_MODULES. There are different >> >> >> >>> >> queues >> >> >> >>> >> for >> >> >> >>> >> core modules and addons? >> >> >> >>> > >> >> >> >>> > The HTTP_AUX_FILTER_MODULES is a generic queue, and it's the >> >> >> >>> > only one currently officially supported for 3rd party modules. >> >> >> >>> > >> >> >> >>> > If you want your filter to be called right before/after >> >> >> >>> > postpone >> >> >> >>> > filter, it should be relatively safe to put it into the >> >> >> >>> > HTTP_POSTPONE_FILTER_MODULE variable though (and may be with >> >> >> >>> > some >> >> >> >>> > additional checks to make sure the postpone filter is enabled, >> >> >> >>> > or >> >> >> >>> > just a code to enable it unconditionally). >> >> >> >>> > >> >> >> >>> >> >> >> >>> And this is also valid when compiling nginx with the >> >> >> >>> --add-module >> >> >> >>> flag? >> >> >> >>> How does config file look like? >> >> >> >>> >> >> >> >>> My knowledge is restricted to Emiller's Guide To Nginx Module >> >> >> >>> Development (http://www.evanmiller.org/nginx-modules-guide.html) >> >> >> >>> and a >> >> >> >>> few debugging hours. >> >> >> >> >> >> >> >> Uhm, looking again into auto/modules I think I was wrong, and >> >> >> >> modifying the HTTP_POSTPONE_FILTER_MODULE variable won't work >> >> >> >> (added module config scripts are executed later on), you should >> >> >> >> modify HTTP_FILTER_MODULES variable instead, and put your module >> >> >> >> into a proper position. >> >> >> >> >> >> >> >> Note that the "config" file of a module is just a shell script, >> >> >> >> and you are free to do more or less anything there. >> >> >> >> >> >> >> >> -- >> >> >> >> Maxim Dounin >> >> >> >> http://nginx.org/ >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> >> nginx-devel mailing list >> >> >> >> nginx-devel at nginx.org >> >> >> >> http://mailman.nginx.org/mailman/listinfo/nginx-devel >> >> >> > >> >> >> > >> >> >> > >> >> >> > -- >> >> >> > Paulo A. Silva >> >> >> > http://tech.pauloasilva.com >> >> >> > http://linkedin.com/in/devpauloasilva/ >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Paulo A. Silva >> >> >> http://tech.pauloasilva.com >> >> >> http://linkedin.com/in/devpauloasilva/ >> >> >> >> >> >> _______________________________________________ >> >> >> nginx-devel mailing list >> >> >> nginx-devel at nginx.org >> >> >> http://mailman.nginx.org/mailman/listinfo/nginx-devel >> >> > >> >> > >> >> > >> >> > _______________________________________________ >> >> > nginx-devel mailing list >> >> > nginx-devel at nginx.org >> >> > http://mailman.nginx.org/mailman/listinfo/nginx-devel >> >> >> >> >> >> >> >> -- >> >> Paulo A. Silva >> >> http://tech.pauloasilva.com >> >> http://linkedin.com/in/devpauloasilva/ >> >> >> >> _______________________________________________ >> >> nginx-devel mailing list >> >> nginx-devel at nginx.org >> >> http://mailman.nginx.org/mailman/listinfo/nginx-devel >> > >> > >> > >> > _______________________________________________ >> > nginx-devel mailing list >> > nginx-devel at nginx.org >> > http://mailman.nginx.org/mailman/listinfo/nginx-devel >> >> >> >> -- >> Paulo A. Silva >> http://tech.pauloasilva.com >> http://linkedin.com/in/devpauloasilva/ >> >> _______________________________________________ >> nginx-devel mailing list >> nginx-devel at nginx.org >> http://mailman.nginx.org/mailman/listinfo/nginx-devel > > > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel -- Paulo A. Silva http://tech.pauloasilva.com http://linkedin.com/in/devpauloasilva/ From vl at nginx.com Tue May 27 10:53:56 2014 From: vl at nginx.com (Homutov Vladimir) Date: Tue, 27 May 2014 10:53:56 +0000 Subject: [nginx] Syslog: fixed possible resource leak and more verbose lo... Message-ID: details: http://hg.nginx.org/nginx/rev/aacd994167d3 branches: changeset: 5708:aacd994167d3 user: Vladimir Homutov date: Mon May 26 23:34:44 2014 +0400 description: Syslog: fixed possible resource leak and more verbose logging. Found by Coverity (CID 1215646). diffstat: src/core/ngx_syslog.c | 34 ++++++++++++++++++++++------------ 1 files changed, 22 insertions(+), 12 deletions(-) diffs (66 lines): diff -r 31dd63dcb9ea -r aacd994167d3 src/core/ngx_syslog.c --- a/src/core/ngx_syslog.c Fri May 23 16:37:05 2014 +0400 +++ b/src/core/ngx_syslog.c Mon May 26 23:34:44 2014 +0400 @@ -289,14 +289,6 @@ ngx_syslog_init_peer(ngx_syslog_peer_t * ngx_syslog_dummy_event.log = &ngx_syslog_dummy_log; - cln = ngx_pool_cleanup_add(peer->pool, 0); - if (cln == NULL) { - return NGX_ERROR; - } - - cln->data = peer; - cln->handler = ngx_syslog_cleanup; - fd = ngx_socket(peer->server.sockaddr->sa_family, SOCK_DGRAM, 0); if (fd == (ngx_socket_t) -1) { ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, @@ -307,17 +299,34 @@ ngx_syslog_init_peer(ngx_syslog_peer_t * if (ngx_nonblocking(fd) == -1) { ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, ngx_nonblocking_n " failed"); - return NGX_ERROR; + goto failed; } if (connect(fd, peer->server.sockaddr, peer->server.socklen) == -1) { ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, "connect() failed"); - return NGX_ERROR; + goto failed; } + cln = ngx_pool_cleanup_add(peer->pool, 0); + if (cln == NULL) { + goto failed; + } + + cln->data = peer; + cln->handler = ngx_syslog_cleanup; + peer->conn.fd = fd; return NGX_OK; + +failed: + + if (ngx_close_socket(fd) == -1) { + ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, + ngx_close_socket_n " failed"); + } + + return NGX_ERROR; } @@ -326,7 +335,8 @@ ngx_syslog_cleanup(void *data) { ngx_syslog_peer_t *peer = data; - if (peer->conn.fd != (ngx_socket_t) -1) { - (void) ngx_close_socket(peer->conn.fd); + if (ngx_close_socket(peer->conn.fd) == -1) { + ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, ngx_socket_errno, + ngx_close_socket_n " failed"); } } From mdounin at mdounin.ru Tue May 27 11:08:14 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 27 May 2014 15:08:14 +0400 Subject: Buffers Chain In-Reply-To: References: <20140523134108.GK1849@mdounin.ru> Message-ID: <20140527110814.GE1849@mdounin.ru> Hello! On Tue, May 27, 2014 at 09:15:28AM +0100, Paulo Silva wrote: > Hi, > One more question about buffers chains. > > Is it possible to a body_filter to say to the engine to hang up client > streaming? > Maybe say "keep it in memory" (there are a plenty undocumented > ngx_chain_t members). > > I'm searching for "" on response body and one possibility is to > get, i.e., "" at the > beginning of the next chain. But on the next body_filter call I will > not have access to the previous chain, right? > > I was thinking about telling to the engine to hang up the client > streaming whenever a chain ends with a substring of "". > The purpose is to inject some HTML right before the so I have > to manipulate the content of chain->buf. > > It is not possible to use SSI, because nginx is being used as reverse > proxy and the backend application doesn't know about nginx. Try looking into sub filter module. http://hg.nginx.org/nginx/file/tip/src/http/modules/ngx_http_sub_filter_module.c http://nginx.org/en/docs/http/ngx_http_sub_module.html -- Maxim Dounin http://nginx.org/ From vl at nginx.com Tue May 27 11:56:37 2014 From: vl at nginx.com (Homutov Vladimir) Date: Tue, 27 May 2014 11:56:37 +0000 Subject: [nginx] Syslog: fixed message sending on win32. Message-ID: details: http://hg.nginx.org/nginx/rev/44532046d9d7 branches: changeset: 5709:44532046d9d7 user: Vladimir Homutov date: Tue May 27 15:42:34 2014 +0400 description: Syslog: fixed message sending on win32. diffstat: src/core/ngx_syslog.c | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diffs (14 lines): diff -r aacd994167d3 -r 44532046d9d7 src/core/ngx_syslog.c --- a/src/core/ngx_syslog.c Mon May 26 23:34:44 2014 +0400 +++ b/src/core/ngx_syslog.c Tue May 27 15:42:34 2014 +0400 @@ -317,6 +317,10 @@ ngx_syslog_init_peer(ngx_syslog_peer_t * cln->handler = ngx_syslog_cleanup; peer->conn.fd = fd; + + /* UDP sockets are always ready to write */ + peer->conn.write->ready = 1; + return NGX_OK; failed: From mdounin at mdounin.ru Tue May 27 12:38:17 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 27 May 2014 12:38:17 +0000 Subject: [nginx] Sub filter: fixed subrequests handling. Message-ID: details: http://hg.nginx.org/nginx/rev/9e9c1310d17c branches: changeset: 5710:9e9c1310d17c user: Maxim Dounin date: Tue May 27 16:37:35 2014 +0400 description: Sub filter: fixed subrequests handling. In particular, properly output partial match at the end of a subrequest response (much like we do at the end of a response), and reset/set the last_in_chain flag as appropriate. Reported by KAWAHARA Masashi. diffstat: src/http/modules/ngx_http_sub_filter_module.c | 8 ++++++-- 1 files changed, 6 insertions(+), 2 deletions(-) diffs (39 lines): diff --git a/src/http/modules/ngx_http_sub_filter_module.c b/src/http/modules/ngx_http_sub_filter_module.c --- a/src/http/modules/ngx_http_sub_filter_module.c +++ b/src/http/modules/ngx_http_sub_filter_module.c @@ -305,6 +305,7 @@ ngx_http_sub_body_filter(ngx_http_reques b->last = ctx->copy_end; b->shadow = NULL; b->last_buf = 0; + b->last_in_chain = 0; b->recycled = 0; if (b->in_file) { @@ -374,7 +375,9 @@ ngx_http_sub_body_filter(ngx_http_reques continue; } - if (ctx->buf->last_buf && ctx->looked.len) { + if (ctx->looked.len + && (ctx->buf->last_buf || ctx->buf->last_in_chain)) + { cl = ngx_chain_get_free_buf(r->pool, &ctx->free); if (cl == NULL) { return NGX_ERROR; @@ -394,7 +397,7 @@ ngx_http_sub_body_filter(ngx_http_reques ctx->looked.len = 0; } - if (ctx->buf->last_buf || ctx->buf->flush + if (ctx->buf->last_buf || ctx->buf->flush || ctx->buf->sync || ngx_buf_in_memory(ctx->buf)) { if (b == NULL) { @@ -414,6 +417,7 @@ ngx_http_sub_body_filter(ngx_http_reques } b->last_buf = ctx->buf->last_buf; + b->last_in_chain = ctx->buf->last_in_chain; b->flush = ctx->buf->flush; b->shadow = ctx->buf; From mdounin at mdounin.ru Tue May 27 13:59:42 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 27 May 2014 13:59:42 +0000 Subject: [nginx] nginx-1.7.1-RELEASE Message-ID: details: http://hg.nginx.org/nginx/rev/0351a6d89c3d branches: changeset: 5711:0351a6d89c3d user: Maxim Dounin date: Tue May 27 17:58:08 2014 +0400 description: nginx-1.7.1-RELEASE diffstat: docs/xml/nginx/changes.xml | 118 +++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 118 insertions(+), 0 deletions(-) diffs (128 lines): diff --git a/docs/xml/nginx/changes.xml b/docs/xml/nginx/changes.xml --- a/docs/xml/nginx/changes.xml +++ b/docs/xml/nginx/changes.xml @@ -5,6 +5,124 @@ + + + + +?????????? "$upstream_cookie_...". + + +the "$upstream_cookie_..." variables. + + + + + +?????????? $ssl_client_fingerprint. + + +the $ssl_client_fingerprint variable. + + + + + +????????? error_log ? access_log ?????? ???????????? ???????????? ? syslog. + + +the "error_log" and "access_log" directives now support logging to syslog. + + + + + +???????? ??????-?????? ?????? ????????? ???? ??????? ??? ??????????. + + +the mail proxy now logs client port on connect. + + + + + +?????? ?????? ??? ????????????? ????????? "ssl_stapling".
+??????? Filipe da Silva. +
+ +memory leak if the "ssl_stapling" directive was used.
+Thanks to Filipe da Silva. +
+
+ + + +????????? alias ?????? location'?, ????????? ?????????? ??????????, +???????? ???????????, ???? ?????????????? ????????? if ??? limit_except. + + +the "alias" directive used inside a location given by a regular expression +worked incorrectly if the "if" or "limit_except" directives were used. + + + + + +????????? charset ?? ??????? ????????? ??? ?????? ??????? ????????. + + +the "charset" directive did not set a charset to encoded backend responses. + + + + + +????????? proxy_pass ??? URI ????? ???????????? ???????????? ?????? +????? ????????? ?????????? $args.
+??????? Yichun Zhang. +
+ +a "proxy_pass" directive without URI part might use original request +after the $args variable was set.
+Thanks to Yichun Zhang. +
+
+ + + +? ?????? ????????? none ????????? smtp_auth; +?????? ????????? ? 1.5.6.
+??????? ?????????? ???????????. +
+ +in the "none" parameter in the "smtp_auth" directive; +the bug had appeared in 1.5.6.
+Thanks to Svyatoslav Nikolsky. +
+
+ + + +??? ?????????? ????????????? sub_filter ? SSI +?????? ????? ???????????? ???????. + + +if sub_filter and SSI were used together, +then responses might be transferred incorrectly. + + + + + +nginx ?? ????????? ? ?????????? --with-file-aio ?? Linux/aarch64. + + +nginx could not be built with the --with-file-aio option on Linux/aarch64. + + + +
+ + From mdounin at mdounin.ru Tue May 27 13:59:43 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Tue, 27 May 2014 13:59:43 +0000 Subject: [nginx] release-1.7.1 tag Message-ID: details: http://hg.nginx.org/nginx/rev/adb742cf27f7 branches: changeset: 5712:adb742cf27f7 user: Maxim Dounin date: Tue May 27 17:58:09 2014 +0400 description: release-1.7.1 tag diffstat: .hgtags | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (8 lines): diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -370,3 +370,4 @@ f995a10d4c7e9a817157a6ce7b753297ad32897e 97b47d95e4449cbde976657cf8cbbc118351ffe0 release-1.5.12 fd722b890eabc600394349730a093f50dac31639 release-1.5.13 d161d68df8be32e5cbf72b07db1a707714827803 release-1.7.0 +0351a6d89c3dbcc7a76295024ba6b70e27b9a497 release-1.7.1 From mat999 at gmail.com Tue May 27 14:07:56 2014 From: mat999 at gmail.com (SplitIce) Date: Wed, 28 May 2014 00:07:56 +1000 Subject: [nginx] nginx-1.7.1-RELEASE In-Reply-To: References: Message-ID: >> the "error_log" and "access_log" directives now support logging to syslog. Is this referring to the Open Source edition? It looks to be so. I hope so :) On Tue, May 27, 2014 at 11:59 PM, Maxim Dounin wrote: > details: http://hg.nginx.org/nginx/rev/0351a6d89c3d > branches: > changeset: 5711:0351a6d89c3d > user: Maxim Dounin > date: Tue May 27 17:58:08 2014 +0400 > description: > nginx-1.7.1-RELEASE > > diffstat: > > docs/xml/nginx/changes.xml | 118 > +++++++++++++++++++++++++++++++++++++++++++++ > 1 files changed, 118 insertions(+), 0 deletions(-) > > diffs (128 lines): > > diff --git a/docs/xml/nginx/changes.xml b/docs/xml/nginx/changes.xml > --- a/docs/xml/nginx/changes.xml > +++ b/docs/xml/nginx/changes.xml > @@ -5,6 +5,124 @@ > > > > + > + > + > + > +?????????? "$upstream_cookie_...". > + > + > +the "$upstream_cookie_..." variables. > + > + > + > + > + > +?????????? $ssl_client_fingerprint. > + > + > +the $ssl_client_fingerprint variable. > + > + > + > + > + > +????????? error_log ? access_log ?????? ???????????? ???????????? ? > syslog. > + > + > +the "error_log" and "access_log" directives now support logging to syslog. > + > + > + > + > + > +???????? ??????-?????? ?????? ????????? ???? ??????? ??? ??????????. > + > + > +the mail proxy now logs client port on connect. > + > + > + > + > + > +?????? ?????? ??? ????????????? ????????? "ssl_stapling".
> +??????? Filipe da Silva. > +
> + > +memory leak if the "ssl_stapling" directive was used.
> +Thanks to Filipe da Silva. > +
> +
> + > + > + > +????????? alias ?????? location'?, ????????? ?????????? ??????????, > +???????? ???????????, ???? ?????????????? ????????? if ??? limit_except. > + > + > +the "alias" directive used inside a location given by a regular expression > +worked incorrectly if the "if" or "limit_except" directives were used. > + > + > + > + > + > +????????? charset ?? ??????? ????????? ??? ?????? ??????? ????????. > + > + > +the "charset" directive did not set a charset to encoded backend > responses. > + > + > + > + > + > +????????? proxy_pass ??? URI ????? ???????????? ???????????? ?????? > +????? ????????? ?????????? $args.
> +??????? Yichun Zhang. > +
> + > +a "proxy_pass" directive without URI part might use original request > +after the $args variable was set.
> +Thanks to Yichun Zhang. > +
> +
> + > + > + > +? ?????? ????????? none ????????? smtp_auth; > +?????? ????????? ? 1.5.6.
> +??????? ?????????? ???????????. > +
> + > +in the "none" parameter in the "smtp_auth" directive; > +the bug had appeared in 1.5.6.
> +Thanks to Svyatoslav Nikolsky. > +
> +
> + > + > + > +??? ?????????? ????????????? sub_filter ? SSI > +?????? ????? ???????????? ???????. > + > + > +if sub_filter and SSI were used together, > +then responses might be transferred incorrectly. > + > + > + > + > + > +nginx ?? ????????? ? ?????????? --with-file-aio ?? Linux/aarch64. > + > + > +nginx could not be built with the --with-file-aio option on Linux/aarch64. > + > + > + > +
> + > + > > > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From maxim at nginx.com Tue May 27 14:13:36 2014 From: maxim at nginx.com (Maxim Konovalov) Date: Tue, 27 May 2014 18:13:36 +0400 Subject: [nginx] nginx-1.7.1-RELEASE In-Reply-To: References: Message-ID: <53849D90.6080707@nginx.com> On 5/27/14 6:07 PM, SplitIce wrote: >>> the "error_log" and "access_log" directives now support logging > to syslog. > > Is this referring to the Open Source edition? It looks to be so. I > hope so :) > Yes. See the announce for 1.7.1: http://mailman.nginx.org/pipermail/nginx/2014-May/043755.html -- Maxim Konovalov http://nginx.com From sobakasu at gmail.com Wed May 28 06:56:04 2014 From: sobakasu at gmail.com (Andrew Williams) Date: Wed, 28 May 2014 16:26:04 +0930 Subject: [PATCH] append "-gzip" to etags processed by gzip_filter_module + not modified support Message-ID: this patch modifies the etag header returned by responses processed by the gzip filter, appending "-gzip" to the etag (instead of deleting it). Also, If-None-Match and If-Match requests work against the modified etag. (the gunzip module appends "-gunzip" to the etag). cheers Andrew # HG changeset patch # User Andrew Williams # Date 1401259215 -34200 # Wed May 28 16:10:15 2014 +0930 # Node ID 102149a95d84250e709051b1f2c8b49a1872bd3e # Parent adb742cf27f771771b293cc280f594038e8d9aab append "-gzip" to etags processed by gzip_filter_module + not modified support diff -r adb742cf27f7 -r 102149a95d84 src/http/modules/ngx_http_gunzip_filter_module.c --- a/src/http/modules/ngx_http_gunzip_filter_module.c Tue May 27 17:58:09 2014 +0400 +++ b/src/http/modules/ngx_http_gunzip_filter_module.c Wed May 28 16:10:15 2014 +0930 @@ -115,6 +115,7 @@ static ngx_http_output_header_filter_pt ngx_http_next_header_filter; static ngx_http_output_body_filter_pt ngx_http_next_body_filter; +static ngx_str_t ngx_http_gunzip_etag_suffix = ngx_string("-gunzip"); static ngx_int_t @@ -149,6 +150,13 @@ return ngx_http_next_header_filter(r); } + ngx_int_t etag_filter_result; + ngx_http_update_etag(r, &ngx_http_gunzip_etag_suffix); + if (ngx_http_test_etag_if_match(r, ngx_http_next_header_filter, + &etag_filter_result)) { + return etag_filter_result; + } + ctx = ngx_pcalloc(r->pool, sizeof(ngx_http_gunzip_ctx_t)); if (ctx == NULL) { return NGX_ERROR; @@ -165,7 +173,6 @@ ngx_http_clear_content_length(r); ngx_http_clear_accept_ranges(r); - ngx_http_clear_etag(r); return ngx_http_next_header_filter(r); } diff -r adb742cf27f7 -r 102149a95d84 src/http/modules/ngx_http_gzip_filter_module.c --- a/src/http/modules/ngx_http_gzip_filter_module.c Tue May 27 17:58:09 2014 +0400 +++ b/src/http/modules/ngx_http_gzip_filter_module.c Wed May 28 16:10:15 2014 +0930 @@ -229,6 +229,7 @@ static ngx_str_t ngx_http_gzip_ratio = ngx_string("gzip_ratio"); +static ngx_str_t ngx_http_gzip_etag_suffix = ngx_string("-gzip"); static ngx_http_output_header_filter_pt ngx_http_next_header_filter; static ngx_http_output_body_filter_pt ngx_http_next_body_filter; @@ -280,6 +281,13 @@ return ngx_http_next_header_filter(r); } + ngx_int_t etag_filter_result; + ngx_http_update_etag(r, &ngx_http_gzip_etag_suffix); + if (ngx_http_test_etag_if_match(r, ngx_http_next_header_filter, + &etag_filter_result)) { + return etag_filter_result; + } + ctx = ngx_pcalloc(r->pool, sizeof(ngx_http_gzip_ctx_t)); if (ctx == NULL) { return NGX_ERROR; @@ -306,7 +314,6 @@ ngx_http_clear_content_length(r); ngx_http_clear_accept_ranges(r); - ngx_http_clear_etag(r); return ngx_http_next_header_filter(r); } diff -r adb742cf27f7 -r 102149a95d84 src/http/modules/ngx_http_not_modified_filter_module.c --- a/src/http/modules/ngx_http_not_modified_filter_module.c Tue May 27 17:58:09 2014 +0400 +++ b/src/http/modules/ngx_http_not_modified_filter_module.c Wed May 28 16:10:15 2014 +0930 @@ -12,8 +12,6 @@ static ngx_uint_t ngx_http_test_if_unmodified(ngx_http_request_t *r); static ngx_uint_t ngx_http_test_if_modified(ngx_http_request_t *r); -static ngx_uint_t ngx_http_test_if_match(ngx_http_request_t *r, - ngx_table_elt_t *header); static ngx_int_t ngx_http_not_modified_filter_init(ngx_conf_t *cf); @@ -160,76 +158,6 @@ } -static ngx_uint_t -ngx_http_test_if_match(ngx_http_request_t *r, ngx_table_elt_t *header) -{ - u_char *start, *end, ch; - ngx_str_t *etag, *list; - - list = &header->value; - - if (list->len == 1 && list->data[0] == '*') { - return 1; - } - - if (r->headers_out.etag == NULL) { - return 0; - } - - etag = &r->headers_out.etag->value; - - ngx_log_debug2(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, - "http im:\"%V\" etag:%V", list, etag); - - start = list->data; - end = list->data + list->len; - - while (start < end) { - - if (etag->len > (size_t) (end - start)) { - return 0; - } - - if (ngx_strncmp(start, etag->data, etag->len) != 0) { - goto skip; - } - - start += etag->len; - - while (start < end) { - ch = *start; - - if (ch == ' ' || ch == '\t') { - start++; - continue; - } - - break; - } - - if (start == end || *start == ',') { - return 1; - } - - skip: - - while (start < end && *start != ',') { start++; } - while (start < end) { - ch = *start; - - if (ch == ' ' || ch == '\t' || ch == ',') { - start++; - continue; - } - - break; - } - } - - return 0; -} - - static ngx_int_t ngx_http_not_modified_filter_init(ngx_conf_t *cf) { diff -r adb742cf27f7 -r 102149a95d84 src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c Tue May 27 17:58:09 2014 +0400 +++ b/src/http/ngx_http_core_module.c Wed May 28 16:10:15 2014 +0930 @@ -1851,6 +1851,134 @@ } +// update etag, append suffix +ngx_uint_t +ngx_http_update_etag(ngx_http_request_t *r, ngx_str_t *suffix) +{ + ngx_table_elt_t *etag; + void *old_etag_value; + + etag = r->headers_out.etag; + + if (etag == NULL || suffix == NULL) + return 0; + + old_etag_value = etag->value.data; + etag->value.data = ngx_pnalloc(r->pool, etag->value.len + + suffix->len + 1); + ngx_cpystrn(etag->value.data, old_etag_value, etag->value.len); + etag->value.len = ngx_sprintf(etag->value.data + etag->value.len - 1, + "%s\"", suffix->data) - etag->value.data; + + ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "http updated etag: %s", etag->value.data); + + return 1; +} + + +ngx_int_t +ngx_http_test_etag_if_match(ngx_http_request_t *r, + ngx_http_output_header_filter_pt next_header_filter, + ngx_int_t *etag_filter_result) +{ + if (r->headers_in.if_match + && !ngx_http_test_if_match(r, r->headers_in.if_match)) { + *etag_filter_result = ngx_http_filter_finalize_request + (r, NULL, NGX_HTTP_PRECONDITION_FAILED); + return 1; + } + else if (r->headers_in.if_none_match + && ngx_http_test_if_match(r, r->headers_in.if_none_match)) { + /* not modified */ + r->headers_out.status = NGX_HTTP_NOT_MODIFIED; + r->headers_out.status_line.len = 0; + r->headers_out.content_type.len = 0; + ngx_http_clear_content_length(r); + ngx_http_clear_accept_ranges(r); + + if (r->headers_out.content_encoding) { + r->headers_out.content_encoding->hash = 0; + r->headers_out.content_encoding = NULL; + } + + *etag_filter_result = next_header_filter(r); + return 1; + } + return 0; +} + + +ngx_uint_t +ngx_http_test_if_match(ngx_http_request_t *r, ngx_table_elt_t *header) +{ + u_char *start, *end, ch; + ngx_str_t *etag, *list; + + list = &header->value; + + if (list->len == 1 && list->data[0] == '*') { + return 1; + } + + if (r->headers_out.etag == NULL) { + return 0; + } + + etag = &r->headers_out.etag->value; + + ngx_log_debug2(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, + "http im:\"%V\" etag:%V", list, etag); + + start = list->data; + end = list->data + list->len; + + while (start < end) { + + if (etag->len > (size_t) (end - start)) { + return 0; + } + + if (ngx_strncmp(start, etag->data, etag->len) != 0) { + goto skip; + } + + start += etag->len; + + while (start < end) { + ch = *start; + + if (ch == ' ' || ch == '\t') { + start++; + continue; + } + + break; + } + + if (start == end || *start == ',') { + return 1; + } + + skip: + + while (start < end && *start != ',') { start++; } + while (start < end) { + ch = *start; + + if (ch == ' ' || ch == '\t' || ch == ',') { + start++; + continue; + } + + break; + } + } + + return 0; +} + + ngx_int_t ngx_http_send_response(ngx_http_request_t *r, ngx_uint_t status, ngx_str_t *ct, ngx_http_complex_value_t *cv) diff -r adb742cf27f7 -r 102149a95d84 src/http/ngx_http_core_module.h --- a/src/http/ngx_http_core_module.h Tue May 27 17:58:09 2014 +0400 +++ b/src/http/ngx_http_core_module.h Wed May 28 16:10:15 2014 +0930 @@ -500,7 +500,6 @@ void *ngx_http_test_content_type(ngx_http_request_t *r, ngx_hash_t *types_hash); ngx_int_t ngx_http_set_content_type(ngx_http_request_t *r); void ngx_http_set_exten(ngx_http_request_t *r); -ngx_int_t ngx_http_set_etag(ngx_http_request_t *r); ngx_int_t ngx_http_send_response(ngx_http_request_t *r, ngx_uint_t status, ngx_str_t *ct, ngx_http_complex_value_t *cv); u_char *ngx_http_map_uri_to_path(ngx_http_request_t *r, ngx_str_t *name, @@ -527,6 +526,15 @@ (ngx_http_request_t *r, ngx_chain_t *chain); +ngx_int_t ngx_http_set_etag(ngx_http_request_t *r); +ngx_uint_t ngx_http_update_etag(ngx_http_request_t *r, ngx_str_t *suffix); +ngx_int_t ngx_http_test_etag_if_match(ngx_http_request_t *r, + ngx_http_output_header_filter_pt next_header_filter, + ngx_int_t *etag_filter_result); +ngx_uint_t ngx_http_test_if_match(ngx_http_request_t *r, + ngx_table_elt_t *header); + + ngx_int_t ngx_http_output_filter(ngx_http_request_t *r, ngx_chain_t *chain); ngx_int_t ngx_http_write_filter(ngx_http_request_t *r, ngx_chain_t *chain); From mdounin at mdounin.ru Wed May 28 11:22:51 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 28 May 2014 15:22:51 +0400 Subject: [PATCH] append "-gzip" to etags processed by gzip_filter_module + not modified support In-Reply-To: References: Message-ID: <20140528112251.GX1849@mdounin.ru> Hello! On Wed, May 28, 2014 at 04:26:04PM +0930, Andrew Williams wrote: > this patch modifies the etag header returned by responses processed by > the gzip filter, appending "-gzip" to the etag (instead of deleting > it). Also, If-None-Match and If-Match requests work against the > modified etag. > (the gunzip module appends "-gunzip" to the etag). This approach is wrong, as resources with the same etag are not guaranteed to be equal. See, e.g., here for previous discussions: http://mailman.nginx.org/pipermail/nginx-devel/2012-November/003004.html See here about plans: http://mailman.nginx.org/pipermail/nginx-devel/2014-April/005231.html -- Maxim Dounin http://nginx.org/ From mdounin at mdounin.ru Wed May 28 13:44:38 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 28 May 2014 13:44:38 +0000 Subject: [nginx] Version bump. Message-ID: details: http://hg.nginx.org/nginx/rev/9b0c92f22bbf branches: changeset: 5713:9b0c92f22bbf user: Maxim Dounin date: Wed May 28 17:41:33 2014 +0400 description: Version bump. diffstat: src/core/nginx.h | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diffs (14 lines): diff --git a/src/core/nginx.h b/src/core/nginx.h --- a/src/core/nginx.h +++ b/src/core/nginx.h @@ -9,8 +9,8 @@ #define _NGINX_H_INCLUDED_ -#define nginx_version 1007001 -#define NGINX_VERSION "1.7.1" +#define nginx_version 1007002 +#define NGINX_VERSION "1.7.2" #define NGINX_VER "nginx/" NGINX_VERSION #ifdef NGX_BUILD From mdounin at mdounin.ru Wed May 28 13:44:40 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 28 May 2014 13:44:40 +0000 Subject: [nginx] Configure: added -Wno-deprecated-declarations on OS X. Message-ID: details: http://hg.nginx.org/nginx/rev/80bd391c90d1 branches: changeset: 5714:80bd391c90d1 user: Maxim Dounin date: Wed May 28 17:41:44 2014 +0400 description: Configure: added -Wno-deprecated-declarations on OS X. Previous workaround to avoid warnings on OS X due to deprecated system OpenSSL library (introduced in a3870ea96ccd) no longer works, as the MAC_OS_X_VERSION_MIN_REQUIRED macro is ignored on OS X 10.9 if a compiler used supports __attribute__(availability). diffstat: auto/cc/clang | 5 +++++ auto/cc/gcc | 5 +++++ src/os/unix/ngx_darwin_config.h | 3 --- 3 files changed, 10 insertions(+), 3 deletions(-) diffs (43 lines): diff --git a/auto/cc/clang b/auto/cc/clang --- a/auto/cc/clang +++ b/auto/cc/clang @@ -88,6 +88,11 @@ CFLAGS="$CFLAGS -Wconditional-uninitiali # we have a lot of unused function arguments CFLAGS="$CFLAGS -Wno-unused-parameter" +# deprecated system OpenSSL library on OS X +if [ "$NGX_SYSTEM" = "Darwin" ]; then + CFLAGS="$CFLAGS -Wno-deprecated-declarations" +fi + # stop on warning CFLAGS="$CFLAGS -Werror" diff --git a/auto/cc/gcc b/auto/cc/gcc --- a/auto/cc/gcc +++ b/auto/cc/gcc @@ -158,6 +158,11 @@ case "$NGX_GCC_VER" in CFLAGS="$CFLAGS -Wno-unused-parameter" # 4.2.1 shows the warning in wrong places #CFLAGS="$CFLAGS -Wunreachable-code" + + # deprecated system OpenSSL library on OS X + if [ "$NGX_SYSTEM" = "Darwin" ]; then + CFLAGS="$CFLAGS -Wno-deprecated-declarations" + fi ;; *) diff --git a/src/os/unix/ngx_darwin_config.h b/src/os/unix/ngx_darwin_config.h --- a/src/os/unix/ngx_darwin_config.h +++ b/src/os/unix/ngx_darwin_config.h @@ -9,9 +9,6 @@ #define _NGX_DARWIN_CONFIG_H_INCLUDED_ -#define MAC_OS_X_VERSION_MIN_REQUIRED MAC_OS_X_VERSION_10_0 - - #include #include #include From pluknet at nginx.com Wed May 28 16:19:32 2014 From: pluknet at nginx.com (Sergey Kandaurov) Date: Wed, 28 May 2014 16:19:32 +0000 Subject: [nginx] Fixed config parsing of the last try_files parameter. Message-ID: details: http://hg.nginx.org/nginx/rev/790ba7484bb6 branches: changeset: 5715:790ba7484bb6 user: Sergey Kandaurov date: Wed May 28 20:18:05 2014 +0400 description: Fixed config parsing of the last try_files parameter. Do not taste the last parameter against directory, as otherwise it would result in the trailing slash being cut from the parameter value. Notably, this prevents an internal redirect to an empty URI if the parameter is set to the literal slash: location / { try_files $uri /; } diffstat: src/http/ngx_http_core_module.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diffs (13 lines): diff -r 80bd391c90d1 -r 790ba7484bb6 src/http/ngx_http_core_module.c --- a/src/http/ngx_http_core_module.c Wed May 28 17:41:44 2014 +0400 +++ b/src/http/ngx_http_core_module.c Wed May 28 20:18:05 2014 +0400 @@ -4789,7 +4789,8 @@ ngx_http_core_try_files(ngx_conf_t *cf, tf[i].name = value[i + 1]; if (tf[i].name.len > 0 - && tf[i].name.data[tf[i].name.len - 1] == '/') + && tf[i].name.data[tf[i].name.len - 1] == '/' + && i + 2 < cf->args->nelts) { tf[i].test_dir = 1; tf[i].name.len--; From mdounin at mdounin.ru Wed May 28 18:38:36 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 28 May 2014 22:38:36 +0400 Subject: [PATCH] Core: merge adjacent free slab pages to ameliorate fragmentation from multi-page blocks (Was Re: Help with shared memory usage) In-Reply-To: References: Message-ID: <20140528183836.GG1849@mdounin.ru> Hello! On Sun, May 11, 2014 at 10:13:52PM -0700, Yichun Zhang (agentzh) wrote: > Hello! > > On Mon, Jul 29, 2013 at 10:11 AM, Maxim Dounin wrote: > > Additionally, doing a full merge of all free blocks on a free > > operation looks too much. It might be something we want to do on > > allocation failure, but not on a normal path in > > ngx_slab_free_pages(). And/or something lightweight may be done > > in ngx_slab_free_pages(), e.g., checking if pages following pages > > we are freeing are free too, and merging them in this case. > > > > I'd propose an alternative patch taking the second approach, that is, > merging adjacent free pages (for both the previous and next blocks) in > ngx_slab_free_pages(). This approach has the following advantages: > > 1. It can effectively distribute the merging computations across all > the page free operations, which can prevent potential frequent and > long stalls when actually running out of large enough free blocks > along the "free" list that is already very long for large zones (which > usually consists of tons of one-page blocks upon allocation > failures). > > 2. it can also make multi-page allocations generally faster because > we're merging pages immediately when we can and thus it's more likely > to find large enough free blocks along the (relatively short) free > list for ngx_slab_alloc_pages(). > > The only downside is that I have to introduce an extra field > "prev_slab" (8-byte for x86_64) in ngx_slab_page_t in my patch, which > makes the slab page metadata a bit larger. Below is a patch which does mostly the same without introducing any additional per-page fields. Please take a look if it works for you. # HG changeset patch # User Maxim Dounin # Date 1401302011 -14400 # Wed May 28 22:33:31 2014 +0400 # Node ID 7fb45c6042324e6cd92b0fb230c67a9c8c75681c # Parent 80bd391c90d11de707a05fcd0c9aa2a09c62877f Core: slab allocator defragmentation. Large allocations from a slab pool result in free page blocks being fragmented, eventually leading to a situation when no further allocation larger than a page size are possible from the pool. While this isn't a problem for nginx itself, it is known to be bad for various 3rd party modules. Fix is to merge adjacent blocks of free pages in the ngx_slab_free_pages() function. Prodded by Wandenberg Peixoto and Yichun Zhang. diff --git a/src/core/ngx_slab.c b/src/core/ngx_slab.c --- a/src/core/ngx_slab.c +++ b/src/core/ngx_slab.c @@ -129,6 +129,8 @@ ngx_slab_init(ngx_slab_pool_t *pool) pool->pages->slab = pages; } + pool->last = pool->pages + pages; + pool->log_nomem = 1; pool->log_ctx = &pool->zero; pool->zero = '\0'; @@ -626,6 +628,8 @@ ngx_slab_alloc_pages(ngx_slab_pool_t *po if (page->slab >= pages) { if (page->slab > pages) { + page[page->slab - 1].prev = (uintptr_t) &page[pages]; + page[pages].slab = page->slab - pages; page[pages].next = page->next; page[pages].prev = page->prev; @@ -672,7 +676,8 @@ static void ngx_slab_free_pages(ngx_slab_pool_t *pool, ngx_slab_page_t *page, ngx_uint_t pages) { - ngx_slab_page_t *prev; + ngx_uint_t type; + ngx_slab_page_t *prev, *join; page->slab = pages--; @@ -686,6 +691,53 @@ ngx_slab_free_pages(ngx_slab_pool_t *poo page->next->prev = page->prev; } + join = page + page->slab; + + if (join < pool->last) { + type = join->prev & NGX_SLAB_PAGE_MASK; + + if (type == NGX_SLAB_PAGE && join->next != NULL) { + pages += join->slab; + page->slab += join->slab; + + prev = (ngx_slab_page_t *) (join->prev & ~NGX_SLAB_PAGE_MASK); + prev->next = join->next; + join->next->prev = join->prev; + + join->slab = NGX_SLAB_PAGE_FREE; + join->next = NULL; + join->prev = NGX_SLAB_PAGE; + } + } + + if (page > pool->pages) { + join = page - 1; + type = join->prev & NGX_SLAB_PAGE_MASK; + + if (type == NGX_SLAB_PAGE && join->slab == NGX_SLAB_PAGE_FREE) { + join = (ngx_slab_page_t *) (join->prev & ~NGX_SLAB_PAGE_MASK); + } + + if (type == NGX_SLAB_PAGE && join->next != NULL) { + pages += join->slab; + join->slab += page->slab; + + prev = (ngx_slab_page_t *) (join->prev & ~NGX_SLAB_PAGE_MASK); + prev->next = join->next; + join->next->prev = join->prev; + + page->slab = NGX_SLAB_PAGE_FREE; + page->next = NULL; + page->prev = NGX_SLAB_PAGE; + + page = join; + } + } + + if (pages) { + page[pages].prev = (uintptr_t) page; + } + page->prev = (uintptr_t) &pool->free; page->next = pool->free.next; diff --git a/src/core/ngx_slab.h b/src/core/ngx_slab.h --- a/src/core/ngx_slab.h +++ b/src/core/ngx_slab.h @@ -29,6 +29,7 @@ typedef struct { size_t min_shift; ngx_slab_page_t *pages; + ngx_slab_page_t *last; ngx_slab_page_t free; u_char *start; -- Maxim Dounin http://nginx.org/ From mdounin at mdounin.ru Wed May 28 18:41:16 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Wed, 28 May 2014 22:41:16 +0400 Subject: Help with shared memory usage In-Reply-To: <20140122165150.GP1835@mdounin.ru> References: <20130729171109.GA2130@mdounin.ru> <20130730100931.GD2130@mdounin.ru> <20131006093708.GY62063@mdounin.ru> <20131220164923.GK95113@mdounin.ru> <20140122165150.GP1835@mdounin.ru> Message-ID: <20140528184116.GH1849@mdounin.ru> Hello! On Wed, Jan 22, 2014 at 08:51:50PM +0400, Maxim Dounin wrote: > Hello! > > On Wed, Jan 22, 2014 at 01:39:54AM -0200, Wandenberg Peixoto wrote: > > > Hello Maxim, > > > > did you have opportunity to take a look on this last patch? > > It looks more or less correct, though I don't happy with the > checks done, and there are various style issues. I'm planning to > look into it and build a better version as time permits. Sorry for long delay, see here for an improved patch: http://mailman.nginx.org/pipermail/nginx-devel/2014-May/005406.html -- Maxim Dounin http://nginx.org/ From donatas.abraitis at gmail.com Thu May 29 12:05:15 2014 From: donatas.abraitis at gmail.com (Donatas Abraitis) Date: Thu, 29 May 2014 15:05:15 +0300 Subject: handler before proxy-pass Message-ID: Hello, I want to add a custom 'handler', which would update ngx_os_argv[0] on request. It should be like this: root 18601 0.0 0.0 62272 3812 ? Ss 07:00 0:00 nginx: master process /opt/nginx/bin/nginx -c /opt/nginx/etc/nginx.conf web 18602 0.0 0.0 70372 7904 ? S 07:00 0:00 \_ 183.54.68.10 [test.domain.com] GET /ok.php HTTP/1.1??Host It works if using only nginx for static content, but if having proxy_pass in location directive it doesn't work. Code snippet is like this: { ... sprintf(title, "%s [%s] %s", r->connection->addr_text.data, r->headers_in.server.data, r->request_line.data); memcpy(ngx_os_argv[0], title, MAXBUF); return NGX_DECLINED; } Question is, how to solve this problem to update ngx_os_argv[0] if using together with proxy_pass? Seems it bypass my 'handler' if using proxy_pass. Maybe there is some kind of sequence of loading modules? -- Donatas -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdounin at mdounin.ru Thu May 29 12:29:22 2014 From: mdounin at mdounin.ru (Maxim Dounin) Date: Thu, 29 May 2014 16:29:22 +0400 Subject: handler before proxy-pass In-Reply-To: References: Message-ID: <20140529122922.GK1849@mdounin.ru> Hello! On Thu, May 29, 2014 at 03:05:15PM +0300, Donatas Abraitis wrote: > Hello, > > I want to add a custom 'handler', which would update ngx_os_argv[0] on > request. It should be like this: > > root 18601 0.0 0.0 62272 3812 ? Ss 07:00 0:00 nginx: > master process /opt/nginx/bin/nginx -c /opt/nginx/etc/nginx.conf > web 18602 0.0 0.0 70372 7904 ? S 07:00 0:00 \_ > 183.54.68.10 [test.domain.com] GET /ok.php HTTP/1.1??Host > > It works if using only nginx for static content, but if having proxy_pass > in location directive it doesn't work. This isn't going to work in any case, as there are multiple requests in a single worker process. > Code snippet is like this: > > { > ... > sprintf(title, "%s [%s] %s", r->connection->addr_text.data, > r->headers_in.server.data, r->request_line.data); > memcpy(ngx_os_argv[0], title, MAXBUF); > return NGX_DECLINED; > } > > Question is, how to solve this problem to update ngx_os_argv[0] if using > together with proxy_pass? Seems it bypass my 'handler' if using proxy_pass. > Maybe there is some kind of sequence of loading modules? >From what you write it looks like you've installed your code as a content handler. You should instead use something like postread phase handler, see e.g. realip module for an example. But, well, it's not going to work anyway, see above. -- Maxim Dounin http://nginx.org/ From sobakasu at gmail.com Fri May 30 02:25:40 2014 From: sobakasu at gmail.com (Andrew Williams) Date: Fri, 30 May 2014 11:55:40 +0930 Subject: [PATCH] append "-gzip" to etags processed by gzip_filter_module + not modified support In-Reply-To: <20140528112251.GX1849@mdounin.ru> References: <20140528112251.GX1849@mdounin.ru> Message-ID: Hi, > On Wed, May 28, 2014 at 04:26:04PM +0930, Andrew Williams wrote: > >> this patch modifies the etag header returned by responses processed by >> the gzip filter, appending "-gzip" to the etag (instead of deleting >> it). Also, If-None-Match and If-Match requests work against the >> modified etag. >> (the gunzip module appends "-gunzip" to the etag). > > This approach is wrong, as resources with the same etag are not > guaranteed to be equal. thanks for the feedback - only stripping strong etags looks like a good solution. looking forward to seeing the weak etag code making it in. cheers Andrew From donatas.abraitis at gmail.com Fri May 30 12:55:15 2014 From: donatas.abraitis at gmail.com (Donatas Abraitis) Date: Fri, 30 May 2014 15:55:15 +0300 Subject: nginx-devel Digest, Vol 55, Issue 34 In-Reply-To: References: Message-ID: Hey, I solved my 'problem' with using post_action after proxy_pass. Thank you. On Fri, May 30, 2014 at 3:00 PM, wrote: > Send nginx-devel mailing list submissions to > nginx-devel at nginx.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mailman.nginx.org/mailman/listinfo/nginx-devel > or, via email, send a message with subject or body 'help' to > nginx-devel-request at nginx.org > > You can reach the person managing the list at > nginx-devel-owner at nginx.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of nginx-devel digest..." > > > Today's Topics: > > 1. handler before proxy-pass (Donatas Abraitis) > 2. Re: handler before proxy-pass (Maxim Dounin) > 3. Re: [PATCH] append "-gzip" to etags processed by > gzip_filter_module + not modified support (Andrew Williams) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 29 May 2014 15:05:15 +0300 > From: Donatas Abraitis > To: nginx-devel at nginx.org > Subject: handler before proxy-pass > Message-ID: > LWmOS0XQTVYPDeurBkNpAn63--V_O7Q at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hello, > > I want to add a custom 'handler', which would update ngx_os_argv[0] on > request. It should be like this: > > root 18601 0.0 0.0 62272 3812 ? Ss 07:00 0:00 nginx: > master process /opt/nginx/bin/nginx -c /opt/nginx/etc/nginx.conf > web 18602 0.0 0.0 70372 7904 ? S 07:00 0:00 \_ > 183.54.68.10 [test.domain.com] GET /ok.php HTTP/1.1??Host > > It works if using only nginx for static content, but if having proxy_pass > in location directive it doesn't work. > > Code snippet is like this: > > { > ... > sprintf(title, "%s [%s] %s", r->connection->addr_text.data, > r->headers_in.server.data, r->request_line.data); > memcpy(ngx_os_argv[0], title, MAXBUF); > return NGX_DECLINED; > } > > Question is, how to solve this problem to update ngx_os_argv[0] if using > together with proxy_pass? Seems it bypass my 'handler' if using proxy_pass. > Maybe there is some kind of sequence of loading modules? > > -- > Donatas > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://mailman.nginx.org/pipermail/nginx-devel/attachments/20140529/66347976/attachment-0001.html > > > > ------------------------------ > > Message: 2 > Date: Thu, 29 May 2014 16:29:22 +0400 > From: Maxim Dounin > To: nginx-devel at nginx.org > Subject: Re: handler before proxy-pass > Message-ID: <20140529122922.GK1849 at mdounin.ru> > Content-Type: text/plain; charset=us-ascii > > Hello! > > On Thu, May 29, 2014 at 03:05:15PM +0300, Donatas Abraitis wrote: > > > Hello, > > > > I want to add a custom 'handler', which would update ngx_os_argv[0] on > > request. It should be like this: > > > > root 18601 0.0 0.0 62272 3812 ? Ss 07:00 0:00 nginx: > > master process /opt/nginx/bin/nginx -c /opt/nginx/etc/nginx.conf > > web 18602 0.0 0.0 70372 7904 ? S 07:00 0:00 \_ > > 183.54.68.10 [test.domain.com] GET /ok.php HTTP/1.1??Host > > > > It works if using only nginx for static content, but if having proxy_pass > > in location directive it doesn't work. > > This isn't going to work in any case, as there are multiple > requests in a single worker process. > > > Code snippet is like this: > > > > { > > ... > > sprintf(title, "%s [%s] %s", r->connection->addr_text.data, > > r->headers_in.server.data, r->request_line.data); > > memcpy(ngx_os_argv[0], title, MAXBUF); > > return NGX_DECLINED; > > } > > > > Question is, how to solve this problem to update ngx_os_argv[0] if using > > together with proxy_pass? Seems it bypass my 'handler' if using > proxy_pass. > > Maybe there is some kind of sequence of loading modules? > > >From what you write it looks like you've installed your code as a > content handler. You should instead use something like postread > phase handler, see e.g. realip module for an example. But, well, > it's not going to work anyway, see above. > > -- > Maxim Dounin > http://nginx.org/ > > > > ------------------------------ > > Message: 3 > Date: Fri, 30 May 2014 11:55:40 +0930 > From: Andrew Williams > To: nginx-devel at nginx.org > Subject: Re: [PATCH] append "-gzip" to etags processed by > gzip_filter_module + not modified support > Message-ID: > rth9TG7-zZiQ at mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > Hi, > > > On Wed, May 28, 2014 at 04:26:04PM +0930, Andrew Williams wrote: > > > >> this patch modifies the etag header returned by responses processed by > >> the gzip filter, appending "-gzip" to the etag (instead of deleting > >> it). Also, If-None-Match and If-Match requests work against the > >> modified etag. > >> (the gunzip module appends "-gunzip" to the etag). > > > > This approach is wrong, as resources with the same etag are not > > guaranteed to be equal. > > thanks for the feedback - only stripping strong etags looks like a > good solution. looking forward to seeing the weak etag code making it > in. > cheers > Andrew > > > > ------------------------------ > > _______________________________________________ > nginx-devel mailing list > nginx-devel at nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx-devel > > End of nginx-devel Digest, Vol 55, Issue 34 > ******************************************* > -- Donatas -------------- next part -------------- An HTML attachment was scrubbed... URL: From vbart at nginx.com Fri May 30 15:56:12 2014 From: vbart at nginx.com (Valentin Bartenev) Date: Fri, 30 May 2014 15:56:12 +0000 Subject: [nginx] SPDY: fixed operator precedence in uint16/uint32 write m... Message-ID: details: http://hg.nginx.org/nginx/rev/34d460c5d186 branches: changeset: 5716:34d460c5d186 user: Valentin Bartenev date: Thu May 29 21:15:19 2014 +0400 description: SPDY: fixed operator precedence in uint16/uint32 write macros. Since the type cast has precedence higher than the bit shift operator, all values were truncated to 8 bits. These macros are used to construct header block for SYN_REPLY frame on platforms with strict alignment requirements. As a result, any response that contains a header with name or value longer than 255 bytes was corrupted on such platforms. diffstat: src/http/ngx_http_spdy.h | 13 ++++++++----- 1 files changed, 8 insertions(+), 5 deletions(-) diffs (25 lines): diff -r 790ba7484bb6 -r 34d460c5d186 src/http/ngx_http_spdy.h --- a/src/http/ngx_http_spdy.h Wed May 28 20:18:05 2014 +0400 +++ b/src/http/ngx_http_spdy.h Thu May 29 21:15:19 2014 +0400 @@ -230,13 +230,16 @@ ngx_int_t ngx_http_spdy_send_output_queu #else #define ngx_spdy_frame_write_uint16(p, s) \ - ((p)[0] = (u_char) (s) >> 8, (p)[1] = (u_char) (s), (p) + sizeof(uint16_t)) + ((p)[0] = (u_char) ((s) >> 8), \ + (p)[1] = (u_char) (s), \ + (p) + sizeof(uint16_t)) #define ngx_spdy_frame_write_uint32(p, s) \ - ((p)[0] = (u_char) (s) >> 24, \ - (p)[1] = (u_char) (s) >> 16, \ - (p)[2] = (u_char) (s) >> 8, \ - (p)[3] = (u_char) (s), (p) + sizeof(uint32_t)) + ((p)[0] = (u_char) ((s) >> 24), \ + (p)[1] = (u_char) ((s) >> 16), \ + (p)[2] = (u_char) ((s) >> 8), \ + (p)[3] = (u_char) (s), \ + (p) + sizeof(uint32_t)) #endif From bfranks781 at gmail.com Fri May 30 21:27:36 2014 From: bfranks781 at gmail.com (bfranks781 at gmail.com) Date: Fri, 30 May 2014 14:27:36 -0700 Subject: best approach for content phase handler to run again with event timer & ngx_done? Message-ID: If a content phase handler needs to wait on some potentially delayed result, my understanding is that it should return NGX_DONE so that it is called again. I've been reading through the eval, echo, and http_limit_req modules to see how to integrate an nginx_add_timer event prior to returning NGX_DONE. A short timer event seems reasonable, because the content phase handler isn't waiting on some other event type (subrequest result, timeout, etc). The timer event seems fairly straight-forward -- configure the event in a request context and set the event handler, data and log. I don't really want my timer event handler to do anything -- rather I just want the same content phase handler that had previously returned NGX_DONE to run again. In that case, should my timer event handler actually do anything at all? Is there a best practice for this -- i.e. have it point to the write_event_handler(), call ngx_http_core_run_phases() or ngx_http_run_posted_requests(), etc? Thank you, Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: