wow, this + last-modified and etags would really cut into the reason people use varnish in their nginx + varnish setup.<br><br><div class="gmail_quote">On Fri, Sep 16, 2011 at 3:51 AM, MagicBear <span dir="ltr"><<a href="mailto:magicbearmo@gmail.com">magicbearmo@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Hello, <div>I have wrote a module to make nginx support 304 to decrease bandwidth usage.</div><div>note: I have a newbie for nginx module development, so the above module may have some problem. Welcome to test it and feedback another problem with me.</div>
<div><br></div><div>You can download full patch file from here: <a href="http://m-b.cc/share/proxy_304.txt" target="_blank">http://m-b.cc/share/proxy_304.txt</a></div><div><br><div># User MagicBear <<a href="mailto:magicbearmo@gmail.com" target="_blank">magicbearmo@gmail.com</a>></div>
<div>Upstream:</div><div>add $upstream_last_modified variant.</div><div>add handler for 304 Unmodified.</div><div>Proxy:</div><div>change to send If-Modified-Since header. </div><div><br></div><div>TODO:</div><div>change write TO not block IO.</div>
<div><br></div><div>diff -ruN a/http/modules/ngx_http_proxy_module.c b/http/modules/ngx_http_proxy_module.c</div><div>--- a/http/modules/ngx_http_proxy_module.c 2011-09-15 22:23:03.284431407 +0800</div><div>+++ b/http/modules/ngx_http_proxy_module.c 2011-09-16 01:41:44.654428632 +0800</div>
<div>@@ -543,7 +543,7 @@</div><div> { ngx_string("Connection"), ngx_string("close") },</div><div> { ngx_string("Keep-Alive"), ngx_string("") },</div><div> { ngx_string("Expect"), ngx_string("") },</div>
<div>- { ngx_string("If-Modified-Since"), ngx_string("") },</div><div>+ { ngx_string("If-Modified-Since"), ngx_string("$upstream_last_modified") },</div><div> { ngx_string("If-Unmodified-Since"), ngx_string("") },</div>
<div> { ngx_string("If-None-Match"), ngx_string("") },</div><div> { ngx_string("If-Match"), ngx_string("") },</div><div>diff -ruN a/http/ngx_http_upstream.c b/http/ngx_http_upstream.c</div>
<div>--- a/http/ngx_http_upstream.c 2011-09-15 22:23:03.284431407 +0800</div><div>+++ b/http/ngx_http_upstream.c 2011-09-16 01:41:44.654428632 +0800</div><div>@@ -16,6 +16,8 @@</div><div> ngx_http_upstream_t *u);</div>
<div> static ngx_int_t ngx_http_upstream_cache_status(ngx_http_request_t *r,</div><div> ngx_http_variable_value_t *v, uintptr_t data);</div><div>+static ngx_int_t ngx_http_upstream_last_modified(ngx_http_request_t *r,</div>
<div>+ ngx_http_variable_value_t *v, uintptr_t data);</div><div> #endif</div><div><br></div><div> static void ngx_http_upstream_init_request(ngx_http_request_t *r);</div><div>@@ -342,6 +344,10 @@</div><div> ngx_http_upstream_cache_status, 0,</div>
<div> NGX_HTTP_VAR_NOCACHEABLE, 0 },</div><div><br></div><div>+ { ngx_string("upstream_last_modified"), NULL,</div><div>+ ngx_http_upstream_last_modified, 0,</div><div>+ NGX_HTTP_VAR_NOCACHEABLE, 0 },</div>
<div>+</div><div> #endif</div><div><br></div><div> { ngx_null_string, NULL, NULL, 0, 0, 0 }</div><div>@@ -1618,6 +1624,80 @@</div><div> u->buffer.last = u->buffer.pos;</div><div> }</div><div>
<br></div><div>+#if (NGX_HTTP_CACHE)</div><div>+</div><div>+ if (u->cache_status == NGX_HTTP_CACHE_EXPIRED &&</div><div>+ u->headers_in.status_n == NGX_HTTP_NOT_MODIFIED &&</div>
<div>+ ngx_http_file_cache_valid(u->conf->cache_valid, u->headers_in.status_n))</div><div>+ {</div><div>+ ngx_int_t rc;</div><div>+</div><div>+ rc = u->reinit_request(r);</div>
<div>+</div><div>+ if (rc == NGX_OK) {</div><div>+ u->cache_status = NGX_HTTP_CACHE_BYPASS;</div><div>+ rc = ngx_http_upstream_cache_send(r, u);</div><div>+</div><div>+ time_t now, valid;</div>
<div>+</div><div>+ now = ngx_time();</div><div>+</div><div>+ valid = r->cache->valid_sec;</div><div>+</div><div>+ if (valid == 0) {</div>
<div>+ valid = ngx_http_file_cache_valid(u->conf->cache_valid,</div><div>+ u->headers_in.status_n);</div>
<div>+ if (valid) {</div><div>+ r->cache->valid_sec = now + valid;</div><div>+ }</div><div>+ }</div>
<div>+</div><div>+ if (valid) {</div><div>+ r->cache->last_modified = r->headers_out.last_modified_time;</div><div>+ r->cache->date = now;</div>
<div>+ r->cache->body_start = (u_short) (u->buffer.pos - u->buffer.start);</div><div>+</div><div>+ // update Header</div><div>+ ngx_http_file_cache_set_header(r, u->buffer.start);</div>
<div>+</div><div>+ ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,</div><div>+ "update cache \"%s\" header to new expired." , r->cache->file.name.data);</div>
<div>+</div><div>+ // Reopen file via RW</div><div>+ ngx_fd_t fd = ngx_open_file(r->cache->file.name.data, NGX_FILE_RDWR, NGX_FILE_OPEN, 0);</div>
<div>+</div><div>+ if (fd == NGX_INVALID_FILE) {</div><div>+ ngx_log_error(NGX_LOG_CRIT, r->connection->log, ngx_errno,</div><div>
+ ngx_open_file_n " \"%s\" failed", r->cache->file.name.data);</div>
<div>+ return;</div><div>+ }</div><div>+</div><div>+ // Write cache</div><div>+ if (write(fd, u->buffer.start, sizeof(ngx_http_file_cache_header_t)) < 0)</div>
<div>+ {</div><div>+ ngx_log_error(NGX_LOG_CRIT, r->connection->log, ngx_errno,</div><div>+ "write proxy_cache \"%s\" failed", r->cache->file.name.data);</div>
<div>+ return;</div><div>+ }</div><div>+</div><div>+ if (ngx_close_file(fd) == NGX_FILE_ERROR) {</div>
<div>+ ngx_log_error(NGX_LOG_ALERT, r->connection->log, ngx_errno,</div><div>+ ngx_close_file_n " \"%s\" failed", r->cache->file.name.data);</div>
<div>+ }</div><div>+ ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,</div><div>+ "update cache \"%s\" header to new expired done." , r->cache->file.name.data);</div>
<div>+ } else {</div><div>+ u->cacheable = 0;</div><div>+ r->headers_out.last_modified_time = -1;</div><div>
+ }</div>
<div>+ }</div><div>+</div><div>+ ngx_http_upstream_finalize_request(r, u, rc);</div><div>+ return;</div><div>+ }</div><div>+</div><div>+#endif</div><div>+</div><div> if (ngx_http_upstream_test_next(r, u) == NGX_OK) {</div>
<div> return;</div><div> }</div><div>@@ -4006,6 +4086,32 @@</div><div><br></div><div> return NGX_OK;</div><div> }</div><div>+</div><div>+ngx_int_t</div><div>+ngx_http_upstream_last_modified(ngx_http_request_t *r,</div>
<div>+ ngx_http_variable_value_t *v, uintptr_t data)</div><div>+{</div><div>+ u_char *u;</div><div>+</div><div>+ if (r->upstream == NULL || r->upstream->cache_status == 0 || r->cache==NULL || r->cache->last_modified <= 0) {</div>
<div>+ v->not_found = 1;</div><div>+ return NGX_OK;</div><div>+ }</div><div>+</div><div>+ v->valid = 1;</div><div>+ v->no_cacheable = 0;</div><div>+ v->not_found = 0;</div><div>+ u = ngx_pcalloc(r->pool, 30);</div>
<div>+ if (u == NULL) {</div><div>+ return NGX_ERROR;</div><div>+ }</div><div>+</div><div>+ v->len = 29;</div><div>+ ngx_http_time(u, r->cache->last_modified);</div><div>+ v->data = u;</div>
<div>+</div><div>+ return NGX_OK;</div><div>+}</div><div><br></div><div> #endif</div></div><div><br></div><font color="#888888"><div><br></div><div>MagicBear</div></font><div><div></div><div class="h5"><div><br><div class="gmail_quote">
2011/9/15 magicbear <span dir="ltr"><<a href="mailto:nginx-forum@nginx.us" target="_blank">nginx-forum@nginx.us</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I have run the nginx 1.1.2 via this patch for 7 days, except for one<br>
days have a large DDoS so I restart nginx for several seconds, it was<br>
very stable to work.<br>
Handle about 70million request without problem happen, I think the last<br>
problem may be have a memory corruption, you are right.<br>
I will check that server when have times.<br>
Thanks for your hard work.<br>
<br>
MagicBear<br>
<br>
Maxim Dounin Wrote:<br>
-------------------------------------------------------<br>
<div><div></div><div>> Hello!<br>
><br>
> On Mon, Sep 05, 2011 at 11:42:31PM +0800,<br>
> ビリビリⅤ wrote:<br>
><br>
> > (gdb) fr 0<br>
> > #0 ngx_http_upstream_handler<br>
> (ev=0x7fc45735f8a8)<br>
> > at src/http/ngx_http_upstream.c:915<br>
> > 915 ctx->current_request = r;<br>
> > (gdb) p ngx_cycle->log<br>
> > $1 = (ngx_log_t *) 0x21f19a8<br>
> > (gdb) p *r<br>
> > $2 = {signature = 51686928, connection =<br>
> 0x23b4160, ctx = 0x0,<br>
><br>
> [...]<br>
><br>
> This looks like memory corruption, but<br>
> unfortunately I don't see<br>
> any traces of the real cause. My best quess is<br>
> improper handling<br>
> of proxy_ignore_client_abort as fixed in 1.1.2.<br>
> Please try 1.1.2<br>
> with patches from<br>
><br>
> <a href="http://nginx.org/patches/patch-nginx-keepalive-ful" target="_blank">http://nginx.org/patches/patch-nginx-keepalive-ful</a><br>
> l-5.txt<br>
><br>
> It already includes upstream keepalive module, as<br>
> well as all<br>
> other upstream-keepalive related fixes. See here<br>
> for details:<br>
><br>
> <a href="http://mailman.nginx.org/pipermail/nginx-devel/201" target="_blank">http://mailman.nginx.org/pipermail/nginx-devel/201</a><br>
> 1-September/001147.html<br>
><br>
> Maxim Dounin<br>
><br>
> _______________________________________________<br>
> nginx mailing list<br>
> <a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a><br>
> <a href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a><br>
<br>
</div></div>Posted at Nginx Forum: <a href="http://forum.nginx.org/read.php?2,213207,215217#msg-215217" target="_blank">http://forum.nginx.org/read.php?2,213207,215217#msg-215217</a><br>
<div><div></div><div><br>
_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a></div></div></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a><br></blockquote></div><br>