Is it possible that nginx will not buffer the client body?
Pasi Kärkkäinen
pasik at iki.fi
Fri Feb 22 10:50:52 UTC 2013
On Fri, Feb 22, 2013 at 11:25:24AM +0200, Pasi Kärkkäinen wrote:
> On Fri, Feb 22, 2013 at 10:06:11AM +0800, Weibin Yao wrote:
> > Use the patch I attached in this mail thread instead, don't use the pull
> > request patch which is for tengine.Â
> > Thanks.
> >
>
> Oh sorry I missed that attachment. It seems to apply and build OK.
> I'll start testing it.
>
I added the patch on top of nginx 1.2.7 and enabled the following options:
client_body_postpone_sending 64k;
proxy_request_buffering off;
after that connections through the nginx reverse proxy started failing
with errors like this:
[error] 29087#0: *49 upstream prematurely closed connection while reading response header from upstream
[error] 29087#0: *60 upstream sent invalid header while reading response header from upstream
And the services are unusable.
Commenting out the two config options above makes nginx happy again.
Any idea what causes that? Any tips how to troubleshoot it?
Thanks!
-- Pasi
>
> > 2013/2/22 Pasi KÀrkkÀinen <[1]pasik at iki.fi>
> >
> > On Fri, Jan 18, 2013 at 10:38:21AM +0200, Pasi KÀrkkÀinen wrote:
> > > On Thu, Jan 17, 2013 at 11:15:58AM +0800, ?????? wrote:
> > > > Â Â Yes. It should work for any request method.
> > > >
> > >
> > > Great, thanks, I'll let you know how it works for me. Probably in two
> > weeks or so.
> > >
> >
> > Hi,
> >
> > Adding the tengine pull request 91 on top of nginx 1.2.7 doesn't work:
> >
> > cc1: warnings being treated as errors
> > src/http/ngx_http_request_body.c: In function
> > 'ngx_http_read_non_buffered_client_request_body':
> > src/http/ngx_http_request_body.c:506: error: implicit declaration of
> > function 'ngx_http_top_input_body_filter'
> > make[1]: *** [objs/src/http/ngx_http_request_body.o] Error 1
> > make[1]: Leaving directory `/root/src/nginx/nginx-1.2.7'
> > make: *** [build] Error 2
> >
> > ngx_http_top_input_body_filter() cannot be found from any .c/.h files..
> > Which other patches should I apply?
> >
> > Perhaps this?
> > [2]https://github.com/cfsego/limit_upload_rate/blob/master/for-nginx.patch
> >
> > Thanks,
> > -- Pasi
> >
> > >
> > > > Â Â 2013/1/16 Pasi KÃ*â*¬rkkÃ*â*¬inen <[1][3]pasik at iki.fi>
> > > >
> > > > Â Â Â On Sun, Jan 13, 2013 at 08:22:17PM +0800, ?????? wrote:
> > > > Â Â Â > Ã* Â Ã* This patch should work between nginx-1.2.6 and
> > nginx-1.3.8.
> > > > Â Â Â > Ã* Â Ã* The documentation is here:
> > > >
> > > > Â Â Â > Ã* Â Ã* ## client_body_postpone_sending ##
> > > > Â Â Â > Ã* Â Ã* Syntax: **client_body_postpone_sending** `size`
> > > > Â Â Â > Ã* Â Ã* Default: 64k
> > > > Â Â Â > Ã* Â Ã* Context: `http, server, location`
> > > > Â Â Â > Ã* Â Ã* If you specify the `proxy_request_buffering` or
> > > > Â Â Â > Ã* Â Ã* `fastcgi_request_buffering` to be off, Nginx will
> > send the body
> > > > Â Â Â to backend
> > > > Â Â Â > Ã* Â Ã* when it receives more than `size` data or the
> > whole request body
> > > > Â Â Â has been
> > > > Â Â Â > Ã* Â Ã* received. It could save the connection and reduce
> > the IO number
> > > > Â Â Â with
> > > > Â Â Â > Ã* Â Ã* backend.
> > > > Â Â Â >
> > > > Â Â Â > Ã* Â Ã* ## proxy_request_buffering ##
> > > > Â Â Â > Ã* Â Ã* Syntax: **proxy_request_buffering** `on | off`
> > > > Â Â Â > Ã* Â Ã* Default: `on`
> > > > Â Â Â > Ã* Â Ã* Context: `http, server, location`
> > > > Â Â Â > Ã* Â Ã* Specify the request body will be buffered to the
> > disk or not. If
> > > > Â Â Â it's off,
> > > > Â Â Â > Ã* Â Ã* the request body will be stored in memory and sent
> > to backend
> > > > Â Â Â after Nginx
> > > > Â Â Â > Ã* Â Ã* receives more than `client_body_postpone_sending`
> > data. It could
> > > > Â Â Â save the
> > > > Â Â Â > Ã* Â Ã* disk IO with large request body.
> > > > Â Â Â >
> > > > Â Â Â >
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* Â Ã* Â Note that, if you specify it
> > to be off, the nginx
> > > > Â Â Â retry mechanism
> > > > Â Â Â > Ã* Â Ã* with unsuccessful response will be broken after
> > you sent part of
> > > > Â Â Â the
> > > > Â Â Â > Ã* Â Ã* request to backend. It will just return 500 when
> > it encounters
> > > > Â Â Â such
> > > > Â Â Â > Ã* Â Ã* unsuccessful response. This directive also breaks
> > these
> > > > Â Â Â variables:
> > > > Â Â Â > Ã* Â Ã* $request_body, $request_body_file. You should not
> > use these
> > > > Â Â Â variables any
> > > > Â Â Â > Ã* Â Ã* more while their values are undefined.
> > > > Â Â Â >
> > > >
> > > > Â Â Â Hello,
> > > >
> > > > Â Â Â This patch sounds exactly like what I need aswell!
> > > > Â Â Â I assume it works for both POST and PUT requests?
> > > >
> > > > Â Â Â Thanks,
> > > >
> > > > Â Â Â -- Pasi
> > > >
> > > > Â Â Â > Ã* Â Ã* Â Ã* Hello!
> > > > Â Â Â > Ã* Â Ã* Â Ã* @yaoweibin
> > > > Â Â Â >
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* If you are eager for this feature, you
> > could try my
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* patch:
> > [2][2][4]https://github.com/taobao/tengine/pull/91.
> > > > Â Â Â This patch has
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* been running in our production servers.
> > > > Â Â Â >
> > > > Â Â Â > Ã* Â Ã* Â Ã* what's the nginx version your patch based on?
> > > > Â Â Â > Ã* Â Ã* Â Ã* Thanks!
> > > > Â Â Â > Ã* Â Ã* Â Ã* On Fri, Jan 11, 2013 at 5:17 PM, Ã**Ã*
> > Ã**Ã*°Ã*±Ã*³
> > > > Â Â Â <[3][3][5]yaoweibin at gmail.com> wrote:
> > > > Â Â Â >
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* I know nginx team are working on it. You
> > can wait for it.
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* If you are eager for this feature, you
> > could try my
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* patch:
> > [4][4][6]https://github.com/taobao/tengine/pull/91.
> > > > Â Â Â This patch has
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* been running in our production servers.
> > > > Â Â Â >
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* 2013/1/11 li zJay
> > <[5][5][7]zjay1987 at gmail.com>
> > > > Â Â Â >
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* Hello!
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* is it possible that nginx will not
> > buffer the client
> > > > Â Â Â body before
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* handle the request to upstream?
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* we want to use nginx as a reverse
> > proxy to upload very
> > > > Â Â Â very big file
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* to the upstream, but the default
> > behavior of nginx is to
> > > > Â Â Â save the
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* whole request to the local disk
> > first before handle it
> > > > Â Â Â to the
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* upstream, which make the upstream
> > impossible to process
> > > > Â Â Â the file on
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* the fly when the file is uploading,
> > results in much high
> > > > Â Â Â request
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* latency and server-side resource
> > consumption.
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* Thanks!
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã*
> > _______________________________________________
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* nginx mailing list
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* [6][6][8]nginx at nginx.org
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã*
> > [7][7][9]http://mailman.nginx.org/mailman/listinfo/nginx
> > > > Â Â Â >
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* --
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Weibin Yao
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* Developer @ Server Platform Team of
> > Taobao
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã*
> > _______________________________________________
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* nginx mailing list
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã* [8][8][10]nginx at nginx.org
> > > > Â Â Â > Ã* Â Ã* Â Ã* Â Ã*
> > [9][9][11]http://mailman.nginx.org/mailman/listinfo/nginx
> > > > Â Â Â >
> > > > Â Â Â > Ã* Â Ã* Â Ã*
> > _______________________________________________
> > > > Â Â Â > Ã* Â Ã* Â Ã* nginx mailing list
> > > > Â Â Â > Ã* Â Ã* Â Ã* [10][10][12]nginx at nginx.org
> > > > Â Â Â > Ã* Â Ã* Â Ã*
> > [11][11][13]http://mailman.nginx.org/mailman/listinfo/nginx
> > > > Â Â Â >
> > > > Â Â Â > Ã* Â Ã* --
> > > > Â Â Â > Ã* Â Ã* Weibin Yao
> > > > Â Â Â > Ã* Â Ã* Developer @ Server Platform Team of Taobao
> > > > Â Â Â >
> > > > Â Â Â > References
> > > > Â Â Â >
> > > > Â Â Â > Ã* Â Ã* Visible links
> > > > Â Â Â > Ã* Â Ã* 1. mailto:[12][14]zjay1987 at gmail.com
> > > > Â Â Â > Ã* Â Ã* 2.
> > [13][15]https://github.com/taobao/tengine/pull/91
> > > > Â Â Â > Ã* Â Ã* 3. mailto:[14][16]yaoweibin at gmail.com
> > > > Â Â Â > Ã* Â Ã* 4.
> > [15][17]https://github.com/taobao/tengine/pull/91
> > > > Â Â Â > Ã* Â Ã* 5. mailto:[16][18]zjay1987 at gmail.com
> > > > Â Â Â > Ã* Â Ã* 6. mailto:[17][19]nginx at nginx.org
> > > > Â Â Â > Ã* Â Ã* 7.
> > [18][20]http://mailman.nginx.org/mailman/listinfo/nginx
> > > > Â Â Â > Ã* Â Ã* 8. mailto:[19][21]nginx at nginx.org
> > > > Â Â Â > Ã* Â Ã* 9.
> > [20][22]http://mailman.nginx.org/mailman/listinfo/nginx
> > > > Â Â Â > Ã* Â 10. mailto:[21][23]nginx at nginx.org
> > > > Â Â Â > Ã* Â 11.
> > [22][24]http://mailman.nginx.org/mailman/listinfo/nginx
> > > >
> > > > Â Â Â > _______________________________________________
> > > > Â Â Â > nginx mailing list
> > > > Â Â Â > [23][25]nginx at nginx.org
> > > > Â Â Â > [24][26]http://mailman.nginx.org/mailman/listinfo/nginx
> > > >
> > > > Â Â Â _______________________________________________
> > > > Â Â Â nginx mailing list
> > > > Â Â Â [25][27]nginx at nginx.org
> > > > Â Â Â [26][28]http://mailman.nginx.org/mailman/listinfo/nginx
> > > >
> > > > Â Â --
> > > > Â Â Weibin Yao
> > > > Â Â Developer @ Server Platform Team of Taobao
> > > >
> > > > References
> > > >
> > > > Â Â Visible links
> > > > Â Â 1. mailto:[29]pasik at iki.fi
> > > > Â Â 2. [30]https://github.com/taobao/tengine/pull/91
> > > > Â Â 3. mailto:[31]yaoweibin at gmail.com
> > > > Â Â 4. [32]https://github.com/taobao/tengine/pull/91
> > > > Â Â 5. mailto:[33]zjay1987 at gmail.com
> > > > Â Â 6. mailto:[34]nginx at nginx.org
> > > > Â Â 7. [35]http://mailman.nginx.org/mailman/listinfo/nginx
> > > > Â Â 8. mailto:[36]nginx at nginx.org
> > > > Â Â 9. [37]http://mailman.nginx.org/mailman/listinfo/nginx
> > > > Â 10. mailto:[38]nginx at nginx.org
> > > > Â 11. [39]http://mailman.nginx.org/mailman/listinfo/nginx
> > > > Â 12. mailto:[40]zjay1987 at gmail.com
> > > > Â 13. [41]https://github.com/taobao/tengine/pull/91
> > > > Â 14. mailto:[42]yaoweibin at gmail.com
> > > > Â 15. [43]https://github.com/taobao/tengine/pull/91
> > > > Â 16. mailto:[44]zjay1987 at gmail.com
> > > > Â 17. mailto:[45]nginx at nginx.org
> > > > Â 18. [46]http://mailman.nginx.org/mailman/listinfo/nginx
> > > > Â 19. mailto:[47]nginx at nginx.org
> > > > Â 20. [48]http://mailman.nginx.org/mailman/listinfo/nginx
> > > > Â 21. mailto:[49]nginx at nginx.org
> > > > Â 22. [50]http://mailman.nginx.org/mailman/listinfo/nginx
> > > > Â 23. mailto:[51]nginx at nginx.org
> > > > Â 24. [52]http://mailman.nginx.org/mailman/listinfo/nginx
> > > > Â 25. mailto:[53]nginx at nginx.org
> > > > Â 26. [54]http://mailman.nginx.org/mailman/listinfo/nginx
> > >
> > > > _______________________________________________
> > > > nginx mailing list
> > > > [55]nginx at nginx.org
> > > > [56]http://mailman.nginx.org/mailman/listinfo/nginx
> > >
> > > _______________________________________________
> > > nginx mailing list
> > > [57]nginx at nginx.org
> > > [58]http://mailman.nginx.org/mailman/listinfo/nginx
> >
> > _______________________________________________
> > nginx mailing list
> > [59]nginx at nginx.org
> > [60]http://mailman.nginx.org/mailman/listinfo/nginx
> >
> > --
> > Weibin Yao
> > Developer @ Server Platform Team of Taobao
> >
> > References
> >
> > Visible links
> > 1. mailto:pasik at iki.fi
> > 2. https://github.com/cfsego/limit_upload_rate/blob/master/for-nginx.patch
> > 3. mailto:pasik at iki.fi
> > 4. https://github.com/taobao/tengine/pull/91
> > 5. mailto:yaoweibin at gmail.com
> > 6. https://github.com/taobao/tengine/pull/91
> > 7. mailto:zjay1987 at gmail.com
> > 8. mailto:nginx at nginx.org
> > 9. http://mailman.nginx.org/mailman/listinfo/nginx
> > 10. mailto:nginx at nginx.org
> > 11. http://mailman.nginx.org/mailman/listinfo/nginx
> > 12. mailto:nginx at nginx.org
> > 13. http://mailman.nginx.org/mailman/listinfo/nginx
> > 14. mailto:zjay1987 at gmail.com
> > 15. https://github.com/taobao/tengine/pull/91
> > 16. mailto:yaoweibin at gmail.com
> > 17. https://github.com/taobao/tengine/pull/91
> > 18. mailto:zjay1987 at gmail.com
> > 19. mailto:nginx at nginx.org
> > 20. http://mailman.nginx.org/mailman/listinfo/nginx
> > 21. mailto:nginx at nginx.org
> > 22. http://mailman.nginx.org/mailman/listinfo/nginx
> > 23. mailto:nginx at nginx.org
> > 24. http://mailman.nginx.org/mailman/listinfo/nginx
> > 25. mailto:nginx at nginx.org
> > 26. http://mailman.nginx.org/mailman/listinfo/nginx
> > 27. mailto:nginx at nginx.org
> > 28. http://mailman.nginx.org/mailman/listinfo/nginx
> > 29. mailto:pasik at iki.fi
> > 30. https://github.com/taobao/tengine/pull/91
> > 31. mailto:yaoweibin at gmail.com
> > 32. https://github.com/taobao/tengine/pull/91
> > 33. mailto:zjay1987 at gmail.com
> > 34. mailto:nginx at nginx.org
> > 35. http://mailman.nginx.org/mailman/listinfo/nginx
> > 36. mailto:nginx at nginx.org
> > 37. http://mailman.nginx.org/mailman/listinfo/nginx
> > 38. mailto:nginx at nginx.org
> > 39. http://mailman.nginx.org/mailman/listinfo/nginx
> > 40. mailto:zjay1987 at gmail.com
> > 41. https://github.com/taobao/tengine/pull/91
> > 42. mailto:yaoweibin at gmail.com
> > 43. https://github.com/taobao/tengine/pull/91
> > 44. mailto:zjay1987 at gmail.com
> > 45. mailto:nginx at nginx.org
> > 46. http://mailman.nginx.org/mailman/listinfo/nginx
> > 47. mailto:nginx at nginx.org
> > 48. http://mailman.nginx.org/mailman/listinfo/nginx
> > 49. mailto:nginx at nginx.org
> > 50. http://mailman.nginx.org/mailman/listinfo/nginx
> > 51. mailto:nginx at nginx.org
> > 52. http://mailman.nginx.org/mailman/listinfo/nginx
> > 53. mailto:nginx at nginx.org
> > 54. http://mailman.nginx.org/mailman/listinfo/nginx
> > 55. mailto:nginx at nginx.org
> > 56. http://mailman.nginx.org/mailman/listinfo/nginx
> > 57. mailto:nginx at nginx.org
> > 58. http://mailman.nginx.org/mailman/listinfo/nginx
> > 59. mailto:nginx at nginx.org
> > 60. http://mailman.nginx.org/mailman/listinfo/nginx
>
> > _______________________________________________
> > nginx mailing list
> > nginx at nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx
>
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
More information about the nginx
mailing list