Making http_auth_request_module a first-class citizen? [patch]

[Max]

15 февраля 2012, 18:50 от Maxim Dounin <mdounin at mdounin.ru>:
> Hello!
> 
> On Wed, Feb 15, 2012 at 08:56:49AM -0500, Maxim Khitrov wrote:
> 
> > Hello Maxim,
> > 
> > Back in 2010 you wrote that it's not likely that your
> > http_auth_request_module would make it into nginx core. I'm curious if
> > anything has changed over the past two years?
> > 
> > It's not that compiling this module into nginx is a problem
> > (especially on FreeBSD), but I think a lot of people are inherently
> > weary of depending on 3rd-party modules, since there is no guarantee
> > of continued support.
> > 
> > What do you think about adding your module to the main nginx repository?
> 
> There are no immediate plans, but this may happen somewhere in the 
> future.

Hello fellow Maxims and others,

I took a closer look at the auth_request module source code today and
realized that I was partially wrong about auth_request authorization
subrequests causing the entire requested file to be retrieved from the
backend server. I apologize for the confusion my posts may have
caused. Due to sr->header_only being set to 1, the connection to the
backend server is terminated from within ngx_http_upstream_send_response()
as soon as the HTTP request status code is received.

So the entire file is (usually) not retrieved, but due to the fact that
the connection is prematurely closed, it may take a while for the backend
server to actually detect that the frontend server has closed the
connection and that it should stop sending.

I have done some testing to see what kind of overhead this kind of
abrupt connection termination causes, and according to my test
results, auth_request module authorization requests always generate
at least 65 kb worth of traffic (including the initial TCP three-way
handshake and the final four-way FIN/ACK connection termination
packets). Under heavy load, the amount of traffic generated mainly
by the backend server while sending the requested file in vain
exceeded 500 kb, so it's obvious that this might lead to
significant overhead when large files are sent in response
to auth_request authorization subrequests.

Moreover, each file on the backend server that is accessed this
way has to be opened and prepared for sending, which causes disk
overhead and buffer allocation because the backend server
treats those requests just like all the other GET method
requests. Under heavy load, this overhead can be significant.
Another minor problem is that logs on the backend server
tend to get filled up with constant error messages about
connections being closed prematurely, writev() failures etc.

All of these issues can be avoided simply by using HEAD method
requests for authorization subrequests. According to my
test results, HEAD method authorization subrequests generate
no more than 1310 bytes worth of traffic (including the initial
TCP three-way handshake and the final four-way FIN/ACK connection
termination packets). GET method authorization subrequests, on
the other hand, generate at best 50 times and at worst 400 times
more traffic than HEAD method requests, so files smaller than 64 kb
actually do get retrieved twice from the backend server on each
request, as well as files smaller than 200 kb under heavy load.

HEAD method request responses, on the other hand, remain the same
size even under heavy load, and the  upstream / backend server always
immediately terminates the connection on its side after sending its
 (usually) single-packet response, so there is also no additional disk
overhead, only the size of the requested file is checked, and no additional
buffer allocation is done, while sr->header_only on the frontend server
can be safely set to 0, which also makes it possible for the auth_request
module to be extended to make the use of the proxy_cache and proxy_store
directives possible even for authorization requests because completed
upstream requests would no longer cause response-buffering temporary
files to be closed.

Since HEAD method requests also go through the same access phase
checking as the GET method requests, they are also a valid means
of checking whether an actual GET method request would be
allowed, unless different responses are configured for each
method.

The best thing about it all is that you can make the auth_request
module use HEAD method authorization requests by adding the
following directive to the auth_request authorization subrequest
location block:

proxy_method HEAD;

I have also modified the auth_request module to use HEAD method
authorization subrequests by default. This setting can be
overridden in the configuration file by using the proxy_method
directive, of course.

You can find my auth_request module patch here:

https://nginxyzpro.berlios.de/patch-head.ngx_http_auth_request_module.c.20120215.diff

SIZE (patch-head.ngx_http_auth_request_module.c.20120215.diff) = 5196 bytes

SHA256 (patch-head.ngx_http_auth_request_module.c.20120215.diff) = \
       6d163ec9e11a06bcadd8395042e0a6ef1dc2dfbe0bbfab4cb9d0c4e73e373f75

Any comments will be appreciated.


Max