General Development Inquiry

[Jeff Heisz]

Hmmm, hopefully this works because I subscribed with digest.  Thank
you very much for the feedback, I now had an idea of where to focus (I
had a rough idea but this made it clearer).  And now I know what the
issue is but have no idea how to resolve it.

My module was setting the upstream length as you suggested, hence my
confusion.  So I flooded the ngx_http_upstream code with debug
statements to hunt down why the code around line 3614 (which calls
finalize on the upstream length reaching zero) wasn't working.  Turns
out, the value was -1, which, if I read the code correctly, indicates
that upstream is complete when connection is closed (consistent with
what I was seeing).

More debugging, turns out that my custom process_header method is
setting the correct length to process, but immediately after that can
the nginx server internally calls process_headers, which transfers all
of the values between the upstream and request headers
and...then...sets the upstream length to -1!

Gah!  So now I know what's wrong, but have no idea how to address it!
There's no indicators or flags that I can see in the process_headers
method to have it not reset the length to EOF!

jmh

>
> Message: 2
> Date: Sun, 7 Jun 2020 06:11:24 +0000
> From: Eran Kornblau <eran.kornblau at kaltura.com>
> To: "nginx-devel at nginx.org" <nginx-devel at nginx.org>
> Subject: RE: General Development Inquiry
> Message-ID:
>         <AM0PR04MB64675CEEC436B3BE76051DDFF5840 at AM0PR04MB6467.eurprd04.prod.outlook.com>
>
> Content-Type: text/plain; charset="us-ascii"
>
> > 2) I'm actually working on a second module/system aside from the one mentioned in my previous post.  This might actually be an easier question?  It's using a custom upstream (like the memcache one) to talk a binary protocol to another daemon with requests/responses.  For the most part it works, but there is one oddity that I can't seem to resolve - the daemon can respond with an HTML code/header set and then a response body.  My upstream process_header function is setting up all of the respective HTTP headers (content length, etc.) and buffer positioning to stream the response and NGINX is issuing it to the browser.  However, it doesn't finalize the request, instead waits for more data (which isn't coming, the response is complete) from the daemon and then times out after a minute and closes the connection.
> > My suspicion is that the 'built-in' upstream handing is designed around servers that close the connection after the response is issued.
> > But my daemon doesn't need to do that, so I'm trying to essentially use upstream with keepalive.  Do I also need to provide an output filter to track the response length myself and properly mark the last buffer/end of chain so that the request finalizes without closing the upstream connection?  Or is there some variable that I'm missing?  Any suggested examples to look at?
> >
> >From your description, I'm guessing you didn't set r->upstream->length, you need to set it to the response size you're expecting. The upstream input_filter decrements u->length by the number of bytes read each time, when it reaches zero, the request is finalized.
>
> Eran
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
> ------------------------------
>
> End of nginx-devel Digest, Vol 128, Issue 8
> *******************************************