[PATCH 1 of 3] HTTP: add support for trailers in HTTP responses

[Piotr Sikora]

Hey Maxim,

> As you can see from the quote, it talks about not generating
> "trailer fields that it believes are necessary for the user agent
> to receive".  RFC 2616 is even more clear on this, specifically
> lists two cases when trailers can be generated, section 3.6.1:
>
>    A server using chunked transfer-coding in a response MUST NOT use the
>    trailer for any header fields unless at least one of the following is
>    true:
>
>    a)the request included a TE header field that indicates "trailers" is
>      acceptable in the transfer-coding of the  response, as described in
>      section 14.39; or,
>
>    b)the server is the origin server for the response, the trailer
>      fields consist entirely of optional metadata, and the recipient
>      could use the message (in a manner acceptable to the origin server)
>      without receiving this metadata.  In other words, the origin server
>      is willing to accept the possibility that the trailer fields might
>      be silently discarded along the path to the client.
>
> That is, there clearly two cases when a server can / should send
> trailers:
>
>    - when there is "TE: trailers";
>
>    - when trailers are optional.
>
> Moreover, given that "TE: trailers" does _not_ guarantee anything
> either (see link above), the only remaining case seems to be "when
> trailers are optional".

I disagree, if someone adds trailers in their configuration, then they
are hardly optional, therefore they should be sent in response to
requests with "TE: trailers".

> As per previous discussion, main cases, as listed by you, are:
>
> 1. Checksums, like Content-SHA256.
> 2. Logging, like X-Log-Something for centralized logging on a
>    frontend.
> 3. Trailing status, like X-Status, to provide additional error
>    information to a frontend.
>
> All these uses cases hardly require "TE: trailers", as in all
> cases information seems optional.  Moreover, in cases (2) and (3),
> backend cannot have "TE: trailers" unless it was already present
> in the original request from the client.  Adding such a header
> would contradict RFC 7230, which says:
>
>    The presence of the keyword "trailers" indicates that the client is
>    willing to accept trailer fields in a chunked transfer coding, as
>    defined in Section 4.1.2, on behalf of itself and any downstream
>    clients.  For requests from an intermediary, this implies that
>    either: (a) all downstream clients are willing to accept trailer
>    fields in the forwarded response; or, (b) the intermediary will
>    attempt to buffer the response on behalf of downstream recipients.
>
> So both cases (2) and (3) generally require that "TE: trailers"
> should be ignored, or it won't be possible to implement them.

That's valid argument.

> Current nginx behaviour is to don't emit trailers by default, and
> I don't think we anyhow change this beaviour unless explicitly
> requested via appropriate configuration directives.  This looks
> conservative enough for me, actually.

Yes, of course. What I meant is that if some module produces trailers
and it's enabled in the configuration, then even if there are only few
obscure clients that break because of it, the only option will be to
disable them altogether, instead of only sending them in response to
requests with "TE: trailers".

Anyway, I'll just remove that requirement for now, and you can re-add
it later if/when others complain.

> I can see that r->expect_trailers can be useful to indicate that
> the code wants to use trailers, and would like to force chunked
> encoding if possible.
>
> I don't see why it should be required when encoding is already
> choosen though.  If we know that we can use trailers or don't
> care, we can just add trailers to the response, and assume they
> will be sent if it is possible to do so.

I'm confused, it sounds like you're saying two opposite things...

Are you saying that r->expect_trailers should be removed and we should
send trailers only if transfer encoding is already chunked and not
send trailers otherwise?

> As per HTTP specification, trailers are expected to be sent as
> normal headers when there is no body, for both HTTP/1.1 and
> HTTP/2.  And I suspect that the fact that HTTP/2 allows two
> HEADERS frames without body inbetween is more or less
> unintentional, too.
>
> (...)
>
> I certainly against intruduction of such a difference between
> HTTP/1.1 and HTTP/2 behaviour.

So what do you suggest we do instead? Merge them with headers? Wasn't
that your biggest concern from the beginning?

Also, how would you merge them in upstream? Cacheable responses to
HEAD requests are "terminated" and headers are sent before upstream
sends complete response, so trailers would be dropped there (at least
in the existing code).

Best regards,
Piotr Sikora