[PATCH] HTTP: add support for trailers in HTTP responses

[Maxim Dounin]

Hello!

On Thu, Aug 18, 2016 at 06:12:46PM -0700, Piotr Sikora wrote:

> > By saying "most popular use case" you are talking about something
> > real-world you are aware of?
> 
> Yes.
> 
> Furthermore, I'm aware of a few systems using NGINX that use either
> ugly workarounds or are missing features exactly because NGINX doesn't
> support trailers.

Could you please describe one of the uses cases in details?  
That's what I'm asking all the time, and still didn't get even a 
single description of a real-world use case.

> > And this probably an additional thing to consider when introducing
> > trailers: right now nginx strips all trailers.  Changing this may
> > be a surprise for those who use trailers internally, if any.
> 
> This can be easily mitigated with "proxy_pass_trailers on|off".

Sure.  This is just one item from the long list of various 
implications of trailers.

> > We've already talked about gRPC.
> 
> Yes, and I'm not sure why you keep ignoring it.
> 
> If it's because you assume that it has to be end-to-end HTTP/2 and as
> such trailers aren't the main blocker, then there are reverse proxies
> like grpc-gateway [1] (which, coincidentally, doesn't support
> trailers), that convert HTTP/1.1 requests to a "pure" gRPC requests
> and back... Lack of trailers in NGINX prevents people from writing
> custom upstream gRPC module, which could provide similar functionality
> in NGINX.

The last sentense here contradicts the first one.  If grpc-gateway 
is fine without trailers support, I don't see how lack of trailers 
in nginx prevents one from providing similar functionality in 
nginx.

> However, if it's because "it's Google pushing it's own library", then
> it's neither good nor technical reason for rejecting trailers support.
> 
> [1] https://github.com/grpc-ecosystem/grpc-gateway
> 
> > Fetch API and Server Timing are
> > just specifications being developed which allow use of trailers,
> > not much different from HTTP/1.1 itself.
> 
> Well, they wouldn't be adding it if it was so useless, would they?

The same applies to HTTP/1.1.  Yet trailers are (mostly?) unused 
for many years.  Moreover, it is clearly understood now that 
merging trailer headers to headers as supposed by HTTP/1.1 is a 
security disaster.

> > On the other hand, here is a good discussion of trailers and their
> > security in this Fetch API ticket:
> >
> > https://github.com/whatwg/fetch/issues/34
> 
> Yes, and the consensus is that there are no new security implications,
> as long as you don't magically merge trailers with headers (which, as
> previously stated, I have no intention of doing).

Merging trailers with headers can happen somewhere else as long as 
you pass trailers through.  And it will be perfectly in line with 
HTTP/1.1 specs.

> > And here is a linked HTTPbis ticket to reconsider "TE: trailers"
> > as it looks unneded:
> >
> > https://github.com/httpwg/http11bis/issues/18
> >
> > This is somewhat in line with what I think about it, as previously
> > discussed in this thread.
> 
> I can drop this requirement if you insist, but that's much less
> conservative approach than NGINX usually takes and I expect that some
> obscure HTTP clients will break because of lack of proper support for
> trailer-part in chunked encoding.

The only potentially bad thing that can happen without forcing 
chunked transfer encoding is that trailers configured won't be 
sent if Content-Length is know.  If this is critical for a use 
case, the Content-Length can be explicitly removed with additional 
configuration.

Additionally, this is not something expected to happen when 
proxying, as Content-Length won't be known anyway if there are 
trailers in the upstream response.

On the other hand, forcing chunked transfer encoding based on "TE: 
trailers" looks all the way wrong:

- it will change the behaviour of nginx for such clients, even if 
  there are no other reasons to do so;

- it won't change the behaviour for other HTTP/1.1 clients who 
  actually support trailers but doesn't advertize it via "TE: 
  trailers", and thus trailers will be lost in some cases anyway.

> > I'm still not convinced that trailers are needed.  As previously
> > said, this HTTP feature was mostly unused for 17 years, and
> > this suggests something is wrong with the feature.  So it may be a
> > good idea to postpone this at least till some real user will
> > appear.
> 
> You have real users commenting in this thread, why do you keep
> ignoring them? Other people already expressed interest, commented on
> their internal implementations and/or said that they use workarounds
> because NGINX doesn't support trailers.
> 
> What more do you need?

I'm just asking for a description of at least one real-world use 
case for trailers, to better understand how they are expected to 
be used in real life.

> > In either case, I do not think that added trailers should by
> > itself change transfer encoding used and remove Content-Length.
> 
> Again, why this is an issue with trailers but it's not an issue with gzip?

Gzip is something you explicitly enable in the configuration to 
change the content of some responses.  It changes the content, and 
this in turn results in chunked transfer encoding when using 
HTTP/1.1.

In contrast, trailers is something exists only in case of chunked 
transfer encoding.  Depending on the use case, it may make sense 
to either:

1. once trailers are configured, force chunked encoding and sent 
   trailers (if it is possible to do so); or

2. once trailers are configured, sent them as long as chunked 
   encoding is used.

Obviously enough, second approach is more flexible.  And, as long 
as there is a way to force chunked encoding by other means, allows 
to do exactly the same thing as the first one, plus some additional 
things.

The first approach may be easier and more natural for some use 
cases.  But as there are no real-world use cases described, there 
are no arguments to support this approach.

-- 
Maxim Dounin
http://nginx.org/