[PATCH] HTTP: add support for trailers in HTTP responses

[Piotr Sikora]

Hey Maxim,

> Could you please describe one of the uses cases in details?
> That's what I'm asking all the time, and still didn't get even a
> single description of a real-world use case.

You were definitely given some real-world use cases in this thread already:

http://mailman.nginx.org/pipermail/nginx-devel/2016-June/008441.html
http://mailman.nginx.org/pipermail/nginx-devel/2016-July/008586.html
http://mailman.nginx.org/pipermail/nginx-devel/2016-August/008619.html

But fine, I can repeat a few of those:

1. Checksums: web server or proxy sends "Content-SHA256:
$response_body_sha256" back to the client. Since it doesn't know the
checksum ahead of time (either because it's a streaming proxy or
because it's not feasible to read the whole file twice), it must sent
it in the trailers. Thanks to the Fetch API, JavaScript clients in the
browser can verify that the checksum matches what was received.

2. Logging: in distributed systems, it's not always feasible or even
possible to write logs at each server that was involved in generating
response, so backend servers can pass information to log (processing
time, profiling data, etc.) in the trailers, so that everything can be
aggregated and/or logged in one place, usually by the front-end proxy.

3. Trailing status (Microservices): servers send "Status: OK" in the
trailers after successfully completing request. This is needed,
because a lot of things can go wrong between sending "200 OK" status
code in the headers and completing response.

Is that enough?

> The same applies to HTTP/1.1.  Yet trailers are (mostly?) unused
> for many years.  Moreover, it is clearly understood now that
> merging trailer headers to headers as supposed by HTTP/1.1 is a
> security disaster.

As stated previously, I have no intention of merging them together.

> Merging trailers with headers can happen somewhere else as long as
> you pass trailers through.  And it will be perfectly in line with
> HTTP/1.1 specs.

Yes, but then it's not NGINX's problem, is it?

> The only potentially bad thing that can happen without forcing
> chunked transfer encoding is that trailers configured won't be
> sent if Content-Length is know.  If this is critical for a use
> case, the Content-Length can be explicitly removed with additional
> configuration.

If you already configured NGINX to send and/or pass trailers, then why
would you also need to configure NGINX to send and/or pass trailer
even if Content-Length is present? This doesn't make any sense.

Furthermore, even with such simple configuration:

    location / {
        add_trailer "Content-SHA256" "$request_body_sha256";
        root /var/www;
        gzip on;
    }

clients that support gzip would receive Content-SHA256 trailer, while
those that didn't request gzip would not. Again, this doesn't make any
sense.

> Additionally, this is not something expected to happen when
> proxying, as Content-Length won't be known anyway if there are
> trailers in the upstream response.

That's only assuming HTTP/1.1. There might be other upstream protocols
that produce trailers or trailer-like metadata.

> On the other hand, forcing chunked transfer encoding based on "TE:
> trailers" looks all the way wrong:
>
> - it will change the behaviour of nginx for such clients, even if
>   there are no other reasons to do so;

Like I previously said, I'm happy to add r->expect_trailers, so that
modules that want to emit trailers could signal that, in which case
chunked encoding would be forced only when trailers are expected to
appear.

> - it won't change the behaviour for other HTTP/1.1 clients who
>   actually support trailers but doesn't advertize it via "TE:
>   trailers", and thus trailers will be lost in some cases anyway.

Again, I'm fine dropping this requirement...

> Gzip is something you explicitly enable in the configuration to
> change the content of some responses.  It changes the content, and
> this in turn results in chunked transfer encoding when using
> HTTP/1.1.

You need to explicitly add and/or enable passing of trailers as well.

> In contrast, trailers is something exists only in case of chunked
> transfer encoding.  Depending on the use case, it may make sense
> to either:
>
> 1. once trailers are configured, force chunked encoding and sent
>    trailers (if it is possible to do so); or

Isn't that what I'm proposing?

> 2. once trailers are configured, sent them as long as chunked
>    encoding is used.
>
> Obviously enough, second approach is more flexible.  And, as long
> as there is a way to force chunked encoding by other means, allows
> to do exactly the same thing as the first one, plus some additional
> things.

I honestly don't see what value would this add, other than confusing
people as to why trailers sometimes work and sometimes don't.

Best regards,
Piotr Sikora