HTTP 400 responses

[Pauls V]

Hello,

Problem: Nginx returns 400 on small amount of POST requests (1-5 per minute
from 10k POST requests)
Setup: Nginx in front of Tomcat
Version: 1.8.0 (upgraded from 1.4.6 that had the same problem)
Log format: log_format  main  $remote_addr - $remote_user [$time_local]
$request $status $request_length $body_bytes_sent $request_time
$http_referer $http_user_agent $http_x_forwarded_for $upstream_response_time
$upstream_status;
Main settings:
http {
    proxy_buffers 8 10k;
    proxy_buffer_size 10k;
    large_client_header_buffers 8 10k;
    keepalive_timeout  65;
}
server {
    listen       80;
    server_name  localhost;

    gzip             on;
    gzip_comp_level  1;
    gzip_min_length  50;
    gzip_proxied     any;
    gzip_types       text/css application/javascript application/json
application/xml text/plain;

    location /json/ {
       proxy_pass http://127.0.0.1:8080/tomcat/webApp/;
    }
}


>From access log:
22.22.22.22 - - [19/Oct/2015:10:12:42 +0000] POST /json/some.json HTTP/1.1
400 430 0 0.019  curl/7.35.0 11.11.11.11 - -
22.22.22.22 - - [19/Oct/2015:10:13:32 +0000] POST /json/some.json HTTP/1.1
400 256 0 59.251 - - 11.11.11.11 - -
22.22.22.22 - - [19/Oct/2015:10:14:23 +0000] POST /json/some.json HTTP/1.1
400 257 0 59.711 - - 11.11.11.11 - -
22.22.22.22 - - [19/Oct/2015:10:14:35 +0000] POST /json/some.json HTTP/1.1
400 255 0 59.536 - - 11.11.11.11 - -

What I have checked:
- the log shows that reqest wasn't sent to tomcat. Tested and tomcat
processes all requests under 200ms.
- nothing is written to error_log (tried info and warn)
- the general answer of increasing large_client_header_buffers size doesn't
seem valid here as $request_length is not close to max(in case someone can
suggest a way of replicating these requests then I can test on 2nd
environment)

1. Why is Nginx replying with HTTP 400 in 0.019s for $request_length 430?
2. Why is there up to 60s $request_time? It looks like at 60s ELB kills it
and returns 5xx to client.

Thanks,
Pauls

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,262308,262308#msg-262308