Sporadic long response times with upstream server

[joncard]

This is similar to a previous question, but my log data shows the opposite
problem. I am seeing rare requests that take perhaps 3s or more, but typical
response times are 100ms or less. This is the log entry for one of the
problematic responses:

LOAD_BALANCER_IP - - [20/Feb/2019:13:36:12 +0000] "POST /DevicePost
HTTP/1.1" 200 16 "-" "-" "client-ip-redacted" 3.052 0.002 .

Here is the log format:

log_format timed_main '$remote_addr - $remote_user [$time_local] "$request"
'
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" '
                      '$request_time $upstream_response_time $pipe';

The 0.002 matches what the Node.js logs are reporting from the upstream
server, and 3.052 matches what the Elastic Load Balancer is reporting as the
response time is recorded. This was set up by AWS Elastic Beanstalk as the
(mostly) default configuration for a Node.js application. The only changes
were for this logging entry, and manually setting the Content-Type header
(the client is an embedded device and their HTTP library doesn't include
this header, which creates problems for Node.js Express).

I think this indicates a delay in Node.js, but it is sporadic and I cannot
tell if it is some kind of garbage collection, a caching problem (there
should be no caching), or something else. Any help would be appreciated.

(I'm sorry I don't know how to format the quotes. I can't find any
instructions on what the admin called "quoting conventions typically used on
mailing lists" in the post "Please Read Before Posting in this Forum").

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,283099,283099#msg-283099