Failed disk + proxy_intercept_errors

[chocholo3]

Hi,
In our deployment we do have configuration of proxy cache with multiple hard
drives. Because of performance we don't have any RAID on these devices. That
means we have to handle even a situation when drive dies, sometime.

After disk failure of proxy_cache_path device nginx usually starts serving
users with http500. So I've had an idea we may use proxy_intercept_errors
but I end up with inconsistent state: ~60 files are handled as expected, but
after that every connection is terminated prematurely without a single byte
sent. In access.log there is http 200.

I broke just ext4 FS (dd if=/dev/zero of=/dev/sdc bs=1k count=$((1024*100)))
and I'm using nginx 1.17.7 on Linux

Relevant snippet from my configuration:
```
location ~ ^/mylocation/ {
        set  $spindle_bucket  cache_01;
        include "snippets/spindle_cache_locations_uspinclude";
        proxy_cache             DISK_$spindle_bucket;
             
        proxy_pass              $backend;
        proxy_cache_key         $uri;
        proxy_cache_revalidate  on;
        proxy_cache_use_stale   off;

        recursive_error_pages on;
        proxy_intercept_errors  on;
        error_page 400 401 402 403 405 406 407 408 409 410 411 412 413 414
415 416 417 418 420 422 423 424 426 428 429 431 444 449 450 451 =
@myotherlocation;
        error_page 500 501 502 503 504 505 506 507 508 510 511 =
@myotherlocation;
}

# same as ^/mylocation/ but without proxy_intercept_errors and with a single
spare drive only
location @myotherlocation {
        include "snippets/spindle_cache_locations_uspinclude";
        set                     $spindle_bucket "spare_01";
        proxy_cache             DISK_$spindle_bucket;

        proxy_pass              $backend;
        proxy_cache_key         $uri;
        proxy_cache_revalidate  on;
        proxy_cache_use_stale   error timeout invalid_header updating
http_500 http_502 http_503 http_504;
}
```

In snippets/spindle_cache_locations_uspinclude I do have:
```
        proxy_buffers           64 8k;

        proxy_set_header        Host            our.akamaized.net;

        proxy_cache_valid       200 720d;
        proxy_cache_valid       206 720d;
        #proxy_cache_valid       301 1d;
        #proxy_cache_valid       302 10m;
        #proxy_cache_valid       any 1s;
        proxy_http_version      1.1;
        proxy_set_header        Connection "";
        proxy_set_header        X-Real-IP       $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;

```
```
$ curl -v
http://myserver/mylocation/66666666-1111-1111-0000-01234567/2ab5355f-5508-438d-acfc-686469877fb3.ism/2ab5355f-5508-438d-acfc-686469877fb3-video_2=1481000-177.m4s
*   Trying myipv6...
* TCP_NODELAY set
* Connected to myserver (myipv6) port 80 (#0)
> GET
/mylocation/66666666-1111-1111-0000-01234567/2ab5355f-5508-438d-acfc-686469877fb3.ism/2ab5355f-55$
8-438d-acfc-686469877fb3-video_2=1481000-177.m4s HTTP/1.1
> Host: myserver
> User-Agent: curl/7.52.1
> Accept: */*
>
* Curl_http_done: called premature == 0
* Empty reply from server
* Connection #0 to host myserver left intact
curl: (52) Empty reply from server
```

Am I doing something wrong or is this a bug? Because of the inconsistency I
tend to the 2nd. But I'm not sure at all :-)

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,287005,287005#msg-287005