nginx fastcgi_cache not freeing deleted items causing excessive disk usage

[Maxim Dounin]

Hello!

On Sun, Jun 05, 2016 at 11:32:35AM -0400, b4a456fb-1402 wrote:

> Hello,
> 
> I'm using nginx 1.10.0 and the nginx_fastcgi_cache option. I've noticed that
> with a high amount of requests per second (I'm not sure when it occurs
> exactly, but we have ~2500RPS with about 2Gbit/s of outgoing traffic
> responses) there is an issue with the fastcgi_cache files not being freed.
> lsof shows a huge amount of files as (deleted) but the space it not being
> freed. Eventually the entire partition fills up like this.
> 
> The keys_zone and max_size are both set to 500m
> The fastcgi_cache_valid is set to 1m
> 
> Even when the server is idle for a while (30+ minutes) the space is not
> being freed. Obviously restarting nginx manually immediately frees the
> space.
> 
> Is this a bug or is there a setting I can use to perform some sort of
> garbage collection?

The fact that a file is shown as "deleted" indicates that it was 
deleted but is still open.  This may happen due to legitimate 
reasons - e.g., the file was deleted, but is current being served 
to a client and hence it is still open.  But if you see no changes 
with 30+ minutes without load, there is likely a problem 
somewhere.  You may try debugging it further.

Some basic things to consider:

- Make sure you are not using open_file_cache with some insane 
  parameters.  If you use open_file_cache, comment it out and check 
  if you are still able to reproduce the problem.

- Check if "nginx -V" shows any 3rd party modules.  If it does, 
  try reproducing the problem without them.

- When leaving the server idle, wait to make sure all client 
  connections are actually closed on socket level by clients - 
  that is, no connections in the ESTABLISHED state.  Check lsof 
  when this happens.

-- 
Maxim Dounin
http://nginx.org/