nginx fastcgi_cache not freeing deleted items causing excessive disk usage
mdounin at mdounin.ru
Mon Jun 6 10:45:29 UTC 2016
On Sun, Jun 05, 2016 at 11:32:35AM -0400, b4a456fb-1402 wrote:
> I'm using nginx 1.10.0 and the nginx_fastcgi_cache option. I've noticed that
> with a high amount of requests per second (I'm not sure when it occurs
> exactly, but we have ~2500RPS with about 2Gbit/s of outgoing traffic
> responses) there is an issue with the fastcgi_cache files not being freed.
> lsof shows a huge amount of files as (deleted) but the space it not being
> freed. Eventually the entire partition fills up like this.
> The keys_zone and max_size are both set to 500m
> The fastcgi_cache_valid is set to 1m
> Even when the server is idle for a while (30+ minutes) the space is not
> being freed. Obviously restarting nginx manually immediately frees the
> Is this a bug or is there a setting I can use to perform some sort of
> garbage collection?
The fact that a file is shown as "deleted" indicates that it was
deleted but is still open. This may happen due to legitimate
reasons - e.g., the file was deleted, but is current being served
to a client and hence it is still open. But if you see no changes
with 30+ minutes without load, there is likely a problem
somewhere. You may try debugging it further.
Some basic things to consider:
- Make sure you are not using open_file_cache with some insane
parameters. If you use open_file_cache, comment it out and check
if you are still able to reproduce the problem.
- Check if "nginx -V" shows any 3rd party modules. If it does,
try reproducing the problem without them.
- When leaving the server idle, wait to make sure all client
connections are actually closed on socket level by clients -
that is, no connections in the ESTABLISHED state. Check lsof
when this happens.
More information about the nginx