Memory usage in nginx proxy setup and use of min_uses

[Lucas Rolff]

Hi Maxim!

> - The attack you are considering is not about "poisoning".  At most, it can be used to make the cache less efficient.

Poisoning is probably the wrong word indeed, and since nginx doesn't really handle reaching the limit of keys_zone, it simply starts to return a 500 internal server error. So I don't think it's making the cache less efficient (Other than you won't be able to cache that much), you're ending up breaking nginx because when the keys_zone limit has been reached, nginx simply starts returning 500 internal server error for items that are not already in proxy_cache - if it would do an LRU/LFU on the keys - then yes, you could probably end up with a cache less efficient.

But as it stands currently, if one uses $request_uri an attacker could reach the keys_zone limit, and break all traffic that is not yet cached.
Even if one would not use $request_uri, but some specific argument where a query string that wouldn't directly affect the output, it could cause the same behavior.

An application where avoiding this is hard, is CDNs for example - while ideally one would not use $request_uri as the cache key, it's sometimes required by customer applications.

> At most, you can try to limit the number of keys an attacker will be able to put into keys_zone

If we take an example of a CDN or any kind of reverse proxy; What impact would it have to have a proxy_cache_path for each domain? Lets say we're talking 10000 domains on a single nginx server. Normally one would share one or a couple (fast/slow storage for example), and the domain would be a part of the cache key

If we want to limit (per domain), it would require a proxy_cache_path per domain - which surely would be very flexible, but I also think you're then asking nginx to do a lot more management.

> But using separate inactive timer for keys not reached min_uses won't help here: an attacker who is able to do arbitrary amount of requests will be able to flush all cache items anyway.

Unless nginx very recently implemented that reaching keys_zone limit, will start purging old cache - then no, it would still break the nginx for non-cached requests (returning 500 internal server error). If nginx has started to purge old things if the limit is reached, then sure the attacker would still be able to wipe out the cache.

But let's say we have an "inactive" set to 24+ hours (Which is often used for static files) - an attack where someone would append random query strings - those keys would first be removed after 24 hours (or higher, depending on the limit) - with a separate flag, one could set this counter to something like 60 seconds (So delete the key from memory if the key haven't reached it's min_uses within 60 seconds) - this way, you're still rotating those keys out *a lot* faster.

> In particular, this can be done with limit_req

If we'd limit this to 20 req/s, this would allow a single IP to use up 1.78 million keys in the keys_zone if "inactive" is 24 hours - do this with 10 IPs, we're at 17.8 million.
If we'd flush the keys not reaching min_uses after 1 minute, we'd limit the keys in the keys_zone per IP to 1200 - the attacker can surely keep doing his 20 requests per second, but since we're throwing out things, pretty quickly, we've decreased the "damage" a user can do from 1.78 million keys down to 1200 keys, or even 12000 keys if we'd keep it for 10 minutes.

I still think such feature would be awesome, since it would allow better control (and play nicely with the proxy_cache_min_uses directive); proxy_cache_min_uses directive is often used to prevent excessive storage due to not enough hits; Being able to do the same with the keys_zone data as well as a part of it, would (I think) benefit quite a lot. Since it would solve (or at least help mitigate) the above from happening. It makes things just a tad harder to cause troubles.

Best Regards,
Lucas Rolff

On 17/05/2021, 21.06, "nginx on behalf of Maxim Dounin" <nginx-bounces at nginx.org on behalf of mdounin at mdounin.ru> wrote:

    Hello!

    On Mon, May 17, 2021 at 02:47:33PM +0000, Lucas Rolff wrote:

    > Hi Maxim,
    > 
    > Thanks a lot for your reply!
    > 
    > I'm indeed aware of the ~8k keys per mb of memory, I was just 
    > wondering if it was handled differently when min_uses are in 
    > use, but it does indeed make sense that nginx has to keep track 
    > of it somehow, and the keys zone makes the most sense!
    > 
    > > Much like with any cache item, such keys are removed from the 
    > > keys_zone if no matching requests are seen during the 
    > > "inactive" time
    > 
    > That's a bummer, since that still allows memory "poisoning" - it 
    > would be awesome to have another flag for proxy_cache_path to 
    > control how long keys that have not yet reached min_uses are 
    > kept in SHM.
    > The benefit of this would be to say if min_uses have not been 
    > reached within let's say 5 minutes, then we purge those keys 
    > from SHM to clear up the memory.
    > 
    > For controlling the cache items - ideally we wanna use query 
    > strings as a part of the cache key, but still ideally prevent 
    > memory poisoning as above - the inactive flag for min_uses would 
    > be pretty useful for this - while it won't prevent it fully, 
    > we'd still be able to somewhat control memory even if people are 
    > trying to do the cache/memory poisoning.

    In no particular order:

    - The attack you are considering is not about "poisoning".  At 
      most, it can be used to make the cache less efficient.

    - The goal "to somewhat control memory" looks confusing: the 
      memory used by caching is hard-limited by the keys_zone size, 
      and it is not possible to use more memory than configured.  At 
      most, you can try to limit the number of keys an attacker will be 
      able to put into keys_zone.  But using separate inactive timer for 
      keys not reached min_uses won't help here: an attacker who is able 
      to do arbitrary amount of requests will be able to flush all cache 
      items anyway.

    - Using proxy_cache_min_uses cannot help here, regardless of how 
      it is handled, since nothing stops the attacker from requesting 
      the same resource multiple times.

    In general, I see two basic options to handle things if you don't 
    want one to be able to reduce your cache efficiency:

    1. Strictly limit which resources are to be cached, in particular, 
    by using appropriate proxy_cache_key, as already suggested.

    2. Limit the maximum number of requests an attacker can do, so it 
    won't be able to cause noticeable degradation of cache efficiency.  
    In particular, this can be done with limit_req 
    (http://nginx.org/r/limit_req).

    Also it is always a good idea to make sure your site works fine 
    without caching at all.

    -- 
    Maxim Dounin
    http://mdounin.ru/
    _______________________________________________
    nginx mailing list
    nginx at nginx.org
    http://mailman.nginx.org/mailman/listinfo/nginx