<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><font face="Monaco" size="4">I’m having trouble with an nginx setup built to serve search engines. </font><div><font face="Monaco" size="4"><br></font></div><div><font face="Monaco" size="4">Based on the user agent, all bots are served only from cache. We populate the cache with our own set of spiders so we can control the overall load.</font></div><div><font face="Monaco" size="4"><br></font></div><div><font face="Monaco" size="4">Total cache size is ~450 GB in ~12 million files.</font></div><div><font face="Monaco" size="4"><br></font></div><div><font face="Monaco" size="4">The problem is that about 1/3 of the requests coming in live from the bots are misses, even though the requested page was requested by our spider a mere hour previously.</font></div><div><font face="Monaco" size="4"><br></font></div><div><font face="Monaco" size="4">Configured limits should be safe:</font></div><div><font face="Monaco" size="4"><br></font></div><div><div style="margin: 0px;"><font face="Monaco" size="4"> proxy_cache_path /var/www/cache levels=1:2 keys_zone=my-cache:2500m max_size=800000m inactive=800h;</font></div></div><div style="margin: 0px;"><font face="Monaco" size="4"><br></font></div><div style="margin: 0px;"><font face="Monaco" size="4">Where should I be looking for why these requests were misses?</font></div><div style="margin: 0px;"><font face="Monaco" size="4"><br></font></div><div style="margin: 0px;"><font face="Monaco" size="4">Thanks,</font></div><div style="margin: 0px;"><font face="Monaco" size="4"><br></font></div><div style="margin: 0px;"><font face="Monaco" size="4">— David</font></div></body></html>