I use nginx maps which depending on user agent either block, rate limit or whitelist https://community.centminmod.com/threads/blocking-bad-or-aggressive-bots.6433/ as the list gets large nginx maps just make it easier to manage Posted at Nginx Forum: https://forum.nginx.org/read.php?2,270930,270940#msg-270940