I have a dynamic robots.txt program on GitHub that lets me tell unwanted bots unambiguously not to scan my site. Its overall effect is minor as only the responsible bots pay attention. While scanning my logs, there are a few …

Read more