Robots.txt

Although this GIAC gold paper is not about search engine optimization, or SEO, this paper will explore a key element of SEO, the robots.txt file. This file is often neglected or misunderstood by HTML designers and web server administrators. The robots.txt file will impact your page rank rating with search engine providers. Configuration errors can result in web site revenue losses, not the kind of problem you want resting on your shoulders. A mis-configured robots.txt file can also lead to information disclosure, a foothold to system compromise. A basic understanding of this simple text file can prevent e-commerce problems and security issues. Complex defense solutions may use a robots.txt file in conjunction with scripting and monitoring to thwart hackers and malicious robots by dynamically denying access to the web site or specific parts of the site. Although a robots.txt file is not a security control, the security implications will be explored in the following pages.