File robots.txt
Download: http://nmap.org/svn/scripts/robots.txt.nse
User Summary
Checks for disallowed entries in robots.txt.
The higher the verbosity or debug level, the more disallowed entries are shown.
Script Arguments
http-max-cache-size, http.useragent, pipeline
See the documentation for the http library.Example Usage
nmap -sV -sC <target>
Script Output
80/tcp open http syn-ack | robots.txt: has 156 disallowed entries (40 shown) | /news?output=xhtml& /search /groups /images /catalogs | /catalogues /news /nwshp /news?btcid=*& /news?btaid=*& | /setnewsprefs? /index.html? /? /addurl/image? /pagead/ /relpage/ | /relcontent /sorry/ /imgres /keyword/ /u/ /univ/ /cobrand /custom | /advanced_group_search /googlesite /preferences /setprefs /swr /url /default | /m? /m/? /m/lcb /m/news? /m/setnewsprefs? /m/search? /wml? |_ /wml/? /wml/search?
Requires
Categories: default discovery safe
Author: Eddie Bell
License: Same as Nmap--See http://nmap.org/book/man-legal.html