# robots.txt file for http://mywebsite.co.uk/
 
#User-agent: mirago
#Disallow:
 
#User-agent: naughtyrobot
#Disallow: /
 
User-agent: *
Disallow: /~
Disallow: /images
Disallow: /css
Disallow: /javascript


# The first line, starting with '#', specifies a comment.

# The next two lines specifies that the Mirago robot has nothing disallowed. This means permission is granted to go anywhere on that site. This is optional, as a robot will assume it has permission to access your site if it is not excluded by any 'disallow' directives.

# The next two lines indicates that the robot called 'naughtyrobot' has all relative URLs starting with '/' disallowed. As all relative URL's on a server start with '/', this means the entire site should not be accessed by the robot. N.B. Don't put more than one path on a Disallow line.

# The third paragraph indicates that all other robots should not visit URLs starting with /stay_out or /devproject. It should be noted that the '*' is a special token meaning 'all robots' and is not a regular expression. Instead of 'Disallow: /myproject/*' just put 'Disallow: /myproject'.