Today's lesson: robots.txt
Posted on 26/09/05 23:00
What happens when Google indexes your personal photo folder on your web space?
Something along the lines of this, maybe?
This brings us to the question of how do we prevent Google from finding this crap? one little astonishingly easy thing we could do is to use robots.txt. Being the honorable little critters they are, web spiders (the computers that find everything for Google) will never visit a link that's included in a file called robots.txt, placed in the root of your domain (the root for my domain would be http://obli.net/).
The contents of a robots.txt restricting access to the folder http://obli.net/nope/ would look like this and be placed in http://obli.net/robots.txt:
User-agent: *
Disallow: nope/
Also, as an extra treat, try searching for DSC0000.jpg or PICT0000.jpg on Google image search for some random personal photos, 0000 being replaced with a random sequence of numbers.
Something along the lines of this, maybe?
This brings us to the question of how do we prevent Google from finding this crap? one little astonishingly easy thing we could do is to use robots.txt. Being the honorable little critters they are, web spiders (the computers that find everything for Google) will never visit a link that's included in a file called robots.txt, placed in the root of your domain (the root for my domain would be http://obli.net/).
The contents of a robots.txt restricting access to the folder http://obli.net/nope/ would look like this and be placed in http://obli.net/robots.txt:
User-agent: *
Disallow: nope/
Also, as an extra treat, try searching for DSC0000.jpg or PICT0000.jpg on Google image search for some random personal photos, 0000 being replaced with a random sequence of numbers.
