Robots.txt - What Is It and How Do You Use It?
Encrypting your link and protect the link from viruses, malware, thief, etc! Made your link safe to visit.
Robots.txt files are frequently mentioned every bit existence an of import foundation of a search friendly spider web site. To site owners too small-scale businesses who are novel to search marketing , the robots.txt file tin audio daunting. In reality , it's i of the fastest , simplest ways to brand your site precisely a lilliputian to a greater extent than search engine friendly.
What is Robots.txt?
Robots.txt is a unproblematic text file that sits on the server alongside your spider web site. It's basically your spider web site's way of giving instructions to search engines nigh what how they index your spider web site.
Search Engines tend to expect for the robots.txt file when they start take in a site. They tin take in too index your site whether y'all convey a robots.txt file or not; having i but helps them along the way.
All of the major search engines read too follow the instructions inward a robots.txt file. That way it's a pretty effective way to move past times along content out of the search indexes.
A discussion of warning. While precisely about sites volition state y'all to utilisation robots.txt to block premium content y'all don't desire people to run across , this isn't a proficient idea. While most search engines volition honour your robots.txt file too ignore the content y'all desire to convey blocked , a far safer selection is to enshroud that premium content behind a login. Requiring a username too password to access the content y'all desire hidden from Earth volition produce a much to a greater extent than effective undertaking of keeping both search engines too people out.
What Does Robots.txt Look Like?
The average robots.txt file is i of the simplest pieces of code you'll e'er write or edit.
If y'all desire to convey a robots.txt file for the engines to take in , but don't desire to give them whatever exceptional instructions , but opened upward up a text editor too type inward the following:
User-Agent: *
Disallow:
The "User-Agent" move specifies which search engines y'all are giving the directions to. Using the asterisk way y'all are giving directions to ALL search engines.
The "disallow" move specifies what content y'all don't desire the search engines to index. If y'all don't desire to block the search engines from whatever expanse of your spider web site , y'all but move out this expanse blank.
For most small-scale spider web sites , those 2 unproblematic lines are all y'all truly need.
If your spider web site is a lilliputian flake larger , or y'all convey a lot of folders on your server , y'all may desire to utilisation the robots.txt file to give precisely about instructions nigh which content to avoid.
A proficient instance of this would move a site that has printer-friendly versions of all of their content housed inward a folder called "print-ready." There's no argue for the search engines to index both forms of the content , then it's a proficient persuasion to become ahead too block the engines from indexing the printer-friendly versions.
In this instance , you'd move out the "user-agent" department lone , but would add together the print-ready folder to the "disallow" line. That robots.txt file would expect similar this:
User-Agent: *
Disallow: /print-ready/
It's of import to Federal Reserve annotation the forwards slashes earlier too afterwards the folder name. The search engines volition tack that folder on to the destination of the domain hollo they are visiting.
That way the /print-ready/ file is constitute at www.yourdomain.com/print-ready/. If it's truly constitute at www.yourdomain.com/css/print-ready/ you'll demand to format your robots.txt this way:
User-Agent: *
Disallow: /css/print-ready/
You tin likewise edit the "user-agent" draw to refer to specific search engines. To produce this , you'll demand to expect upward the hollo of a search engine's robot. (For instance , Google's robot is called "googlebot" too Yahoo's is called "slurp.")
If y'all desire to fix your robots.txt file to give instructions ONLY to Google , y'all would format it similar this:
User-Agent: googlebot
Disallow: /css/print-ready/
How produce I Put Robots.txt on my Site?
Once you've written your robots.txt file to reverberate the directions y'all desire to give the search engines , y'all but relieve the text file every bit "robots.txt" too upload it to the source folder of your spider web site.
It's that simple.
