Beginners Guide To Robots txt Files For SEO class video tutorials 2019

How does robots.txt work?

Search engines have two main jobs:

Crawling the web to discover content;
Indexing that content so that it can be served up to searchers who are looking for information.

Robots.text file

Robots, Crawlers, Spiders etc are all recognized with the same. These crawlers indexes our website in their search engine databases. Google robots are called as Googlebot, Yahoo robots are called as Yahoo Slurp and MSN robots are called as MSN bot.

robots.txt is a text file which is created at the time of creating a website. This name is case sensitive, maintain all the letters to be lower case.

We give the instruction to the search engine through this file robots.txt file. robots.txt file guides the search engines whether to include or exclude the particular file or directory in its database. We create this file to hide the sensitive information from displaying them in the search engine results.

We place this file robots.txt in the root directory. That means we place this file besides the index file of our website

User-agent: *
Allow: /
Sitemap: https://www.----.com/sitemap.xml

Disallow :

User-agent: [user-agent name]
Disallow: [URL string not to be crawled]

Website Link

Images

Image Folders

how to check robots.txt - mydomain.com/robots.txt