{"id":2670,"date":"2013-04-17T11:56:08","date_gmt":"2013-04-17T11:56:08","guid":{"rendered":"http:\/\/blog.esds.co.in\/?p=2670"},"modified":"2021-03-12T09:30:42","modified_gmt":"2021-03-12T09:30:42","slug":"what-is-the-difference-between-robot-spider-and-crawler","status":"publish","type":"post","link":"https:\/\/www.esds.co.in\/blog\/what-is-the-difference-between-robot-spider-and-crawler\/","title":{"rendered":"What Is The Difference Between Robot, Spider, And Crawler"},"content":{"rendered":"\n<h2 class=\"has-text-align-center wp-block-heading\"><span class=\"ez-toc-section\" id=\"Difference_between_a_spider_crawler_and_robots\"><\/span>Difference between a spider, crawler, and robots<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large\"><img decoding=\"async\" src=\"http:\/\/www.esds.co.in\/blog\/wp-content\/uploads\/2013\/04\/google-spider-bots.jpg\" alt=\"google-spider-bots\"\/><\/figure><\/div>\n\n\n\n<p style=\"text-align: justify;\">Increasingly, the sites are modernizing and trying to keep up on top of search results. However, you need to invest in technology to achieve better positioning. Due to the considerable increase of material available on the web, it is essential to determine its existence so as to remain competitive. A site that is ranking in the search will surely be benefited.<\/p><div id=\"ez-toc-container\" class=\"ez-toc-v2_0_76 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.esds.co.in\/blog\/what-is-the-difference-between-robot-spider-and-crawler\/#Difference_between_a_spider_crawler_and_robots\" >Difference between a spider, crawler, and robots<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.esds.co.in\/blog\/what-is-the-difference-between-robot-spider-and-crawler\/#Crawler\" >Crawler<\/a><\/li><\/ul><\/nav><\/div>\n\n\n\n\n<p><strong>As a definition, we have:<\/strong><\/p>\n\n\n\n<h2 class=\"has-text-align-center wp-block-heading\"><span class=\"ez-toc-section\" id=\"Crawler\"><\/span><strong>Crawler<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p style=\"text-align: justify;\">Also known as Robot, Bot, or Spider. These are programs used by search engines to explore the Internet and automatically download web content available on websites. They capture the text of the pages and the links found and thus enable search engine users to find new pages. Methodically, it exposes content and deems irrelevant content in the source code of sites, and stores the rest in the database. It is software developed to perform a scan on the internet in a systematic manner through information perceived as relevant to their function. One of the bases of the Search Engines, they are responsible for the indexing of websites and storing them in the database of search engines.<\/p>\n\n\n\n<p style=\"text-align: justify;\">The process that executes a web crawler is called Web crawling or spidering. Many sites, in particular search engines, use crawlers to maintain an updated database. Web crawlers are mainly used to create a copy of all the visited pages for post-processing by a search engine that will index the downloaded pages to provide faster searches. Crawlers can also be used for automated maintenance tasks on a website, such as checking links or validating HTML code. The crawlers can also be used to obtain specific types of information from Web pages, such as mining addresses emails (most commonly for spam).<\/p>\n\n\n\n<p style=\"text-align: justify;\">The search engine crawlers generally seek information about permissions on the content. There are two ways to block a decent crawler from indexing a particular page (and the links contained therein). The first, and most common, is through the robots.txt file. The other way is through the meta robots tag with the value &#8220;index&#8221; or &#8220;no follow&#8221;, used to not index (the page itself) and not below (the links in the page), respectively. There is also a third possibility, much less exploited, which is using the rel = &#8220;nofollow&#8221; for links, indicating that the link, in particular, should not be followed.<\/p>\n\n\n\n<p class=\"has-text-align-center\"><strong>The Robots Perform In Three Basic Actions:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>First they find the pages of the site (process called crawling or spidering) and build a list of words and phrases found in every page;<\/li><li>With this list they create a database and find the exact pages they should seek by entering the query in search option and the database organized by general features found in its pages. The machine enters the site in general database is called indexer;<\/li><li>After that, the robot is able to find the site when the end user type a word or phrase. This step is called query processor.<\/li><\/ul>\n\n\n\n<p style=\"text-align: justify;\">As we can see, behind any search performed on the internet, there are a number of mechanisms that work together to provide a satisfactory result to the user. The process seems somewhat complex, however, nothing noticeable to us mere information seekers.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-full\"><a href=\"https:\/\/www.esds.co.in\/\"><img loading=\"lazy\" decoding=\"async\" width=\"1257\" height=\"598\" src=\"https:\/\/www.esds.co.in\/blog\/wp-content\/uploads\/2021\/02\/ESDS-1.jpg\" alt=\"ESDS 1\" class=\"wp-image-11843\" srcset=\"https:\/\/www.esds.co.in\/blog\/wp-content\/uploads\/2021\/02\/ESDS-1.jpg 1257w, https:\/\/www.esds.co.in\/blog\/wp-content\/uploads\/2021\/02\/ESDS-1-300x143.jpg 300w, https:\/\/www.esds.co.in\/blog\/wp-content\/uploads\/2021\/02\/ESDS-1-1024x487.jpg 1024w, https:\/\/www.esds.co.in\/blog\/wp-content\/uploads\/2021\/02\/ESDS-1-150x71.jpg 150w, https:\/\/www.esds.co.in\/blog\/wp-content\/uploads\/2021\/02\/ESDS-1-660x314.jpg 660w\" sizes=\"auto, (max-width: 1257px) 100vw, 1257px\" \/><\/a><\/figure><\/div>\n","protected":false},"excerpt":{"rendered":"<p>Difference between a spider, crawler, and robots Increasingly, the sites are modernizing and trying to keep up on top of search results. However, you need to invest in technology to achieve better positioning. Due to the considerable increase of material available on the web, it is essential to determine its existence so as to remain&#8230; <\/p>\n<div class=\"clear\"><\/div>\n<p><a href=\"https:\/\/www.esds.co.in\/blog\/what-is-the-difference-between-robot-spider-and-crawler\/\" class=\"gdlr-button small excerpt-read-more\">Read More<\/a><\/p>\n","protected":false},"author":5,"featured_media":11843,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[187],"tags":[922,1348,434,921,924,923],"class_list":["post-2670","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-general","tag-crawler","tag-google","tag-google-bots","tag-roobot","tag-search-engine","tag-website"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.esds.co.in\/blog\/wp-json\/wp\/v2\/posts\/2670","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.esds.co.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.esds.co.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.esds.co.in\/blog\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/www.esds.co.in\/blog\/wp-json\/wp\/v2\/comments?post=2670"}],"version-history":[{"count":7,"href":"https:\/\/www.esds.co.in\/blog\/wp-json\/wp\/v2\/posts\/2670\/revisions"}],"predecessor-version":[{"id":11933,"href":"https:\/\/www.esds.co.in\/blog\/wp-json\/wp\/v2\/posts\/2670\/revisions\/11933"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.esds.co.in\/blog\/wp-json\/wp\/v2\/media\/11843"}],"wp:attachment":[{"href":"https:\/\/www.esds.co.in\/blog\/wp-json\/wp\/v2\/media?parent=2670"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.esds.co.in\/blog\/wp-json\/wp\/v2\/categories?post=2670"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.esds.co.in\/blog\/wp-json\/wp\/v2\/tags?post=2670"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}