{"id":10186,"date":"2018-05-31T11:00:54","date_gmt":"2018-05-31T11:00:54","guid":{"rendered":"http:\/\/www.esds.co.in\/blog\/?p=10186"},"modified":"2020-01-07T10:42:07","modified_gmt":"2020-01-07T10:42:07","slug":"link-crawling","status":"publish","type":"post","link":"https:\/\/www.esds.co.in\/blog\/link-crawling\/","title":{"rendered":"Link Crawling"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Introduction\"><\/span><strong>Introduction<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p><p style=\"text-align: justify;\">Link crawling is the process of capturing all the pages (URLs\/links) present on a website. It helps us know how many and which pages are there in our website. Site owners can also crosscheck whether these page are legitimate or not.<\/p><div id=\"ez-toc-container\" class=\"ez-toc-v2_0_76 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.esds.co.in\/blog\/link-crawling\/#Introduction\" >Introduction<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.esds.co.in\/blog\/link-crawling\/#ESDS_VTMScan_Technique\" >ESDS VTMScan Technique<\/a><\/li><\/ul><\/nav><\/div>\n\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"ESDS_VTMScan_Technique\"><\/span>ESDS <strong>VTMScan Technique<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p><p style=\"text-align: justify;\">We crawl all the pages which are present on a website by actually visiting the pages and links which are present in the view source of the page. In crawling we capture URLs\/Pages by following methods:<\/p>\n\n\n\n<p><p style=\"text-align: justify;\"><strong>Page Visit:<\/strong>&nbsp;We visit each and every page and links present on them via this recursive process. If we get 200, 403 response then we add this in URL file.<\/p>\n\n\n\n<p><p style=\"text-align: justify;\"><strong>Admin Buster:<\/strong>&nbsp;We append admin page keywords such as index.EXT, admin.EXT, login.htm, login\/, login.EXT, login\/login.EXT, adm\/, siteadmin.EXT, etc. After scanning the URL for e.g. http:\/\/demotest.com\/admin.php if we get 200, 403 response then we add this in URL file.<\/p>\n\n\n\n<p><p style=\"text-align: justify;\"><strong>Directory Buster:<\/strong>&nbsp;We append admin page keywords such as index, images, wimages, imgs, img, iconset, icons, home, etc. After scanning the URL for e.g. http:\/\/demotest.com\/admin if we get 200, 403 response then we add this in URL file.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Introduction Link crawling is the process of capturing all the pages (URLs\/links) present on a website. It helps us know how many and which pages are there in our website. Site owners can also crosscheck whether these page are legitimate or not. ESDS VTMScan Technique We crawl all the pages which are present on a&#8230; <\/p>\n<div class=\"clear\"><\/div>\n<p><a href=\"https:\/\/www.esds.co.in\/blog\/link-crawling\/\" class=\"gdlr-button small excerpt-read-more\">Read More<\/a><\/p>\n","protected":false},"author":81,"featured_media":10191,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[1271],"tags":[1914,1943,1985,1984,1939],"class_list":["post-10186","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-security-2","tag-esds-vtmscan","tag-esds-vtmscan-vulnerability-scanner","tag-esds-vtmscanner","tag-link-crawling","tag-vtm-scanner"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.esds.co.in\/blog\/wp-json\/wp\/v2\/posts\/10186","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.esds.co.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.esds.co.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.esds.co.in\/blog\/wp-json\/wp\/v2\/users\/81"}],"replies":[{"embeddable":true,"href":"https:\/\/www.esds.co.in\/blog\/wp-json\/wp\/v2\/comments?post=10186"}],"version-history":[{"count":3,"href":"https:\/\/www.esds.co.in\/blog\/wp-json\/wp\/v2\/posts\/10186\/revisions"}],"predecessor-version":[{"id":11160,"href":"https:\/\/www.esds.co.in\/blog\/wp-json\/wp\/v2\/posts\/10186\/revisions\/11160"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.esds.co.in\/blog\/wp-json\/wp\/v2\/media\/10191"}],"wp:attachment":[{"href":"https:\/\/www.esds.co.in\/blog\/wp-json\/wp\/v2\/media?parent=10186"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.esds.co.in\/blog\/wp-json\/wp\/v2\/categories?post=10186"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.esds.co.in\/blog\/wp-json\/wp\/v2\/tags?post=10186"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}