{"id":4713,"date":"2017-09-09T03:31:51","date_gmt":"2017-09-08T19:31:51","guid":{"rendered":"https:\/\/www.darknet.org.uk\/?p=4713"},"modified":"2017-09-09T03:32:12","modified_gmt":"2017-09-08T19:32:12","slug":"dcrawl-web-crawler-unique-domains","status":"publish","type":"post","link":"https:\/\/www.darknet.org.uk\/2017\/09\/dcrawl-web-crawler-unique-domains\/","title":{"rendered":"dcrawl &#8211; Web Crawler For Unique Domains"},"content":{"rendered":"<p>dcrawl is a simple, but smart, multithreaded web crawler for randomly gathering huge lists of unique domain names.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.darknet.org.uk\/wp-content\/uploads\/2017\/09\/dcrawl-Web-Crawler-For-Unique-Domains-640x418.png\" alt=\"dcrawl - Web Crawler For Unique Domains\" width=\"640\" height=\"418\" class=\"aligncenter size-medium wp-image-4714\" srcset=\"https:\/\/www.darknet.org.uk\/wp-content\/uploads\/2017\/09\/dcrawl-Web-Crawler-For-Unique-Domains-640x418.png 640w, https:\/\/www.darknet.org.uk\/wp-content\/uploads\/2017\/09\/dcrawl-Web-Crawler-For-Unique-Domains.png 740w\" sizes=\"(max-width: 640px) 100vw, 640px\" \/><\/p>\n<script async src=\"https:\/\/pagead2.googlesyndication.com\/pagead\/js\/adsbygoogle.js?client=ca-pub-3033787195489589\"\r\n     crossorigin=\"anonymous\"><\/script>\r\n<!-- Darknet Responsive Post Ad Links [previously link ad unit] -->\r\n<ins class=\"adsbygoogle\"\r\n     style=\"display:block\"\r\n     data-ad-client=\"ca-pub-3033787195489589\"\r\n     data-ad-slot=\"5456620019\"\r\n     data-ad-format=\"auto\"><\/ins>\r\n<script>\r\n     (adsbygoogle = window.adsbygoogle || []).push({});\r\n<\/script>\n<h2>How does dcrawl work?<\/h2>\n<p>dcrawl takes one site URL as input and detects all <code>a href=<\/code> links in the site&#8217;s body. Each found link is put into the queue. Successively, each queued link is crawled in the same way, branching out to more URLs found in links on each site&#8217;s body.<\/p>\n<h3>dcrawl Web Crawler Features<\/h3>\n<ul>\n<li>Branching out only to predefined number of links found per one hostname.<\/li>\n<li>Maximum number of allowed different hostnames per one domain (avoids subdomain crawling hell e.g. blogspot.com).<\/li>\n<li>Can be restarted with same list of domains &#8211; last saved domains are added to the URL queue.<\/li>\n<li>Crawls only sites that return text\/html Content-Type in HEAD response.<\/li>\n<li>Retrieves site body of maximum 1MB size.<\/li>\n<li>Does not save inaccessible domains.<\/li>\n<\/ul>\n<h2>dcrawl Usage<\/h3>\n<pre>    ___                          __\r\n  __| _\/________________ __  _  _|  |\r\n \/ __ |\/ ___\\_  __ \\__  \\\\ \\\/ \\\/ \/  |\r\n\/ \/_\/ \\  \\___|  | \\\/\/ __ \\\\     \/|  |__\r\n\\____ |\\___  >__|  (____  \/\\\/\\_\/ |____\/\r\n     \\\/    \\\/           \\\/       v.1.0\r\n\r\nusage: dcrawl -url URL -out OUTPUT_FILE -t THREADS\r\n\r\n  -ms int\r\n        maximum different subdomains for one domain (def. 10) (default 10)\r\n  -mu int\r\n        maximum number of links to spider per hostname (def. 5) (default 5)\r\n  -out string\r\n        output file to save hostnames to\r\n  -t int\r\n        number of concurrent threads (def. 8) (default 8)\r\n  -url string\r\n        URL to start scraping from\r\n  -v bool\r\n        verbose (default false)<\/pre>\n<script async src=\"https:\/\/pagead2.googlesyndication.com\/pagead\/js\/adsbygoogle.js?client=ca-pub-3033787195489589\"\r\n     crossorigin=\"anonymous\"><\/script>\r\n<!-- Darknet Responsive Post Ad Links [previously link ad unit] -->\r\n<ins class=\"adsbygoogle\"\r\n     style=\"display:block\"\r\n     data-ad-client=\"ca-pub-3033787195489589\"\r\n     data-ad-slot=\"5456620019\"\r\n     data-ad-format=\"auto\"><\/ins>\r\n<script>\r\n     (adsbygoogle = window.adsbygoogle || []).push({});\r\n<\/script>\n<p>Example:<\/p>\n<pre>go build dcrawl.go\r\n.\/dcrawl -url http:\/\/wired.com -out ~\/domain_lists\/domains1.txt -t 8<\/pre>\n<p>There are other tools which do similar things, or could be scripted together recursively to perform a similar kind of task &#8211; but nothing this focused. Examples would be:<\/p>\n<p>&#8211; <a href=\"https:\/\/www.darknet.org.uk\/2014\/05\/host-extract-enumerate-iphost-patterns-web-page\/\">Host-Extract \u2013 Enumerate All IP\/Host Patterns In A Web Page<\/a><br \/>\n&#8211; <a href=\"https:\/\/www.darknet.org.uk\/2016\/04\/recon-ng-web-reconnaissance-framework\/\">Recon-ng \u2013 Web Reconnaissance Framework<\/a><\/p>\n<p>You can download dcrawl go web crawler here:<\/p>\n<p><a href=\"https:\/\/github.com\/kgretzky\/dcrawl\/archive\/master.zip\">dcrawl-master.zip<\/a><\/p>\n<p>Or read more <a href=\"https:\/\/github.com\/kgretzky\/dcrawl\">here<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>dcrawl is a simple, but smart, multithreaded web crawler for randomly gathering huge lists of unique domain names. How does dcrawl work? dcrawl takes one site URL as input and detects all a href= links in the site&#8217;s body. Each found link is put into the queue. Successively, each queued link is crawled in the [&hellip;]<\/p>\n","protected":false},"author":25,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_seopress_robots_primary_cat":"","_seopress_titles_title":"","_seopress_titles_desc":"dcrawl is a simple, but smart, multithreaded web crawler for randomly gathering huge lists of unique domain names. It will branch out indefinitely.","_seopress_robots_index":"","_genesis_hide_title":false,"_genesis_hide_breadcrumbs":false,"_genesis_hide_singular_image":false,"_genesis_hide_footer_widgets":false,"_genesis_custom_body_class":"","_genesis_custom_post_class":"","_genesis_layout":"","footnotes":""},"categories":[9],"tags":[8140,2345,3105],"featured_image_src":null,"featured_image_src_square":null,"author_info":{"display_name":"Darknet","author_link":"https:\/\/www.darknet.org.uk\/author\/darknet\/"},"_links":{"self":[{"href":"https:\/\/www.darknet.org.uk\/wp-json\/wp\/v2\/posts\/4713"}],"collection":[{"href":"https:\/\/www.darknet.org.uk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.darknet.org.uk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.darknet.org.uk\/wp-json\/wp\/v2\/users\/25"}],"replies":[{"embeddable":true,"href":"https:\/\/www.darknet.org.uk\/wp-json\/wp\/v2\/comments?post=4713"}],"version-history":[{"count":0,"href":"https:\/\/www.darknet.org.uk\/wp-json\/wp\/v2\/posts\/4713\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.darknet.org.uk\/wp-json\/wp\/v2\/media?parent=4713"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.darknet.org.uk\/wp-json\/wp\/v2\/categories?post=4713"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.darknet.org.uk\/wp-json\/wp\/v2\/tags?post=4713"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}