dcrawl – Web Crawler For Unique Domains

Outsmart Malicious Hackers


dcrawl is a simple, but smart, multithreaded web crawler for randomly gathering huge lists of unique domain names.

dcrawl - Web Crawler For Unique Domains


How does dcrawl work?

dcrawl takes one site URL as input and detects all a href= links in the site’s body. Each found link is put into the queue. Successively, each queued link is crawled in the same way, branching out to more URLs found in links on each site’s body.

dcrawl Web Crawler Features

  • Branching out only to predefined number of links found per one hostname.
  • Maximum number of allowed different hostnames per one domain (avoids subdomain crawling hell e.g. blogspot.com).
  • Can be restarted with same list of domains – last saved domains are added to the URL queue.
  • Crawls only sites that return text/html Content-Type in HEAD response.
  • Retrieves site body of maximum 1MB size.
  • Does not save inaccessible domains.

dcrawl Usage


Example:

There are other tools which do similar things, or could be scripted together recursively to perform a similar kind of task – but nothing this focused. Examples would be:

Host-Extract – Enumerate All IP/Host Patterns In A Web Page
Recon-ng – Web Reconnaissance Framework

You can download dcrawl go web crawler here:

dcrawl-master.zip

Or read more here.

Posted in: Hacking Tools

, ,


Latest Posts:


Terabytes Of US Military Social Media Spying S3 Data Exposed Terabytes Of US Military Social Media Spying S3 Data Exposed
Once again the old, default Amazon AWS S3 settings are catching people out, the US Military has left terabytes of social media spying S3 data exposed.
SNIFFlab - Create Your Own MITM Test Environment SNIFFlab – Create Your Own MITM Test Environment
SNIFFlab is a set of scripts in Python that enable you to create your own MITM test environment for packet sniffing through a WiFi access point.
Skype Log Viewer Download - View Logs on Windows Skype Log Viewer Download – View Logs on Windows
Skype Log Viewer allows you to download and view the Skype history and log files, on Windows, without actually downloading the Skype client itself.
Ethereum Parity Bug Destroys Over $250 Million In Tokens Ethereum Parity Bug Destroys Over $250 Million In Tokens
If you are into cryptocurrency or blockchain at all, you will have heard about the Ethereum Parity Bug that has basically binned $280 Million + ETH.
WPSeku - Black-Box Remote WordPress Security Scanner WPSeku – Black-Box Remote WordPress Security Scanner
WPSeku is a black box WordPress Security scanner that can be used to scan remote WordPress installations to find security issues and vulnerabilities.
Malaysia Telco Hack - Corporations Spill 46 Million Records Malaysia Telco Hack – Corporations Spill 46 Million Records
The Malaysia Telco Hack has been blowing up in the news with over 42 Million Records being leaked including IMEI numbers, SIM details and home addresses.


Comments are closed.