dcrawl – Web Crawler For Unique Domains

Outsmart Malicious Hackers


dcrawl is a simple, but smart, multithreaded web crawler for randomly gathering huge lists of unique domain names.

dcrawl - Web Crawler For Unique Domains


How does dcrawl work?

dcrawl takes one site URL as input and detects all a href= links in the site’s body. Each found link is put into the queue. Successively, each queued link is crawled in the same way, branching out to more URLs found in links on each site’s body.

dcrawl Web Crawler Features

  • Branching out only to predefined number of links found per one hostname.
  • Maximum number of allowed different hostnames per one domain (avoids subdomain crawling hell e.g. blogspot.com).
  • Can be restarted with same list of domains – last saved domains are added to the URL queue.
  • Crawls only sites that return text/html Content-Type in HEAD response.
  • Retrieves site body of maximum 1MB size.
  • Does not save inaccessible domains.

dcrawl Usage


Example:

There are other tools which do similar things, or could be scripted together recursively to perform a similar kind of task – but nothing this focused. Examples would be:

Host-Extract – Enumerate All IP/Host Patterns In A Web Page
Recon-ng – Web Reconnaissance Framework

You can download dcrawl go web crawler here:

dcrawl-master.zip

Or read more here.

Posted in: Hacking Tools

, ,


Latest Posts:


CloudFrunt - Identify Misconfigured CloudFront Domains CloudFrunt – Identify Misconfigured CloudFront Domains
CloudFrunt is a Python-based tool for identifying misconfigured CloudFront domains, it uses DNS and looks for CNAMEs which may be allowed to be associated with CloudFront distributions.
Airbash - Fully Automated WPA PSK Handshake Capture Script Airbash – Fully Automated WPA PSK Handshake Capture Script
Airbash is a POSIX-compliant, fully automated WPA PSK handshake capture script aimed at penetration testing, it is compatible with Bash and Android Shell.
XXEinjector - Automatic XXE Injection Tool For Exploitation XXEinjector – Automatic XXE Injection Tool For Exploitation
XXEinjector is an XXE Injection Tool that automates retrieving files using direct and out of band methods. Directory listing only works in Java applications.
Yahoo! Fined 35 Million USD For Late Disclosure Of Hack Yahoo! Fined 35 Million USD For Late Disclosure Of Hack
Ah Yahoo! in trouble again, this time the news is Yahoo! fined for 35 million USD by the SEC for the 2 year delayed disclosure of the massive hack, we actually reported on the incident in 2016 when it became public.
Drupwn - Drupal Enumeration Tool & Security Scanner Drupwn – Drupal Enumeration Tool & Security Scanner
Drupwn is a Python-based Drupal Enumeration Tool that also includes an exploit mode, which can check for and exploit relevant CVEs.
MyEtherWallet DNS Hack Causes 17 Million USD User Loss MyEtherWallet DNS Hack Causes 17 Million USD User Loss
Big news in the crypto scene this week was that the MyEtherWallet DNS Hack that occured managed to collect about $17 Million USD worth of Ethereum in just a few hours.


Comments are closed.