dcrawl – Web Crawler For Unique Domains

The New Acunetix V12 Engine


dcrawl is a simple, but smart, multithreaded web crawler for randomly gathering huge lists of unique domain names.

dcrawl - Web Crawler For Unique Domains


How does dcrawl work?

dcrawl takes one site URL as input and detects all a href= links in the site’s body. Each found link is put into the queue. Successively, each queued link is crawled in the same way, branching out to more URLs found in links on each site’s body.

dcrawl Web Crawler Features

  • Branching out only to predefined number of links found per one hostname.
  • Maximum number of allowed different hostnames per one domain (avoids subdomain crawling hell e.g. blogspot.com).
  • Can be restarted with same list of domains – last saved domains are added to the URL queue.
  • Crawls only sites that return text/html Content-Type in HEAD response.
  • Retrieves site body of maximum 1MB size.
  • Does not save inaccessible domains.

dcrawl Usage


Example:

There are other tools which do similar things, or could be scripted together recursively to perform a similar kind of task – but nothing this focused. Examples would be:

Host-Extract – Enumerate All IP/Host Patterns In A Web Page
Recon-ng – Web Reconnaissance Framework

You can download dcrawl go web crawler here:

dcrawl-master.zip

Or read more here.

Posted in: Hacking Tools

, ,


Latest Posts:


testssl.sh - Test SSL Security Including Ciphers, Protocols & Detect Flaws testssl.sh – Test SSL Security Including Ciphers, Protocols & Detect Flaws
testssl.sh is a free command line tool to test SSL security, it checks a server's service on any port for the support of TLS/SSL ciphers, protocols as well as recent cryptographic flaws and more.
Four Year Old libSSH Bug Leaves Servers Wide Open Four Year Old libssh Bug Leaves Servers Wide Open
A fairly serious 4-year old libssh bug has left servers vulnerable to remote compromise, fortunately, the attack surface isn't that big as neither OpenSSH or the GitHub implementation are affected.
CHIPSEC - Platform Security Assessment Framework CHIPSEC – Platform Security Assessment Framework For Firmware Hacking
CHIPSEC is a platform security assessment framework for PCs including hardware, system firmware (BIOS/UEFI), and platform components for firmware hacking.
How To Recover When Your Website Got Hacked How To Recover When Your Website Got Hacked
The array of easily available Hacking Tools out there now is astounding, combined with self-propagating malware, people often come to me when their website got hacked and they don't know what to do, or even where to start.
HTTrack - Website Downloader Copier & Site Ripper Download HTTrack – Website Downloader Copier & Site Ripper Download
HTTrack is a free and easy-to-use offline browser utility which acts as a website downloader and a site ripper for copying websites and downloading them for offline viewing.
sshLooter - Script To Steal SSH Passwords sshLooter – Script To Steal SSH Passwords
sshLooter is a Python script using a PAM module to steal SSH passwords by logging the password and notifying the admin of the script via Telegram when a user logs in.


Comments are closed.