dcrawl – Web Crawler For Unique Domains

The New Acunetix V12 Engine


dcrawl is a simple, but smart, multithreaded web crawler for randomly gathering huge lists of unique domain names.

dcrawl - Web Crawler For Unique Domains


How does dcrawl work?

dcrawl takes one site URL as input and detects all a href= links in the site’s body. Each found link is put into the queue. Successively, each queued link is crawled in the same way, branching out to more URLs found in links on each site’s body.

dcrawl Web Crawler Features

  • Branching out only to predefined number of links found per one hostname.
  • Maximum number of allowed different hostnames per one domain (avoids subdomain crawling hell e.g. blogspot.com).
  • Can be restarted with same list of domains – last saved domains are added to the URL queue.
  • Crawls only sites that return text/html Content-Type in HEAD response.
  • Retrieves site body of maximum 1MB size.
  • Does not save inaccessible domains.

dcrawl Usage


Example:

There are other tools which do similar things, or could be scripted together recursively to perform a similar kind of task – but nothing this focused. Examples would be:

Host-Extract – Enumerate All IP/Host Patterns In A Web Page
Recon-ng – Web Reconnaissance Framework

You can download dcrawl go web crawler here:

dcrawl-master.zip

Or read more here.

Posted in: Hacking Tools

, ,


Latest Posts:


Malcom - Malware Communication Analyzer Malcom – Malware Communication Analyzer
Malcom is a Malware Communication Analyzer designed to analyze a system's network communication using graphical representations of network traffic.
WepAttack - WLAN 802.11 WEP Key Hacking Tool WepAttack – WLAN 802.11 WEP Key Hacking Tool
WepAttack is a WLAN open source Linux WEP key hacking tool for breaking 802.11 WEP keys using a wordlist based dictionary attack.
Eraser - Windows Secure Erase Hard Drive Wiper Eraser – Windows Secure Erase Hard Drive Wiper
Eraser is a hard drive wiper for Windows which allows you to run a secure erase and completely remove sensitive data from your hard drive by overwriting it several times with carefully selected patterns.
Insecure software versions are a problem Web Security Stats Show XSS & Outdated Software Are Major Problems
Netsparker just published some anonymized Web Security Stats about the security vulnerabilities their online solution identified on their users’ web applications and web services during the last 3 years.
CTFR - Abuse Certificate Transparency Logs For HTTPS Subdomains CTFR – Abuse Certificate Transparency Logs For HTTPS Subdomains
CTFR is a Python-based tool to Abuse Certificate Transparency Logs to get subdomains from a HTTPS website in a few seconds.
testssl.sh - Test SSL Security Including Ciphers, Protocols & Detect Flaws testssl.sh – Test SSL Security Including Ciphers, Protocols & Detect Flaws
testssl.sh is a free command line tool to test SSL security, it checks a server's service on any port for the support of TLS/SSL ciphers, protocols as well as recent cryptographic flaws and more.


Comments are closed.