dcrawl – Web Crawler For Unique Domains

Use Netsparker


dcrawl is a simple, but smart, multithreaded web crawler for randomly gathering huge lists of unique domain names.

dcrawl - Web Crawler For Unique Domains


How does dcrawl work?

dcrawl takes one site URL as input and detects all a href= links in the site’s body. Each found link is put into the queue. Successively, each queued link is crawled in the same way, branching out to more URLs found in links on each site’s body.

dcrawl Web Crawler Features

  • Branching out only to predefined number of links found per one hostname.
  • Maximum number of allowed different hostnames per one domain (avoids subdomain crawling hell e.g. blogspot.com).
  • Can be restarted with same list of domains – last saved domains are added to the URL queue.
  • Crawls only sites that return text/html Content-Type in HEAD response.
  • Retrieves site body of maximum 1MB size.
  • Does not save inaccessible domains.

dcrawl Usage


Example:

There are other tools which do similar things, or could be scripted together recursively to perform a similar kind of task – but nothing this focused. Examples would be:

Host-Extract – Enumerate All IP/Host Patterns In A Web Page
Recon-ng – Web Reconnaissance Framework

You can download dcrawl go web crawler here:

dcrawl-master.zip

Or read more here.

Posted in: Hacking Tools

, ,


Latest Posts:


dcipher - Online Hash Cracking Using Rainbow & Lookup Tables dcipher – Online Hash Cracking Using Rainbow & Lookup Tables
dcipher is a JavaScript-based online hash cracking tool to decipher hashes using online rainbow & lookup table attack services.
HTTP Security Considerations - An Introduction To HTTP Basics HTTP Security Considerations – An Introduction To HTTP Basics
HTTP is ubiquitous now with pretty much everything being powered by an API, a web application or some kind of cloud-based HTTP driven infrastructure. With that HTTP Security becomes paramount and to secure HTTP you have to understand it.
Cangibrina - Admin Dashboard Finder Tool Cangibrina – Admin Dashboard Finder Tool
Cangibrina is a Python-based multi platform admin dashboard finder tool which aims to obtain the location of website dashboards by using brute-force, wordlists etc.
Enumall - Subdomain Discovery Using Recon-ng & AltDNS Enumall – Subdomain Discovery Using Recon-ng & AltDNS
Enumall is a Python-based tool that helps you do subdomain discovery using only one command by combining the abilities of Recon-ng and AltDNS.
RidRelay - SMB Relay Attack For Username Enumeration RidRelay – SMB Relay Attack For Username Enumeration
RidRelay is a Python-based tool to enumerate usernames on a domain where you have no credentials by using a SMB Relay Attack with low privileges.
NetBScanner - NetBIOS Network Scanner NetBScanner – NetBIOS Network Scanner
NetBScanner is a NetBIOS network scanner tool that scans all computers in the IP addresses range you choose, using the NetBIOS protocol.


Comments are closed.