dcrawl – Web Crawler For Unique Domains


dcrawl is a simple, but smart, multithreaded web crawler for randomly gathering huge lists of unique domain names.

dcrawl - Web Crawler For Unique Domains


How does dcrawl work?

dcrawl takes one site URL as input and detects all a href= links in the site’s body. Each found link is put into the queue. Successively, each queued link is crawled in the same way, branching out to more URLs found in links on each site’s body.

dcrawl Web Crawler Features

  • Branching out only to predefined number of links found per one hostname.
  • Maximum number of allowed different hostnames per one domain (avoids subdomain crawling hell e.g. blogspot.com).
  • Can be restarted with same list of domains – last saved domains are added to the URL queue.
  • Crawls only sites that return text/html Content-Type in HEAD response.
  • Retrieves site body of maximum 1MB size.
  • Does not save inaccessible domains.

dcrawl Usage


Example:

There are other tools which do similar things, or could be scripted together recursively to perform a similar kind of task – but nothing this focused. Examples would be:

Host-Extract – Enumerate All IP/Host Patterns In A Web Page
Recon-ng – Web Reconnaissance Framework

You can download dcrawl go web crawler here:

dcrawl-master.zip

Or read more here.

Posted in: Hacking Tools

, ,


Latest Posts:


SharpHose - Asynchronous Password Spraying Tool SharpHose – Asynchronous Password Spraying Tool
SharpHose is an asynchronous password spraying tool in C# for Windows environments that takes into consideration fine-grained password policies and can be run over Cobalt Strike's execute-assembly.
Axiom - Pen-Testing Server For Collecting Bug Bounties Axiom – Pen-Testing Server For Collecting Bug Bounties
Project Axiom is a set of utilities for managing a small dynamic infrastructure setup for bug bounty, basically a pen-testing server out of the box with 1-line.
Quasar RAT - Windows Remote Administration Tool Quasar RAT – Windows Remote Administration Tool
Quasar is a fast and light-weight Windows remote administration tool coded in C#. Used for user support through day-to-day administrative work to monitoring.
Pingcastle - Active Directory Security Assessment Tool Pingcastle – Active Directory Security Assessment Tool
PingCastle is a Active Directory Security Assessment Tool designed to quickly assess the Active Directory security level based on a risk and maturity framework.
Second Order - Subdomain Takeover Scanner Tool Second Order – Subdomain Takeover Scanner Tool
Second Order Subdomain Takeover Scanner Tool scans web apps for second-order subdomain takeover by crawling the application and collecting URLs (and other data)
Binwalk - Firmware Security Analysis & Extraction Tool Binwalk – Firmware Security Analysis & Extraction Tool
Binwalk is a fast and easy to use Python-based firmware security analysis tool that allows for firmware analysis, reverse engineering & extracting of firmware.


Comments are closed.