dcrawl – Web Crawler For Unique Domains


dcrawl is a simple, but smart, multithreaded web crawler for randomly gathering huge lists of unique domain names.

dcrawl - Web Crawler For Unique Domains


How does dcrawl work?

dcrawl takes one site URL as input and detects all a href= links in the site’s body. Each found link is put into the queue. Successively, each queued link is crawled in the same way, branching out to more URLs found in links on each site’s body.

dcrawl Web Crawler Features

  • Branching out only to predefined number of links found per one hostname.
  • Maximum number of allowed different hostnames per one domain (avoids subdomain crawling hell e.g. blogspot.com).
  • Can be restarted with same list of domains – last saved domains are added to the URL queue.
  • Crawls only sites that return text/html Content-Type in HEAD response.
  • Retrieves site body of maximum 1MB size.
  • Does not save inaccessible domains.

dcrawl Usage


Example:

There are other tools which do similar things, or could be scripted together recursively to perform a similar kind of task – but nothing this focused. Examples would be:

Host-Extract – Enumerate All IP/Host Patterns In A Web Page
Recon-ng – Web Reconnaissance Framework

You can download dcrawl go web crawler here:

dcrawl-master.zip

Or read more here.

Posted in: Hacking Tools

, ,


Latest Posts:


GKE Auditor - Detect Google Kubernetes Engine Misconfigurations GKE Auditor – Detect Google Kubernetes Engine Misconfigurations
GKE Auditor is a Java-based tool to detect Google Kubernetes Engine misconfigurations, it aims to help security & dev teams streamline the configuration process
zANTI - Android Wireless Hacking Tool Free Download zANTI – Android Wireless Hacking Tool Free Download
zANTI is an Android Wireless Hacking Tool that functions as a mobile penetration testing toolkit that lets you assess the risk level of a network using mobile.
HELK - Open Source Threat Hunting Platform HELK – Open Source Threat Hunting Platform
The Hunting ELK or simply the HELK is an Open-Source Threat Hunting Platform with advanced analytics capabilities such as SQL declarative language, graphing etc
trape - OSINT Analysis Tool For People Tracking Trape – OSINT Analysis Tool For People Tracking
Trape is an OSINT analysis tool, which allows people to track and execute intelligent social engineering attacks in real-time.
Fuzzilli - JavaScript Engine Fuzzing Library Fuzzilli – JavaScript Engine Fuzzing Library
Fuzzilii is a JavaScript engine fuzzing library, it's a coverage-guided fuzzer for dynamic language interpreters based on a custom intermediate language.
OWASP APICheck - HTTP API DevSecOps Toolset OWASP APICheck – HTTP API DevSecOps Toolset
APICheck is an HTTP API DevSecOps toolset, it integrates existing tools, creates execution chains easily and is designed for integration with 3rd parties.


Comments are closed.