dcrawl – Web Crawler For Unique Domains


dcrawl is a simple, but smart, multithreaded web crawler for randomly gathering huge lists of unique domain names.

dcrawl - Web Crawler For Unique Domains


How does dcrawl work?

dcrawl takes one site URL as input and detects all a href= links in the site’s body. Each found link is put into the queue. Successively, each queued link is crawled in the same way, branching out to more URLs found in links on each site’s body.

dcrawl Web Crawler Features

  • Branching out only to predefined number of links found per one hostname.
  • Maximum number of allowed different hostnames per one domain (avoids subdomain crawling hell e.g. blogspot.com).
  • Can be restarted with same list of domains – last saved domains are added to the URL queue.
  • Crawls only sites that return text/html Content-Type in HEAD response.
  • Retrieves site body of maximum 1MB size.
  • Does not save inaccessible domains.

dcrawl Usage


Example:

There are other tools which do similar things, or could be scripted together recursively to perform a similar kind of task – but nothing this focused. Examples would be:

Host-Extract – Enumerate All IP/Host Patterns In A Web Page
Recon-ng – Web Reconnaissance Framework

You can download dcrawl go web crawler here:

dcrawl-master.zip

Or read more here.

Posted in: Hacking Tools

, ,


Latest Posts:


APT-Hunter - Threat Hunting Tool via Windows Event Log APT-Hunter – Threat Hunting Tool via Windows Event Log
APT-Hunter is a threat hunting tool for windows event logs made from the perspective of the purple team mindset to provide detection for APT movements hidden in the sea of windows event logs.
GitLab Watchman - Audit Gitlab For Sensitive Data & Credentials GitLab Watchman – Audit Gitlab For Sensitive Data & Credentials
GitLab Watchman is an app that uses the GitLab API to audit GitLab for sensitive data and credentials exposed internally, this includes code, commits, wikis etc
GKE Auditor - Detect Google Kubernetes Engine Misconfigurations GKE Auditor – Detect Google Kubernetes Engine Misconfigurations
GKE Auditor is a Java-based tool to detect Google Kubernetes Engine misconfigurations, it aims to help security & dev teams streamline the configuration process
zANTI - Android Wireless Hacking Tool Free Download zANTI – Android Wireless Hacking Tool Free Download
zANTI is an Android Wireless Hacking Tool that functions as a mobile penetration testing toolkit that lets you assess the risk level of a network using mobile.
HELK - Open Source Threat Hunting Platform HELK – Open Source Threat Hunting Platform
The Hunting ELK or simply the HELK is an Open-Source Threat Hunting Platform with advanced analytics capabilities such as SQL declarative language, graphing etc
trape - OSINT Analysis Tool For People Tracking Trape – OSINT Analysis Tool For People Tracking
Trape is an OSINT analysis tool, which allows people to track and execute intelligent social engineering attacks in real-time.


Comments are closed.