dcrawl – Web Crawler For Unique Domains


dcrawl is a simple, but smart, multithreaded web crawler for randomly gathering huge lists of unique domain names.

dcrawl - Web Crawler For Unique Domains


How does dcrawl work?

dcrawl takes one site URL as input and detects all a href= links in the site’s body. Each found link is put into the queue. Successively, each queued link is crawled in the same way, branching out to more URLs found in links on each site’s body.

dcrawl Web Crawler Features

  • Branching out only to predefined number of links found per one hostname.
  • Maximum number of allowed different hostnames per one domain (avoids subdomain crawling hell e.g. blogspot.com).
  • Can be restarted with same list of domains – last saved domains are added to the URL queue.
  • Crawls only sites that return text/html Content-Type in HEAD response.
  • Retrieves site body of maximum 1MB size.
  • Does not save inaccessible domains.

dcrawl Usage


Example:

There are other tools which do similar things, or could be scripted together recursively to perform a similar kind of task – but nothing this focused. Examples would be:

Host-Extract – Enumerate All IP/Host Patterns In A Web Page
Recon-ng – Web Reconnaissance Framework

You can download dcrawl go web crawler here:

dcrawl-master.zip

Or read more here.

Posted in: Hacking Tools

, ,


Latest Posts:


zBang - Privileged Account Threat Detection Tool zBang – Privileged Account Threat Detection Tool
zBang is a risk assessment tool for Privileged Account Threat Detection on a scanned network, organizations & red teams can use it to identify attack vectors
Memhunter - Automated Memory Resident Malware Detection Memhunter – Automated Memory Resident Malware Detection
Memhunter is an Automated Memory Resident Malware Detection tool for the hunting of memory resident malware at scale, improving threat hunter analysis process.
Sandcastle - AWS S3 Bucket Enumeration Tool Sandcastle – AWS S3 Bucket Enumeration Tool
Sandcastle is an Amazon AWS S3 Bucket Enumeration Tool, formerly known as bucketCrawler. The script takes a target's name as the stem argument (e.g. shopify).
Astra - API Automated Security Testing For REST Astra – API Automated Security Testing For REST
Astra is a Python-based tool for API Automated Security Testing, REST API penetration testing is complex due to continuous changes in existing APIs.
Judas DNS - Nameserver DNS Poisoning Attack Tool Judas DNS – Nameserver DNS Poisoning Attack Tool
Judas DNS is a Nameserver DNS Poisoning Attack Tool which functions as a DNS proxy server built to be deployed in place of a taken over nameserver to perform targeted exploitation.
dsniff Download - Tools for Network Auditing & Password Sniffing dsniff Download – Tools for Network Auditing & Password Sniffing
Dsniff download is a collection of tools for network auditing & penetration testing. Dsniff, filesnarf, mailsnarf, msgsnarf, URLsnarf, and WebSpy passively monitor a network


Comments are closed.