dcrawl – Web Crawler For Unique Domains

Outsmart Malicious Hackers


dcrawl is a simple, but smart, multithreaded web crawler for randomly gathering huge lists of unique domain names.

dcrawl - Web Crawler For Unique Domains


How does dcrawl work?

dcrawl takes one site URL as input and detects all a href= links in the site’s body. Each found link is put into the queue. Successively, each queued link is crawled in the same way, branching out to more URLs found in links on each site’s body.

dcrawl Web Crawler Features

  • Branching out only to predefined number of links found per one hostname.
  • Maximum number of allowed different hostnames per one domain (avoids subdomain crawling hell e.g. blogspot.com).
  • Can be restarted with same list of domains – last saved domains are added to the URL queue.
  • Crawls only sites that return text/html Content-Type in HEAD response.
  • Retrieves site body of maximum 1MB size.
  • Does not save inaccessible domains.

dcrawl Usage


Example:

There are other tools which do similar things, or could be scripted together recursively to perform a similar kind of task – but nothing this focused. Examples would be:

Host-Extract – Enumerate All IP/Host Patterns In A Web Page
Recon-ng – Web Reconnaissance Framework

You can download dcrawl go web crawler here:

dcrawl-master.zip

Or read more here.

Posted in: Hacking Tools

, ,


Latest Posts:


OWASP ZSC - Obfuscated Code Generator Tool OWASP ZSC – Obfuscated Code Generator Tool
OWASP ZSC is an open source obfuscated code generator tool in Python which lets you generate customized shellcodes and convert scripts to an obfuscated script.
A Look Back At 2017 – Tools & News Highlights A Look Back At 2017 – Tools & News Highlights
So here we are in 2018, taking a look back at 2017, quite a year it was. Here is a quick rundown of some of the best hacking/security tools released in 2017, the biggest news stories and the 10 most viewed posts on Darknet as a bonus.
Spectre & Meltdown Checker - Vulnerability Mitigation Tool For Linux Spectre & Meltdown Checker – Vulnerability Mitigation Tool For Linux
Spectre & Meltdown Checker is a simple shell script to tell if your Linux installation is vulnerable against the 3 "speculative execution" CVEs that were made public early 2018.
Hijacker - Reaver For Android Wifi Hacker App Hijacker – Reaver For Android Wifi Hacker App
Hijacker is a native GUI which provides Reaver for Android along with Aircrack-ng, Airodump-ng and MDK3 making it a powerful Wifi hacker app.
Sublist3r - Fast Python Subdomain Enumeration Tool Sublist3r – Fast Python Subdomain Enumeration Tool
Sublist3r is a Python-based tool designed to enumerate subdomains of websites using OSINT. It helps penetration testers and bug hunters collect and gather subdomains for the domain they are targeting.
coWPAtty Download - Audit Pre-shared WPA Keys coWPAtty Download – Audit Pre-shared WPA Keys
coWPAtty is a C-based tool for running a brute-force dictionary attack against WPA-PSK and audit pre-shared WPA keys.


Comments are closed.