dcrawl – Web Crawler For Unique Domains

The New Acunetix V12 Engine


dcrawl is a simple, but smart, multithreaded web crawler for randomly gathering huge lists of unique domain names.

dcrawl - Web Crawler For Unique Domains


How does dcrawl work?

dcrawl takes one site URL as input and detects all a href= links in the site’s body. Each found link is put into the queue. Successively, each queued link is crawled in the same way, branching out to more URLs found in links on each site’s body.

dcrawl Web Crawler Features

  • Branching out only to predefined number of links found per one hostname.
  • Maximum number of allowed different hostnames per one domain (avoids subdomain crawling hell e.g. blogspot.com).
  • Can be restarted with same list of domains – last saved domains are added to the URL queue.
  • Crawls only sites that return text/html Content-Type in HEAD response.
  • Retrieves site body of maximum 1MB size.
  • Does not save inaccessible domains.

dcrawl Usage


Example:

There are other tools which do similar things, or could be scripted together recursively to perform a similar kind of task – but nothing this focused. Examples would be:

Host-Extract – Enumerate All IP/Host Patterns In A Web Page
Recon-ng – Web Reconnaissance Framework

You can download dcrawl go web crawler here:

dcrawl-master.zip

Or read more here.

Posted in: Hacking Tools

, ,


Latest Posts:


DeepSound - Audio Steganography Tool DeepSound – Audio Steganography Tool
DeepSound is an audio steganography tool and audio converter that hides secret data into audio files, the application also enables you to extract from files.
2019 High Severity Vulnerabilities What are the MOST Critical Web Vulnerabilities in 2019?
So what is wild on the web this year? Need to know about the most critical web vulnerabilities in 2019 to protect your organization?
GoBuster - Directory/File & DNS Busting Tool in Go GoBuster – Directory/File & DNS Busting Tool in Go
GoBuster is a tool used to brute-force URIs (directories and files) in web sites and DNS subdomains (inc. wildcards) - a directory/file & DNS busting tool.
BDFProxy - Patch Binaries via MITM - BackdoorFactory + mitmProxy BDFProxy – Patch Binaries via MiTM – BackdoorFactory + mitmproxy
BDFProxy allows you to patch binaries via MiTM with The Backdoor Factory combined with mitmproxy enabling on the fly patching of binary downloads
Domained - Multi Tool Subdomain Enumeration Domained – Multi Tool Subdomain Enumeration
Domained is a multi tool subdomain enumeration tool that uses several subdomain enumeration tools and wordlists to create a unique list of subdomains.
Acunetix Vulnerability Scanner For Linux Now Available Acunetix Vulnerability Scanner For Linux Now Available
Acunetix Vulnerability Scanner For Linux is now available, now you get all of the functionality of Acunetix, with all of the dependability of Linux.


Comments are closed.