dcrawl – Web Crawler For Unique Domains

Use Netsparker


dcrawl is a simple, but smart, multithreaded web crawler for randomly gathering huge lists of unique domain names.

dcrawl - Web Crawler For Unique Domains


How does dcrawl work?

dcrawl takes one site URL as input and detects all a href= links in the site’s body. Each found link is put into the queue. Successively, each queued link is crawled in the same way, branching out to more URLs found in links on each site’s body.

dcrawl Web Crawler Features

  • Branching out only to predefined number of links found per one hostname.
  • Maximum number of allowed different hostnames per one domain (avoids subdomain crawling hell e.g. blogspot.com).
  • Can be restarted with same list of domains – last saved domains are added to the URL queue.
  • Crawls only sites that return text/html Content-Type in HEAD response.
  • Retrieves site body of maximum 1MB size.
  • Does not save inaccessible domains.

dcrawl Usage


Example:

There are other tools which do similar things, or could be scripted together recursively to perform a similar kind of task – but nothing this focused. Examples would be:

Host-Extract – Enumerate All IP/Host Patterns In A Web Page
Recon-ng – Web Reconnaissance Framework

You can download dcrawl go web crawler here:

dcrawl-master.zip

Or read more here.

Posted in: Hacking Tools

, ,


Latest Posts:


snallygaster - Scan For Secret Files On HTTP Servers snallygaster – Scan For Secret Files On HTTP Servers
snallygaster is a Python-based tool that can help you to scan for secret files on HTTP servers, files that are accessible that shouldn't be public and can pose a s
Portspoof - Spoof All Ports Open & Emulate Valid Services Portspoof – Spoof All Ports Open & Emulate Valid Services
The primary goal of the Portspoof program is to enhance your system security through a set of new camouflage techniques which spoof all ports open and also emulate valid services on every port.
Cambridge Analytica Facebook Data Scandal Cambridge Analytica Facebook Data Scandal
One of the biggest stories of the year so far has been the scandal surrounding Cambridge Analytica that came out after a Channel 4 expose that demonstrated the depths they are willing to go to profile voters, manipulate elections and much more.
GetAltName - Discover Sub-Domains From SSL Certificates GetAltName – Discover Sub-Domains From SSL Certificates
GetAltName it's a little script to discover sub-domains that can extract Subject Alt Names for SSL Certificates directly from HTTPS websites which can provide you with DNS names or virtual servers.
Memcrashed - Memcached DDoS Exploit Tool Memcrashed – Memcached DDoS Exploit Tool
Memcrashed is a Memcached DDoS exploit tool written in Python that allows you to send forged UDP packets to a list of Memcached servers obtained from Shodan.
QualysGuard - Vulnerability Management Tool QualysGuard – Vulnerability Management Tool
QualysGuard is a web-based vulnerability management tool provided by Qualys, Inc, which was the first company to deliver vulnerability management services as a SaaS-based web-service.


Comments are closed.