dcrawl – Web Crawler For Unique Domains


dcrawl is a simple, but smart, multithreaded web crawler for randomly gathering huge lists of unique domain names.

dcrawl - Web Crawler For Unique Domains


How does dcrawl work?

dcrawl takes one site URL as input and detects all a href= links in the site’s body. Each found link is put into the queue. Successively, each queued link is crawled in the same way, branching out to more URLs found in links on each site’s body.

dcrawl Web Crawler Features

  • Branching out only to predefined number of links found per one hostname.
  • Maximum number of allowed different hostnames per one domain (avoids subdomain crawling hell e.g. blogspot.com).
  • Can be restarted with same list of domains – last saved domains are added to the URL queue.
  • Crawls only sites that return text/html Content-Type in HEAD response.
  • Retrieves site body of maximum 1MB size.
  • Does not save inaccessible domains.

dcrawl Usage


Example:

There are other tools which do similar things, or could be scripted together recursively to perform a similar kind of task – but nothing this focused. Examples would be:

Host-Extract – Enumerate All IP/Host Patterns In A Web Page
Recon-ng – Web Reconnaissance Framework

You can download dcrawl go web crawler here:

dcrawl-master.zip

Or read more here.

Posted in: Hacking Tools

, ,


Latest Posts:


Socialscan - Command-Line Tool To Check For Email And Social Media Username Usage Socialscan – Command-Line Tool To Check For Email And Social Media Username Usage
socialscan is an accurate command-line tool to check For email and social media username usage on online platforms, given an email address or username,
CFRipper - CloudFormation Security Scanning & Audit Tool CFRipper – CloudFormation Security Scanning & Audit Tool
CFRipper is a Python-based Library and CLI security analyzer that functions as an AWS CloudFormation security scanning and audit tool
CredNinja - Test Credential Validity of Dumped Credentials or Hashes CredNinja – Test Credential Validity of Dumped Credentials or Hashes
CredNinja is a tool to quickly test credential validity of dumped credentials (or hashes) across an entire network or domain very efficiently.
assetfinder - Find Related Domains and Subdomains assetfinder – Find Related Domains and Subdomains
assetfinder is a Go-based tool to find related domains and subdomains that are related to a given domain from a variety of sources including Facebook and more.
Karkinos - Beginner Friendly Penetration Testing Tool Karkinos – Beginner Friendly Penetration Testing Tool
Karkinos is a light-weight Beginner Friendly Penetration Testing Tool, which is basically a 'Swiss Army Knife' for pen-testing and/or hacking CTF's.
Aclpwn.Py - Exploit ACL Based Privilege Escalation Paths in Active Directory Aclpwn.Py – Exploit ACL Based Privilege Escalation Paths in Active Directory
Aclpwn.py is a tool that interacts with BloodHound< to identify and exploit ACL based privilege escalation paths.


Comments are closed.