Skip to content
HackIndex.io

Web Crawling and Spidering

6 min read Created Sep 13, 2026 Last updated Sep 13, 2026

Web crawling follows links and parses responses to build a map of the application's structure. It complements content discovery (which guesses paths via wordlists) and passive recon (which collects archived URLs without touching the target). Use crawling first to discover the visible attack surface, then brute-force for hidden paths that crawling misses. Feed discovered URLs into parameter crawling to find hidden GET and POST parameters on each endpoint. Modern crawlers also parse JavaScript files to extract API endpoints, hardcoded paths, and internal URLs that never appear in HTML.

Hakrawler

Hakrawler is a fast Go-based crawler that accepts URLs from stdin. It extracts links, form actions, and script sources. Useful for quick discovery when you need results fast and do not need deep JavaScript parsing.

┌──(kali㉿kali)-[~]
└─$ echo "http://$TARGET_IP:$PORT" | hakrawler -d 3 -insecure 
http://10.10.10.50/login
http://10.10.10.50/dashboard
http://10.10.10.50/api/v1/users
http://10.10.10.50/assets/app.js
http://10.10.10.50/uploads/
http://10.10.10.50/static/config.js
Explain command
-d 3 Crawl depth, how many link levels to follow from the start URL.
-insecure Ignore TLS certificate errors for self-signed certs.

Pipe the output to sort -u to deduplicate results. Filter for interesting patterns like API paths, upload directories, and JavaScript files that may contain hardcoded endpoints.

┌──(kali㉿kali)-[~]
└─$ echo "http://$TARGET_IP:$PORT" | hakrawler -d 3 -insecure | grep -E '(api/|/v[0-9]/|\.js$)' | sort -u 
http://10.10.10.50/api/v1/users
http://10.10.10.50/assets/app.js
http://10.10.10.50/static/config.js

Katana

Katana (ProjectDiscovery) is the most capable modern crawler. It handles standard HTML crawling and also supports headless browser mode for JavaScript-heavy single-page applications that render content dynamically. The -jc flag enables JavaScript crawling, which parses JS files for embedded URLs and API routes.

┌──(kali㉿kali)-[~]
└─$ katana -u http://$TARGET_IP:$PORT -d 3 -jc -o crawl-results.txt 
   __        __
  / /_____ _/ /_____ _____  ___ _
 / '_/ _  / __/ _  / _ \/ _  /
/_/\_\_,_/\__/\_,_/_//_/\_,_/

http://10.10.10.50/
http://10.10.10.50/login
http://10.10.10.50/register
http://10.10.10.50/api/v2/health
http://10.10.10.50/api/v2/users/profile
http://10.10.10.50/static/main.chunk.js
http://10.10.10.50/admin/dashboard
Explain command
-u Target URL to crawl.
-d 3 Maximum crawl depth.
-jc Enable JavaScript file crawling to extract embedded URLs.
-o crawl-results.txt Write discovered URLs to a file.

For single-page applications built with React, Angular, or Vue, the standard crawler misses routes that only exist in client-side JavaScript. Use headless mode to render pages in a real browser and discover these routes.

┌──(kali㉿kali)-[~]
└─$ katana -u http://$TARGET_IP:$PORT -d 3 -jc -headless -o crawl-headless.txt
Explain command
-headless Use a headless browser to render JavaScript before extracting URLs.

Headless mode is slower but discovers routes that only appear after JavaScript execution. Compare results between standard and headless mode to identify dynamically loaded content.

Gospider

Gospider combines crawling with integration for external sources like Wayback Machine, CommonCrawl, and OTX. This surfaces archived URLs and historical paths that no longer appear in the live application but may still be accessible on the server.

┌──(kali㉿kali)-[~]
└─$ gospider -s http://$TARGET_IP:$PORT -d 3 --other-source -o gospider-output 
[url] - [code-200] - http://10.10.10.50/
[url] - [code-200] - http://10.10.10.50/about
[url] - [code-200] - http://10.10.10.50/contact
[javascript] - http://10.10.10.50/static/app.bundle.js
[linkfinder] - http://10.10.10.50/api/internal/debug
[wayback] - http://10.10.10.50/old-admin/
[wayback] - http://10.10.10.50/backup.zip
Explain command
-s Target site URL.
-d 3 Maximum crawl depth.
--other-source Include results from Wayback Machine, CommonCrawl, and other archived sources.

The Wayback results are especially valuable. Archived paths like /old-admin/ and /backup.zip may still be served by the web server even though they are no longer linked from any live page. Always verify these against the live target.

JavaScript File Analysis

JavaScript files frequently contain hardcoded API endpoints, internal URLs, API keys, and authentication tokens. After crawling, collect all discovered JS files and analyze them for interesting strings. LinkFinder automates endpoint extraction from JavaScript files using regex pattern matching.

┌──(kali㉿kali)-[~]
└─$ python3 linkfinder.py -i http://$TARGET_IP:$PORT/static/app.bundle.js -o cli 
/api/v2/users/profile
/api/v2/admin/settings
/api/v2/files/upload
/api/internal/debug
/graphql
/ws/notifications
Explain command
-i Input URL of the JavaScript file to analyze.
-o cli Output results to the terminal.

For a quick manual approach without LinkFinder, download JS files and grep for common API patterns.

┌──(kali㉿kali)-[~]
└─$ curl -sk "http://$TARGET_IP:$PORT/static/app.bundle.js" | grep -oE '"(/[a-zA-Z0-9_/-]+)"' | sort -u 
"/api/internal/debug"
"/api/v2/admin/settings"
"/api/v2/files/upload"
"/api/v2/users/profile"
"/graphql"

Also search for hardcoded secrets, API keys, and tokens in JS files. Look for patterns like apiKey, secret, token, password, Authorization, and Bearer.

┌──(kali㉿kali)-[~]
└─$ curl -sk "http://$TARGET_IP:$PORT/static/app.bundle.js" | grep -oiE '(api_?key|secret|token|password|authorization|bearer)["\x27]?\s*[:=]\s*["\x27][^"\x27]+' | head -20 
apiKey: "sk_live_abc123def456"
authorization: "Bearer eyJhbGciOi..."

Comparing Results

Each crawler has strengths. Hakrawler is fast for basic link extraction. Katana handles JavaScript-heavy applications with headless mode. Gospider pulls in historical data from web archives. Running all three and merging the results gives the most complete picture of the attack surface.

┌──(kali㉿kali)-[~]
└─$ cat crawl-results.txt gospider-output/* | grep -oE 'https?://[^ ]+' | sort -u > all-urls.txt
┌──(kali㉿kali)-[~]
└─$ wc -l all-urls.txt 
247 all-urls.txt

From the merged URL list, extract unique paths and parameters for further testing. Feed discovered paths into a directory brute-forcer to find additional content in the same directory structures. Pass parameters to injection testing tools. JavaScript files go to LinkFinder for endpoint extraction. This pipeline maximizes coverage before moving into vulnerability discovery.

┌──(kali㉿kali)-[~]
└─$ cat all-urls.txt | unfurl paths | sort -u > unique-paths.txt
┌──(kali㉿kali)-[~]
└─$ head -10 unique-paths.txt 
/admin/dashboard
/api/internal/debug
/api/v1/users
/api/v2/admin/settings
/api/v2/files/upload
/api/v2/health
/api/v2/users/profile
/backup.zip
/login
/old-admin/
┌──(kali㉿kali)-[~]
└─$ cat all-urls.txt | unfurl keys | sort -u 
id
page
q
redirect
token
url
user

Parameters like url and redirect are immediate candidates for SSRF and open redirect testing. id and user suggest IDOR opportunities. q and page are common injection points for SQLi and XSS. Each parameter type maps to a specific set of vulnerability tests covered in the vulnerability discovery phase.

References