# Web Crawling and Spidering

> Systematically map web application structure by discovering pages, directories, and endpoints through automated crawling and link analysis.

- **URL:** https://hackindex.io/services/web/enumeration/web-crawling
- **Path:** HTTP/HTTPS > Enumeration
- **Up:** [Enumeration](https://hackindex.io/services/web/enumeration.md)
- **Published:** 2026-09-13
- **Updated:** 2026-09-13

Web crawling follows links and parses responses to build a map of the application's structure. It complements [content discovery](https://hackindex.io/services/web/enumeration/content-discovery) (which guesses paths via wordlists) and passive recon (which collects archived URLs without touching the target). Use crawling first to discover the visible attack surface, then brute-force for hidden paths that crawling misses. Feed discovered URLs into [parameter crawling](https://hackindex.io/services/web/enumeration/parameter-crawling) to find hidden GET and POST parameters on each endpoint. Modern crawlers also parse JavaScript files to extract API endpoints, hardcoded paths, and internal URLs that never appear in HTML.

## Hakrawler

Hakrawler is a fast Go-based crawler that accepts URLs from stdin. It extracts links, form actions, and script sources. Useful for quick discovery when you need results fast and do not need deep JavaScript parsing.

**Basic hakrawler crawl**

```bash
echo "http://$TARGET_IP:$PORT" | hakrawler -d 3 -insecure
# http://10.10.10.50/login
# http://10.10.10.50/dashboard
# http://10.10.10.50/api/v1/users
# http://10.10.10.50/assets/app.js
# http://10.10.10.50/uploads/
# http://10.10.10.50/static/config.js
```

Pipe the output to `sort -u` to deduplicate results. Filter for interesting patterns like API paths, upload directories, and JavaScript files that may contain hardcoded endpoints.

**Filter for API endpoints and JS files**

```bash
echo "http://$TARGET_IP:$PORT" | hakrawler -d 3 -insecure | grep -E '(api/|/v[0-9]/|\.js$)' | sort -u
# http://10.10.10.50/api/v1/users
# http://10.10.10.50/assets/app.js
# http://10.10.10.50/static/config.js
```

## Katana

Katana (ProjectDiscovery) is the most capable modern crawler. It handles standard HTML crawling and also supports headless browser mode for JavaScript-heavy single-page applications that render content dynamically. The `-jc` flag enables JavaScript crawling, which parses JS files for embedded URLs and API routes.

**Standard katana crawl with JS parsing**

```bash
katana -u http://$TARGET_IP:$PORT -d 3 -jc -o crawl-results.txt
#    __        __
#   / /_____ _/ /_____ _____  ___ _
#  / '_/ _  / __/ _  / _ \/ _  /
# /_/\_\_,_/\__/\_,_/_//_/\_,_/
#
# http://10.10.10.50/
# http://10.10.10.50/login
# http://10.10.10.50/register
# http://10.10.10.50/api/v2/health
# http://10.10.10.50/api/v2/users/profile
# http://10.10.10.50/static/main.chunk.js
# http://10.10.10.50/admin/dashboard
```

For single-page applications built with React, Angular, or Vue, the standard crawler misses routes that only exist in client-side JavaScript. Use headless mode to render pages in a real browser and discover these routes.

**Headless mode for SPAs**

```bash
katana -u http://$TARGET_IP:$PORT -d 3 -jc -headless -o crawl-headless.txt
```

Headless mode is slower but discovers routes that only appear after JavaScript execution. Compare results between standard and headless mode to identify dynamically loaded content.

## Gospider

Gospider combines crawling with integration for external sources like Wayback Machine, CommonCrawl, and OTX. This surfaces archived URLs and historical paths that no longer appear in the live application but may still be accessible on the server.

**Gospider with external sources**

```bash
gospider -s http://$TARGET_IP:$PORT -d 3 --other-source -o gospider-output
# [url] - [code-200] - http://10.10.10.50/
# [url] - [code-200] - http://10.10.10.50/about
# [url] - [code-200] - http://10.10.10.50/contact
# [javascript] - http://10.10.10.50/static/app.bundle.js
# [linkfinder] - http://10.10.10.50/api/internal/debug
# [wayback] - http://10.10.10.50/old-admin/
# [wayback] - http://10.10.10.50/backup.zip
```

The Wayback results are especially valuable. Archived paths like `/old-admin/` and `/backup.zip` may still be served by the web server even though they are no longer linked from any live page. Always verify these against the live target.

## JavaScript File Analysis

JavaScript files frequently contain hardcoded API endpoints, internal URLs, API keys, and authentication tokens. After crawling, collect all discovered JS files and analyze them for interesting strings. LinkFinder automates endpoint extraction from JavaScript files using regex pattern matching.

**Extract endpoints with LinkFinder**

```bash
python3 linkfinder.py -i http://$TARGET_IP:$PORT/static/app.bundle.js -o cli
# /api/v2/users/profile
# /api/v2/admin/settings
# /api/v2/files/upload
# /api/internal/debug
# /graphql
# /ws/notifications
```

For a quick manual approach without LinkFinder, download JS files and grep for common API patterns.

**Manual JS endpoint extraction**

```bash
curl -sk "http://$TARGET_IP:$PORT/static/app.bundle.js" | grep -oE '"(/[a-zA-Z0-9_/-]+)"' | sort -u
# "/api/internal/debug"
# "/api/v2/admin/settings"
# "/api/v2/files/upload"
# "/api/v2/users/profile"
# "/graphql"
```

Also search for hardcoded secrets, API keys, and tokens in JS files. Look for patterns like `apiKey`, `secret`, `token`, `password`, `Authorization`, and `Bearer`.

**Search for secrets in JS files**

```bash
curl -sk "http://$TARGET_IP:$PORT/static/app.bundle.js" | grep -oiE '(api_?key|secret|token|password|authorization|bearer)["\x27]?\s*[:=]\s*["\x27][^"\x27]+' | head -20
# apiKey: "sk_live_abc123def456"
# authorization: "Bearer eyJhbGciOi..."
```

## Comparing Results

Each crawler has strengths. Hakrawler is fast for basic link extraction. Katana handles JavaScript-heavy applications with headless mode. Gospider pulls in historical data from web archives. Running all three and merging the results gives the most complete picture of the attack surface.

**Merge and deduplicate results from multiple crawlers**

```bash
cat crawl-results.txt gospider-output/* | grep -oE 'https?://[^ ]+' | sort -u > all-urls.txt
wc -l all-urls.txt
# 247 all-urls.txt
```

From the merged URL list, extract unique paths and parameters for further testing. Feed discovered paths into a directory brute-forcer to find additional content in the same directory structures. Pass parameters to injection testing tools. JavaScript files go to LinkFinder for endpoint extraction. This pipeline maximizes coverage before moving into vulnerability discovery.

**Extract unique paths for directory brute-forcing**

```bash
cat all-urls.txt | unfurl paths | sort -u > unique-paths.txt
head -10 unique-paths.txt
# /admin/dashboard
# /api/internal/debug
# /api/v1/users
# /api/v2/admin/settings
# /api/v2/files/upload
# /api/v2/health
# /api/v2/users/profile
# /backup.zip
# /login
# /old-admin/
```

**Extract unique parameters for injection testing**

```bash
cat all-urls.txt | unfurl keys | sort -u
# id
# page
# q
# redirect
# token
# url
# user
```

Parameters like `url` and `redirect` are immediate candidates for SSRF and open redirect testing. `id` and `user` suggest IDOR opportunities. `q` and `page` are common injection points for SQLi and XSS. Each parameter type maps to a specific set of vulnerability tests covered in the vulnerability discovery phase.

## References

- [hakrawler](https://github.com/hakluke/hakrawler) - Go-based web crawler for quick URL discovery.
- [katana](https://github.com/projectdiscovery/katana) - Crawler with JavaScript parsing and headless browser support.
- [gospider](https://github.com/jaeles-project/gospider) - Fast web spider with Wayback Machine and CommonCrawl integration.
- [LinkFinder](https://github.com/GerbenJavado/LinkFinder) - Extracts endpoints from JavaScript files using regex.