# Web Content Discovery

> Techniques for discovering hidden files, directories, and endpoints on web servers using fuzzing tools, wordlists, and response analysis.

- **URL:** https://hackindex.io/services/web/enumeration/content-discovery
- **Path:** HTTP/HTTPS > Enumeration
- **Up:** [Enumeration](https://hackindex.io/services/web/enumeration.md)
- **Published:** 2026-01-23
- **Updated:** 2026-04-24

Content discovery finds endpoints the app doesn't link to: admin panels, backups, APIs, debug paths, and unlinked apps. Complement it with [web crawling](https://hackindex.io/services/web/enumeration/web-crawling) to map the visible attack surface first. Run it after [tech fingerprinting](https://hackindex.io/services/web/enumeration/http-tech-fingerprinting) so you know which extensions and wordlists match the stack. Discovery output feeds into [parameter crawling](https://hackindex.io/services/web/enumeration/parameter-crawling) and [exposed git directory checks](https://hackindex.io/services/web/vulnerability-discovery/check-exposed-git-directory).

## Check robots.txt and sitemap.xml

Before running a wordlist scan, check what the application exposes willingly. `robots.txt` often reveals admin paths, API directories, and staging endpoints that the site owner tried to hide from search engines. `sitemap.xml` maps out the full URL structure:

```bash
curl -sk http://$TARGET_IP:$PORT/robots.txt
# User-agent: *
# Disallow: /admin/
# Disallow: /api/internal/
# Disallow: /backup/
# Allow: /
```

```bash
curl -sk http://$TARGET_IP:$PORT/sitemap.xml
```

Every `Disallow` entry is a path worth testing directly. Combine the results with wordlist output to avoid duplicating effort.

## ffuf Baseline

ffuf is the default choice. Fast, flexible filtering, and handles custom 404s cleanly:

```bash
ffuf -w /usr/share/seclists/Discovery/Web-Content/common.txt -u http://$TARGET_IP:$PORT/FUZZ -mc 200,204,301,302,307,401,403
```

301/302 to a new path is real content. 401/403 is still interesting: it's a valid endpoint behind auth. If everything returns a hit, you're seeing a custom 404 and need size filtering.

## Handle Custom 404s with Size Filtering

Measure the error response size, then exclude it:

```bash
curl -sk http://$TARGET_IP:$PORT/thisshouldnotexist | wc -c
# 4242
```

```bash
ffuf -w /usr/share/seclists/Discovery/Web-Content/common.txt -u http://$TARGET_IP:$PORT/FUZZ -mc 200,204,301,302,307,401,403 -fs 4242
```

Replace 4242 with the actual size. For apps returning varying sizes on 404, filter by word count with `-fw` instead.

## Stack-Matched Extensions

Match extensions to what fingerprinting revealed. Wrong extensions waste significant time on large wordlists:

#### PHP / Apache

```bash
ffuf -w /usr/share/seclists/Discovery/Web-Content/common.txt -u http://$TARGET_IP:$PORT/FUZZ -e .php,.php5,.phtml,.html,.txt,.bak,.old,.zip -mc 200,204,301,302,307,401,403
```

#### ASP.NET / IIS

```bash
ffuf -w /usr/share/seclists/Discovery/Web-Content/common.txt -u http://$TARGET_IP:$PORT/FUZZ -e .asp,.aspx,.ashx,.asmx,.config,.txt,.bak -mc 200,204,301,302,307,401,403
```

#### Java / Tomcat

```bash
ffuf -w /usr/share/seclists/Discovery/Web-Content/common.txt -u http://$TARGET_IP:$PORT/FUZZ -e .jsp,.jspx,.do,.action,.txt,.xml -mc 200,204,301,302,307,401,403
```

## Recursive Discovery

ffuf handles recursion but feroxbuster is faster and more reliable for deep trees:

```bash
ffuf -w /usr/share/seclists/Discovery/Web-Content/common.txt -u http://$TARGET_IP:$PORT/FUZZ -mc 200,204,301,302,307,401,403 -recursion -recursion-depth 2
```

```bash
feroxbuster -u http://$TARGET_IP:$PORT/ -w /usr/share/seclists/Discovery/Web-Content/common.txt -x php,html,txt,js,json -t 30 -k
```

```bash
feroxbuster -u http://$TARGET_IP:$PORT/ -w /usr/share/seclists/Discovery/Web-Content/common.txt -x php,html,txt,js,json -C 404,302 -o ferox_results.txt
```

## Authenticated Discovery

Run a second pass authenticated to find endpoints only visible after login:

```bash
ffuf -w /usr/share/seclists/Discovery/Web-Content/common.txt -u http://$TARGET_IP:$PORT/FUZZ -H "Cookie: session=REPLACE_ME" -mc 200,204,301,302,307,401,403
```

```bash
ffuf -w /usr/share/seclists/Discovery/Web-Content/common.txt -u http://$USER:$PASSWORD@$TARGET_IP:$PORT/FUZZ -mc 200,204,301,302,307,401,403
```

## Larger Wordlists for Deeper Web Content Discovery

When common.txt misses things, upgrade the wordlist. raft and directory-list are heavier but surface more results:

```bash
ffuf -w /usr/share/seclists/Discovery/Web-Content/raft-medium-directories.txt -u http://$TARGET_IP:$PORT/FUZZ -mc 200,204,301,302,307,401,403
```

```bash
ffuf -w /usr/share/seclists/Discovery/Web-Content/directory-list-2.3-medium.txt -u http://$TARGET_IP:$PORT/FUZZ -mc 200,204,301,302,307,401,403 -t 100
```

Wordlist selection by use case:

- `common.txt`: fast first pass, covers standard paths and filenames
- `raft-medium-directories.txt`: broader directory names from real-world crawls
- `directory-list-2.3-medium.txt`: large general-purpose list, slow but thorough
- `big.txt`: exhaustive, use when medium lists miss and you have time
- `quickhits.txt`: targeted checks for config files, backups, and admin paths

### Tech-Specific Wordlists

After fingerprinting the stack, swap in a matching wordlist from SecLists. These surface paths that generic lists miss:

- `Discovery/Web-Content/CGIs.txt`: CGI scripts on Apache, common in legacy apps
- `Discovery/Web-Content/IIS.fuzz.txt`: IIS-specific paths, config files, and handler mappings
- `Discovery/Web-Content/apache.txt`: Apache-specific directories and config paths
- `Discovery/Web-Content/nginx.txt`: nginx status pages, config paths, and common misconfigs
- `Discovery/Web-Content/tomcat.txt`: Tomcat manager, host-manager, and deployment paths
- `Discovery/Web-Content/AdobeCQ-AEM.txt`: Adobe Experience Manager nodes and servlets

Run the generic list first, then the tech-specific one. Overlap is minimal.

## Gobuster and dirsearch as Alternatives

> [!NOTE]
> dirb is deprecated. Wfuzz is still useful for multi-payload fuzzing (e.g., fuzzing path and parameter simultaneously with multiple FUZZ keywords) and built-in encoder chains. For standard single-wordlist directory brute-forcing, ffuf and feroxbuster are faster.

```bash
gobuster dir -u http://$TARGET_IP:$PORT/ -w /usr/share/seclists/Discovery/Web-Content/common.txt -x php,html,txt,js,json --random-agent -t 50 -o gobuster.txt
```

```bash
dirsearch -u http://$TARGET_IP:$PORT/ -w /usr/share/seclists/Discovery/Web-Content/common.txt -e php,html,txt,js,json -i 200,204,301,302,307,401,403 -t 50 -o dirsearch.txt
```

## References

- [ffuf](https://github.com/ffuf/ffuf) - Matchers, filters, recursion, and output formats
- [feroxbuster](https://github.com/epi052/feroxbuster) - Recursive forced browsing and filtering options
- [Gobuster](https://github.com/OJ/gobuster) - dir mode usage and flags
- [SecLists](https://github.com/danielmiessler/SecLists) - Web content wordlists including raft, common, and directory-list