Skip to content
HackIndex.io

Web Content Discovery

8 min read Created Jan 23, 2026 Last updated Apr 24, 2026

Content discovery finds endpoints the app doesn't link to: admin panels, backups, APIs, debug paths, and unlinked apps. Complement it with web crawling to map the visible attack surface first. Run it after tech fingerprinting so you know which extensions and wordlists match the stack. Discovery output feeds into parameter crawling and exposed git directory checks.

Check robots.txt and sitemap.xml

Before running a wordlist scan, check what the application exposes willingly. robots.txt often reveals admin paths, API directories, and staging endpoints that the site owner tried to hide from search engines. sitemap.xml maps out the full URL structure:

┌──(kali㉿kali)-[~]
└─$ curl -sk http://$TARGET_IP:$PORT/robots.txt 
User-agent: *
Disallow: /admin/
Disallow: /api/internal/
Disallow: /backup/
Allow: /
Explain command
-sk Silent mode, skip TLS certificate verification.
┌──(kali㉿kali)-[~]
└─$ curl -sk http://$TARGET_IP:$PORT/sitemap.xml

Every Disallow entry is a path worth testing directly. Combine the results with wordlist output to avoid duplicating effort.

ffuf Baseline

ffuf is the default choice. Fast, flexible filtering, and handles custom 404s cleanly:

┌──(kali㉿kali)-[~]
└─$ ffuf -w /usr/share/seclists/Discovery/Web-Content/common.txt -u http://$TARGET_IP:$PORT/FUZZ -mc 200,204,301,302,307,401,403
Explain command
-w /usr/share/seclists/Discovery/Web-Content/common.txt Wordlist file path used for fuzzing input.
-u http://$TARGET_IP:$PORT/FUZZ Target URL with FUZZ keyword marking the injection point.
$TARGET_IP:$PORT Variable placeholders for the target host IP and port.
-mc 200,204,301,302,307,401,403 Match only responses with these specific HTTP status codes.

301/302 to a new path is real content. 401/403 is still interesting: it's a valid endpoint behind auth. If everything returns a hit, you're seeing a custom 404 and need size filtering.

Handle Custom 404s with Size Filtering

Measure the error response size, then exclude it:

┌──(kali㉿kali)-[~]
└─$ curl -sk http://$TARGET_IP:$PORT/thisshouldnotexist | wc -c 
4242
Explain command
-s Silent mode; suppresses progress meter and error messages.
-k Allow insecure TLS connections, skipping certificate verification.
http://$TARGET_IP:$PORT/thisshouldnotexist Target URL with variable host/port to a non-existent path.
┌──(kali㉿kali)-[~]
└─$ ffuf -w /usr/share/seclists/Discovery/Web-Content/common.txt -u http://$TARGET_IP:$PORT/FUZZ -mc 200,204,301,302,307,401,403 -fs 4242
Explain command
-w /usr/share/seclists/Discovery/Web-Content/common.txt Wordlist file path used for fuzzing input.
-u http://$TARGET_IP:$PORT/FUZZ Target URL with FUZZ keyword as the injection point.
$TARGET_IP:$PORT Variable placeholders for the target host IP and port.
-mc 200,204,301,302,307,401,403 Match only responses with these HTTP status codes.
-fs 4242 Filter out responses with a body size of 4242 bytes.

Replace 4242 with the actual size. For apps returning varying sizes on 404, filter by word count with -fw instead.

Stack-Matched Extensions

Match extensions to what fingerprinting revealed. Wrong extensions waste significant time on large wordlists:

┌──(kali㉿kali)-[~]
└─$ ffuf -w /usr/share/seclists/Discovery/Web-Content/common.txt -u http://$TARGET_IP:$PORT/FUZZ -e .php,.php5,.phtml,.html,.txt,.bak,.old,.zip -mc 200,204,301,302,307,401,403
Explain command
-w /usr/share/seclists/Discovery/Web-Content/common.txt Wordlist file used for fuzzing input.
-u http://$TARGET_IP:$PORT/FUZZ Target URL with FUZZ keyword marking the injection point.
$TARGET_IP:$PORT Variable placeholders for the target host IP and port.
-e .php,.php5,.phtml,.html,.txt,.bak,.old,.zip File extensions to append to each wordlist entry.
-mc 200,204,301,302,307,401,403 Match only responses with these HTTP status codes.
┌──(kali㉿kali)-[~]
└─$ ffuf -w /usr/share/seclists/Discovery/Web-Content/common.txt -u http://$TARGET_IP:$PORT/FUZZ -e .asp,.aspx,.ashx,.asmx,.config,.txt,.bak -mc 200,204,301,302,307,401,403
Explain command
-w /usr/share/seclists/Discovery/Web-Content/common.txt Wordlist file to use for fuzzing input.
-u http://$TARGET_IP:$PORT/FUZZ Target URL with FUZZ keyword as the injection point.
$TARGET_IP:$PORT Variable placeholders for target host IP and port.
-e .asp,.aspx,.ashx,.asmx,.config,.txt,.bak Append these extensions to each wordlist entry during fuzzing.
-mc 200,204,301,302,307,401,403 Match only responses with these HTTP status codes.
┌──(kali㉿kali)-[~]
└─$ ffuf -w /usr/share/seclists/Discovery/Web-Content/common.txt -u http://$TARGET_IP:$PORT/FUZZ -e .jsp,.jspx,.do,.action,.txt,.xml -mc 200,204,301,302,307,401,403
Explain command
-w /usr/share/seclists/Discovery/Web-Content/common.txt Wordlist file to use for fuzzing input.
-u http://$TARGET_IP:$PORT/FUZZ Target URL with FUZZ as the injection point placeholder.
$TARGET_IP:$PORT Variable placeholders for the target host IP and port.
-e .jsp,.jspx,.do,.action,.txt,.xml Append these extensions to each wordlist entry during fuzzing.
-mc 200,204,301,302,307,401,403 Match only responses with these specific HTTP status codes.

Recursive Discovery

ffuf handles recursion but feroxbuster is faster and more reliable for deep trees:

┌──(kali㉿kali)-[~]
└─$ ffuf -w /usr/share/seclists/Discovery/Web-Content/common.txt -u http://$TARGET_IP:$PORT/FUZZ -mc 200,204,301,302,307,401,403 -recursion -recursion-depth 2
Explain command
-w /usr/share/seclists/Discovery/Web-Content/common.txt Wordlist file to use for fuzzing input.
-u http://$TARGET_IP:$PORT/FUZZ Target URL with FUZZ keyword marking the injection point.
$TARGET_IP:$PORT Placeholder for target host IP address and port number.
-mc 200,204,301,302,307,401,403 Match only responses with these HTTP status codes.
-recursion Enable recursive fuzzing on discovered directories.
-recursion-depth 2 Limit recursive fuzzing to a maximum depth of 2 levels.
┌──(kali㉿kali)-[~]
└─$ feroxbuster -u http://$TARGET_IP:$PORT/ -w /usr/share/seclists/Discovery/Web-Content/common.txt -x php,html,txt,js,json -t 30 -k
Explain command
-u http://$TARGET_IP:$PORT/ Target URL to perform directory/file brute-forcing against.
-w /usr/share/seclists/Discovery/Web-Content/common.txt Wordlist file used for brute-forcing paths and filenames.
-x php,html,txt,js,json File extensions to append to each wordlist entry during fuzzing.
-t 30 Number of concurrent threads to use for scanning.
-k Disable TLS certificate validation for HTTPS targets.
┌──(kali㉿kali)-[~]
└─$ feroxbuster -u http://$TARGET_IP:$PORT/ -w /usr/share/seclists/Discovery/Web-Content/common.txt -x php,html,txt,js,json -C 404,302 -o ferox_results.txt
Explain command
-u Target URL to begin directory/file brute-forcing against.
http://$TARGET_IP:$PORT/ Base URL with variable host and port placeholders.
-w Wordlist file path to use for brute-forcing.
/usr/share/seclists/Discovery/Web-Content/common.txt Common web content wordlist from SecLists collection.
-x File extensions to append to each wordlist entry during fuzzing.
php,html,txt,js,json Extensions to test: PHP, HTML, plain text, JavaScript, and JSON.
-C HTTP status codes to filter out (exclude) from results.
404,302 Suppresses responses with Not Found and Redirect status codes.
-o Output file path to save scan results.
ferox_results.txt File where discovered results will be written.

Authenticated Discovery

Run a second pass authenticated to find endpoints only visible after login:

┌──(kali㉿kali)-[~]
└─$ ffuf -w /usr/share/seclists/Discovery/Web-Content/common.txt -u http://$TARGET_IP:$PORT/FUZZ -H "Cookie: session=REPLACE_ME" -mc 200,204,301,302,307,401,403
Explain command
-w /usr/share/seclists/Discovery/Web-Content/common.txt Wordlist file to use for fuzzing input.
-u http://$TARGET_IP:$PORT/FUZZ Target URL with FUZZ keyword marking the injection point.
$TARGET_IP:$PORT Placeholder for the target host IP and port number.
-H "Cookie: session=REPLACE_ME" Adds a custom Cookie header with a session token to each request.
-mc 200,204,301,302,307,401,403 Match only responses with these specific HTTP status codes.
┌──(kali㉿kali)-[~]
└─$ ffuf -w /usr/share/seclists/Discovery/Web-Content/common.txt -u http://$USER:$PASSWORD@$TARGET_IP:$PORT/FUZZ -mc 200,204,301,302,307,401,403
Explain command
-w /usr/share/seclists/Discovery/Web-Content/common.txt Wordlist file to use for fuzzing.
-u http://$USER:$PASSWORD@$TARGET_IP:$PORT/FUZZ Target URL with HTTP Basic Auth credentials embedded in the URL.
-mc 200,204,301,302,307,401,403 Match HTTP response codes to include in results.

Larger Wordlists for Deeper Web Content Discovery

When common.txt misses things, upgrade the wordlist. raft and directory-list are heavier but surface more results:

┌──(kali㉿kali)-[~]
└─$ ffuf -w /usr/share/seclists/Discovery/Web-Content/raft-medium-directories.txt -u http://$TARGET_IP:$PORT/FUZZ -mc 200,204,301,302,307,401,403
Explain command
-w /usr/share/seclists/Discovery/Web-Content/raft-medium-directories.txt Wordlist file used for fuzzing directory names.
-u http://$TARGET_IP:$PORT/FUZZ Target URL with FUZZ keyword marking the injection point.
$TARGET_IP:$PORT Variable placeholders for the target host IP and port.
-mc 200,204,301,302,307,401,403 Match only responses with these specific HTTP status codes.
┌──(kali㉿kali)-[~]
└─$ ffuf -w /usr/share/seclists/Discovery/Web-Content/directory-list-2.3-medium.txt -u http://$TARGET_IP:$PORT/FUZZ -mc 200,204,301,302,307,401,403 -t 100
Explain command
-w /usr/share/seclists/Discovery/Web-Content/directory-list-2.3-medium.txt Wordlist file to use for fuzzing input.
-u http://$TARGET_IP:$PORT/FUZZ Target URL with FUZZ keyword marking the injection point.
$TARGET_IP:$PORT Placeholder for the target host IP and port number.
-mc 200,204,301,302,307,401,403 Match only responses with these HTTP status codes.
-t 100 Number of concurrent threads to use for requests.

Wordlist selection by use case:

  • common.txt: fast first pass, covers standard paths and filenames
  • raft-medium-directories.txt: broader directory names from real-world crawls
  • directory-list-2.3-medium.txt: large general-purpose list, slow but thorough
  • big.txt: exhaustive, use when medium lists miss and you have time
  • quickhits.txt: targeted checks for config files, backups, and admin paths

Tech-Specific Wordlists

After fingerprinting the stack, swap in a matching wordlist from SecLists. These surface paths that generic lists miss:

  • Discovery/Web-Content/CGIs.txt: CGI scripts on Apache, common in legacy apps
  • Discovery/Web-Content/IIS.fuzz.txt: IIS-specific paths, config files, and handler mappings
  • Discovery/Web-Content/apache.txt: Apache-specific directories and config paths
  • Discovery/Web-Content/nginx.txt: nginx status pages, config paths, and common misconfigs
  • Discovery/Web-Content/tomcat.txt: Tomcat manager, host-manager, and deployment paths
  • Discovery/Web-Content/AdobeCQ-AEM.txt: Adobe Experience Manager nodes and servlets

Run the generic list first, then the tech-specific one. Overlap is minimal.

Gobuster and dirsearch as Alternatives

Notice

dirb is deprecated. Wfuzz is still useful for multi-payload fuzzing (e.g., fuzzing path and parameter simultaneously with multiple FUZZ keywords) and built-in encoder chains. For standard single-wordlist directory brute-forcing, ffuf and feroxbuster are faster.

┌──(kali㉿kali)-[~]
└─$ gobuster dir -u http://$TARGET_IP:$PORT/ -w /usr/share/seclists/Discovery/Web-Content/common.txt -x php,html,txt,js,json --random-agent -t 50 -o gobuster.txt
Explain command
dir Use directory/file brute-forcing mode.
-u Target URL to enumerate.
http://$TARGET_IP:$PORT/ Base URL with variable host and port placeholders.
-w Wordlist file path to use for brute-forcing.
/usr/share/seclists/Discovery/Web-Content/common.txt Common web content wordlist from SecLists.
-x File extensions to append to each wordlist entry.
php,html,txt,js,json Extensions to try: php, html, txt, js, and json.
--random-agent Use a random User-Agent string per request.
-t Number of concurrent threads to use.
50 Thread count value: 50 parallel requests.
-o Write output results to a specified file.
gobuster.txt Output file where results will be saved.
┌──(kali㉿kali)-[~]
└─$ dirsearch -u http://$TARGET_IP:$PORT/ -w /usr/share/seclists/Discovery/Web-Content/common.txt -e php,html,txt,js,json -i 200,204,301,302,307,401,403 -t 50 -o dirsearch.txt
Explain command
-u http://$TARGET_IP:$PORT/ Target URL to perform directory brute-forcing against.
-w /usr/share/seclists/Discovery/Web-Content/common.txt Wordlist file used for directory and file enumeration.
-e php,html,txt,js,json File extensions to append to each wordlist entry during scan.
-i 200,204,301,302,307,401,403 Only include responses matching these HTTP status codes.
-t 50 Number of concurrent threads to use during the scan.
-o dirsearch.txt Write scan output/results to the specified file.

References