How to Scan Websites for Interesting Directories & Files with Gobuster

One of the first steps in attacking a web application is enumerating hidden directories and files. Doing so can often yield valuable information that makes it easier to execute a precise attack, leaving less room for errors and wasted time. There are many tools available to do this, but not all of them are created equally. Gobuster, a directory scanner written in Go, is definitely worth exploring.

Traditional directory brute-force scanners like DirBuster and DIRB work just fine, but can often be slow and prone to errors. Gobuster is a Go implementation of these tools and is offered in a convenient command-line format.

The main advantage Gobuster has over other directory scanners is speed. As a programming language, Go is known to be fast. It also has excellent support for concurrency so that Gobuster can take advantage of multiple threads for faster processing.

The one downfall of Gobuster, though, is the lack of recursive directory searching. For directories more than one level deep, another scan will be needed, unfortunately. Often this isn't that big of a deal, and other scanners can step up and fill in the gaps for Gobuster in this area.

• Don't Miss: Use Websploit to Scan Websites for Hidden Directories

Gobuster offers a simple command-line interface that just works. It has some useful options, but not so many that it's easy to get bogged down in the details. All in all, it's a great tool that is effective and fast. In this tutorial, we'll be exploring it with DVWA (Damn Vulnerable Web App) as the target and Kali Linux as the attacking machine. You can follow along with those or use a similar testing configuration.

Step 1: Install Gobuster

The first thing we can do is create a working directory to keep things neat, then change into it.

~# mkdir gobuster
~# cd gobuster/

Next, we need to install Gobuster since it is not included on Kali by default.

~/gobuster# apt-get install gobuster

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  gobuster
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/1,532 kB of archives.
After this operation, 4,963 kB of additional disk space will be used.
Selecting previously unselected package gobuster.
(Reading database ... 412624 files and directories currently installed.)
Preparing to unpack .../gobuster_2.0.1-1_amd64.deb ...
Unpacking gobuster (2.0.1-1) ...
Setting up gobuster (2.0.1-1) ...
Processing triggers for man-db (2.8.5-2) ...

Then, simply type gobuster in the terminal to run the tool.

~/gobuster# gobuster

2019/05/06 11:43:08 [!] 2 errors occurred:
    * WordList (-w): Must be specified (use `-w -` for stdin)
    * Url/Domain (-u): Must be specified

We can see it gives us a couple of errors. It needs at least two parameters (-u for the URL, and -w for the wordlist) to run properly. We can also display the help menu with the -h flag.

~/gobuster# gobuster -h

Usage of gobuster:
  -P string
        Password for Basic Auth (dir mode only)
  -U string
        Username for Basic Auth (dir mode only)
  -a string
        Set the User-Agent string (dir mode only)
  -c string
        Cookies to use for the requests (dir mode only)
  -cn
        Show CNAME records (dns mode only, cannot be used with '-i' option)
  -e    Expanded mode, print full URLs
  -f    Append a forward-slash to each directory request (dir mode only)
  -fw
        Force continued operation when wildcard found
  -i    Show IP addresses (dns mode only)
  -k    Skip SSL certificate verification
  -l    Include the length of the body in the output (dir mode only)
  -m string
        Directory/File mode (dir) or DNS mode (dns) (default "dir")
  -n    Don't print status codes
  -np
        Don't display progress
  -o string
        Output file to write results to (defaults to stdout)
  -p string
        Proxy to use for requests [http(s)://host:port] (dir mode only)
  -q    Don't print the banner and other noise
  -r    Follow redirects
  -s string
        Positive status codes (dir mode only) (default "200,204,301,302,307,403")
  -t int
        Number of concurrent threads (default 10)
  -to duration
        HTTP Timeout in seconds (dir mode only) (default 10s)
  -u string
        The target URL or Domain
  -v    Verbose output (errors)
  -w string
        Path to the wordlist
  -x string
        File extension(s) to search for (dir mode only)

Step 2: Install Some Extra Wordlists

Wordlists on Kali are located in the /usr/share/wordlists directory.

~/gobuster# ls /usr/share/wordlists/

dirb  dirbuster  dnsmap.txt  fasttrack.txt  fern-wifi  metasploit  nmap.lst  rockyou.txt.gz  sqlmap.txt  wfuzz

The dirb and dirbuster ones are fine, but there is another wordlist I like to use for directory brute-forcing. There is a whole repository of useful wordlists on GitHub called SecLists.

The "common.txt" wordlist contains a good number of common directory names.

We can download the raw file into our current directory using the wget utility.

~/gobuster# wget https://raw.githubusercontent.com/danielmiessler/SecLists/master/Discovery/Web-Content/common.txt

--2019-05-06 11:46:40--  https://raw.githubusercontent.com/danielmiessler/SecLists/master/Discovery/Web-Content/common.txt
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.148.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.148.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 35744 (35K) [text/plain]
Saving to: ‘common.txt’

common.txt                                            100%[======================================================================================================================>]  34.91K  --.-KB/s    in 0.03s

2019-05-06 11:46:40 (1.24 MB/s) - ‘common.txt’ saved [35744/35744]

Alternatively, if we wanted to install the whole SecLists repository, we can do so with the package manager.

~/gobuster# apt-get install seclists

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  seclists
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/221 MB of archives.
After this operation, 795 MB of additional disk space will be used.
Selecting previously unselected package seclists.
(Reading database ... 412629 files and directories currently installed.)
Preparing to unpack .../seclists_2019.1-0kali1_all.deb ...
Unpacking seclists (2019.1-0kali1) ...
Setting up seclists (2019.1-0kali1) ...

That will install the entirety of SecLists in the /usr/share directory. It contains a ton of content, and if you're looking for a go-to set of wordlists, SecLists will likely be the last thing you'll ever need.

/usr/share/seclists# ls

Discovery  Fuzzing  IOCs  Miscellaneous  Passwords  Pattern-Matching  Payloads  README.md  Usernames  Web-Shells

Step 3: Scan Directories & Files

Now that everything is set up and installed, we're ready to use Gobuster. Let's run it against our target with the default parameters.

~/gobuster# gobuster -u http://10.10.0.50/dvwa/ -w common.txt

=====================================================
Gobuster v2.0.1              OJ Reeves (@TheColonial)
=====================================================
[+] Mode         : dir
[+] Url/Domain   : http://10.10.0.50/dvwa/
[+] Threads      : 10
[+] Wordlist     : common.txt
[+] Status codes : 200,204,301,302,307,403
[+] Timeout      : 10s
=====================================================
2019/05/06 11:50:25 Starting gobuster
=====================================================
/.htaccess (Status: 403)
/.htpasswd (Status: 403)
/.hta (Status: 403)
/README (Status: 200)
/config (Status: 301)
/docs (Status: 301)
/about (Status: 302)
/external (Status: 301)
/favicon.ico (Status: 200)
/cmd (Status: 200)
/index (Status: 302)
/index.php (Status: 302)
/php.ini (Status: 200)
/instructions (Status: 302)
/logout (Status: 302)
/robots (Status: 200)
/robots.txt (Status: 200)
/login (Status: 200)
/phpinfo (Status: 302)
/phpinfo.php (Status: 302)
/setup (Status: 200)
/security (Status: 302)
=====================================================
2019/05/06 11:50:38 Finished
=====================================================

We can see it gives us some information about the tool in the banner, and then it kicks off the directory discovery process. It returns the names of the directories and files as well as their status codes.

We can have it return the length of the body with the -l flag, which can be useful for enumeration.

~/gobuster# gobuster -u http://10.10.0.50/dvwa/ -w common.txt -l

=====================================================
Gobuster v2.0.1              OJ Reeves (@TheColonial)
=====================================================
[+] Mode         : dir
[+] Url/Domain   : http://10.10.0.50/dvwa/
[+] Threads      : 10
[+] Wordlist     : common.txt
[+] Status codes : 200,204,301,302,307,403
[+] Show length  : true
[+] Timeout      : 10s
=====================================================
2019/05/06 11:52:41 Starting gobuster
=====================================================
/.hta (Status: 403) [Size: 292]
/.htaccess (Status: 403) [Size: 297]
/.htpasswd (Status: 403) [Size: 297]
/README (Status: 200) [Size: 4934]
/config (Status: 301) [Size: 319]
/docs (Status: 301) [Size: 317]
/external (Status: 301) [Size: 321]
/favicon.ico (Status: 200) [Size: 1406]
2019/05/06 11:52:52 [!] Get http://10.10.0.50/dvwa/about: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
/php.ini (Status: 200) [Size: 148]
2019/05/06 11:52:54 [!] Get http://10.10.0.50/dvwa/cmd: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
/robots (Status: 200) [Size: 26]
/robots.txt (Status: 200) [Size: 26]
2019/05/06 11:52:59 [!] Get http://10.10.0.50/dvwa/index: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2019/05/06 11:52:59 [!] Get http://10.10.0.50/dvwa/index.php: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2019/05/06 11:52:59 [!] Get http://10.10.0.50/dvwa/instructions: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2019/05/06 11:53:00 [!] Get http://10.10.0.50/dvwa/login: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2019/05/06 11:53:00 [!] Get http://10.10.0.50/dvwa/logout: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
/phpinfo.php (Status: 302) [Size: 0]
/phpinfo (Status: 302) [Size: 0]
/security (Status: 302) [Size: 0]
/setup (Status: 200) [Size: 3549]
=====================================================
2019/05/06 11:53:01 Finished
=====================================================

Usually, if something is zero bytes, it isn't even worth looking into. It can save loads of time, especially when dealing with a large website or a large number of directories. If we only want specific status codes to be displayed, we can do that using the -s flag followed by the code we want.

~/gobuster# gobuster -u http://10.10.0.50/dvwa/ -w common.txt -s 200

=====================================================
Gobuster v2.0.1              OJ Reeves (@TheColonial)
=====================================================
[+] Mode         : dir
[+] Url/Domain   : http://10.10.0.50/dvwa/
[+] Threads      : 10
[+] Wordlist     : common.txt
[+] Status codes : 200
[+] Timeout      : 10s
=====================================================
2019/05/06 11:54:16 Starting gobuster
=====================================================
/README (Status: 200)
/favicon.ico (Status: 200)
/cmd (Status: 200)
/php.ini (Status: 200)
/login (Status: 200)
/robots (Status: 200)
/robots.txt (Status: 200)
/setup (Status: 200)
=====================================================
2019/05/06 11:54:27 Finished
=====================================================

Use commas to specify multiple codes.

~/gobuster# gobuster -u http://10.10.0.50/dvwa/ -w common.txt -s 200,301

=====================================================
Gobuster v2.0.1              OJ Reeves (@TheColonial)
=====================================================
[+] Mode         : dir
[+] Url/Domain   : http://10.10.0.50/dvwa/
[+] Threads      : 10
[+] Wordlist     : common.txt
[+] Status codes : 200,301
[+] Timeout      : 10s
=====================================================
2019/05/06 11:54:58 Starting gobuster
=====================================================
/README (Status: 200)
/config (Status: 301)
/docs (Status: 301)
/external (Status: 301)
/favicon.ico (Status: 200)
/cmd (Status: 200)
/php.ini (Status: 200)
/login (Status: 200)
/robots (Status: 200)
/robots.txt (Status: 200)
/setup (Status: 200)
=====================================================
2019/05/06 11:55:10 Finished
=====================================================

Let's say we just wanted a quick way to view the directories, without the extra noise of the banner and status codes. Use the -q flag to hide the banner, and the -n flag to hide the status codes.

~/gobuster# gobuster -u http://10.10.0.50/dvwa/ -w common.txt -q -n

/.hta
/.htpasswd
/.htaccess
/README
/config
/docs
/external
/favicon.ico
/about
/cmd
/php.ini
/index
/index.php
/instructions
/logout
/robots
/robots.txt
/login
/phpinfo.php
/phpinfo
/setup
/security

Another useful feature is the ability to save the results to a file. Use the -o flag to specify the output file.

~/gobuster# gobuster -u http://10.10.0.50/dvwa/ -w common.txt -o results

Now the results are saved and can be viewed at a later time.

~/gobuster# cat results

/.hta (Status: 403)
/.htaccess (Status: 403)
/.htpasswd (Status: 403)
/README (Status: 200)
/config (Status: 301)
/docs (Status: 301)
/external (Status: 301)
/favicon.ico (Status: 200)
/about (Status: 302)
/cmd (Status: 200)
/php.ini (Status: 200)
/instructions (Status: 302)
/index (Status: 302)
/logout (Status: 302)
/index.php (Status: 302)
/login (Status: 200)
/robots (Status: 200)
/robots.txt (Status: 200)
/phpinfo (Status: 302)
/phpinfo.php (Status: 302)
/security (Status: 302)
/setup (Status: 200)

Wrapping Up

In this tutorial, we learned about Gobuster, a directory brute-force scanner written in the Go programming language. First, we learned how to install the tool, as well as some useful wordlists not found on Kali by default. Next, we ran it against our target and explored some of the various options it ships with. The bottom line: Gobuster is a fast and powerful directory scanner that should be an essential part of any hacker's repertoire, and now you know how to use it. Go!

Cover image by Pixabay/Pexels; Screenshots by drd_/Null Byte