Hack Like a Pro:

Hack Like a Pro: How to Hack Web Apps, Part 2 (Website Spidering with WebScarab)

Welcome back, my tenderfoot hackers!

Now that we have begun this trip down web app hacking lane, we need to first address target reconnaissance. Like any hack, reconnaissance is critical. (Are you tired of me saying that yet?)

There is no better telltale sign of a script-kiddie than a hacker who runs willy-nilly into trying to hack/exploit without doing proper recon. They say they don't "have time" to do proper recon, then when they are invariably unsuccessful, they scratch their head and ask, "Why didn't this hack work?"

For a professional hacker, reconnaissance is often 70% or more of the time we spend on a hack. Since each hack/exploit is specific to a vulnerability, and the vulnerability is specific to the OS, the ports, the apps, the technologies used, and even the language, hacking without recon is simply an exercise in futility.

Web App Reconnaissance

I think it's important to note here that web app reconnaissance is a process and not a tool. Before attacking a web app, you need to gather as much information as possible. In some cases, you will get conflicting information, and when that happens, you may need to run another reconnaissance tool or technique.

Here are some of the tools and tutorials that I have already covered that you can use for reconnaissance.

  • Operating System - The underlying operating system of the target can often be determined by using Nmap, Xprobe2, P0f, or Netcraft.
  • Web Server - The underlying web server can often be determined by Netcraft, banner grabbing with Netcat, Httprint, or Shodan.
  • Web Technologies - The underlying technologies can be determined with Netcraft.
  • DNS - You may need to perform DNS recon to find hidden servers.
  • Wikto - This is an excellent tool for finding so much information on the website including finding hidden directories and Google hacking.
  • DirBuster - OWASP's tool maps nearly every directory in a website and often finds hidden or unknown directories in a website.
  • Maltego - This tool is great for many of the above tasks, as well as social networking relationships.
  • Httrack - This tool enables us to make a copy of the website for online reconnaissance and analysis before exploitation.

Website Spidering

Before a website attack or penetration test, we need to spider the site. Many of the tools we use to attack a site need a map of the website in order to do their work. We could manually spider the site by simply navigating to each page and saving it, but fortunately, we have tools that can save us time and automate this process. The tool we will use here is called WebScarab by OWASP. It's built into Kali, so no need to download or install anything.

Step 1: Fire Up Kali & Open WebScarab

Let's begin by firing up Kali and then navigate to Applications -> Kali Linix -> Web Applications -> Web Crawlers -> webscarab.

Step 2: WebScarab

When we click on the webscarab option, it opens with a GUI interface like that below. As you can see, WebScarab has many web reconnaissance features, but here we will focusing on its ability to spider a website. In later tutorials, we will explore some of its other capabilities.

Step 3: Configure Your Browser

Before we begin spidering a website, we need make certain that your browser is configured properly. By default, WebScarab uses a proxy on on 8008. You can change it by clicking on the "Proxy" tab, but for now, let's keep the default setting and make certain that our browser is using the same setting.

For more information on configuring the proxy setting in IceWeasel, see my tutorial on THC-Hydra and Burp Suite.

Step 4: Point WebScarab at a Website

Now, to see how WebScarab can spider a site, let's point it at our favorite website, WonderHowTo. In the "Allowed Domains" window, simply type in www.wonderhowto.com.

Next, go to your browser, in this case IceWeasel, and navigate to www.wonderhowto.com. When we do so, WebScarab will begin to populate the main window with every web address linked on that page. Note that the webpages are arranged in alphabetical order.

Let's navigate down a bit to the Ns and find https://null-byte.wonderhowto.com/. We can click on it and expand all the links within Null Byte, our favorite hacker training site.

In this way, we can now see every webpage and link on the target website. In a future tutorial, when we begin the attack phase, we will see how we can actually use this information.

Keep coming back, my tenderfoot hackers, as we explore web app hacking!

Cover image via Shutterstock, OWASP


Am I the only one who actually enjoys recon?


Nope, I'm with you there. It's fun stalking people in my case. lol

Although it is interesting to be able to find out so much about a target with these tools at first, ocassionally, constant recon does get slightly tedious. I come home knackered and do not wish to spend hours of my evening performing recon but as OTW said, recon = most crucial part.

Maybe I'm just lazy

Thanks ...
i have a question ...
how to start crawling webste that protected by cloud flare?
scanners ,sqlmap,live http headers plugin ... didn't work ! because it protected by cloud flare ...
please help me to resolve it.

Zap has the ajax spider that might help. You can manually go through each page and every link in the source code. You'll learn a lot about how the site works. Then try guessing every other possible path name. Like if there is a /wp-uploads/2015/ directory there very well could be a /wp-uploads/2014/ directory (for example).

Hi,DILL,thx for your suggestion,I try that but once I clicked attack, I got 403 error, how can I fix that,and any tool I can use?(the web I spider is https format)

Without knowing the details of what you are doing it will be hard to help, but. When you say attack are you just entering the URL into Zap's quick attack option? Or are you actually running the ajax spider through Zap's attack feature in the sites menu?

i followed the above when i navigate to the url,webscarab does not populate

how to install plug-n-hack for iceweasal on kali sana?

PS: iceweasal gives me addon unsigned error when i try to install it.

Share Your Thoughts

  • Hot
  • Latest