How To: Use WebTech to Discover What Technologies a Website Uses

Correctly identifying the underlying technologies that run on a website gives pentesters a considerable advantage when preparing an attack. Whether you're testing out the defenses of a large corporation or playing the latest CTF, figuring out what technologies a site uses is a crucial pen-tester skill.

Knowing the technology and codebase used to build a site can speed things up by eliminating potential attack vectors or exploits that we know won't work. But it can also reduce the chances of your penetration being detected by raising fewer alarms.

Today we will be exploring a tool called WebTech to bring these technologies to light.

Why WebTech?

WebTech is an open-source Python tool used to identify web technologies. You can utilize it in a variety of ways, including as a command line tool, as a Burp Suite extension, and as a Python library for scripting.

There are other tools available to accomplish this goal, such as Wappalyzer, a commonly used browser extension, or other online alternatives like W3Techs or this one from Pentest-Tools. But WebTech stands out in a way that is extremely modular and easy to use, especially when it comes to Python scripting.

The reconnaissance phase is essential in the ethical hacking or penetration testing process. The old saying "measure twice, cut once" holds here.

Install & Basic Usage

In order to use WebTech, we must first install it. While you can use WebTech on any operating system that supports Python, I'll be showing you here how it works in Kali Linux (or any other Debian-based distro).

First, make sure you have Python installed on your device — we can check with the which command:

~# which python

/usr/bin/python

If you don't see any output from this, install it with the package manager:

~# apt-get install python

Then, install pip, a package management system for Python, with the following command:

~# apt-get install python-pip

Now we can finally install WebTech:

~# pip install webtech

Collecting webtech
  Downloading https://files.pythonhosted.org/packages/a7/66/3bd231369ca661e76fa863546c2d7d8c73fd214fc018dcee37ff32a368d8/webtech-1.2.7.tar.gz (103kB)
    100% |████████████████████████████████| 112kB 1.5MB/s
Requirement already satisfied: requests in /usr/lib/python2.7/dist-packages (from webtech) (2.21.0)
Building wheels for collected packages: webtech
  Running setup.py bdist_wheel for webtech ... done
  Stored in directory: /root/.cache/pip/wheels/36/0d/d6/67a0bbbfd449ecb578cac82c098668ef032dbd513640257c94
Successfully built webtech
Installing collected packages: webtech
Successfully installed webtech-1.2.7

Simply type webtech in the terminal to display its usage and options:

~# webtech

No URL(s) given!
Usage: webtech [options]

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -u URLS, --urls=URLS  url(s) to scan
  --urls-file=URLS_FILE, --ul=URLS_FILE
                        url(s) list file to scan
  --user-agent=USER_AGENT, --ua=USER_AGENT
                        use this user agent
  --random-user-agent, --rua
                        use a random user agent
  --database-file=DATABASE_FILE, --db=DATABASE_FILE
                        custom database file
  --json, --oj          output json-encoded report
  --grep, --og          output grepable report
  --update-db, --udb    force update of remote db files
  --timeout=TIMEOUT     maximum timeout for scrape requests

The most basic usage of the tool is with the -u flag to specify a URL to scan:

~# webtech -u https://null-byte.wonderhowto.com/

Target URL: https://null-byte.wonderhowto.com/
Detected technologies:
    - jQuery 1.7
    - Google Font API
    - comScore
Detected the following interesting custom headers:
    - Server: WonderHowTo
    - X-UA-Compatible: IE=Edge,chrome=1
    - X-Server-Name: APP02

Here we can see the technologies it detected that are used by the site, as well as a few interesting headers. This information can be useful when preparing an attack, since cutting down unneeded extra variables can drastically decrease the time needed to be successful.

This tool also lets us specify a custom user agent, which can sometimes be utilized to probe the site for different responses depending on how it's set up. User agents are a means of identifying the browser and OS to the web server, sent as a string of text in HTTP headers.

Here is a database of virtually any user agent you can think of to help with your probing.

Find the user agent you want to test and add it into this string after your URL --ua='[USERAGENTCODE]' instead of USERAGENTCODE. Here's what that looks like:

~# webtech -u https://null-byte.wonderhowto.com/ --ua='Mozilla/5.0 (Linux; Android 6.0.1; SM-G920V Build/MMB29K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.98 Mobile Safari/537.36'

Target URL: https://null-byte.wonderhowto.com/
Detected technologies:
    - jQuery 1.7
    - Google Font API
    - comScore
Detected the following interesting custom headers:
    - Server: WonderHowTo
    - X-UA-Compatible: IE=Edge,chrome=1
    - X-Server-Name: APP02

Here we set the user agent to mimic a Samsung Galaxy S6, although we did not get any different results. Sometimes, for example, if there is a bug in a specific browser, the server will send a different response depending on the user agent.

We could also set a random user agent if we wanted to, using the --random-user-agent flag, which will randomly select a user agent to use.

WebTech's command line usage is definitely helpful when identifying the underlying technologies of a website, but where it really shines is its ability to be used in scripts.

Scripting with WebTech

You can use WebTech in any Python script by merely importing the library. This flexibility is the most useful feature of the tool, in my opinion, since you can integrate it into any other script where determining what a website is built on is important.

For example, WebTech could be used as part of a larger fingerprinting script alongside port scans and service enumeration. It would also be useful in exploit scripts, where the details of the exploit change slightly depending on the platform.

Let's write a quick demo script to show it in action. Create your Python file with your text editor of choice — in this case, I'll use nano since it is easy:

~# nano scan.py

The first line should tell the script how to run by pointing to our Python binary:

#!/usr/bin/python

Next, we need an import statement to import the WebTech library:

import webtech

Then we can start a new instance with the wt variable:

wt = webtech.WebTech()

And start a scan of the desired URL (I had to also set a short timeout or it wouldn't work) and save that to the results variable:

results = wt.start_from_url('https://null-byte.wonderhowto.com/', timeout=1)

Finally, we can print the results of the scan to the screen:

print results

The final script should look like this:

#!/usr/bin/python

import webtech

wt = webtech.WebTech()

results = wt.start_from_url('https://null-byte.wonderhowto.com/', timeout=1)
print results

We can now run our script with the python command we set up above:

~# python scan.py

Target URL: https://null-byte.wonderhowto.com/
Detected technologies:
    - jQuery 1.7
    - Google Font API
    - comScore
Detected the following interesting custom headers:
    - Server: WonderHowTo
    - X-UA-Compatible: IE=Edge,chrome=1
    - X-Server-Name: APP03

Here we can see that we've obtained the same results as before. Keep in mind, this was just a simple proof-of-concept — we could make this a lot more robust if we wanted.

What this does is prove how potentially useful WebTech could be when integrating it into other scripts or tools.

Wrapping Up

In this article, we learned about WebTech — a Python tool used to identify website technologies.

We saw how to install it with pip and run it from the command line with ease. We also explored its true power, the ability to be used as a Python library, by writing our little script. WebTech makes it easy to get to know your target during the recon phase, giving you the advantage when planning an attack.