Correctly identifying the underlying technologies that run on a website gives pentesters a considerable advantage when preparing an attack. Whether you're testing out the defenses of a large corporation or playing the latest CTF, figuring out what technologies a site uses is a crucial pen-tester skill.
Knowing the technology and codebase used to build a site can speed things up by eliminating potential attack vectors or exploits that we know won't work. But it can also reduce the chances of your penetration being detected by raising fewer alarms.
Today we will be exploring a tool called WebTech to bring these technologies to light.
WebTech is an open-source Python tool used to identify web technologies. You can utilize it in a variety of ways, including as a command line tool, as a Burp Suite extension, and as a Python library for scripting.
There are other tools available to accomplish this goal, such as Wappalyzer, a commonly used browser extension, or other online alternatives like W3Techs or this one from Pentest-Tools. But WebTech stands out in a way that is extremely modular and easy to use, especially when it comes to Python scripting.
The reconnaissance phase is essential in the ethical hacking or penetration testing process. The old saying "measure twice, cut once" holds here.
In order to use WebTech, we must first install it. While you can use WebTech on any operating system that supports Python, I'll be showing you here how it works in Kali Linux (or any other Debian-based distro).
First, make sure you have Python installed on your device — we can check with the which command:
~# which python /usr/bin/python
If you don't see any output from this, install it with the package manager:
~# apt-get install python
Then, install pip, a package management system for Python, with the following command:
~# apt-get install python-pip
Now we can finally install WebTech:
~# pip install webtech Collecting webtech Downloading https://files.pythonhosted.org/packages/a7/66/3bd231369ca661e76fa863546c2d7d8c73fd214fc018dcee37ff32a368d8/webtech-1.2.7.tar.gz (103kB) 100% |████████████████████████████████| 112kB 1.5MB/s Requirement already satisfied: requests in /usr/lib/python2.7/dist-packages (from webtech) (2.21.0) Building wheels for collected packages: webtech Running setup.py bdist_wheel for webtech ... done Stored in directory: /root/.cache/pip/wheels/36/0d/d6/67a0bbbfd449ecb578cac82c098668ef032dbd513640257c94 Successfully built webtech Installing collected packages: webtech Successfully installed webtech-1.2.7
Simply type webtech in the terminal to display its usage and options:
~# webtech No URL(s) given! Usage: webtech [options] Options: --version show program's version number and exit -h, --help show this help message and exit -u URLS, --urls=URLS url(s) to scan --urls-file=URLS_FILE, --ul=URLS_FILE url(s) list file to scan --user-agent=USER_AGENT, --ua=USER_AGENT use this user agent --random-user-agent, --rua use a random user agent --database-file=DATABASE_FILE, --db=DATABASE_FILE custom database file --json, --oj output json-encoded report --grep, --og output grepable report --update-db, --udb force update of remote db files --timeout=TIMEOUT maximum timeout for scrape requests
The most basic usage of the tool is with the -u flag to specify a URL to scan:
~# webtech -u https://null-byte.wonderhowto.com/ Target URL: https://null-byte.wonderhowto.com/ Detected technologies: - jQuery 1.7 - Google Font API - comScore Detected the following interesting custom headers: - Server: WonderHowTo - X-UA-Compatible: IE=Edge,chrome=1 - X-Server-Name: APP02
Here we can see the technologies it detected that are used by the site, as well as a few interesting headers. This information can be useful when preparing an attack, since cutting down unneeded extra variables can drastically decrease the time needed to be successful.
This tool also lets us specify a custom user agent, which can sometimes be utilized to probe the site for different responses depending on how it's set up. User agents are a means of identifying the browser and OS to the web server, sent as a string of text in HTTP headers.
Here is a database of virtually any user agent you can think of to help with your probing.
Find the user agent you want to test and add it into this string after your URL --ua='[USERAGENTCODE]' instead of USERAGENTCODE. Here's what that looks like:
~# webtech -u https://null-byte.wonderhowto.com/ --ua='Mozilla/5.0 (Linux; Android 6.0.1; SM-G920V Build/MMB29K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.98 Mobile Safari/537.36' Target URL: https://null-byte.wonderhowto.com/ Detected technologies: - jQuery 1.7 - Google Font API - comScore Detected the following interesting custom headers: - Server: WonderHowTo - X-UA-Compatible: IE=Edge,chrome=1 - X-Server-Name: APP02
Here we set the user agent to mimic a Samsung Galaxy S6, although we did not get any different results. Sometimes, for example, if there is a bug in a specific browser, the server will send a different response depending on the user agent.
We could also set a random user agent if we wanted to, using the --random-user-agent flag, which will randomly select a user agent to use.
WebTech's command line usage is definitely helpful when identifying the underlying technologies of a website, but where it really shines is its ability to be used in scripts.
You can use WebTech in any Python script by merely importing the library. This flexibility is the most useful feature of the tool, in my opinion, since you can integrate it into any other script where determining what a website is built on is important.
For example, WebTech could be used as part of a larger fingerprinting script alongside port scans and service enumeration. It would also be useful in exploit scripts, where the details of the exploit change slightly depending on the platform.
Let's write a quick demo script to show it in action. Create your Python file with your text editor of choice — in this case, I'll use nano since it is easy:
~# nano scan.py
The first line should tell the script how to run by pointing to our Python binary:
Next, we need an import statement to import the WebTech library:
Then we can start a new instance with the wt variable:
wt = webtech.WebTech()
And start a scan of the desired URL (I had to also set a short timeout or it wouldn't work) and save that to the results variable:
results = wt.start_from_url('https://null-byte.wonderhowto.com/', timeout=1)
Finally, we can print the results of the scan to the screen:
The final script should look like this:
#!/usr/bin/python import webtech wt = webtech.WebTech() results = wt.start_from_url('https://null-byte.wonderhowto.com/', timeout=1) print results
We can now run our script with the python command we set up above:
~# python scan.py Target URL: https://null-byte.wonderhowto.com/ Detected technologies: - jQuery 1.7 - Google Font API - comScore Detected the following interesting custom headers: - Server: WonderHowTo - X-UA-Compatible: IE=Edge,chrome=1 - X-Server-Name: APP03
Here we can see that we've obtained the same results as before. Keep in mind, this was just a simple proof-of-concept — we could make this a lot more robust if we wanted.
What this does is prove how potentially useful WebTech could be when integrating it into other scripts or tools.
In this article, we learned about WebTech — a Python tool used to identify website technologies.
We saw how to install it with pip and run it from the command line with ease. We also explored its true power, the ability to be used as a Python library, by writing our little script. WebTech makes it easy to get to know your target during the recon phase, giving you the advantage when planning an attack.
Want to start making money as a white hat hacker? Jump-start your hacking career with our 2020 Premium Ethical Hacking Certification Training Bundle from the new Null Byte Shop and get over 60 hours of training from cybersecurity professionals.
Other worthwhile deals to check out: