Hack Like a Pro: How Antivirus Software Works & How to Evade It, Pt. 1

How Antivirus Software Works & How to Evade It, Pt. 1

Welcome back, my tenderfoot hackers!

As hackers, we often are required to get past antivirus (AV) software or other security measures. To do so effectively, we need to have some understanding of how AV software works. In this tutorial, we will take a cursory view of how AV software works so that you can better strategize on how to evade detection by it.

The antivirus software industry is huge and well-funded. As you would expect, they employ a large number of malware detection and identification personnel. (I will use the preferred term "malware" to mean all bad or malicious software including viruses, spyware, rootkits, worms, etc.)

Many of these people have a hacker or virus-making background, so the things we are doing here are not mysterious to them. (Eugene Kaspersky, the founder Kaspersky Labs, is rumored to have worked at one time as a hacker/cryptographer for Russia's KGB. Many others in the industry have a background in virus/worm creation.)

The AV industry attempts to collect as much information as they can about threats new and old and package that information into their software. The older a tool or technique is, the greater the chance they will have a mechanism to detect it.

Types of AV Detection Mechanisms:


The key to making good AV software is to have a complete database of all malware signatures. These signatures are the essential part of the malware that distinguishes it from other software. That's why its key for users to keep those databases updated daily. As new malware is introduced daily into the wild, an out-of-date database of signatures can become nearly useless.

For a better concept of what these signature databases are like, you might take a look at Snort, an open-source IDS that detects malware and attacks on the wire. The beauty of working with Snort is that its signature database is open and viewable to anyone.

In the screenshot below, you can see some of the rules or signatures from the web-attacks.rules file from Snort. To better understand malware signatures, you may want to take some time to study these "signatures." It's important to note that these rules don't look for the entire file, but simply the single element of the malware that is unique or its "signature." Obviously, this is much more efficient.

The AV software companies maintain individuals and honeypots around the world that are constantly searching for new malware, and when they find it, they try to condense its signature down to the essentials. In addition, they have users all over the globe who send them suspicious malware they find.

These signatures, as I'm sure you guessed, are only effective on known malware. A zero-day attack would not have a signature and therefore would likely go undetected by the antivirus software.

One of the exceptions to this is that some malware production tools leave a signature on their output, so even a new piece of malware might be detected by AV if the tools used to create it have a signature that the AV developers have identified and coded for. A good example of this that many of you have found is the msfvenom module in Metasploit. The template that creates the payloads has a signature, so no matter how we re-encode our payload, it still has a known signature. The key to defeat the AV with this module is to use a new template.


In some cases, it is not possible to have a signature for all malware, and in those cases, the AV developers attempt to deploy heuristic techniques to detect malware.

Wikipedia defines heuristic techniques as "...any approach to problem solving, learning, or discovery that employs a practical methodology not guaranteed to be optimal or perfect, but sufficient for the immediate goals. Where finding an optimal solution is impossible or impractical, heuristic methods can be used to speed up the process of finding a satisfactory solution."

James Whitcomb Riley once said, "When I see a bird that walks like a duck and swims like a duck and quacks like a duck, I call that bird a duck." That principle summarizes heuristics succinctly. If it looks like malware and behaves like malware, it's probably malware.

The AV software developers look for telltale signs that the software's structure or behavior is malicious and then treat it like malware, even if no signature exists. For instance, if a file begins to replace several system files, it's probably malware. If a piece of software is trying make a TCP connection back to a known malicious IP address, it's probably malware. The beauty of this strategy is that it works with zero-day malware just as well as known malware. The drawbacks of this strategy are that it requires more compute cycles and the false positives that are often produced by innocuous software.


Some people create a separate category of behavior-based malware detection, but I consider it to be a subcategory of heuristics. Behavior is part of the "walks like a duck" mentioned above.


In some cases, suspicious software can be run in a virtual machine environment to see what it will do before it is installed on the system. This is referred to as "sandboxing." In this way, the AV software can study what it does or tries to do and then determine whether it is safe without endangering the entire system.

Image via Lowe's

Due to the time and resources necessary, this isn't a practical way of dealing with all software and files, but for particularly suspicious ones, it can be effective.

Evading AV Software:

First, let's state clearly that AV software is not perfect. Even with all their resources and skill, they still miss known malware. Depending what package you are using, detection of known malware can be as low as 40% and as high as 98 percent. Even in this best case scenario, that means that 1 out of 50 pieces of known malware gets past the AV software.


As most of you know, msfvenom is a module in Metasploit that enables us to re-encode our payloads to evade AV software. Due to the ubiquity of Metasploit, as soon as a new encoding scheme is developed by Metasploit, the AV makers jump on it and produce a signature (they would be foolish if they didn't).

As I mentioned above, the AV makers don't really need to have a signature for every re-encoding by msfvenom, they only need a signature for the template. This means that msfvenom has limited effectiveness without an original template.

Testing Your Malware with Virus Total

If you want to test whether you malware, rootkit, virus, or whatever will be detected by AV software, you can submit it to VirusTotal, which will use over 60 commercial software packages to determine whether your malware will be detected. You can keep re-encoding your malware until VirusTotal indicates it will not be detected.

Unfortunately, if you are successful in being undetected by any of these AV packages, VirusTotal will present your malware to the AV industry to develop a signature. Despite this, there will still be a window before the AV developers add the signature and the end users update their signature database.

Veil Evasion

Veil-Evasion is a software package for changing the signature of your payload executables so that they can bypass AV software. Veil is now in the Kali repository and can be installed simply by typing:

kali > apt-get install veil

I will be doing a tutorial on its use in the near future.

Web Mail Detection

Many of you have struggled sending malware via web-based mail, such Gmail, Yahoo, Hotmail, etc. Of course, those companies maintain their legitimacy by deploying one of the major AV companies to scan any attachments you might try to send. In most cases, trying to send a malicious PDF, Word document, or other file will likely be detected by these services and stopped.

One of the beauties of using a Linux distribution is the ability to set up your own SMTP server and use it to send your email. There are numerous SMTP servers available for Linux that you can download and install including atmail, Exim, qmail, Postfix, and sendmail. Look for my tutorial coming soon on installing the Exim SMTP server in my Linux Basics series.

Having your own SMTP server would resolve any issues with sending malicious payloads but, of course, will not resolve issues on the receiving end.

Build Your Own

Ultimately, the best way to evade AV and other security devices is to build your own exploit/payload. That is our goal in my Exploit Building series, where we are working to develop our own unique exploit that will be undetectable by any AV or security device.

Keep coming back, my tenderfoot hackers, as we explore more ways to evade security and build our own exploits!

Just updated your iPhone? You'll find new features for Podcasts, News, Books, and TV, as well as important security improvements and fresh wallpapers. Find out what's new and changed on your iPhone with the iOS 17.5 update.

Cover image via Shutterstock


I was just thinking about AV and such when I remembered you had promised us more material on this. Sure enough I pop on NullByte and you have one up! Thank you sir OTW.

Thanks Sir OTW,

Just as promised....looking forward to other related goodies


Yup, VirusTotal is both a gift and a curse.

I haven't messed with Veil yet. I'll have to try that later today.
Amazing tutorial! Kudos, +1, Tweet, etc. I can't wait for more on the exploit-building series.

Quick question: are buffer overflows detected as malware? All they are are scripts or programs that contain many characters (as that is what a computer would see, nothing harmful), right? Is what gets it caught is the shellcode?

C|H of C3

I am by no means an expert, but I would assume it would be the signature of whatever payload you used the buffer overflow vector to inject into the system. Things like nope sleds get picked up because they are usually only seen in malware, as well as certain byte patterns that are machine code for reverse connections, etc.

Can't we just keep on sending non-malicious exe files to virustotal (They will obviously get reported as undetectable viruses) and upload the actual virus to be tested among the crowd.


No, that's not the point. The point of using VirusTotal is to see if your malware is detected.

That is not how it works. It sends everything to be checked and whether it is in a crowd or not does not matter.

Veil is pretty fantastic. I have been looking into the framework and it is slightly unnerving at how easy it is to use, and how it can give considerable power to newcomers. That being said, the field is a game of evolution, and what fuels progress is competition on each side. Looking forward to seeing Veil articles in the future, OTW.

Honestly your tutorials are amazing. I was wondering though, if you were planning to carry on your series on exploit building anytime soon.

Thanks, Robyn

Excited for the exploit building series :)

Not really, it was designed to be helpful to white-hats, not black-hats.

I have heard there are some online virus scanners who dosen't give those tested malicious files to AV companies.is it totaly truth?

Yeah, not all upload like VirusTotal. However, if something is repeatedly scanned, the AV will likely put a flag on it.

Hello Sir once more,

First off I want to thank you for your time and for adding the Awards and Certifications. I have learned 90% of my Network Security knowledge here.

Now, just an hour ago my friend asked me to aid him in removing some persistant virus, as I did. The AV recognized it as W32/Injector.BQLO!tr but I didn't pay much attention. I sucessfully removed it in safe mode, but later on got me thinking. That was not the type of malware that's delivered through any of Kali's tools, Since I've tested many of them and they got detected as Trojan.Backdoor or Trojan.RozenaWin32, even the injector ones that download themseves from the cloud and go straight into memory instead of a specified process. Then i went to foriguard.com and saw that BQLO has the leading activity. Is it a custom coded virus? I don't think it is, because if it was custom made it wouldn't have such a high activity. So I'm wondering, How are these types of viruses produced, and what's so special about them that they've got 20% activity above others. Perhaps this is a question for a completely separate guide, in which case I won't rush you. I am here to learn, and that requires patience. Thank you for your time.

what about andoroid

Extremely well written .I was thinking what if we send an metasploit created exploit file to our victim via usb drive combined with the autorun program of notepad (or i presume in any text editor ) and disable antivirus program ( also in notepad and both hidden in pen drive) Will this work cuz i heard disable antivirus virus disables antivirus before they even detect it. And of course we need to conduct reconnaissance first

Share Your Thoughts

  • Hot
  • Latest