Edit: If you want to see how deep this rabbit hole goes, check out our Black Hat / DEF CON presentation: https://www.youtube.com/watch?v=x97ejtv56xw

In a recent post, we talked a little bit about detecting obfuscated PowerShell through the use of PowerShell’s tokenizer - tackling, as an example, the highly irregular variable names generated by MetaSploit’s PowerShell encoder. Obfuscation has been around as long as computer programs have, so the rise of obfuscated PowerShell scripts shouldn’t be much of a surprise. Obfuscated VBScript, Perl, Ruby, Python, and of course assembly language are very common. A great example comes from this blog: http://perl-users.jp/articles/advent-calendar/2010/sym/11, which relies heavily on dynamic evaluation through Invoke-Expression:

Enabling ScriptBlock logging in PowerShell v5 is an incredibly effective way to gain insight into this style of technique:

Obfuscation through Invoke-Expression or basic things like variable names are one thing, but what happens when people really start obfuscating the mechanics of the scripts themselves? At DerbyCon this year, Daniel Bohannon (Blue Teamer Par Excellence) recently gave a really great presentation: “Invoke-Obfuscation: PowerShell obFUsk8tion Techniques & How To (Try To) D"“e`Tec`T ‘Th’+‘em’ ”.

In that presentation, he dropped gems like this that rely heavily on the Format operator:

That obfuscation doesn’t rely on the Invoke-Expression command, so would show up basically the same in script block logging. Along with the crazy Invoke-Expression stuff, these examples demonstrate, without a doubt, that relying on string matching alone to detect evil is a fool’s errand.

But here’s the thing. Obfuscated scripts aren’t normal. Anybody looking at an obfuscated script knows that they’re not normal. They stick out like a sore thumb. This alone can be used as an incredibly rich signal that somebody is trying to avoid getting caught, in the same way that all self-respecting SOCs look for the events that come from an attacker turning off Antivirus.

In the previous post, we looked at the letter frequency of variable names, but what if we expanded on that approach to look at all characters in a script? The Invoke-Expression based script above relied entirely on 16 characters. The script that relied on PowerShell’s Format operator relied heavily on quoting and brace characters.

But how do you know what’s abnormal across “all PowerShell scripts”?

On the PowerShell team, one thing we often use for questions like this is a corpus that we created by downloading everything we could get our grubby little hands on. So, let’s take a look at character frequencies using Measure-CharacterFrequency.

PS C:\PowerShellCorpus\PoshCode> $globalFrequency = Measure-CharacterFrequency *.ps1
PS C:\PowerShellCorpus\PoshCode> $globalFrequency | Select -First 20

Name Percent
---- -------
E      9.912
T      7.414
A      5.512
R       5.43
S      5.303
I      5.041
N      5.025
O      4.944
L      3.509
M        3.3
C      3.191
$      3.076
P      2.914
D      2.753
U       2.29
-      1.955
.      1.917
"      1.822
F      1.626
G      1.526

Now, compare that to some of these other ones:

## The Token-based obfuscation that relies on the Format operator
PS > Measure-CharacterFrequency C:\temp\tokenall.ps1 | Select -First 10

Name Percent
---- -------
'     20.175
{      7.456
}      7.456
,      5.702
E      3.947
T      3.509
N      3.509
"      3.509
(       3.07
)       3.07

## The one that relies on Invoke-Expression
PS > Measure-CharacterFrequency C:\temp\symbolic.ps1 | Select -First 10

Name Percent
---- -------
$     21.808
{     21.659
}     21.659
+     13.313
"      7.452
=      2.832
[      2.086
(      1.689
;       1.54
)      1.341

The difference is huge, and unmistakable. But how do we compare these sets in a robust and reliable way? We steal from the field of Information Retrieval, that’s how!

The field of Information Retrieval has long used a technique called vector similarity / cosine similarity to compare two sets of things. For example, the similarity of two documents, or how closely a search matches a document.

That’s a lot of complicated math-looking symbols - but it turns out it’s very easy to calculate. Here’s an example, using Measure-VectorSimilarity.

PS > Measure-VectorSimilarity @(1..10) @(4..15)
0.639

So, let’s automate a vector similarity comparison. Take random selection of scripts from PoshCode, dump in some obfuscated ones, and see if anything sticks out based on the vector similarity score:

[C:\PowerShellCorpus\PoshCode]
PS > md c:\temp\randomscripts
PS > dir | Get-Random -Count 20 | Copy-Item -Destination C:\temp\randomscripts
PS > copy C:\temp\symbolic.ps1 C:\temp\randomscripts
PS > copy C:\temp\tokenall.ps1 C:\temp\randomscripts
PS > dir C:\temp\randomscripts\ | % {
>>>     $scriptFrequency = $_ | Measure-CharacterFrequency.ps1
>>>     $sim = Measure-VectorSimilarity $globalFrequency $scriptFrequency 
>>>             -KeyProperty Name -ValueProperty Percent
>>>     [PSCustomObject] @{ Name = $_.Name; Similarity = $sim }
>>> }

Name                                     Similarity
----                                     ----------
43a28a15-5023-4feb-a71f-abe95aa0f2a6.ps1      0.957
Export-PSCredential_4.ps1                     0.979
Get-BogonList_1.ps1                           0.925
Get-Netstat _1.9.ps1                           0.89
Get-Parameter_8.ps1                           0.959
group-byobject_4.ps1                          0.939
IADsDNWithBinary Cmdlet_1.ps1                 0.924
Import-ExcelToSQL_2.ps1                       0.961
Invoke-Sql_2.ps1                              0.979
List AddRemovePrograms.ps1                    0.961
Lock-WorkStation.ps1                          0.905
Monitor-FileSize_1.ps1                        0.974
symbolic.ps1                                  0.157 <<<<<<<<<<<<<<<
Reverse filename sequenc.ps1                  0.874
scriptable telnet client_2.ps1                0.967
Set Active Sync DeviceID.ps1                  0.955
SharePoint Large Lists_1.ps1                  0.944
Show-Sample_1.ps1                             0.919
Start-Verify.ps1                              0.923
tokenall.ps1                                  0.379 <<<<<<<<<<<<<<<

In fact, if you graph the similarity scores of the nearly 3500 scripts from PoshCode, only 2% of them have a similarity score less than 80%. And almost all of them are legitimately obfuscated for fun.

The difference is unmistakable. If you’re currently trying to detect malicious script-based content, be sure to also look for indicators of script obfuscation. Reliable techniques exist, and you’re likely running blind without them.