This is the
second of a three-part series on using PowerShell for audit and compliance
measurements. These blog posts supplement the material presented in the free
webcast series "PowerShell for Audit,
Compliance, and Security Automation and Visualization".
If you are using VMWare products like vSphere Hypervisor (ESXi) and vCenter server in your on-premise environment, you may need to test them for compliance with enterprise policies or security benchmarks. Sticking with the theme of this series, in this post I will show you how to use the PowerCLI suite of PowerShell modules to take measurements of your VMWare systems.
Installing the PowerCLI Modules
You can install the PowerCLI software as a package from PSGallery using this command:
Install-Module -Name VMware.PowerCLI
PowerCLI is really a suite of script-based modules which provide the ability to query and manage your VMWare infrastructure. I can view the modules included with the installed module using the Get-Module cmdlet:
Get-Module -Name VMWare*
As an auditor and compliance consultant, I generally use only the Get-* cmdlets included with most modules. PowerCLI includes a LOT of Get- cmdlets. My version of PowerCLI includes 307 different Get cmdlets.
Common VMWare Configuration
While most of your compliance testing will focus on your hypervisor (ESXi) host settings, most enterprises don't manage the ESXi hosts directly. Instead, they setup a vCenter server, often as a VM running in the virtualized environment. You may have heard your engineers refer to the vCenter Server Appliance (VCSA) being used to manage the hosts. Your technical staff might use the web interface on the vCenter server to control what happens on the hosts. You will use PowerCLI to run your tests against the vCenter server and the hosts it manages. A common architecture might like this:
Compliance Testing VMWare
With the large number of available cmdlets, the biggest problem you are likely to face with VMWare compliance measurements is finding the right combination of commands to use to gather the information you need. I'll work through several examples to demonstrate how you can go about your data gathering. We'll start with authenticating to the vCenter server.
Store Credentials Securely
For most of your VMWare compliance testing, you will connect and authenticate to the vCenter server, and then use that connection to query settings on the vCenter server and the hosts it manages. If you are working interactively at the console, you can use the Get-Credential cmdlet to get a PSCredential object, which contains the username as a string, and the password as a SecureString object (which encrypts the password in memory). If you are building scripts to run automatically, especially under PowerShell Core on non-Windows systems, you'll need a secure way of saving credentials for access in the script. If you work in a DevOps environment where you have a password vault (like Hashicorp Vault), you can save credentials there. If not, you can save your password in a symmetrically encrypted file.
Start by getting your credentials in memory using Get-Credential:
$cred= Get-Credential
Next, you can create a symmetric key file to be used to encrypt your password. Use the .NET cryptographic random number generator to build a key of an appropriate size - 256 bits in this example. Be sure that you set the permissions on the key file so that only authorized users can read it, since it will be used to decrypt your password!
#Create a new key byte array and fill it
$Key = New-Object Byte[] 16
[Security.Cryptography.RNGCryptoServiceProvider]::Create().GetBytes($Key)#Send to the key file</p>
$Key | Out-File .\keyFileThen, you can save the password to a file, encrypted with the key.
$cred.Password | ConvertFrom-SecureString -Key $Key | Out-File .\esxiPassword
Now, whenever you need credentials for the vCenter server, you can build a PSCredential object using a username and your saved password.
$key = Get-Content .\keyFile
$encryptedPwd = Get-Content .\esxiPassword
$cred = New-Object System.Management.Automation.PSCredential -ArgumentList 'auditor@sec557.local',`
($encryptedPwd | ConvertTo-SecureString -Key $key)
Connect-VIServer -Server 10.50.7.33 -Credential $cred
Note that the "auditor@sec557.local" user in these examples has been granted a custom role called "AuditRole" on the vCenter server. AuditRole is a clone of the built-in "Read-only" role with the "Global settings" permission added for the vCenter server and all its child objects. vCenter allows for very granular control over the access provided to users.
Verify ESXi Version
Now that we've connected to the vCenter server, let's start our data gathering with a very common management requirement: ensure that all ESXi hosts are running a current, supported, version of the software. Today, versions 6.5, 6.7 and 7.0 are all supported. 6.5 and 6.7 were released and will lose support simultaneously. 6.5 supports some older CPUs than 6.7, which means that if you see 6.5 in the datacenter, you may want to follow up on the hardware in use to see if it can run the newer versions of ESXi.
Let's query the vCenter server to get the version information for all the ESXi hypervisors it manages:
Get-VMHost | Select-Object Name, Version, Build
Next, you could research the version numbers using this VMWare knowledge base article, which lists every ESXi version back to ESXi version 4.0, released in 2009.
You could do a similar check for the vCenter server by querying it directly. The vCenter server(s) you are currently connected to is available through the DefaultVIServers global variable.
$Global:DefaultVIServers | select Name, Version, Build
The VMWare knowledge base article for vCenter versions is your resource for checking whether your version is current.
Other Patching Compliance Checks
Patch data can be retrieved with the ESXCLI cmdlets, which emulate the results you would receive if were to run commands in the command line interface (CLI) at the host's console. To check patching on an ESXi host, you need to get a list of the "VIBs" (vSphere Installation Bundles) installed on the machine.
(Get-VMHost -Name 10.50.7.31 | Get-EsxCli).software.vib.list()We frequently take two measurements related to patching: the "patch age" of a system is the number of days since the last patch was installed, and "patch velocity" measures the number of patches installed per date. Patch velocity can be obtained with a query like this:
(Get-VMHost -Name 10.50.7.31 | Get-EsxCli).software.vib.list() |
Group-Object InstallDate
This host has only been patched twice, In November 2019, 82 packages were installed, probably when the host was deployed. Then in December of 2020 (maybe when the administrators heard an auditor was coming!) another 41 packages were installed. If the enterprise's policies call for regular patching, this host is probably out of compliance.
To calculate patch age, we will get the InstallDate of the most recently install patch and measure the number of days between then and now. We can use a TimeSpan PowerShell object to see how many days it has been since the host was last patched.
$lastPatchDate = ((Get-VMHost -Name 10.50.7.31 | Get-EsxCli).software.vib.list() |
Sort-Object InstallDate -Descending | Select-Object -First 1).InstallDate</
$patchAge = (New-TimeSpan -Start $lastPatchDate -End (Get-Date)).Days</
$patchAge<
This host was last patched 34 days ago.
Verify Host Service Settings
Many security policies and benchmarks require ESXi servers to be configured to use NTP for time synchronization. To check this, you'll need to look at two things:
- Is the host set to use the correct NTP server?
- Is the NTP service enabled and running?
Let's start with checking the service status on the two hosts in the lab.
Get-VMHost | Get-VMHostService |
Where-Object {$_.key -eq "ntpd"} |
Select-Object VMHost, Label, Key, Policy, Running, Required
Notice that both hosts have the NTP service disabled and not running! It really doesn't matter what NTP server the hosts are set to use, but let's check the settings anyway. The NTP server setting can be queried using a cmdlet built for the task. I usually combine the service configuration and server settings into a single big query. I'm using PowerShell's "calculated properties" here to organize the results:
Get-VMHost | Sort Name | Select Name, `
@{Name="NTPServer";Expression={$_ | Get-VMHostNtpServer}}, `
@{N="ServiceRunning";E={(Get-VmHostService -VMHost $_ |
Where-Object {$_.key-eq "ntpd"} ).Running}}, `
@{N="ServiceRequired";E={(Get-VmHostService -VMHost $_ |
Where-Object {$_.key-eq "ntpd"} ).Required}}
Notice that the second server has no NTP server configured at all.
DNS Server Settings
When there is not a cmdlet to directly query a setting, you can often find what you need in the ExtensionData property returned by the Get-VMHost cmdlet. Here's a quick example of querying the DNS server settings for my lab servers.
(Get-VMHost).ExtensionData.Config.Network.DNSConfig
Honorable Mentions
I'd be remiss if I didn't mention a couple of other tools which have been helpful in auditing and analyzing VMWare.
Robware RVTools
The first is Robware RVTools. It's Windows .NET application with a heavily tabbed interface. Simply point it at your vCenter server, login, and view all sorts of inventory information from your infrastructure.
One of my favorite features of RVTools is the vHealth tab, which can show you common compliance issues without having to run any queries.
The other feature I love is the ability to export the data from RVTools to a tabbed Excel spreadsheet:
AsBuiltReport PowerShell Module
Finally, I leave you with the ASBuiltReport PowerShell modules. This ever-growing set of tools can be run against several different infrastructure technologies, including VMWare. After installing the modules, you can build a nice HTML or Word document which details your infrastructure.
New-AsBuiltReport -Report VMware.vSphere -Target 10.50.7.33 -Format HTML -OutputPath 'C:\Users\auditor\Documents\' -Timestamp -Credential $credThe resulting HTML report is very nicely formatted and helpful for understanding the configuration and inventory of your datacenter environment. Just a couple of samples from my home lab are included below.
ABOUT THE AUTHOR
Clay is the president of Risenhoover Consulting, Inc., an IT management consulting firm based in Durant, Oklahoma. Founded in 2003, RCI provides IT audit and IT management consulting services to clients in multiple sectors. Clay is a licensed Certified Public Accountant (CPA) with the Certified Information Technology Professional (CITP) designation. Since 2013, Clay has been involved with the SANS Institute and is the author and instructor for AUD507: Auditing & Monitoring Networks, Perimeters, and Systems and SEC557: Cloud Security and Continuous Compliance, as well as a Faculty Research Advisor (FRA) for the SANS Technology Institute. Learn More About Clay.
