Tags:
Archive File Management In PowerShell
PowerShell 5.0 includes two cmdlets for working with compressed Zip files: Compress-Archive and Expand-Archive. However, these cmdlets do not support encryption, are relatively slow, cannot handle other archive formats, cannot peek at file listings inside of Zip archives without doing extraction, and cannot handle files larger than 2 GB (which is a big problem for archiving log files).
For this article, it's the lack of encryption support which is the real deal-killer. Is there a better alternative?
7-Zip Is Better (and Free)
7-Zip is a free, open source, cross-platform, very fast, archive file manager. It supports a wide variety of formats (like 7z, zip, tar, wim, iso, rar, and rpm) and can be run from the command line or as a graphical application. Unlike the old Zip archives, modern 7-Zip archives (in 7z format) can be up to 16,000,000 TB in size!
Archive Encryption (Zip Format)
Most importantly, 7-Zip supports archive encryption. Both the Zip and 7z archive formats can be encrypted with a user-supplied passphrase.
Zip archives can be encrypted with the original "ZipCrypto" stream cipher (also known as the PKZIP cipher) or with AES.
The original ZipCrypto/PKZIP method is the most widely compatible, but is fatally flawed (overview and academic paper on the problems). ZipCrypto/PKZIP encryption should never be used when security is the primary concern. When compatibility across operating systems and archival tools is more important than security, then use the Zip format (not 7z) either with no encryption or with ZipCrypto/PKZIP encryption (not AES).
Zip archives can also be encrypted with 256-bit AES, which is much more secure. However, there is a compatibility problem: Zip with AES is not supported by many operating systems or other archival tools; for example, Windows File Explorer does not support AES-encrypted Zip files and probably never will.
A third problem with AES-encrypted Zip files is that some vendors will encrypt the names and paths of files inside the Zip archive, while others do not. Beyond the compatibility problem, file names and folder paths are "metadata" which can be very revealing all by themselves, even if the file contents are encrypted.
Imagine capturing an archive that contained files like "DivorceAttorneys.xls" or "\Contracts\2016\Panama-Gov\OffShoreAccounts.pdf". If you are uploading encrypted Zip files to Amazon or Azure, it's possible that file names and paths are being extracted and indexed, perhaps by Cortana in OneDrive, even though you intend to keep the contents of the Zip files 100% private. When you e-mail encrypted Zip files to others, or upload/download such files through proxy servers, it is also possible that the e-mail gateways and proxy servers are examining and logging the plaintext file names and paths in the otherwise-encrypted Zip files too. In some countries, just having suspicious file names could land you in jail.
A fourth problem is that, if you attempt to extract files from an encrypted Zip file using the wrong password, such as with a typo, you risk overwriting any existing files of the same names with a zero-byte files, thus effectively deleting the existing files! This could accidentally destroy irreplaceable data.
Hence, for both compatibility and security reasons, it's best to never use Zip format encrypted archives, even when AES is the cipher.
Archive Encryption (7z Format)
7-Zip also supports the 7z or "SevenZip" archive format (*.7z file name extension). The 7z format is technically superior to Zip as an archive, and 7z files can also be encrypted with 256-bit AES in CBC mode. The AES key is derived by hashing a user-supplied passphrase with SHA-256 several times.
When 7z archives are encrypted, you have the option to also encrypt the file names and paths inside the archive. You should always encrypt file/path metadata, unless there is some compelling reason not to do so. Unfortunately, this is not the default in 7-Zip, you have to choose to check the "Encrypt file names" box in 7-Zip (or use the -EncryptFilenames switch in PowerShell). But once you do choose this option, 7-Zip remembers your choice the next time you run 7-Zip by setting the following registry value (which could be set pre-emptively with a .reg file or through Group Policy):
Windows Registry Editor Version 5.00 [HKEY_CURRENT_USER\SOFTWARE\7-Zip\Compression] "EncryptHeaders"=dword:00000001
While the AES key is always 256 bits in size, the strength of the encryption is not always equivalent to a random 256-bit key. It depends on the length and randomness of the passphrase entered by the user when encrypting the archive.
A yardstick cryptographers use to measure encryption strength is "bits of entropy", and you can read about how entropy is calculated for keys derived from human passwords, which includes suggestions about how to choose a good passphrase. In short, the passphrase should be at least 25 characters long with some complexity. 7-Zip supports passphrases in excess of 1000 characters. The real solution is to use a password manager application (like KeePass) to secure and provide convenient access to your 7-Zip passphrases. (More on this below.)
Unfortunately, there is no "Passphrase Policy" that can be enforced for 7-Zip. There is nothing to stop a user from choosing a one-character password.
Hence, when security is the primary goal, only use the 7z archive format (not Zip), always check the box to "Encrypt file names" (or use the -EncryptFilenames switch in PowerShell), and use a complex passphrase that is at least 25 characters in length. The longer and more random the passphrase, the better it is for the quality of the encryption.
7-Zip PowerShell Module
7-Zip can be "wrapped" by PowerShell for very convenient command-line access and scripting use. A popular PowerShell module for this is 7Zip4PowerShell, which can be installed for free from the PowerShell Gallery, or, if you have an older version of PowerShell, downloaded from GitHub. (You must have at least PowerShell version 2.0.)
If you have PowerShell 5.0 or later, install 7Zip4PowerShell from the PSGallery over the Internet like this:
Install-Module -Name 7Zip4PowerShell -Verbose
To see the new commands provided by the module:
Get-Command -Module 7Zip4PowerShell
To copy all the *.log files in the present directory into a 7z-compressed archive:
dir *.log | Compress-7Zip -ArchiveFileName logbackup.7z
To copy the F:\Temp folder and all its subdirectories and files into a traditional Zip archive that is compatible with Windows, Mac and Linux:
Compress-7Zip -Path F:\Temp -ArchiveFileName backup.zip -Format Zip
To open an archive in the graphical 7-Zip application for viewing, just invoke or "execute" the archive's file name at the command line:
.\logbackup.7z
(Note: You can associate other archive file name extensions with 7-Zip by pulling down the Tools menu in 7-Zip and selecting Options.)
To see details about the files inside an archive without actually extracting them:
Get-7Zip -ArchiveFileName logbackup.7z Get-7Zip -ArchiveFileName archive.zip | Format-Table FileName,Size
To extract everything from an archive into the present directory ("."):
Expand-7Zip -ArchiveFileName archive.zip -TargetPath .
(If you want to go beyond the above basic operations, see this command reference and the -CustomInitialization parameter. You have access to all the features of 7-Zip through the PowerShell wrapper, it's just that not all of them are exposed as separate parameter names — there would be far too many, it would be clutter for 99% of users.)
To archive and encrypt a folder and everything underneath it with a passphrase:
Compress-7Zip -Path .\DataFolder -ArchiveFileName backup.7z -Format SevenZip -Password "SomeLONG&randuumP@ssf8zzaize" -EncryptFilenames
Notice in the above that the archive format is SevenZip (creates a *.7z file) and the -EncryptFilenames switch is used. As discussed above, this combination should be considered mandatory. If you do not encrypt file names, and you attempt to extract files from the encrypted 7z archive using the wrong password (perhaps accidentally) then you risk overwriting any existing files with the same names with empty files, thus deleting the contents of those files! This does not happen when the -EncryptFilenames switch is always used.
To decrypt and extract the files from a 7z archive to the C:\Data folder:
Expand-7Zip -ArchiveFileName backup.7z -Password "SomeLONG&randuumP@ssf8zzaize" -TargetPath C:\Data
In a PowerShell script, the passphrase and other arguments could be stored as variables:
$Key = "iLFH&s9a>P=e9AcaCh_TaGIni<pre>$Key = "iLFH&s9a>P=e9AcaCh_TaGIni#####replaceparse10#####gt;+e#^s=%#PZ2Vc1&~sM-PXT)Km{(REM?<LR^p~!"
Expand-7Zip -ArchiveFileName backup.7z -Password $Key -TargetPath C:\Data</pre>gt;+e#^s=%#PZ2Vc1&~sM-PXT)Km{(REM?<LR^p~!"
Expand-7Zip -ArchiveFileName backup.7z -Password $Key -TargetPath C:\Data
But we don't want to hard-code decryption keys into scripts, so how could we safely get the key string into the variable? And if the key string is 50+ random characters, it's just too long to enter by hand each time.
KeePass for 7-Zip and PowerShell
The 7-Zip encryption passphrase can be over 1000 characters in length. This is too long for a human to enter by hand, but a 50-character passphrase might be stored in a password manager application, like KeePass.
Because KeePass can also be scripted with PowerShell, this opens up new possibilities in which KeePass secures a passphrase for 7-Zip, 7-Zip encrypts gigabytes of sensitive data, and PowerShell provides the automation to glue it all together.
(Here is another KeePass module for PowerShell in GitHub, soon to be in the PSGallery too.)
For example, you could encrypt 500GB of your personal files using 7-Zip and an encryption passphrase stored in KeePass, then upload that archive to Amazon Glacier or Azure Cool Blob Storage for pennies per month. Because your data is encrypted locally, you don't have to trust Amazon or Microsoft. Because you're using PowerShell to automate the process, it can be done quickly and conveniently. And because the encryption passphrase is stored in KeePass, the passphrase does not need to be hard-coded into any plaintext scripts.
(When archiving a large number of data files, it may be best to first make copies of those files to a temp folder, archive the copies from the temp folder, then securely delete the temp folder. Using the built-in ROBOCOPY.EXE utility you can copy just the files you want to archive using a variety of command-line switches. On Server 2016 and later, check out Storage Replica vs. ROBOCOPY too.)
In summary, when encrypting small chunks of data, like credit card numbers and passphrases, KeePass can be scripted with PowerShell. When encrypting gigabytes of data, 7-Zip can be scripted with PowerShell too. Combining these tools and PowerShell together, we can automate the solution to many data encryption problems (for free). And because KeePass and 7-Zip are cross-platform, we can hopefully interoperate with Mac and Linux users too.
SANS SEC505: Securing Windows and PowerShell Automation
To learn more about using PowerShell for security, please consider my SANS Institute six-day course on Securing Windows and PowerShell Automation (SEC505). Hope to see you at a conference!