homepage
Open menu
Go one level top
  • Train and Certify
    • Overview
    • Get Started in Cyber
    • Courses
    • GIAC Certifications
    • Training Roadmap
    • OnDemand
    • Live Training
    • Summits
    • Cyber Ranges
    • College Degrees & Certificates
    • Scholarship Academies
    • NICE Framework
    • Specials
  • Manage Your Team
    • Overview
    • Group Purchasing
    • Why Work with SANS
    • Build Your Team
    • Hire Cyber Talent
    • Team Development
    • Private Training
    • Security Awareness Training
    • Leadership Training
    • Industries
  • Resources
    • Overview
    • Internet Storm Center
    • White Papers
    • Webcasts
    • Tools
    • Newsletters
    • Blog
    • Podcasts
    • Posters & Cheat Sheets
    • Summit Presentations
    • Security Policy Project
  • Focus Areas
    • Cyber Defense
    • Cloud Security
    • Digital Forensics & Incident Response
    • Industrial Control Systems
    • Cyber Security Leadership
    • Offensive Operations
  • Get Involved
    • Overview
    • Join the Community
    • Work Study
    • Teach for SANS
    • CISO Network
    • Partnerships
    • Sponsorship Opportunities
  • About
    • About SANS
    • Our Founder
    • Instructors
    • Mission
    • Diversity
    • Awards
    • Contact
    • Frequently Asked Questions
    • Customer Reviews
    • Press
  • SANS Sites
    • GIAC Security Certifications
    • Internet Storm Center
    • SANS Technology Institute
    • Security Awareness Training
  • Search
  • Log In
  • Join
    • Account Dashboard
    • Log Out
  1. Home >
  2. Blog >
  3. PowerShell Hash Integrity Verifier To Detect File System Changes
370x370_Jason-Fossen.jpg
Jason Fossen

PowerShell Hash Integrity Verifier To Detect File System Changes

June 8, 2015

Using tools like Tripwire, MD5DEEP and MD5SUM to hash files to detect file system changes is a well-known security technique.

But that's not what this article is about.

This article is about how to quickly compare two files which contain hundreds of thousands of hashes of the same directories made at two different times. The script in this article compares these two files and extracts the files which are new, missing or changed.

Comparing Files with Hashes

When hashing large numbers of files in PowerShell, the output can be exported to a comma-delimited (CSV) text file for easy archival and comparison at a later time, for example (this is one command that wraps to multiples lines):

dir -Path C:\Data -File -Recurse |
Get-FileHash -Algorithm MD5 |
Export-Csv -Path C:\Temp\Output.csv

And with MD5DEEP or SHA256DEEP, the output can be redirected to a TXT file:

md5deep64.exe -r C:\Data > Output.txt

But how can we compare two files filled with hundreds of thousands of hashes to detect file system changes? PowerShell does include another cmdlet, Compare-Object, which can be used to compare arrays of any type, but unfortunately it is awkward to use and slow. Is there something better which has been optimized just for file hashes?

You can download a script, Compare-FileHashesList.ps1, to quickly compare two files of hash data to show what files are new, which files have gone missing, and which files have changed since the first hash dump was created. The script is in the public domain and can be found in the SEC505 zip file, Day5 folder, at BlueTeamPowerShell.com.

(SEC505 is the SANS six-day course on "Securing Windows and PowerShell Automation". The script is from one of the hands-on labs in the course. There are many other PowerShell scripts in the SEC505 zip file too.)

Using the Compare-FileHashesList.ps1 Script

To use the script, first save two hash dump files of the same folder(s) using a command similar to the ones shown above with either Get-FileHash or one of the *DEEP tools. Choose any hashing algorithm you wish. In real life, you might make a hash dump every night or every weekend in order to track file system changes. To just test the script, though, you can make one hash dump (before.csv), make a copy of it (after.csv), then edit that copy in Notepad (after.csv) in order to have some changes to detect, e.g., delete a few lines, change some file names, and change some of the hexadecimal hash values. This will be good enough to get started.

Once you have your two hash dumps, compare them with the script like this (make sure to save the file with a ".csv" file name extension when using Export-Csv or with a ".txt" extension when using MD5DEEP):

.\Compare-FileHashesList.ps1 -ReferenceFile .\before.csv -DifferenceFile .\after.csv 

The output will look similar to the following:

Status Path
——- ——
Changed C:\Data\file1.exe
New C:\Data\doc44.doc
Missing C:\Data\file3.ps1

If the status of a file is "Changed", then it's hash is different in the two dump files. If the status is "New", then the file does not exist in the first file (before.csv), but does exist in the second file (after.csv). If it is "Missing", then the file existed in the first (before.csv), but no longer exists in the second (after.csv). (The files do not have to be named "before" and "after" of course, these are just examples.)

By default, files whose hashes have not changed are not output by the script, but they can be shown too: just add the -IncludeUnchanged switch when you run the script. Just like when we originally hashed the target files, so the output of the script can also be saved to a third CSV file using the Export-CSV cmdlet.

If you only need to see a summary of the changes, use the -SummaryOnly switch, like this:

.\Compare-FileHashesList.ps1 -ReferenceFile .\before.csv -DifferenceFile .\after.csv -SummaryOnly

The summary does not display each new, missing or changed file separately. The output looks like this:

StartTime : 8/17/2021 9:18:48 PM
FinishTime : 8/17/2021 9:18:50 PM
RunTimeInSeconds : 1.915
TotalDifferences : 12
New : 3
Missing : 4
Changed : 5

If you use the -Verbose switch, then the same summary details are displayed, but these details are not actually a part of the output stream of file objects, so they won't mess up your code when you pipe the output of this script into other scripts or export the output to another CSV file.

To see the embedded help in the script, run:

Get-Help -Full .\Compare-FileHashesList.ps1

Performance and Choice of Hashing Algorithm

MD5 is an obsolete hashing algorithm, but it is much faster than SHA-256. For most file integrity-checking purposes, MD5 is still fine, but you'll have to decide what is more important: performance or reducing the probability of a hash collision. (On my test machine, which has a solid state drive, using MD5 with the Get-FileHash cmdlet will hash data at an average rate of 408MB/sec, while SHA-256 only does 103MB/sec.)

The performance of the Compare-FileHashesList.ps1 script is not affected by the choice of hashing algorithm. Performance is mainly dependent on the number of lines in the two files. (On my test machine, which has an Intel Core i7-4790 at 3.60GHz, two large CSV files can be compared at 77,000 entries/second with PowerShell 5.0 on Windows 10.) Also, using the -IncludeUnchanged switch imposes a performance penalty too, so avoid this switch when you don't need this output.

You can choose any tool for producing the hashes. You can use Get-FileHash, MD5DEEP, or any other tool that outputs in the same format as MD5DEEP. If you have a choice, save your hashes to a file using UTF-8 encoding to reduce the size of the file.

Try to use PowerShell 3.0 or later to benefit from in-memory compilation of some of the code in the script with the DLR. PowerShell 5.0 on Windows 10 gives the best performance so far, probably because of under-the-hood DLR improvements.

Doing the comparison of these hash lists in PowerShell is convenient because the output of the script is an array of objects (not raw text that requires parsing), scripts can be easily edited and customized (unlike binaries), and PowerShell scripts are easier to use with PowerShell remoting when you want to hash files on large numbers of systems over the network.

Revision History:

[10.Jun.2015] Support for MD5DEEP was added and the script was renamed.
[3.Jul.2015] Increased performance by 40% (lesson learned: don't use Compare-Object with large arrays!)
[9.Jul.2015] Increased performance by 600% (lesson learned: don't use ConvertFrom-Csv!)
[18.May.2021] Fixed download link.

Share:
TwitterLinkedInFacebook
Copy url Url was copied to clipboard
Subscribe to SANS Newsletters
Receive curated news, vulnerabilities, & security awareness tips
United States
Canada
United Kingdom
Spain
Belgium
Denmark
Norway
Netherlands
Australia
India
Japan
Singapore
Afghanistan
Aland Islands
Albania
Algeria
American Samoa
Andorra
Angola
Anguilla
Antarctica
Antigua and Barbuda
Argentina
Armenia
Aruba
Austria
Azerbaijan
Bahamas
Bahrain
Bangladesh
Barbados
Belarus
Belize
Benin
Bermuda
Bhutan
Bolivia
Bonaire, Sint Eustatius, and Saba
Bosnia And Herzegovina
Botswana
Bouvet Island
Brazil
British Indian Ocean Territory
Brunei Darussalam
Bulgaria
Burkina Faso
Burundi
Cambodia
Cameroon
Cape Verde
Cayman Islands
Central African Republic
Chad
Chile
China
Christmas Island
Cocos (Keeling) Islands
Colombia
Comoros
Cook Islands
Costa Rica
Croatia (Local Name: Hrvatska)
Curacao
Cyprus
Czech Republic
Democratic Republic of the Congo
Djibouti
Dominica
Dominican Republic
East Timor
East Timor
Ecuador
Egypt
El Salvador
Equatorial Guinea
Eritrea
Estonia
Ethiopia
Falkland Islands (Malvinas)
Faroe Islands
Fiji
Finland
France
French Guiana
French Polynesia
French Southern Territories
Gabon
Gambia
Georgia
Germany
Ghana
Gibraltar
Greece
Greenland
Grenada
Guadeloupe
Guam
Guatemala
Guernsey
Guinea
Guinea-Bissau
Guyana
Haiti
Heard And McDonald Islands
Honduras
Hong Kong
Hungary
Iceland
Indonesia
Iraq
Ireland
Isle of Man
Israel
Italy
Jamaica
Jersey
Jordan
Kazakhstan
Kenya
Kingdom of Saudi Arabia
Kiribati
Korea, Republic Of
Kosovo
Kuwait
Kyrgyzstan
Lao People's Democratic Republic
Latvia
Lebanon
Lesotho
Liberia
Liechtenstein
Lithuania
Luxembourg
Macau
Macedonia
Madagascar
Malawi
Malaysia
Maldives
Mali
Malta
Marshall Islands
Martinique
Mauritania
Mauritius
Mayotte
Mexico
Micronesia, Federated States Of
Moldova, Republic Of
Monaco
Mongolia
Montenegro
Montserrat
Morocco
Mozambique
Myanmar
Namibia
Nauru
Nepal
Netherlands Antilles
New Caledonia
New Zealand
Nicaragua
Niger
Nigeria
Niue
Norfolk Island
Northern Mariana Islands
Oman
Pakistan
Palau
Palestine
Panama
Papua New Guinea
Paraguay
Peru
Philippines
Pitcairn
Poland
Portugal
Puerto Rico
Qatar
Reunion
Romania
Russian Federation
Rwanda
Saint Bartholemy
Saint Kitts And Nevis
Saint Lucia
Saint Martin
Saint Vincent And The Grenadines
Samoa
San Marino
Sao Tome And Principe
Senegal
Serbia
Seychelles
Sierra Leone
Sint Maarten
Slovakia (Slovak Republic)
Slovenia
Solomon Islands
South Africa
South Georgia and the South Sandwich Islands
South Sudan
Sri Lanka
St. Helena
St. Pierre And Miquelon
Suriname
Svalbard And Jan Mayen Islands
Swaziland
Sweden
Switzerland
Taiwan
Tajikistan
Tanzania
Thailand
Togo
Tokelau
Tonga
Trinidad And Tobago
Tunisia
Turkey
Turkmenistan
Turks And Caicos Islands
Tuvalu
Uganda
Ukraine
United Arab Emirates
United States Minor Outlying Islands
Uruguay
Uzbekistan
Vanuatu
Vatican City
Venezuela
Vietnam
Virgin Islands (British)
Virgin Islands (U.S.)
Wallis And Futuna Islands
Western Sahara
Yemen
Yugoslavia
Zambia
Zimbabwe

Recommended Training

  • SEC565: Red Team Operations and Adversary Emulation
  • SEC555: SIEM with Tactical Analytics
  • AUD507: Auditing & Monitoring Networks, Perimeters & Systems

Tags:
  • Cyber Defense

Related Content

Blog
PowerShell-Blue-Team.jpg
Cyber Defense
July 11, 2021
Instructor Spotlight: Jason Fossen for SEC505 PowerShell
Meet Jason Fossen, SANS Fellow and author of SEC505: Securing Windows and PowerShell Automation
370x370_Jason-Fossen.jpg
Jason Fossen
read more
Blog
PowerShell-Blue-Team.jpg
Cyber Defense
June 6, 2016
PowerShell 7-Zip Module Versus Compress-Archive with Encryption
Archive File Management In PowerShell PowerShell 5.0 includes two cmdlets for working with compressed Zip files: Compress-Archive and Expand-Archive. However, these cmdlets do not support encryption, are relatively slow, cannot handle other archive formats, cannot peek at file listings inside of...
370x370_Jason-Fossen.jpg
Jason Fossen
read more
Blog
PowerShell-Blue-Team.jpg
Cyber Defense
June 1, 2016
PowerShell Function to Send UDP SYSLOG Message Packets
The syslog protocol is very widely used for sending UDP log messages over the network to a centralized logging server, typically running UNIX or Linux. Today, virtually every Security Information Event Management (SIEM) product supports inbound syslog as well, despite the security shortcomings of...
370x370_Jason-Fossen.jpg
Jason Fossen
read more
  • Register to Learn
  • Courses
  • Certifications
  • Degree Programs
  • Cyber Ranges
  • Job Tools
  • Security Policy Project
  • Posters & Cheat Sheets
  • White Papers
  • Focus Areas
  • Cyber Defense
  • Cloud Security
  • Cyber Security Leadership
  • Digital Forensics
  • Industrial Control Systems
  • Offensive Operations
Subscribe to SANS Newsletters
Receive curated news, vulnerabilities, & security awareness tips
United States
Canada
United Kingdom
Spain
Belgium
Denmark
Norway
Netherlands
Australia
India
Japan
Singapore
Afghanistan
Aland Islands
Albania
Algeria
American Samoa
Andorra
Angola
Anguilla
Antarctica
Antigua and Barbuda
Argentina
Armenia
Aruba
Austria
Azerbaijan
Bahamas
Bahrain
Bangladesh
Barbados
Belarus
Belize
Benin
Bermuda
Bhutan
Bolivia
Bonaire, Sint Eustatius, and Saba
Bosnia And Herzegovina
Botswana
Bouvet Island
Brazil
British Indian Ocean Territory
Brunei Darussalam
Bulgaria
Burkina Faso
Burundi
Cambodia
Cameroon
Cape Verde
Cayman Islands
Central African Republic
Chad
Chile
China
Christmas Island
Cocos (Keeling) Islands
Colombia
Comoros
Cook Islands
Costa Rica
Croatia (Local Name: Hrvatska)
Curacao
Cyprus
Czech Republic
Democratic Republic of the Congo
Djibouti
Dominica
Dominican Republic
East Timor
East Timor
Ecuador
Egypt
El Salvador
Equatorial Guinea
Eritrea
Estonia
Ethiopia
Falkland Islands (Malvinas)
Faroe Islands
Fiji
Finland
France
French Guiana
French Polynesia
French Southern Territories
Gabon
Gambia
Georgia
Germany
Ghana
Gibraltar
Greece
Greenland
Grenada
Guadeloupe
Guam
Guatemala
Guernsey
Guinea
Guinea-Bissau
Guyana
Haiti
Heard And McDonald Islands
Honduras
Hong Kong
Hungary
Iceland
Indonesia
Iraq
Ireland
Isle of Man
Israel
Italy
Jamaica
Jersey
Jordan
Kazakhstan
Kenya
Kingdom of Saudi Arabia
Kiribati
Korea, Republic Of
Kosovo
Kuwait
Kyrgyzstan
Lao People's Democratic Republic
Latvia
Lebanon
Lesotho
Liberia
Liechtenstein
Lithuania
Luxembourg
Macau
Macedonia
Madagascar
Malawi
Malaysia
Maldives
Mali
Malta
Marshall Islands
Martinique
Mauritania
Mauritius
Mayotte
Mexico
Micronesia, Federated States Of
Moldova, Republic Of
Monaco
Mongolia
Montenegro
Montserrat
Morocco
Mozambique
Myanmar
Namibia
Nauru
Nepal
Netherlands Antilles
New Caledonia
New Zealand
Nicaragua
Niger
Nigeria
Niue
Norfolk Island
Northern Mariana Islands
Oman
Pakistan
Palau
Palestine
Panama
Papua New Guinea
Paraguay
Peru
Philippines
Pitcairn
Poland
Portugal
Puerto Rico
Qatar
Reunion
Romania
Russian Federation
Rwanda
Saint Bartholemy
Saint Kitts And Nevis
Saint Lucia
Saint Martin
Saint Vincent And The Grenadines
Samoa
San Marino
Sao Tome And Principe
Senegal
Serbia
Seychelles
Sierra Leone
Sint Maarten
Slovakia (Slovak Republic)
Slovenia
Solomon Islands
South Africa
South Georgia and the South Sandwich Islands
South Sudan
Sri Lanka
St. Helena
St. Pierre And Miquelon
Suriname
Svalbard And Jan Mayen Islands
Swaziland
Sweden
Switzerland
Taiwan
Tajikistan
Tanzania
Thailand
Togo
Tokelau
Tonga
Trinidad And Tobago
Tunisia
Turkey
Turkmenistan
Turks And Caicos Islands
Tuvalu
Uganda
Ukraine
United Arab Emirates
United States Minor Outlying Islands
Uruguay
Uzbekistan
Vanuatu
Vatican City
Venezuela
Vietnam
Virgin Islands (British)
Virgin Islands (U.S.)
Wallis And Futuna Islands
Western Sahara
Yemen
Yugoslavia
Zambia
Zimbabwe
  • © 2022 SANS™ Institute
  • Privacy Policy
  • Contact
  • Careers
  • Twitter
  • Facebook
  • Youtube
  • LinkedIn