homepage
Open menu
Go one level top
  • Train and Certify
    Train and Certify

    Immediately apply the skills and techniques learned in SANS courses, ranges, and summits

    • Overview
    • Courses
      • Overview
      • Full Course List
      • By Focus Areas
        • Cloud Security
        • Cyber Defense
        • Cybersecurity and IT Essentials
        • DFIR
        • Industrial Control Systems
        • Offensive Operations
        • Management, Legal, and Audit
      • By Skill Levels
        • New to Cyber
        • Essentials
        • Advanced
        • Expert
      • Training Formats
        • OnDemand
        • In-Person
        • Live Online
      • Course Demos
    • Training Roadmaps
      • Skills Roadmap
      • Focus Area Job Roles
        • Cyber Defence Job Roles
        • Offensive Operations Job Roles
        • DFIR Job Roles
        • Cloud Job Roles
        • ICS Job Roles
        • Leadership Job Roles
      • NICE Framework
        • Security Provisionals
        • Operate and Maintain
        • Oversee and Govern
        • Protect and Defend
        • Analyze
        • Collect and Operate
        • Investigate
        • Industrial Control Systems
    • GIAC Certifications
    • Training Events & Summits
      • Events Overview
      • Event Locations
        • Asia
        • Australia & New Zealand
        • Latin America
        • Mainland Europe
        • Middle East & Africa
        • Scandinavia
        • United Kingdom & Ireland
        • United States & Canada
      • Summits
    • OnDemand
    • Get Started in Cyber
      • Overview
      • Degree and Certificate Programs
      • Scholarships
    • Cyber Ranges
  • Manage Your Team
    Manage Your Team

    Build a world-class cyber team with our workforce development programs

    • Overview
    • Why Work with SANS
    • Group Purchasing
    • Build Your Team
      • Team Development
      • Assessments
      • Private Training
      • Hire Cyber Professionals
      • By Industry
        • Health Care
        • Industrial Control Systems Security
        • Military
    • Leadership Training
  • Security Awareness
    Security Awareness

    Increase your staff’s cyber awareness, help them change their behaviors, and reduce your organizational risk

    • Overview
    • Products & Services
      • Security Awareness Training
        • EndUser Training
        • Phishing Platform
      • Specialized
        • Developer Training
        • ICS Engineer Training
        • NERC CIP Training
        • IT Administrator
      • Risk Assessments
        • Knowledge Assessment
        • Culture Assessment
        • Behavioral Risk Assessment
    • OUCH! Newsletter
    • Career Development
      • Overview
      • Training & Courses
      • Professional Credential
    • Blog
    • Partners
    • Reports & Case Studies
  • Resources
    Resources

    Enhance your skills with access to thousands of free resources, 150+ instructor-developed tools, and the latest cybersecurity news and analysis

    • Overview
    • Webcasts
    • Free Cybersecurity Events
      • Free Events Overview
      • Summits
      • Solutions Forums
      • Community Nights
    • Content
      • Newsletters
        • NewsBites
        • @RISK
        • OUCH! Newsletter
      • Blog
      • Podcasts
      • Summit Presentations
      • Posters & Cheat Sheets
    • Research
      • White Papers
      • Security Policies
    • Tools
    • Focus Areas
      • Cyber Defense
      • Cloud Security
      • Digital Forensics & Incident Response
      • Industrial Control Systems
      • Cyber Security Leadership
      • Offensive Operations
  • Get Involved
    Get Involved

    Help keep the cyber community one step ahead of threats. Join the SANS community or begin your journey of becoming a SANS Certified Instructor today.

    • Overview
    • Join the Community
    • Work Study
    • Teach for SANS
    • CISO Network
    • Partnerships
    • Sponsorship Opportunities
  • About
    About

    Learn more about how SANS empowers and educates current and future cybersecurity practitioners with knowledge and skills

    • SANS
      • Overview
      • Our Founder
      • Awards
    • Instructors
      • Our Instructors
      • Full Instructor List
    • Mission
      • Our Mission
      • Diversity
      • Scholarships
    • Contact
      • Contact Customer Service
      • Contact Sales
      • Press & Media Enquiries
    • Frequent Asked Questions
    • Customer Reviews
    • Press
    • Careers
  • Contact Sales
  • SANS Sites
    • GIAC Security Certifications
    • Internet Storm Center
    • SANS Technology Institute
    • Security Awareness Training
  • Search
  • Log In
  • Join
    • Account Dashboard
    • Log Out
  1. Home >
  2. Blog >
  3. PowerShell Hash Integrity Verifier To Detect File System Changes
370x370_Jason-Fossen.jpg
Jason Fossen

PowerShell Hash Integrity Verifier To Detect File System Changes

June 8, 2015

Using tools like Tripwire, MD5DEEP and MD5SUM to hash files to detect file system changes is a well-known security technique.

But that's not what this article is about.

This article is about how to quickly compare two files which contain hundreds of thousands of hashes of the same directories made at two different times. The script in this article compares these two files and extracts the files which are new, missing or changed.

Comparing Files with Hashes

When hashing large numbers of files in PowerShell, the output can be exported to a comma-delimited (CSV) text file for easy archival and comparison at a later time, for example (this is one command that wraps to multiples lines):

dir -Path C:\Data -File -Recurse |
Get-FileHash -Algorithm MD5 |
Export-Csv -Path C:\Temp\Output.csv

And with MD5DEEP or SHA256DEEP, the output can be redirected to a TXT file:

md5deep64.exe -r C:\Data > Output.txt

But how can we compare two files filled with hundreds of thousands of hashes to detect file system changes? PowerShell does include another cmdlet, Compare-Object, which can be used to compare arrays of any type, but unfortunately it is awkward to use and slow. Is there something better which has been optimized just for file hashes?

You can download a script, Compare-FileHashesList.ps1, to quickly compare two files of hash data to show what files are new, which files have gone missing, and which files have changed since the first hash dump was created. The script is in the public domain and can be found in the SEC505 zip file, Day5 folder, at BlueTeamPowerShell.com.

(SEC505 is the SANS six-day course on "Securing Windows and PowerShell Automation". The script is from one of the hands-on labs in the course. There are many other PowerShell scripts in the SEC505 zip file too.)

Using the Compare-FileHashesList.ps1 Script

To use the script, first save two hash dump files of the same folder(s) using a command similar to the ones shown above with either Get-FileHash or one of the *DEEP tools. Choose any hashing algorithm you wish. In real life, you might make a hash dump every night or every weekend in order to track file system changes. To just test the script, though, you can make one hash dump (before.csv), make a copy of it (after.csv), then edit that copy in Notepad (after.csv) in order to have some changes to detect, e.g., delete a few lines, change some file names, and change some of the hexadecimal hash values. This will be good enough to get started.

Once you have your two hash dumps, compare them with the script like this (make sure to save the file with a ".csv" file name extension when using Export-Csv or with a ".txt" extension when using MD5DEEP):

.\Compare-FileHashesList.ps1 -ReferenceFile .\before.csv -DifferenceFile .\after.csv 

The output will look similar to the following:

Status Path
——- ——
Changed C:\Data\file1.exe
New C:\Data\doc44.doc
Missing C:\Data\file3.ps1

If the status of a file is "Changed", then it's hash is different in the two dump files. If the status is "New", then the file does not exist in the first file (before.csv), but does exist in the second file (after.csv). If it is "Missing", then the file existed in the first (before.csv), but no longer exists in the second (after.csv). (The files do not have to be named "before" and "after" of course, these are just examples.)

By default, files whose hashes have not changed are not output by the script, but they can be shown too: just add the -IncludeUnchanged switch when you run the script. Just like when we originally hashed the target files, so the output of the script can also be saved to a third CSV file using the Export-CSV cmdlet.

If you only need to see a summary of the changes, use the -SummaryOnly switch, like this:

.\Compare-FileHashesList.ps1 -ReferenceFile .\before.csv -DifferenceFile .\after.csv -SummaryOnly

The summary does not display each new, missing or changed file separately. The output looks like this:

StartTime : 8/17/2021 9:18:48 PM
FinishTime : 8/17/2021 9:18:50 PM
RunTimeInSeconds : 1.915
TotalDifferences : 12
New : 3
Missing : 4
Changed : 5

If you use the -Verbose switch, then the same summary details are displayed, but these details are not actually a part of the output stream of file objects, so they won't mess up your code when you pipe the output of this script into other scripts or export the output to another CSV file.

To see the embedded help in the script, run:

Get-Help -Full .\Compare-FileHashesList.ps1

Performance and Choice of Hashing Algorithm

MD5 is an obsolete hashing algorithm, but it is much faster than SHA-256. For most file integrity-checking purposes, MD5 is still fine, but you'll have to decide what is more important: performance or reducing the probability of a hash collision. (On my test machine, which has a solid state drive, using MD5 with the Get-FileHash cmdlet will hash data at an average rate of 408MB/sec, while SHA-256 only does 103MB/sec.)

The performance of the Compare-FileHashesList.ps1 script is not affected by the choice of hashing algorithm. Performance is mainly dependent on the number of lines in the two files. (On my test machine, which has an Intel Core i7-4790 at 3.60GHz, two large CSV files can be compared at 77,000 entries/second with PowerShell 5.0 on Windows 10.) Also, using the -IncludeUnchanged switch imposes a performance penalty too, so avoid this switch when you don't need this output.

You can choose any tool for producing the hashes. You can use Get-FileHash, MD5DEEP, or any other tool that outputs in the same format as MD5DEEP. If you have a choice, save your hashes to a file using UTF-8 encoding to reduce the size of the file.

Try to use PowerShell 3.0 or later to benefit from in-memory compilation of some of the code in the script with the DLR. PowerShell 5.0 on Windows 10 gives the best performance so far, probably because of under-the-hood DLR improvements.

Doing the comparison of these hash lists in PowerShell is convenient because the output of the script is an array of objects (not raw text that requires parsing), scripts can be easily edited and customized (unlike binaries), and PowerShell scripts are easier to use with PowerShell remoting when you want to hash files on large numbers of systems over the network.

Revision History:

[10.Jun.2015] Support for MD5DEEP was added and the script was renamed.
[3.Jul.2015] Increased performance by 40% (lesson learned: don't use Compare-Object with large arrays!)
[9.Jul.2015] Increased performance by 600% (lesson learned: don't use ConvertFrom-Csv!)
[18.May.2021] Fixed download link.

Share:
TwitterLinkedInFacebook
Copy url Url was copied to clipboard
Subscribe to SANS Newsletters
Receive curated news, vulnerabilities, & security awareness tips
United States
Canada
United Kingdom
Spain
Belgium
Denmark
Norway
Netherlands
Australia
India
Japan
Singapore
Afghanistan
Aland Islands
Albania
Algeria
American Samoa
Andorra
Angola
Anguilla
Antarctica
Antigua and Barbuda
Argentina
Armenia
Aruba
Austria
Azerbaijan
Bahamas
Bahrain
Bangladesh
Barbados
Belarus
Belize
Benin
Bermuda
Bhutan
Bolivia
Bonaire, Sint Eustatius, and Saba
Bosnia And Herzegovina
Botswana
Bouvet Island
Brazil
British Indian Ocean Territory
Brunei Darussalam
Bulgaria
Burkina Faso
Burundi
Cambodia
Cameroon
Cape Verde
Cayman Islands
Central African Republic
Chad
Chile
China
Christmas Island
Cocos (Keeling) Islands
Colombia
Comoros
Cook Islands
Costa Rica
Croatia (Local Name: Hrvatska)
Curacao
Cyprus
Czech Republic
Democratic Republic of the Congo
Djibouti
Dominica
Dominican Republic
East Timor
East Timor
Ecuador
Egypt
El Salvador
Equatorial Guinea
Eritrea
Estonia
Ethiopia
Falkland Islands (Malvinas)
Faroe Islands
Fiji
Finland
France
French Guiana
French Polynesia
French Southern Territories
Gabon
Gambia
Georgia
Germany
Ghana
Gibraltar
Greece
Greenland
Grenada
Guadeloupe
Guam
Guatemala
Guernsey
Guinea
Guinea-Bissau
Guyana
Haiti
Heard And McDonald Islands
Honduras
Hong Kong
Hungary
Iceland
Indonesia
Iraq
Ireland
Isle of Man
Israel
Italy
Jamaica
Jersey
Jordan
Kazakhstan
Kenya
Kiribati
Korea, Republic Of
Kosovo
Kuwait
Kyrgyzstan
Lao People's Democratic Republic
Latvia
Lebanon
Lesotho
Liberia
Liechtenstein
Lithuania
Luxembourg
Macau
Macedonia
Madagascar
Malawi
Malaysia
Maldives
Mali
Malta
Marshall Islands
Martinique
Mauritania
Mauritius
Mayotte
Mexico
Micronesia, Federated States Of
Moldova, Republic Of
Monaco
Mongolia
Montenegro
Montserrat
Morocco
Mozambique
Myanmar
Namibia
Nauru
Nepal
Netherlands Antilles
New Caledonia
New Zealand
Nicaragua
Niger
Nigeria
Niue
Norfolk Island
Northern Mariana Islands
Oman
Pakistan
Palau
Palestine
Panama
Papua New Guinea
Paraguay
Peru
Philippines
Pitcairn
Poland
Portugal
Puerto Rico
Qatar
Reunion
Romania
Russian Federation
Rwanda
Saint Bartholemy
Saint Kitts And Nevis
Saint Lucia
Saint Martin
Saint Vincent And The Grenadines
Samoa
San Marino
Sao Tome And Principe
Saudi Arabia
Senegal
Serbia
Seychelles
Sierra Leone
Sint Maarten
Slovakia
Slovenia
Solomon Islands
South Africa
South Georgia and the South Sandwich Islands
South Sudan
Sri Lanka
St. Helena
St. Pierre And Miquelon
Suriname
Svalbard And Jan Mayen Islands
Swaziland
Sweden
Switzerland
Taiwan
Tajikistan
Tanzania
Thailand
Togo
Tokelau
Tonga
Trinidad And Tobago
Tunisia
Turkey
Turkmenistan
Turks And Caicos Islands
Tuvalu
Uganda
Ukraine
United Arab Emirates
United States Minor Outlying Islands
Uruguay
Uzbekistan
Vanuatu
Vatican City
Venezuela
Vietnam
Virgin Islands (British)
Virgin Islands (U.S.)
Wallis And Futuna Islands
Western Sahara
Yemen
Yugoslavia
Zambia
Zimbabwe

By providing this information, you agree to the processing of your personal data by SANS as described in our Privacy Policy.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Recommended Training

  • SEC660: Advanced Penetration Testing, Exploit Writing, and Ethical Hacking
  • SEC511: Continuous Monitoring and Security Operations
  • SEC586: Blue Team Operations: Defensive PowerShell

Tags:
  • Cyber Defense

Related Content

Blog
CD_Blog_HowtoautomateinAzure_Part1_2.jpg
Cyber Defense, Cloud Security
October 11, 2022
How to Automate in Azure Using PowerShell - Part 1
In this post, we’ll cover how to automate the assessment and reporting of your cloud security configuration opportunities.
370x370_josh-johnson.jpg
Josh Johnson
read more
Blog
powershell_option_340x340.jpg
Cybersecurity and IT Essentials, Cyber Defense, Penetration Testing and Red Teaming
August 4, 2022
Month of PowerShell - Discoveries from the Month of PowerShell
A final wrap-up from the Month of PowerShell: discoveries, recommendations, complaints, and successes.
370x370_Joshua-Wright.jpg
Joshua Wright
read more
Blog
powershell_option_340x340.jpg
Cybersecurity and IT Essentials, Cyber Defense, Penetration Testing and Red Teaming
July 31, 2022
Month of PowerShell - Keyboard Shortcuts Like a Boss
Let's look at several keyboard shortcuts to speed up your PowerShell sessions.
370x370_Joshua-Wright.jpg
Joshua Wright
read more
  • Register to Learn
  • Courses
  • Certifications
  • Degree Programs
  • Cyber Ranges
  • Job Tools
  • Security Policy Project
  • Posters & Cheat Sheets
  • White Papers
  • Focus Areas
  • Cyber Defense
  • Cloud Security
  • Cybersecurity Leadership
  • Digital Forensics
  • Industrial Control Systems
  • Offensive Operations
Subscribe to SANS Newsletters
Receive curated news, vulnerabilities, & security awareness tips
United States
Canada
United Kingdom
Spain
Belgium
Denmark
Norway
Netherlands
Australia
India
Japan
Singapore
Afghanistan
Aland Islands
Albania
Algeria
American Samoa
Andorra
Angola
Anguilla
Antarctica
Antigua and Barbuda
Argentina
Armenia
Aruba
Austria
Azerbaijan
Bahamas
Bahrain
Bangladesh
Barbados
Belarus
Belize
Benin
Bermuda
Bhutan
Bolivia
Bonaire, Sint Eustatius, and Saba
Bosnia And Herzegovina
Botswana
Bouvet Island
Brazil
British Indian Ocean Territory
Brunei Darussalam
Bulgaria
Burkina Faso
Burundi
Cambodia
Cameroon
Cape Verde
Cayman Islands
Central African Republic
Chad
Chile
China
Christmas Island
Cocos (Keeling) Islands
Colombia
Comoros
Cook Islands
Costa Rica
Croatia (Local Name: Hrvatska)
Curacao
Cyprus
Czech Republic
Democratic Republic of the Congo
Djibouti
Dominica
Dominican Republic
East Timor
East Timor
Ecuador
Egypt
El Salvador
Equatorial Guinea
Eritrea
Estonia
Ethiopia
Falkland Islands (Malvinas)
Faroe Islands
Fiji
Finland
France
French Guiana
French Polynesia
French Southern Territories
Gabon
Gambia
Georgia
Germany
Ghana
Gibraltar
Greece
Greenland
Grenada
Guadeloupe
Guam
Guatemala
Guernsey
Guinea
Guinea-Bissau
Guyana
Haiti
Heard And McDonald Islands
Honduras
Hong Kong
Hungary
Iceland
Indonesia
Iraq
Ireland
Isle of Man
Israel
Italy
Jamaica
Jersey
Jordan
Kazakhstan
Kenya
Kiribati
Korea, Republic Of
Kosovo
Kuwait
Kyrgyzstan
Lao People's Democratic Republic
Latvia
Lebanon
Lesotho
Liberia
Liechtenstein
Lithuania
Luxembourg
Macau
Macedonia
Madagascar
Malawi
Malaysia
Maldives
Mali
Malta
Marshall Islands
Martinique
Mauritania
Mauritius
Mayotte
Mexico
Micronesia, Federated States Of
Moldova, Republic Of
Monaco
Mongolia
Montenegro
Montserrat
Morocco
Mozambique
Myanmar
Namibia
Nauru
Nepal
Netherlands Antilles
New Caledonia
New Zealand
Nicaragua
Niger
Nigeria
Niue
Norfolk Island
Northern Mariana Islands
Oman
Pakistan
Palau
Palestine
Panama
Papua New Guinea
Paraguay
Peru
Philippines
Pitcairn
Poland
Portugal
Puerto Rico
Qatar
Reunion
Romania
Russian Federation
Rwanda
Saint Bartholemy
Saint Kitts And Nevis
Saint Lucia
Saint Martin
Saint Vincent And The Grenadines
Samoa
San Marino
Sao Tome And Principe
Saudi Arabia
Senegal
Serbia
Seychelles
Sierra Leone
Sint Maarten
Slovakia
Slovenia
Solomon Islands
South Africa
South Georgia and the South Sandwich Islands
South Sudan
Sri Lanka
St. Helena
St. Pierre And Miquelon
Suriname
Svalbard And Jan Mayen Islands
Swaziland
Sweden
Switzerland
Taiwan
Tajikistan
Tanzania
Thailand
Togo
Tokelau
Tonga
Trinidad And Tobago
Tunisia
Turkey
Turkmenistan
Turks And Caicos Islands
Tuvalu
Uganda
Ukraine
United Arab Emirates
United States Minor Outlying Islands
Uruguay
Uzbekistan
Vanuatu
Vatican City
Venezuela
Vietnam
Virgin Islands (British)
Virgin Islands (U.S.)
Wallis And Futuna Islands
Western Sahara
Yemen
Yugoslavia
Zambia
Zimbabwe

By providing this information, you agree to the processing of your personal data by SANS as described in our Privacy Policy.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
  • © 2023 SANS™ Institute
  • Privacy Policy
  • Contact
  • Careers
  • Twitter
  • Facebook
  • Youtube
  • LinkedIn