FOR577: LINUX Incident Response and Threat Hunting

  • In Person (6 days)
  • Online
36 CPEs

Linux powers the Internet, and most webservers, databases, and even super-computing clusters today run on it. This means that our valuable, sensitive, and personal data is likely to be stored on a device running Linux. Unfortunately, our adversaries know this. Attacks against Linux platforms are increasing, so responders need the tools and techniques to combat them. FOR577 teaches the skills needed to identify, analyze, and respond to attacks on Linux platforms and how to use threat hunting techniques to find even the stealthiest attacker.

What You Will Learn

FOR577: Linux Threat Hunting & Incident Response provides responders and threat hunting teams with advanced skills to hunt down, identify, counter, and recover from a wide range of threats within enterprise networks, including advanced persistent threat (APT) nation-state adversaries, organized crime syndicates, and hactivism. Constantly updated, the course addresses today's incidents by teaching the hands-on incident response and threat hunting tactics and techniques that elite responders and hunters are successfully using to combat real-world breach cases.

FOR577 teaches the skills needed to identify, analyze, and respond to attacks on Linux platforms and how to use threat hunting techniques to find the stealthy attackers who can bypass existing controls. The concepts taught are built on common foundations in that we gather evidence, analyze it, and make decisions based on this analysis, all the while focusing on the specifics of the Linux platform. By using the tools built into the SANS SIFT Workstation, the course provides an all-inclusive solution that enables responders to quickly and effectively react to sophisticated intrusions.

During the course you will work through a number of exercises culminating in a final capstone, challenge built around a realistic attack with endpoint evidence, log data, and other artifacts you will encounter during day-to-day incident response activities. You will uncover evidence of an advanced threat actor working through a multiple-phase attack, going from reconnaissance to initial intrusion, then moving laterally throughout the organization's network. During the capstone you will bring together everything you have learned during the course and present your findings and recommendations on how security can be improved.

You Will Be Able To

  • Use the tools, techniques, and procedures necessary to effectively hunt, detect, and contain a variety of adversaries and to remediate incidents
  • Hunt through and perform incident response on Linux systems using the SIFT Workstation
  • Identify and track malware beaconing outbound to its command and control (C2) channel via analytical techniques.
  • Determine how the breach occurred by identifying the beachhead and spear phishing attack mechanisms
  • Track user and attacker activity second-by-second on the system you are analyzing through in-depth timeline and super-timeline analysis
  • Identify lateral movement and pivots within your enterprise, showing how attackers transition from system to system without detection.
  • Track data movement as the attackers collect critical data and shift those data to exfiltration collection points
  • Recover and analyze archives and archive files (.rar, .tar, etc.) used by APT-like attackers to exfiltrate sensitive data from the enterprise network
  • Use collected data to perform effective remediation across the entire enterprise.

Business Takeaways

  • Understand attacker tradecraft in order to perform proactive compromise assessments
  • Upgrade detection capabilities by having a better understanding of novel attack techniques and available forensic artifacts, and by focusing on critical attack paths
  • Develop threat intelligence to track targeted adversaries and prepare for future intrusion events
  • Build advanced forensics skills to counter anti-forensics and data hiding from technical subjects for use in both internal and external investigations.

Course Topics

  • Advanced use of a wide range of best-of-breed open-source tools in the SIFT Workstation to perform incident response and digital forensics
  • Hunting and responding to advanced adversaries such as nation-state actors, organized crime, and hacktivists
  • Threat hunting techniques that will aid in quicker identification of breaches
  • Rapid incident response analysis and breach assessment
  • An incident response and intrusion forensics methodology
  • Evidence collection, including disk and memory, during incident response and threat hunting
  • Internal lateral movement analysis and detection
  • Rapid and deep-dive timeline creation and analysis
  • Adversary threat intelligence development, indicators of compromise, and usage
  • Cyber-kill chain strategies
  • Step-by-step tactics and procedures to respond to and investigate intrusion cases

What You Will Receive With This Course

  • SIFT Workstation

This course uses the SIFT Workstation extensively to teach incident responders and forensic analysts how to investigate and respond to sophisticated attacks. The workstation contains hundreds of free and open-source tools, easily matching any modern forensic and incident response commercial response tool suite. A virtual machine is used with many of the hands-on class exercises. Features of the SIFT Workstation include:

  • Ubuntu Linux LTS base
  • 64-bit base system
  • Better memory utilization
  • Auto-DFIR package update and customizations
  • Latest forensics tools and techniques
  • VMware Appliance ready to tackle forensics
  • Cross-compatibility between Linux and Windows
  • Expanded file system support (NTFS, HFS, EXFAT, and more)
  • Electronic Download Package
    • Case images (disk and memory) from systems compromised by an APT intrusion
    • SIFT Workstation virtual machines, tools, and documentation
    • Exercise book is over 250 pages long with detailed step-by-step instructions and examples to help you become a master incident responder.

Syllabus (36 CPEs)

Download PDF
  • Overview

    Incident responders and threat hunters should be armed with the latest tools, techniques, and processes (TTPs) to identify, track, and contain advanced adversaries and to remediate incidents. It is important that our DFIR knowledge includes our own TTPs and those used by our adversaries. Section 1 introduces the fundamentals of incident response and then looks at the specific needs to carry out our duties in a Linux environment. The section starts by examining the reasons why we need incident response and presents SANS' six-step incident response methodology as it applies to an enterprise's response to a targeted attack.

    This section will also introduce the Stark Skunkworks intrusion scenario, which sets the stage for our lab exercises and capstone challenge. This is followed by looking at how, as incident responders, we can use the Linux command line to our advantage and analyze common activity such as installing specific software packages.

    We finish the section by looking at the importance of developing cyber threat intelligence to impact the adversaries' kill chain. We'll demonstrate forensic live response techniques and tactics that can be applied both to single systems and across the entire enterprise.

    • SIFT Workstation orientation
    • Situational awareness in incident response: Understanding Stark Skunkworks
    • Introduction to Linux commands and how to use them in Digital Forensics and Incident Response (DFIR)
    • Reviewing package management evidence
    • Threat intelligence and threat hunting
    • Why Incident Response Is Needed
      • Who are our adversaries?
      • The current state of Linux intrusions
    • The Incident Response Process
      • Preparation: Key tools, techniques, and procedures that an incident response team needs to respond properly to intrusions
      • Identification/Scoping: Proper scoping of an incident and detecting all compromised systems in the enterprise
      • Containment/Intelligence Development: Restricting access, monitoring, and learning about the adversary in order to develop threat intelligence
      • Eradication/Remediation: Determining and executing key steps that must be taken to help stop the current incident and then move to real-time remediation
      • Recovery: Recording the threat intelligence to be used in the event of a similar adversary returning to the enterprise
      • Avoiding "Whack-A-Mole" Incident Response: Going beyond immediate eradication without proper incident scoping/containment
    • SRL Skunkworks
      • Introduction to the course scenario
      • Client background
    • Introduction to Linux
      • Linux basics
      • DFIR challenges
      • The distro problem
      • Linux terminal basics
    • Package Management
      • Distro differences
      • Package management tool differences
      • Manual analysis
    • Threat Intelligence and Host-based Threat Hunting
      • Hunting vs. reactive response
      • Intelligence-driven incident response
      • Building a continuous incident response/threat hunting capability
      • Forensic analysis versus threat hunting across endpoints
      • Threat hunt team roles
  • Overview

    Disk evidence collection and analysis skills are crucial for incident responders, forensic investigators, and threat hunters because they allow for identifying the source and scope of a security breach. Digital forensic experts need to collect and preserve data from disk storage devices such as hard drives, solid-state drives, and USB drives in order to determine how an attack occurred, what data was accessed or stolen, and who was responsible. Without this critical evidence, it is challenging to reconstruct the events leading up to the breach and determine the necessary steps to prevent similar incidents from happening in the future.

    Moreover, disk analysis skills help responders and investigators identify the type of malware or malicious code used in the attack. This information is essential to determine the tactics, techniques, and procedures used by the attackers and their motivations. By analyzing the data stored on disks, responders and investigators can identify suspicious files, unusual network traffic patterns, and other indicators of compromise. They can then use this information to develop countermeasures to mitigate the risk of further attacks.

    Fundamentally, the ability to collect evidence from disks is critical for DFIR because most digital evidence is stored on disk storage devices, making the devices an essential source of information for responders, investigators, and threat hunters. Even if you just need to collect log data, being able to collect it from a disk image opens up opportunities for a broad range of incident response solutions. In addition, disk storage devices often hold deleted files and other remnants of past activities, which can provide valuable clues to the sequence of events leading up to an incident.

    • Introduction to the Sleuth Kit
    • Reviewing filesystem data
    • Disk evidence collection
    • Reviewing operating system filesystems
    • The Sleuth Kit
      • Introduction and the layers model
      • Filesystem layer tools
      • Filename layer tools
      • Metadata layer tools
      • Data units layer tools
      • Application layer tools
    • Linux File Systems
      • Overview
      • Basic structures - superblocks and inodes
      • Ext family
      • XFS family
      • Manually extracting data
    • Disk Evidence Collection
      • Physical vs. virtual systems
      • dd
      • dcfldd
      • dc3dd
      • Ewfacquire
    • Image Mounting
      • RAW/Simple files
      • E01 format evidence files
      • Complex files
    • Operating System File Structures
      • File system hierarchy
      • Boot file locations
      • Binary file locations
      • Configuration file locations
      • Devices and driver file locations
      • Shared libraries
      • User profiles
      • Optionally installed files
      • Temporary file locations
      • Runtime data
    • File System Artifacts
      • Hunting tips
      • Areas to investigate
  • Overview

    Section 3 looks at how to use the data logged by the operating system to profile the device and analyze boot sequences, kernel activity, logins and user events. The section covers default log data, Auditd (although this isn't enabled by default on all Linux distros, you should definitely consider turning it on) and the Operating System Journal.

    Log data is a fundamental evidence source for incident response and threat hunting. It allows investigators to understand what happened and when it happened. Using built-in capabilities, we can peel back the actions of our adversaries and, with well-configured logging, make it almost impossible for an attacker to completely hide from our investigation. Unfortunately, Linux logging can be significantly different from what we are used to -- especially if we have come from a Windows DFIR background. Significant issues faced by investigators include the different ways Linux distro's log data and a mix between UTC and local timestamps. This section will look at strategies you can implement to manage and mitigate these issues.

    • System and log profiling
    • Reviewing system logs
    • Analyzing authentication logs
    • Reviewing webserver logs
    • Reviewing database logs
    • AuditD logs the Journal
    • Device Profiling
      • Evidence management

      • Confirm the device
      • Check time zones
      • Validate the distro
    • Linux Logs
      • Linux logging basics
      • Log analysis strategies
      • Syslog and Logrotate
      • Global system logs -- logging the kernel, boot processes, system messages and background services
      • Authentication logs -- authentication, privilege use, binary and plain text log formats
      • Application logs -- webservers, databases, filesharing and firewall logs
    • Auditd

      • Introduction
      • Log file format
      • Analysis techniques
    • The Operating System Journal
      • Introduction
      • How the journal works
      • What gets logged
      • Analysis techniques
  • Overview

    Section 4 expands on the knowledge we have built so far and introduces tools and techniques to respond to intrusions in larger enterprises. The section starts by looking at how to scale your response and some of the tools that can assist with this. This topic is then developed further as we move into Endpoint Detection and Response (EDR) solutions for the Linux environment and introduce two alternatives to expensive commercial EDR tools -- OSSEC and Velociraptor. We'll cover how to configure and deploy both tools, enabling you to make sure that all your Linux devices have good quality monitoring and response capabilities.

    Finally, this section looks at Linux memory structures and how to collect volatile data for analysis. Given that this can be a complex process, and that analytical tools today are still not what they should be, we also look at using live response techniques to view this data on a target system. This has the added benefit of being something we can leverage through EDR tools, reducing the time and bandwidth required to capture memory from systems where the installed RAM could be running in the hundreds of gigabytes.

    • EDR Tools
    • Capturing RAM
    • Live memory analysis
    • Enterprise Response
      • Introduction
      • Problems and solutions
      • Tools to consider
    • Endpoint Detection and Response (EDR)
      • Introduction
      • Linux EDR issues
      • Alternatives to commercial EDR
      • OSSEC deployment and use
      • Velociraptor deployment and use
    • Linux Memory and DFIR
      • Why memory matters
      • Memory acquisition with AVML
      • Memory locations on the filesystem
    • Live memory analysis
      • Reviewing /proc
      • Live response workflow
  • Overview

    This course section builds on the previous sections by looking at how we can use our increased knowledge to enhance our incident response work. We start by looking at triage, which is essential for any modern incident response, especially in large enterprises. We introduce the concept of rapidly assessing systems to make quick decisions about which devices need further investigation. This approach allows us to quickly work through large environments and focus our investigative efforts where they provide maximum value. We'll also look at freely available tools that help facilitate triage and improve response times.

    The section then moves to looking at timeline generation. Timelines are arguably an incident responder's superpower, allowing you to uncover some of the deepest secrets about an attack. We will look at two basic methods for building timelines and how to analyze them effectively.

    Once we understand the timelines, we will look at how attackers try to defeat them, then examine the most common anti-forensic techniques and how incident responders can minimize their impact on the investigation. We close the section with a broad discussion on how to make incident response in Linux better.

    • Running triage tools
    • Triage assessment
    • Filesystem timelines
    • Super timeline creation
    • Super timeline analysis
    • Triage and DFIR Tools
      • Introduction and concepts
      • Workflow
      • Collecting the data
      • Open-source triage tools
      • CyLR
      • GRR
      • Velociraptor offline collectors
      • Dissect
      • Triage with UAC
      • Build your own triage scripts
    • Timelines
      • Introduction
      • Types of timelines
      • Filesystem timeline creation and analysis
      • Super-timeline creation and analysis
      • Targeted timeline creation
    • Anti-Forensics
      • What to look for
      • Timestamp manipulation
      • Recovering deleted files
    • Improving Incident Response
      • Workflows
      • Hardening the environment
  • Overview

    This incredibly rich and realistic Intrusion Forensic Challenge is based on a real-world advanced persistent threat (APT) group. It brings together techniques learned throughout the course and tests your newly acquired skills in a case that simulates an attack by an advanced adversary. The challenge is based on a real intrusion into a Linux enterprise environment. You will be asked to uncover how the systems were compromised in the initial intrusion, find other systems the adversary moved to laterally, and identify intellectual property stolen via data exfiltration. This capstone exercise will enable you to leave the course with hands-on experience investigating realistic attacks, curated by a cadre of instructors with decades of experience fighting advanced threats from attackers ranging from nation-states to financial crime syndicates and hactivist groups.

    • Work in incident response teams to analyze multiple systems in an enterprise network
    • Learn to identify and track attacker actions across a multi-device environment finding initial exploitation, reconnaissance, persistence, privilege escalation, lateral movement, and data theft/exfiltration
    • Witness and participate in a team-based approach to incident response
    • Discover evidence of some of the most common and sophisticated attacks in the wild, including custom nation-state malware.
    • Each team will be asked to answer key questions, just as they would during a real breach in their organizations, in critical areas outlined below:
    Identification and Scoping:
    • When did the APT group breach our network?
    • How did the attackers get into the environment?
    • What systems were compromised?
    • What accounts and privileges did the attackers attain on each system?
    • When and how did the attackers first laterally move to each system?
    Containment and Threat Intelligence Gathering:
    • Once on other systems, what did the attackers look for on each system?
    • What data was exfiltrated and how? Determine what was stolen (recover any archives exfiltrated, find encoding passwords, and extract the contents to verify extracted data) and perform damage assessments.
    • Collect and list all malware used in the attack.
    • Develop and present security intelligence or an indicator of compromise for the APT group "beacon" malware for both host- and network-based enterprise scoping. What specific indicators exist for the use of this malware?
    Remediation and Recovery:
    • What accounts need password changes? Did any malicious accounts get created?
    • Based on the attacker techniques and tools discovered during the incident, what are the recommended steps to remediate and recover from this incident?
      • What systems need to be rebuilt?
      • What IP addresses need to be blocked?
      • What countermeasures should we deploy to slow or stop these attackers if they come back?
      • What recommendations would you make to detect these intruders in our network again?


FOR577 is an advanced incident response course that focuses on the Linux operating system. We do not cover basic forensic techniques or introductory attacker techniques. Students are not expected to have detailed understanding of Linux, but it is strongly recommended that they have at least the level of knowledge provided by the SANS SEC401 or SEC406 courses.

Laptop Requirements


A properly configured system is required to fully participate in this course. If you do not carefully read and follow these instructions, you will likely leave the class unsatisfied because you will not be able to participate in hands-on exercises that are essential to this course. Therefore, we strongly urge you to arrive with a system meeting all the requirements specified for the course.

As a summary, you can use any operating system that also can install and run VMware virtualization products. Please note, macOS computers with M1 or M2 chips are not currently supported and cannot run the virtual machines provided for this course.

Please download and install VMware Workstation 15 or VMware Fusion 7 or higher versions on your system before the start of the class. If you do not own a licensed copy of VMware Workstation or Fusion, you can download a free 30-day trial copy from VMware. VMware will send you a time-limited serial number if you register for the trial on its website.

This is common sense, but we will say it anyway: Back up your system before class. Better yet, do not have any sensitive data stored on the system. SANS cannot be responsible for your system or data.

  • CPU: 64-bit Intel i5/i7 x64 2.0+ GHz (4th generation or above) processor or higher-based system is mandatory for this class (Important - Please Read: a 64-bit system processor is mandatory)
  • CRITICAL NOTE: Apple systems using the M1 or M2 processor line cannot perform the necessary virtualization functionality and therefore cannot in any way be used for this course.
  • BIOS settings must be set to enable virtualization technology, such as "Intel-VT." Be absolutely certain you can access your BIOS if it is password protected, in case changes are necessary.
  • 16 GB of RAM or more is required.
  • 350GB of free storage space or more is required.
  • At least one available USB 3.0 Type-A port. A Type-C to Type-A adapter may be necessary for newer laptops. Some endpoint protection software prevents the use of USB devices, so test your system with a USB drive before class.
  • Wireless networking (802.11 standard) is required. There is no wired Internet access in the classroom. Local Administrator Access is required. This is absolutely required. Don't let your IT team tell you otherwise. If your company will not permit this access for the duration of the course, then you should make arrangements to bring a different laptop.
  • Local Administrator Access is required. This is absolutely required. Don't let your IT team tell you otherwise. If your company will not permit this access for the duration of the course, then you should make arrangements to bring a different laptop.
  • PLEASE NOTE: Do NOT use the version of the SIFT Workstation downloaded from the Internet. We will provide a custom FOR577 version specifically configured for training on Day 1 of the course.
MANDATORY FOR577 SYSTEM SOFTWARE REQUIREMENTS (Please install the following before the beginning of the class):
  1. Install VMware Workstation 15 or VMware Fusion 7 (or a higher version)
  2. Download and install 7Zip on your host.

Additional notes:

  • Your course media is delivered via download from the SANS "Course Material Downloads" page. The media files for class will be large, in the 40 - 50 GB range. You need to allow plenty of time for the download to complete. Internet connections and speeds vary greatly and are dependent on many different factors. Therefore, it is not possible to give an estimate of the length of time it will take to download your materials. Please start your course media downloads as soon as you get the link. You will need your course media immediately on the first day of class. Waiting until the night before the class starts to begin your download has a high probability of failure.
  • SANS has begun providing printed materials in PDF form. This course uses an electronic workbook in addition to the PDFs. We have found that a second monitor and/or a tablet device can be useful for keeping the class materials visible while the instructor is presenting or while you are working on lab exercises.
  • Bring/install any other forensic tool you feel could be useful (EnCase, FTK, etc.). For the final challenge at the end of the course, you can utilize any forensic tool, including commercial capabilities. If you have any dongles, licensed software, you are free to use them.
  • Again, DO NOT use the version of the SIFT Workstation downloaded from the Internet. We will provide you with a version specifically configured for the FOR577 materials on Day 1 of the course.

If you have additional questions about the laptop specifications, please contact

Author Statement

"Linux is a mainstream operating system found in almost every enterprise. It is used to host critical services and store sensitive personal and financial data, and it powers the underlying infrastructure we use on a day-to-day basis, making it a high-value target for our adversaries. Additionally, there is often a perception that Linux is 'more secure' than other operating systems, which results in less thorough security tool coverage. These two elements combine to make Linux intrusions both increasingly common and harder for our Security Operations Center/Incident Response teams to fully respond to. In one recent incident, attackers installed a persistence mechanism in a company's firewall that remained undiscovered during Windows-focused response and remediation activities.

"All cybersecurity defenders need to have the knowledge to deal with attacks on every platform in our environments. This means it is essential to understand how to collect and analyze digital evidence from Linux systems to determine the extent of the damage and identify the root cause of an incident. By analyzing the digital evidence, defenders can identify indicators of compromise and determine the tools, techniques, and processes used by the attacker. This information can be used to develop countermeasures and prevent similar attacks from occurring in the future."

-Taz Wake

Register for FOR577