FOR577: LINUX Incident Response and Threat Hunting

Overview

Incident responders and threat hunters should be armed with the latest tools, techniques, and processes (TTPs) to identify, track, and contain advanced adversaries and to remediate incidents. It is important that our DFIR knowledge includes our own TTPs and those used by our adversaries. Section 1 introduces the fundamentals of incident response and then looks at the specific needs to carry out our duties in a Linux environment. The section starts by examining the reasons why we need incident response and presents SANS' six-step incident response methodology as it applies to an enterprise's response to a targeted attack. We also consider how AI is starting to reshape DFIR work and where it fits in a responder's toolkit.

This section will also introduce the Stark Skunkworks intrusion scenario, which sets the stage for our lab exercises and capstone challenge. This is followed by looking at how, as incident responders, we can use the Linux command line to our advantage during investigations. Next, we look at the typical attacker behaviors for initial access, lateral movement, and persistence during intrusions into Linux-based systems.

We finish the section by looking in depth into the way Linux stores files and records common adversary behavior. We'll demonstrate forensic live response techniques and tactics that can be applied both to single systems and across the entire enterprise.

Full Lab Details

• SIFT Workstation orientation
• Situational awareness in incident response: Stark Skunkworks Introduction
• Introduction to Linux commands and how to use them in Digital Forensics and Incident Response (DFIR)
• Conducting an initial attack assessment
• Reviewing Operating System data during an intrusion

Full Topic Details

• Incident Response Foundations
• Who are our adversaries?
• The current state of Linux intrusions
• The Incident Response Process
• Preparation: Key tools, techniques, and procedures that an incident response team needs to respond properly to intrusions
• Identification/Scoping: Proper scoping of an incident and detecting all compromised systems in the enterprise
• Containment/Intelligence Development: Restricting access, monitoring, and learning about the adversary in order to develop threat intelligence
• Eradication/Remediation: Determining and executing key steps that must be taken to help stop the current incident and then move to real-time remediation
• Recovery: Recording the threat intelligence to be used in the event of a similar adversary returning to the enterprise
• Avoiding "Whack-A-Mole" Incident Response: Going beyond immediate eradication without proper incident scoping/containment
• SRL Skunkworks
• Introduction to the course scenario
• Client background
• Introduction to Linux
• What is Linux?
• DFIR challenges
• The distro problem
• Cyber Attacks and Linux
• The Unified Kill Chain
• Linux Attacks
• Linux Command Basics
• Common commands
• DFIR techniques
• Operating System File Structures
• File system hierarchy
• Boot file locations
• Binary file locations
• Configuration file locations
• Devices and driver file locations
• Shared libraries
• User profiles
• Optionally installed files
• Temporary file locations
• Runtime data
• File System Artifacts
• Hunting tips
• Areas to investigate
• User accounts and authentication
• Shell history files
• Running processes
• Network connections
• Linux persistence
• Investigating Windows Subsystem for Linux

Overview

Disk evidence collection and analysis skills are essential for incident responders, forensic investigators, and threat hunters because most digital evidence still resides on storage devices. Being able to acquire data soundly from both physical and virtual systems, and to mount the resulting RAW and E01 images, ensures that evidence is preserved and ready for examination. On Linux, this also means understanding the file systems most commonly encountered, such as ext4, XFS, and Btrfs, and how their superblock and inode structures, timestamps, and metadata can be interpreted with The Sleuth Kit or, where necessary, recovered by hand to reconstruct what happened on a system.

Knowing how software is installed and run is equally important when investigating a Linux intrusion. Package management data records what has been added to a system, and being able to read package formats and metadata, verify digital signatures, and hunt through this data across Debian, Red Hat, and SUSE distributions helps responders identify unexpected or malicious software. Linux executables tell a similar story: using ELF review tools and capability analysis, investigators can determine what a binary is and what it is able to do without ever running it, which is critical when assessing suspected malware.

Cyber threat intelligence and threat hunting turn this evidence into proactive defense. Understanding what threat intelligence is, where it comes from, and how indicators map to the Pyramid of Pain allows responders to recognize adversary behavior rather than isolated artifacts. Threat hunting builds on this, using hypothesis-led and intelligence-led approaches, increasingly supported by AI-assisted query generation and tools such as YARA, to search an environment for attackers who have bypassed existing controls before an incident is formally declared.

Full Topic Details

• The Sleuth Kit
• Introduction and the layers model
• Filesystem layer tools
• Filename layer tools
• Metadata layer tools
• Data units layer tools
• Application layer tools
• Linux File Systems
• Overview
• Basic structures - superblocks and inodes
• Ext family filesystems
• XFS Filesystems
• Btrfs filesystems
• Manually extracting data
• Disk Evidence Collection
• Physical vs. virtual systems
• dd
• dcfldd
• dc3dd
• Ewfacquire
• Collecting data over the wire
• Image Mounting
• RAW/Simple files
• E01 format evidence files
• Complex files
• Mounting VHDX/VMDK files
• Mounting LVM systems
• Package Management
• Distro differences
• Package management tool differences
• Manual package analysis on Debian and Red Hat systems
• Examining Linux Executable Files
• ELF formats
• Investigating ELF files
• Examining capabilities with Capa
• Threat Intelligence and Host-based Threat Hunting
• What is threat intelligence
• Sources of threat intelligence
• Introduction to threat hunting
• Hypothesis driven threat hunts
• Using AI to support threat hunting
• Intelligence led threat hunting
• Using Yara

Overview

Log data is a fundamental evidence source for incident response and threat hunting, allowing investigators to establish what happened on a system and when. With well-configured logging, it becomes very difficult for an attacker to operate without leaving a trace. Linux logging, however, can differ significantly from what responders with a Windows background expect: distributions record data in different ways and locations, and timestamps may be stored in a mix of UTC and local time. Knowing how to profile a system, confirming its identity, timezone, and distribution, and understanding how syslog, the systemd journal, and log rotation work, are therefore the starting points for any sound log analysis.

From this foundation, investigators can work through the logs that matter most during an intrusion. System logs record boot sequences, kernel messages, and the services running on a host, while authentication and authorization logs, including the binary wtmp and btmp records, reveal who logged in, when, and what privileges they used. The Linux Auditing Daemon (Auditd) extends this further and, when configured well, provides a detailed account of system activity that is valuable for both live response and offline analysis.

Many Linux systems exist to run applications, and the logs those applications produce are frequently where evidence of an attack first appears. Knowing how to locate and interpret logs from web servers such as Nginx and Apache, databases such as MySQL, MariaDB, and PostgreSQL, file-sharing services such as vsftpd and Samba, and host firewalls allows responders to trace adversary activity through the services an organization actually depends on.

Full Lab Details

• System and Log Profiling
• Investigating the journal
• Analyzing Authentication Logs
• Analyzing Audit logs
• Auditd and Elasticsearch
• Reviewing Webserver Logs
• Reviewing Database Logs
• Reviewing Host Firewall Logs

Full Topic Details

• Device Profiling
• Evidence management
• Confirm the device
• Check time zones
• Check network configuration
• Validate the distro
• Linux Logging
• Linux logging basics
• Linux Journal basics
• Log analysis strategies
• Logging configurations and Logrotate
• Understanding the Journal and Syslog
• How the journal works
• What gets logged
• Journal analysis techniques
• Global system logs—logging the kernel, boot processes, system messages and background services
• Authentication Logs
• Authentication and privilege use
• binary and plain text log formats
• Auditd
• Introduction
• Log file format
• Analysis techniques
• Application Logs
• Webserver logs
• Database logs
• Filesharing application logs
• Host-based firewall logs

Overview

Section 4 expands on the knowledge we have built so far and introduces tools and techniques to respond to intrusions in larger enterprises. The section starts by looking at how to scale your response and some of the tools that can assist with this. This topic is then developed further as we move into Endpoint Detection and Response (EDR) solutions for the Linux environment and introduce two alternatives to expensive commercial EDR tools—OSSEC and Velociraptor. We'll cover how to configure and deploy both tools, enabling you to make sure that all your Linux devices have good quality monitoring and response capabilities.

Finally, this section looks at Linux memory structures and how to collect volatile data for analysis. Given that this can be a complex process, and that analytical tools today are still not what they should be, we also look at using live response techniques to view this data on a target system. This has the added benefit of being something we can leverage through EDR tools, reducing the time and bandwidth required to capture memory from systems where the installed RAM could be running in the hundreds of gigabytes.

Full Topic Details

• Enterprise Response
• Introduction
• Problems and solutions
• Tools to consider
• Endpoint Detection and Response (EDR)
• Introduction
• Linux EDR issues
• Alternatives to commercial EDR
• OSSEC deployment and use
• Velociraptor deployment and use
• Linux Memory and DFIR
• Why memory matters
• Memory acquisition with AVML
• Memory locations on the filesystem
• Live memory analysis
• Reviewing /proc
• Live response workflow

Overview

Artificial intelligence and large language model (LLM) tools have moved rapidly into the enterprise, from cloud services and self-hosted models to the AI coding assistants now embedded in developer workflows. Each of these introduces a new attack surface and a new source of evidence that most responders have never had to examine. Investigating them starts with understanding how they are deployed, whether cloud, local, or hybrid, and where useful data sits within the AI application stack, so that the model, its implementation, the host system, and the runtime can each be examined for evidence.

Because these tools store conversation history, configuration, and tool activity in predictable places, knowing where to look is much of the work. Investigators learn the evidence locations and artifacts left by widely used assistants such as Claude Code, Cursor, GitHub Copilot, and Google's Gemini CLI, as well as self-hosted platforms such as Ollama, including conversation and retrieval-augmented generation (RAG) data and how to verify that a model has not been tampered with. The growing use of autonomous AI agents, which can take actions of their own, adds a further dimension that responders need to be able to investigate.

While the underlying incident response process does not change, the ways these systems are attacked do. Prompt injection, including indirect injection through data the model consumes, can subvert an AI tool's behavior, while sensitive data can be exposed or exfiltrated through ordinary use or through abuse of connected data sources. Supply chain risks, model tampering, and compromised plugins extend the problem further. Recognizing the indicators of these attacks, scoping an LLM incident, collecting the right live-response data, and correlating a prompt with its effect on the host are the skills that turn a standard investigation into an effective response to an AI-related incident.

Attackers also work to hide their activity, and Linux offers many opportunities to do so. Examining the most common anti-forensic techniques, including tampering with shell history, manipulating file timestamps with tools such as touch, and deleting evidence, shows both how these methods work and how to detect them. Where files have been destroyed, recovery tools such as Foremost, Scalpel, PhotoRec, and tsk_recover can often bring them back, allowing an investigation to continue despite an attacker's efforts.

Finally, good incident response is as much about preparation as reaction. Repeatable DFIR playbooks, AI-assisted tabletop exercises, system hardening, forensic readiness, and effective monitoring all help an organization respond faster and more consistently the next time, and to keep improving with each iteration.

Full Topic Details

• Investigating AI and LLM Tools
• The expanding AI/LLM attack surface and why it is difficult to investigate
• Deployment models: cloud, local, and hybrid
• Mapping evidence across the AI application stack: model, implementation, system, and runtime layers
• AI Coding Assistants
• Evidence locations and artifacts for Claude Code, Cursor, GitHub Copilot, Windsurf, and Gemini CLI
• Model Control Protocol (MCP) architecture and its forensic artifacts
• Worked examples: Claude Code history and Cursor's state.vscdb
• Self-Hosted LLM Platforms
• Ollama architecture and local LLM evidence sources
• Conversation and retrieval-augmented generation (RAG) evidence
• Verifying model integrity and investigating autonomous AI agents
• What Is Different About LLM Incidents
• LLM threat categories and attack surface
• Prompt injection, including indirect injection, and its indicators
• Sensitive data exposure, exfiltration, and RAG abuse (including reading vector databases)
• Supply chain risks, model tampering, and plugin/tool compromise
• Responding to LLM Incidents
• Scoping and triaging an LLM incident
• Local LLM live-response commands and agent evidence collection
• Correlating prompt, tool, and host effect; containment decisions
• Linux Anti-Forensics
• Shell history file attacks
• Timestamp manipulation with touch, and how to detect it
• Recovering deleted files with Foremost, Scalpel, PhotoRec, and tsk_recover
• Improving Incident Response
• Building and iterating DFIR playbooks
• AI-assisted tabletop exercises
• System hardening, forensic readiness, and monitoring

Overview

This realistic intrusion forensics challenge is based on a real-world advanced persistent threat (APT) group and a genuine intrusion into a Linux enterprise environment. It brings together the techniques learned throughout the course and tests your skills against an attack by an advanced adversary. Working as part of an incident response team, you will identify the first system compromised and how the attackers got in, trace their lateral movement across the network and the accounts and privileges they gained, and recover and analyze the malware they used to develop indicators of compromise for host- and network-based scoping. You will also determine what data was exfiltrated and how, decide what is needed to eradicate the attacker, and recommend how to recover from and prevent a similar incident in the future. Finally, you will present your findings to a technical CISO, just as you would during a real breach, leaving the course with hands-on experience of investigating a sophisticated Linux intrusion from end to end.

Full Topic Details

• Work in incident response teams to analyze multiple systems in an enterprise network
• Learn to identify and track attacker actions across a multi-device environment finding initial exploitation, reconnaissance, persistence, privilege escalation, lateral movement, and data theft/exfiltration
• Witness and participate in a team-based approach to incident response
• Discover evidence of some of the most common and sophisticated attacks in the wild, including custom nation-state malware.
• Each team will be asked to answer key questions, just as they would during a real breach in their organizations, in critical areas outlined below:
• Identification and Scoping:
• When did the APT group breach our network?
• How did the attackers get into the environment?
• What systems were compromised?
• What accounts and privileges did the attackers attain on each system?
• When and how did the attackers first laterally move to each system?
• Containment and Threat Intelligence Gathering:
• Once on other systems, what did the attackers look for on each system?
• What data was exfiltrated and how? Determine what was stolen (recover any archives exfiltrated, find encoding passwords, and extract the contents to verify extracted data) and perform damage assessments.
• Collect and list all malware used in the attack.
• Develop and present security intelligence or an indicator of compromise for the APT group "beacon" malware for both host- and network-based enterprise scoping. What specific indicators exist for the use of this malware?
• Remediation and Recovery:
• What accounts need password changes? Did any malicious accounts get created?
• Based on the attacker techniques and tools discovered during the incident, what are the recommended steps to remediate and recover from this incident?
• What systems need to be rebuilt?
• What IP addresses need to be blocked?
• What countermeasures should we deploy to slow or stop these attackers if they come back?
• What recommendations would you make to detect these intruders in our network again?