FOR608: Enterprise-Class Incident Response & Threat Hunting

Overview

The FOR608: Enterprise-Class Incident Response & Threat Hunting course begins with discussions on current cyber defense concerns, and how incident responders and threat hunters can take a more active role in detection and response. Collaboration within the team and the community are a focus, as we look to incorporate shared knowledge from sources like the MITRE ATT&CK(R) framework. Furthermore, we discuss taking an active defense approach to slow attackers and facilitate detection. Specific to active detection, the use of honeypots, honey tokens, and canaries are covered, along with ways to deploy them opportunistically. This type of tripwire in the network provides defenders and responders needed visibility to find and respond to intrusions quickly.

When a compromise does occur, which is an unfortunate but inevitable truth, we continue the discussion with a focus on the processes and techniques that allow for efficient handling of intrusions. Concepts such as leading the response, managing team members, evaluating resource constraints, documenting findings and communicating with stakeholders are covered in detail. The purpose-built Aurora documentation tool is presented as a means for tracking the investigation phases, from initial detection to scoping, containment, indicator development, and remediation.

We continue the discussion by examining an alert triggered in our example company (Stark Research Labs) as a pivot point into a potential attack. We present rapid response options for targeted data collections at scale for such situations. In particular, we discuss KAPE and Velociraptor as options for standalone triage imaging tools. In the case of Velociraptor, it can be installed on a persistent client-server basis, but also as a standalone collector. We demonstrate how to use it in either case to collect critical artifacts for tracking the adversary's progress. Rapidly post-processing the acquired data for analysis is another important piece of the puzzle. Solutions are presented to quickly take the collected artifact files and process them for analysis in Timesketch, Elasticsearch, or for individual artifact review.

Section one wraps up with a deeper-dive look at a possibly compromised host from Stark Research Labs. Triage data was collected by company personnel has been processed into a timeline and the data imported into Timesketch, a tool developed by the Google IR team. We utilize Timesketch as a powerful platform for scalable and collaborative analysis of forensic data.

Full Topic Details

• Incident Response and Threat Hunting in the Enterprise
• Taking an Active Defense approach to threat hunting and detection
• Using Active Defense concepts of Deny, Disrupt, and Degrade for attacker containment
• Using the Active Defense concept of Deception for detection
• Pros and cons of using honeypots
• Pros and cons of canary / honey tokens
• Deploying canary tokens into an environment for intrusion detection
• Managing Large-Scale Response
• Fostering key principles of successful response within the team and organization
• Structuring teams, roles, and responsibilities
• Leading the response
• Managing resources
• Combining incident response and project management disciplines
• Effective documentation and communication for tracking and reporting incidents
• Introduction to IR documentation tools and techniques, including Aurora, an open-source incident response documentation utility
• Rapid Response Triage
• Learn about KAPE for Windows artifact acquisition and parsing
• Execute Velociraptor to quickly acquire forensic artifacts from Windows, Linux, and Mac
• Create custom acquisition packages for Velociraptor
• Post-process results for timeline analysis using Timesketch, Elasticsearch, or CSV files
• Scalable and Collaborative Analysis with Timesketch
• Using Timesketch to perform deep-dive analysis across multiple hosts with multiple analysts
• Annotate, label, bookmark events of interest to create custom timeline views
• Apply analytics and visualizers to assist analysis
• Create stories to convey findings

Overview

We begin Section two with an examination of key threat intelligence concepts, including developing and implementing threat intelligence internally. External projects such as the MITRE ATT&CK(R) Matrix are reviewed as a common source of shared threat intelligence. We also discuss both MISP and OpenCTI as two comprehensive threat intel platforms for ingesting, tracking, and sharing threat intelligence.

Section two then pivots into response mode. We begin collecting evidence at scale to scope a potential intrusion against our example company, Stark Research Labs (SRL). SRL has Endpoint Detection and Response (EDR) tooling in place, and we leverage that data to assist scoping. However, attackers sometimes bypass or otherwise subvert EDR technology, so a discussion of common bypass techniques is presented. This provides students with both awareness of EDR limitations, as well as training to look for anomalous activities within the EDR log data.

Moving beyond the analysis of commonly logged artifacts, we introduce the open-source Velociraptor tool as a powerful platform for incident response and threat hunting at scale. Velociraptor is adept at pulling forensic artifacts from across the enterprise, as well as providing analysts with a tool to deep-dive individual hosts of interest. We show Velociraptor to be a flexible tool useful for a number of situations, as well as for a number of operating systems and architectures.

One of many useful features of Velociraptor is its ability to push collected data into Elasticsearch. Elasticsearch is another powerful and flexible tool appropriate for any responder's toolkit. As such, we discuss how to ingest and process various data types, including data from Velociraptor, from the PowerShell IR framework "Kansa", and from the "Log2timeline" tool. We then use Elasticsearch's SQL API and newer ES|QL query language to run fast searches and aggregations against ingested data. A bonus lab guides students through the process to create dashboards and visualizations in Kibana for a more traditional approach to searching and analyzing forensic data using the Elastic Stack.

We conclude Section two with a discussion of how Artificial Intelligence (AI) intersects with digital forensics and incident response (DFIR), emphasizing effective techniques such as context control, retrieval-augmented generation (RAG), and customized AI solutions to support research and learning. We explore practical methods for integrating AI into DFIR tools through Model Context Protocol (MCP) servers and the emerging role of agentic AI in investigative workflows. We also compare cloud-based and local Large Language Model (LLM) deployments for DFIR operations and discuss strategies for safely and accurately incorporating them into enterprise DFIR practices. Finally, several attacks against AI are covered, including prompt-injection attacks and MCP server compromises. These attack vectors are important to understand as AI continues to permeate enterprise IT.

Full Topic Details

• Intel-Driven Incident Response
• Understand the importance of cyber threat intelligence in threat hunting and incident response, as well as to the organization’s overall security program
• Review various sources of threat intelligence and integrating it with the IR process
• Leverage cyber threat intelligence available from MITRE's ATT&CK(R) Matrix
• Discuss developing and managing intelligence in your organization
• EDR and EDR Bypass
• Review the capabilities and challenges of endpoint detection and response (EDR)
• Analyze Sysmon telemetry and log events for incident scoping/identification
• Discuss attacker techniques for subverting and bypassing EDR tooling
• Scaling Incident Response with Velociraptor
• Describe the various use cases for Velociraptor
• Learn to customize Velociraptor Query Language (VQL) analyzers ("artifacts")
• Rapidly deploying Velociraptor in a client-server configuration
• Performing hunts and acquiring forensic evidence
• Use Velociraptor notebooks for effective post-processing and analysis
• Export results to Elasticsearch, Splunk, or CSV flat-files for external analysis
• Scaling Analysis with the Elastic Stack
• Utilize the Elastic Stack (aka ELK) to ingest and analyze logs
• Ingest structured and freeform data types into the Elastic Stack
• Use dashboards, histograms, graphs, and saved searches to locate attacker TTPs quickly
• Utilize the Elasticsearch SQL API and new ES|QL piped query language for fast data aggregation and filtering
• Integrating AI into DFIR
• Leveraging AI effectively for confident DFIR research and learning using context control, RAG, and customized AI solutions
• Integrating AI into DFIR tools using MCP
• Agentic AI in DFIR
• Cloud vs Local AI for DFIR investigations
• Introduction to AI attacks and Hunting for Prompt Injection and MCP Server infections

Overview

Section three transitions to more traditional host-based forensic artifact analysis. The section starts with a look at some of the latest techniques for attacking Windows systems, including the all-too-common ransomware attack. As part of looking for precursors to ransomware attacks, as well as other targeted attacks, we spend time focusing on attackers' use of "living-off-the-land" techniques to avoid detection. There are many clever ways attackers leverage built-in binaries and scripts (aka "LOLBAS" "Living-Off-the-Land Binaries and Scripts") to accomplish their goals without bringing custom malware onto the host. Learning to proactively detect and retroactively analyze these techniques is critical to investigating many modern-day intrusions.

To facilitate more rapid detection and response, we focus on the use of actionable cyber threat intelligence using Sigma rules. We use Sigma rules to hunt for suspicious activity in logs ingested into Elasticsearch. Another exercise focuses on leveraging Sigma rules packaged into the Hayabusa tool for searching Windows event logs directly. These are powerful techniques which help responders solve cases more quickly at scale.

Following the discussion on Windows threats, the remaining part of Section 3 focuses on Linux incident response and analysis. Many organizations, large and small, have Linux systems present in their environment. Although intrusions against Linux do not make the headlines as often, it's no secret that attackers regularly exploit vulnerable Linux systems to establish and maintain footholds in victim organizations.

FOR608 discusses common vulnerabilities in Linux systems and configurations, then covers common attacker exploits targeting these systems. Privilege escalation, persistence, and lateral movement are techniques we frequently associate with attacks against Windows environments, but they apply equally to Linux as well.

Our Linux discussion continues with coverage of DFIR fundamentals when analyzing Linux systems. Topics that are critical, but often cause confusion, include differences among Linux distributions, Linux file systems, the Logical Volume Manager, key log file locations, and more. Strategies are presented to handle both initial triage and deeper forensic analysis of Linux systems. Searching for unexpected logins, suspicious new files or altered files, and outliers in application logs are just a few of the techniques used to locate malicious behavior. We conclude the section with best practices for hardening systems, enhancing logging configurations, and adding monitoring capabilities to aid future investigations. Providing students with the ability to investigate Linux intrusions is a key goal of FOR608. Upon completion of the course, students will leave with important new skills and techniques for responding to large-scale intrusions across diverse enterprise networks.

Full Topic Details

• Modern Attacks Against Windows
• Fileless malware in the wild
• Common "LOLBAS" activity, including precursors to ransomware attacks
• Hunting amongst the noise for suspicious "LOLBAS" usage
• Detect and Respond to Modern Attacks
• Leverage the Sigma Project for threat detection against log data
• Build new Sigma rules for maintaining and sharing threat intelligence
• Perform Sigma rule searches against centralized log servers
• Perform Sigma rule searches directly against Windows event logs
• Introduction to Linux
• History of Linux
• Ubiquitous nature of Linux
• Challenges organizations face with managing, securing, and monitoring Linux systems
• Modern Attacks Against Linux
• Exploiting vulnerable applications or operating system services
• Misconfigurations or unpatched services lead to successful attacks
• Attacker techniques for accomplishing the attack lifecycle, including privilege escalation, persistence, lateral movement, and exfiltration
• Linux DFIR Fundamentals
• Understanding primary differences in file systems
• EXT3, EXT4, XFS file system overviews
• Understanding the Logical Volume Manager (LVM2)
• Available timestamps in Linux file systems (comparing EXT3, EXT4, XFS, Btrfs, ZFS)
• Typical Linux file system directory hierarchy
• Linux Log Analysis
• Common logs and locations
• IR strategy for log analysis
• Reviewing logon activity
• Mining application logs for suspicious events
• Linux Triage Collection and Forensic Readiness
• Collecting key configuration files
• Collecting artifact-rich logs
• Scripting collection for simplicity and consistency
• Hardening Linux configurations
• Improving audit policies
• Adding endpoint security tooling

Overview

By this point in the course, students have undertaken a wide-range of tasks, including collecting of host-based data, deployment of live-response tools to catch attackers "in the act", and utilizing "big-data" analysis platforms to find suspicious activity at scale. Students have also taken a deep dive into the Linux operating system and discovered important ways to respond to the inevitable attacks against these systems.

In the next module, we move on to look at key aspects of the Apple macOS operating system. These hosts have become more prevalent in many enterprise networks. Therefore, it's important that incident responders have some understanding and training for responding to such systems. Before diving into the incident response techniques, we discuss the history and current ecosystem of macOS and Apple mobile devices. We then move into important topics such as the Apple Filesystem (APFS), the file and directory structure, and important file types for Mac analysis such as the Property List (plist) configuration file.

After a discussion of the fundamentals, we turn our attention to the challenges and opportunities for responding to macOS incidents. Questions such as how best to acquire disk and triage data, how to review those acquisitions, and which logs and other artifacts are most useful in spotting suspicious activity are all covered in detail.

After establishing a solid foundation for Linux and Mac forensic analysis, we then turn our attention to the concept of containerized microservices. Containers are a popular way to deploy applications and services in a reliable and repeatable way. The most common platform for containers is Docker, which is where we focus our attention in FOR608.

Discussions on the architecture and management of Docker containers help students understand where to focus their analysis. A specific triage workflow is also covered to arm analysts with a repeatable process for quick and effective response against individual containers as well as the container host.

Full Topic Details

• macOS Foundations
• A history of Apple operating systems
• Apple in the enterprise
• Apple Filesystems
• APFS characteristics
• macOS timestamps
• macOS file & directory structure
• Key file types such as Property List (.plist) files
• Mac Incident Response
• Challenges with forensic acquisitions
• Options for mounting disk images
• Profiling users and system configurations
• Review common persistence methods
• Log analysis for macOS
• Containers in the Enterprise
• Conceptual overview of containers
• Containers vs. virtual machines
• Introduction to Docker
• Attacks against containers
• Forensic challenges of containerized environments
• DFIR for Containers
• Metadata collection and analysis
• Using snapshots to save containerized files
• Log analysis for Docker
• Gather ephemeral data
• Review image files and history

Overview

This section is focused on responding to incidents in the major cloud platforms from Microsoft and Amazon. Although the analysis focuses on those platforms, we cover log analysis techniques, architecture designs, and automation initiatives that can be applied to just about any cloud provider. We also cover ways attackers leverage cloud environments and the artifacts that might be left behind in such cases.

Cloud environments provide unique challenges for incident response, but some exciting opportunities too. A quick intro into these factors will start the day. Once again, we find that the MITRE ATT&CK(R) framework is useful for organizing our defenses and detections - specifically the Cloud Matrix.

Moving into Microsoft 365 (M365) and Azure, several popular SaaS offerings are discussed. These include Entra ID (formerly known as Azure AD) for the underlying identity provider service, which supports other hosted services such as Exchange, SharePoint, and Teams. Many organizations subscribe to these services, and predictably, attackers have become proficient at finding weaknesses in their implementations. We therefore review a variety of common attack scenarios against M365 and Azure. Log analysis is critical to solving these cases, so log acquisition and review is a major focus for discussion. Specifically, we look for suspicious user logon and email activity from the Unified Audit Logs (UAL) as a common method for detection. The Entra ID Audit log and others are useful resources as well.

Important for any incident response is the Recovery phase, which typically includes implementing security enhancements to detect or prevent similar attacks in the future. Therefore, we cover some of the more useful security enhancements in M365 and Entra ID.

The second part of the section delves into the Amazon Web Services (AWS) cloud platform. Its general architecture and components are covered to provide a solid foundation for those new to AWS. We then go into detail on the many logs and services that provide critical detection and analysis data for responders. This includes CloudTrail logs, VPC flow logs, GuardDuty alerts, and more.

The section concludes with discussions on architecting for response in the cloud for faster and more effective analysis. This involves setting up security accounts for a secure enclave within AWS. Template VMs (AMIs) are also recommended for performing analysis against volume snapshots, network packet captures, and log data. Finally, we look at common IR tasks that can be automated and how to do it rather seamlessly using AWS Lamda and Step Functions. While the solutions presented in this section are AWS-centric, the concepts can (and should) be applied to almost any cloud platform with significant use by an organization.

Full Topic Details

• DFIR in the Cloud
• Cloud service models (IaaS, PaaS, SaaS)
• Cloud forensics vs. traditional forensics
• MITRE ATT&CK(R) Cloud Matrix
• Incident Response in Azure & M365
• M365/O365 SaaS offerings
• Azure IaaS and PaaS platform
• Entra ID (formerly Azure AD) architecture
• Common attack scenarios
• Important log sources & log extraction
• Investigating suspicious user logons and email activity
• Securing M365 & Azure
• Attackers in the Cloud
• Investigating attacks that leverage the cloud
• Discover host-based artifacts from attacker's cloud infrastructure
• AWS Foundations
• Organizational and account hierarchy
• AWS Identity and Access Management (IAM)
• Authentication and identity types
• AWS regions and API endpoints
• AWS computing, storage, and networking constructs
• Incident Response in AWS
• AWS incident domains
• Critical log sources such as CloudTrail, CloudWatch, and S3 access logs
• Threat detection and response services such as GuardDuty and Detective
• Network analysis with VPC flow logs and traffic mirroring
• Architecting for analysis in the cloud
• Acquiring logs and snapshots
• Planning and practicing likely scenarios
• IR Automation in AWS
• Identifying tasks for automation
• Using AWS VM templates (AMIs) for quick response
• Leveraging AWS Lamda and Step Functions for automation and orchestration