What You Will Learn
This is a foundational course in open-source intelligence (OSINT) gathering that will move quickly through many areas of the field. While the course is an entry point for people wanting to learn about OSINT, the concepts and tools taught are far from basic. The goal is to provide the foundational knowledge for students to be successful in their fields, whether they are cyber defenders, threat intelligence analysts, private investigators, insurance fraud investigators, intelligence analysts, law enforcement personnel, or people who are curious about OSINT.
Many people think that using their favorite Internet search engine is enough to find the data they need to do their work, without realizing that most of the Internet is not indexed by search engines. SEC487 teaches students effective methods to find the unlinked data. You will learn real-world skills and techniques to scour the massive amounts of data found on the Internet. Once you have this information, SEC487 will show you how to ensure that it is corroborated, how to analyze what you gathered, and how to make sure it is useful to your customers.
With over 25 real-world exercises using the live Internet and dark web to reinforce the course material, and with quizzes and other activities to test knowledge, the SEC487 course does not just provide you materials but also helps you learn them. The course teaches students how to use specific tools and techniques to accomplish their investigative goals, focusing on processes through flow charts that map out procedures for most of the course techniques.
This Course Will Prepare You To:
- Create an OSINT process
- Conduct OSINT investigations in support of a wide range of customers
- Understand the data collection life cycle
- Create a secure platform for data collection
- Analyze customer collection requirements
- Capture and record data
- Create sock puppet accounts
- Harvest web data
- Perform searches for people
- Access social media data
- Assess a remote location using online cameras and maps
- Examine geolocated social media
- Research businesses
- Collect data from the dark web
What You Will Receive with This Course:
- A Digital Download Package with a custom Linux virtual machine. Labs will be run from this platform.
- An electronic workbook (inside the virtual machine) containing interactive labs.
Syllabus (36 CPEs)
The first section of the course seeks to get all students speaking the same language and understanding core concepts. We will introduce the common terms and techniques to be used throughout the course. With such a diverse set of students taking the SEC487 course, establishing this common ground for all students is not only useful for discussions but is imperative to move forward.
The concepts covered focus on topics students need to examine and prepare for before they begin collecting OSINT data, including discussions about what OSINT is, setting up an OSINT collection platform, how to document and analyze OSINT data objectively, and the use of research accounts and sock puppets.
- Overview of OSINT
- What is OSINT?
- Who uses OSINT and why?
The Intelligence Process
- What is it and how does it apply to OSINT?
- Creating and Understanding the OSINT Process Stages
- Goals of OSINT Collection
- Setting Up an OSINT Platform
- Using virtual OSINT systems and mobile emulators
- Understanding issues that could decrease investigator anonymity
- Using VPNs for OSINT work
- Leveraging different web browsers and browser add-ons and extensions
- How to record data within OSINT investigations
- Examination of link analysis tools, Mind Map applications, and activity-recording programs
- Sock Puppets
- What is an OSINT sock puppet or false identity?
- When and how to use sock puppets effectively in investigations
- How to create a sock puppet
- Issues that could get your sock puppet account disabled
- Data Analysis
- How to analyze data obtained from the Internet
- Types of logic and reasoning
- Identification of and methods to reduce logical fallacies and bias
- Network theory and link analysis techniques
In most of their assessments, OSINT investigators perform certain techniques such as querying search engines, analyzing images, and examining files for metadata. These core OSINT skills are the focus for this course section, which flows from finding data to downloading it, analyzing what it means, and then moving back to the Internet to discover other places where it can be found online.
Search engines play a large role in the indexing of data on the Internet, and for that reason Section 2 starts with a detailed look at how search engines work and how to use them. Following that, students will learn techniques to retrieve files and web data rapidly and safely through command-line and web-based tools. With a firm understanding of how to gather files and data, students will learn how to analyze image content and extract metadata from those files. This naturally leads the conversation to imagery and mapping sites students can use to examine remote locations, discover video footage that can be used in their work, and geolocation techniques.
- Leveraging Search Engines
- Preparation for using search engines
- Using advanced search operators
Harvesting Web Data
- Techniques and tools to download files from Internet sources
File Metadata Analysis
- Extracting and validating metadata from files
Reverse Image Searching
- What reverse image searching is and how to use it in OSINT investigations
- How to analyze images to geolocate and extract meaningful data points
- Imagery and Maps
- Exploration of how to use maps and imagery in OSINT work
- Comparison of different imagery data sources
- Multiple methods of extracting and translating foreign text
Humans generate online data. They post, share photos and videos at certain locations, and discuss topics that may be important in your OSINT investigations. Many investigators focus their entire assessment on what people do and where they do it. For others, human activity may be a smaller portion of their work.
Regardless of how often your work focuses on OSINT data about people, Section 3 teaches students the core people investigation skills they need. The flow of Section 3 starts with data about people, such as email addresses and usernames, and turns to how to use those data points to discover user activities. Since these activities are usually discovered in social media platforms, a large portion of Section 3 is devoted to examining social media data.
- Searching for email addresses
- Analysis of usernames for meaning
- Leveraging usernames to connect users to activities on multiple sites
Avatars and Image Searching
- Analyzing avatars to discover meaning and other locations where they are used
Addresses and Phone Numbers
- How to use addresses and phone numbers to discover additional data about targets
- People Search Engines
- What are people search engines?
- Where in the OSINT process are they useful?
Introduction to Social Media
- General discussion on common techniques used in most social media exploration
- An in-depth module revealing simple and advanced search and extraction techniques within the Facebook.com social media platform
- An in-depth module revealing simple and advanced search and extraction techniques within the Twitter.com social media platform
- Exploration of common and novel methods to tie Internet data to locations on Earth
Section 4 explores more computer-focused sources of OSINT data and gives investigators the skills to research Internet domains, IP addresses, and websites. This course section reveals new techniques to investigators that mainly focus on human activities and social media. For students with strong skills in information technology and cybersecurity, Section 4 reveals new tools and techniques that will enhance their investigative approaches to domain and website investigations.
Since websites are prime targets for OSINT investigators to research, the section begins with an examination of how to research these locations, progresses to discovering data on websites, and finishes by teaching students how to analyze the servers that run the sites. Many websites are tied to domains, so the courseware shifts to techniques to discover the owners of domains and where those domains are registered. Since domains are usually tied to IP addresses, students learn how to research and understand where IP addresses are and how to use them to find online data. Continuing to follow the connected information, IP addresses are tied to computer infrastructure that may be hosting non-website data. Students will learn techniques to research all aspects of a website, from what is displayed on it down to the systems it is hosted on.
- Website Investigations
- Understanding how to use third-party data to explore website content
- Active discovery of website data
- Analysis of the infrastructure that runs a website
- What is WHOIS and how can WHOIS data be used in OSINT work?
- What is DNS and how can understanding DNS records help OSINT investigations?
- How to research and geolocate IP addresses
- Discovering and analyzing Internet-facing hosts
- Exploring how to use collected wireless data in OSINT assessments
The two main topics for Section 5 are business OSINT and the dark web. Students will learn how to take a business name and discover, through official and unofficial sources, who runs the business, where the company does its work, and what people think about that companys brand and reputation.
The course section then turns to the dark web. Students will learn how several dark webs work, why people use them, and how to access them in their OSINT investigations. Students will also learn how the Tor dark web network works, what software to use to reach Tor onion services, and how to research data inside Tor.
Section 5 continues by showing students how to harvest and interact with online data efficiently using automated websites and tools, then reveals how breach data can be used in OSINT investigations.
The end of this section features a massive exercise called the Solo Capture-the-Flag (CTF). This challenge helps students practice the tools and skills they learned in the course in a fun, challenging exercise. Through a semi-guided walkthrough that touches on many of the concepts taught throughout the course, students will work through CTF challenge questions at their own speed. Setting aside time to work through the OSINT processes discussed in class in an organized manner reinforces key concepts and allows students to practice executing OSINT processes, procedures, and techniques.
- Business OSINT
- Analyzing online business registrations and documents
- Examining the resources companies use in their work
Surface, Deep, and Dark Webs
- What are they and why does it matter in OSINT work?
- Overview of Several Dark Webs
- Comparison of a few major dark web networks, why people use them, and how to perform OSINT in those networks
- What is Tor?
- How can it be used by investigators and by their targets?
- Techniques for investigating data found in Tor
- Using applications to work more efficiently
- Breach Data
- Ethical analysis of breach data use
- Investigation into how breach data can augment OSINT work
The capstone for SEC487 is a group event that brings together everything that students have learned throughout the course. This is not a canned Capture-the-Flag event where specific flags are planted and teams must find them. It is a competition where each team will collect specific OSINT data about certain live, online targets. The output from this work will be turned in as a deliverable to the client (the instructor and fellow classmates). This multi-hour, hands-on event reinforces what the students practiced in the Solo CTF and adds the complexity of performing OSINT assessments under pressure and in a group.
GIAC Open Source Intelligence
“As the first and only non-vendor specific, industry-wide OSINT certification, the GIAC Open Source Intelligence (GOSI) certification represents a huge milestone in the worlds of open source intelligence and cyber reconnaissance. It creates a marker from which students can be recognized for their achievements and competence in the OSINT field of study. Whether they are performing social media analysis of a target or just “fancy googling,” the GOSI certification shows they have a strong foundation in OSINT.” - Micah Hoffman, SEC487 Course Author
Open Source Intelligence Methodologies and Frameworks
OSINT Data Collection, Analysis, and Reporting
Harvesting Data from the Dark Web
Basic computer knowledge is required for this course.
Important! Bring your own system configured according to these instructions!
A properly configured system is required to fully participate in this course. If you do not carefully read and follow these instructions, you will likely leave the class unsatisfied because you will not be able to participate in hands-on exercises that are essential to this course. Therefore, we strongly urge you to arrive with a system meeting all the requirements specified for the course.
A properly configured system is required for each student participating in this course. Before coming to class, carefully read and follow these instructions exactly.
Host Operating System: Latest version of Windows 10, macOS 10.15.x or later, or Linux that also can install and run the VMware virtualization products described below. Note: Apple systems using the M1 processor cannot perform the necessary virtualization at this time and cannot be used for this course.
It is necessary to fully update your host operating system prior to the class to ensure you have the right drivers and patches installed to utilize the latest USB 3.0 devices.
Those who use a Linux host must also be able to access exFAT partitions using the appropriate kernel or FUSE modules.
You also must have 8 GB of RAM or higher for the VM to function properly in the class.
It is critical that your CPU and operating system support 64-bit so that our 64-bit guest virtual machine will run on your laptop.
In addition to having 64-bit capable hardware, AMD-V, Intel VT-x, or the equivalent must be enabled in BIOS/UEFI.
Download and install either VMware Workstation Pro 15.5.x, VMware Player 15.5.x, or Fusion 11.5.x or higher versions before class. If you do not own a licensed copy of VMware Workstation or Fusion, you can download a free 30-day trial copy from VMware. VMware will send you a time-limited serial number if you register for the trial on its website.
Other virtualization software, such as VirtualBox and Hyper-V, are not appropriate because of compatibility and troubleshooting problems you might encounter during class.
VMware Workstation Pro and VMware Player on Windows 10 is not compatible with Windows 10 Credential Guard and Device Guard technologies. Please disable these capabilities for the duration of the class, if they are enabled on your system.
MANDATORY SEC487 SYSTEM REQUIREMENTS:
- CPU: 64-bit 2.0+ GHz processor or higher-based system is mandatory for this course (Important - Please Read: a 64-bit system processor is mandatory)
- BIOS/UEFI: VT-x, AMD-V, or the equivalent must be enabled in the BIOS/UEFI
- RAM: 8 GB (gigabytes) of RAM or higher is mandatory for this course (Important - Please Read: 8 GB of RAM or higher is mandatory)
- Wireless Ethernet 802.11 G/N/AC
- USB 3.0 port (courseware provided via USB)
- Disk: 30 gigabytes of free disk space
- VMware Workstation Pro 15.5.X+, VMware Player 15.5.X+ or Fusion 11.5+
- Privileged access to the host operating system with the ability to disable security tools
- A Linux virtual machine will be provided in class
Your course media will be delivered via download. The media files for class can be large, some in the 10-15 GB range. You need to allow plenty of time for the download to complete. Internet connections and speed vary greatly and are dependent on many different factors. Therefore, it is not possible to give an estimate of the length of time it will take to download your materials. Please start your course media downloads as you get the link. You will need your course media immediately on the first day of class. Waiting until the night before the class starts to begin your download has a high probability of failure.
SANS has begun providing printed materials in PDF form. Additionally, certain classes are using an electronic workbook in addition to the PDFs. The number of classes using eWorkbooks will grow quickly. In this new environment, we have found that a second monitor and/or a tablet device can be useful for keeping the class materials visible while the instructor is presenting or while you are working on lab exercises.
"I have always been intrigued by the types and amount of data that are available on the Internet. From researching the best restaurants in a foreign town to watching people via video cameras, it all fascinates me. As the Internet evolved, more high-quality, real-time resources became available and every day was like a holiday, with new and wondrous tools and sites coming online and freely accessible.
"At a certain point, I was no longer in awe of the great resources on the web and, instead, transitioned to being surprised that people would post images of themselves performing illegal acts or in compromising positions, or that a user profile would contain such explicit, detailed content. My wonder shifted to concern for these people. What I found was that, if you looked in the right places, you could find almost anything about a person, a network, or a company. Piecing together seemingly random pieces of data into meaningful stories became my passion and, ultimately, the reason for this course.
"I recognized that the barrier to creating excellent open-source intelligence reports was not that there was no free data on the Internet. It was that there was too much data on the Internet. The challenge transitioned from 'how do I find something' to 'how do I find only what I need.' This course was born from this need to help others learn the tools and techniques to effectively gather and analyze OSINT data from the Internet." - Micah Hoffman