SEC497: Practical Open-Source Intelligence (OSINT)

Overview

Before diving into tools and techniques to find, gather, and process information, the course starts with a discussion of how to undertake these activities as safely and effectively as possible. This section begins with an overview of the OSINT process and tips on avoiding analytical pitfalls. We then move into Operational Security or OPSEC. This isn't just a list of browser plugins and VPN providers: we'll look at what truly matters when managing attribution. Most of us don't have unlimited budgets and can't achieve "perfect OPSEC" - the good news is that most of us don't have to! We'll discuss the major risks and ways to mitigate them within a reasonable budget.

A big part of OSINT is going to new sites and downloading files and information. We'll discuss free online resources that can be used to determine if a site is a known security risk before you visit it and to analyze files to determine if they have malicious content. We'll discuss how canary tokens work and create our own with a hands-on lab. Canary tokens are files that can give away our actual IP address and other information about our systems without our knowledge.

Creating fictitious accounts (aka sock puppets) has gotten tougher over the past few years, with many sites requiring criteria like a real phone number, facial image, etc. We'll discuss the issues and cover current methods for creating these accounts.

The course section wraps up by examining two tools that can improve your organization and efficiency. Hunchly is a fantastic tool for cataloging online research, and Obsidian is an effective open-source tool for note-taking and various other uses. We'll also cover report writing.

Many OSINT students have improving Linux skills on their to-do list, so at the end of the section there is an optional lab for Linux command line practice. This gives students who would like to work on these skills the opportunity to do so in a controlled environment.

Exercises

• Managing Your Attribution
• Dealing with Potential Malware
• Canary Tokens
• Hunchly
• Obsidian
• [Optional] Linux Command Line Practice

Topics

• The OSINT Process
• Avoiding Analytical Pitfalls
• OPSEC
• Dealing with Potential Malware
• Canary Tokens
• Creating Accounts
• Hunchly
• Effective Note Taking
• Report Writing
• Introduction to Linux

Overview

Section Two presents a range of fundamental skills that all OSINT practitioners should have, regardless of the industry they work in. We'll start with a brief overview of curated lists of OSINT resources and quickly move into understanding the fundamentals of how the web works and utilizing search engines effectively. We'll cover methods to find other sites owned and operated by the same individuals, how to see content that the site owners may not want you to see, and, as always, the OPSEC implications and how to do undertake these tasks safely. We'll also cover the why and how of setting up persistent monitoring alerts.

Multiple methods will be presented to archive content from websites, view historical content from websites, and get other sites to visit websites on your behalf. We'll talk about collecting and preserving Internet data and how to convert raw data into useable formats for processing and analysis. Well discuss how to gather useful intelligence from metadata, even if the data initially appear insignificant or do not appear at all, and look at useful sites for mapping, imagery, and analysis.

The course section will then turn to image analysis, with a discussion of methodology, tools that can help us, and some real-world examples. From there, we'll move into facial recognition and real-world examples and resources we can use to find people online. We'll conclude with a discussion about translation resources.

At the end of the section there will be an optional capstone. Participants will start off with raw chat logs from a Russian ransomware group and go through the process of converting the logs into a usable format for analysis.

Exercises

• Search
• Instant Data Scraper
• Metadata
• Reverse Image Search
• Facial Recognition
• Translation
• [Optional] Day 2 Capstone

Topics

• OSINT Link and Bookmark Collections
• Web Fundamentals and Search Engines
• Web Archives and Proxy Sites
• Collecting and Processing Web Data
• Metadata
• Mapping
• Image Analysis and Reverse Image Searches
• Facial Recognition
• Translations

Overview

Section Three of the course focuses on investigating individuals or groups. We'll start by discussing privacy and then get into techniques to research usernames and email addresses across popular sites to discover an individuals accounts. The section then covers how to determine if email addresses are potentially tied to fraud and the places where the individual(s) connected to the email addresses may have been.

Many OSINT investigations start with a selector such as a phone number or address and require that the researcher tie that selector to an individual or group. We'll cover numerous resources and techniques you can use to do this, including some that can help identify the owner of a prepaid phone number.

The remainder of the section will focus on social media sites, including advanced Facebook searches and ways to see deleted Twitter data and analyze Twitter bots. We'll also cover methods to view content on social media sites when you don't have an account on that site; searching and analyzing alternative social media sites; geolocation of social media data; and trends, sentiment, and reputation.

Exercises

• Researching Usernames
• Keybase
• Email
• Twitter
• Twitter Bot Analysis

Topics

• Privacy
• Usernames
• Email Addresses
• Addresses and Phone Numbers
• Introduction to Social Media
• Facebook
• Twitter
• Other Social Media Sites
• Geolocation
• Trends, Sentiment, and Bots

Overview

Section Four covers investigating websites, IP addresses, and other infrastructure, including the cloud. For students who don't consider themselves tech savvy, we'll take the time to explain what the elements are and how they work, and well provide numerous real-world examples of how these elements have helped in investigations. This course section is critical even for analysts who don't focus on technical topics because understanding how these technical elements work reduces the likelihood of falling down rabbit holes during their research.

For students who focus more on technology topics, such as those who work in Cyber Threat Intelligence, we'll cover a variety of tools and resources to learn as much as we can about such topics as:

• Where in the world an IP address is located, and whether it is a VPN/proxy/Tor node
• Why IPv6 is different and why that might be good for your investigation
• Historical WHOIS data
• Understanding DNS records
• Why certificate transparency is excellent for defenders&and attackers
• What we can learn from email headers
• Subdomains for enumeration and, potentially, de-anonymization
• Technology-focused search engines like Shodan and Censys
• Cyber Threat Intelligence
• Finding sensitive data in the cloud

This course section is a mix of understanding how things work, studying real-world examples and case studies, looking at some cutting-edge research, and using tools in creative ways to achieve things most people did not know were possible.

Exercises

• IP Address Research
• WHOIS
• DNS
• Amass and Eyewitness
• Censys and Shodan
• Buckets of Fun

Topics

• IP Addresses
• Common Ports
• WHOIS
• DNS
• Certificate Transparency
• Email Headers
• Subdomains
• Technology-Focused Search Engines
• Cyber Threat Intelligence
• Cloud

Overview

Section Five is a fun mix of topics ranging from researching businesses and transitions to covering wireless for OSINT, including using Wi-Fi names to enrich digital forensics data and research locations. We'll also explore different types of breach data and how it can be used for various OSINT and cyber defender purposes.

If you work in OSINT long enough, a giant pile of data will eventually be placed in front of you, and someone will ask you whats in it. Depending on your job, this may already be a regular occurrence. This section will cover how to triage and search large datasets effectively and quickly using free or cheap resources.

We'll also take a deep dive into the dark web, covering how it works, how we can find things, and what we can expect to find. We'll examine a case study of breach data hitting the dark web and tricks we can use to speed up dark web downloads. We'll also have a short section on cryptocurrency that mainly focuses on a resource that allows us to track cryptocurrency transactions with a focus on web 3.0 and NFTs.

As the course section winds down, we'll talk about different automation options that require no programming. The final portion of the section is called "path forward" and covers a variety of resources that can help you continue your OSINT learning journey.

Exercises

• Business
• Wireless
• Bulk Data Triage
• Tor and PGP
• Breach Data

Topics

• Researching Businesses
• Wireless
• Breach Data
• Dealing with Large Datasets
• Dark Web
• Cryptocurrency
• Automation
• Path Forward

Overview

The capstone for the SEC497 course is a multi-hour capture the flag event which allows students to work together in small groups to create a threat assessment for a fictional client. Preparing this assessment will require that students use the skills learned throughout the course on a variety of real-world sites. The assessment will be delivered to the client (the instructor), who will provide feedback to each group.