When approaching incident response in an environment you may not be very familiar with, the biggest challenge is often knowing what to look for. This is especially true when you’re facing dozens of data sources with hundreds of event types. As an incident responder, it’s your job to determine what is legitimate and what is not among a sea of activity, which is easier said than done. This blog post and its accompanying cheat sheet provide guidance on some key events of interest that can be a starting point for reviewing Google Workspace activity in the scope of an incident.
Many of the events we highlight in this blog post and the cheat sheet occur as part of normal business operations, so it's up to your analysis as an incident responder to find the behavior that stands out or can be tied to existing evidence.
Gmail and Google Chat
Gmail is one of the key services leveraged in Google Workspace organizations, providing all email services needed to carry out business. Google Chat may not be as widely used as Gmail but we cover it here because, like Gmail, it provides a way to communicate with end users and share files. Through both these services, threat actors obtain a method of social engineering for either initial access or lateral movement.
As mentioned, both services provide the ability to share files, which can lead to further phishing attempts or malware distribution in the scope of threat actor behavior. Luckily, activities involving attachments are logged. In the Gmail audit logs we can find both Attachment download and Attachment link click events. For Google Chat, we not only see Attachment downloaded events, but also Attachment uploaded. In Gmail, we will see Link click events which can tell a similar story to attachments being shared.
Some basic event types that will be broadly observed but provide key insights into communication patterns are the Send and Open events in Gmail, as well as the Direct message started, Room member added, and Invite accept events in Google Chat. All of these events indicate a communication channel being established and can provide a root cause in phishing attempts.
Lastly, because Google performs spam analysis on emails, as well as providing users the ability to report spam emails, events such as Late spam classification and User spam classification may help unearth suspicious emails.
User and Admin Actions
Once an account is compromised, we must track down how that compromised account was abused. One of the first places to look is the User Accounts audit log, which provides records of both sign-in attempts and settings changes.
Logins show us initial access and any follow on access activity. Because of the significance of authentication activity to understanding an attack, most events in this log provide value. Failed login, Successful login, Login challenge, Login verification, and Logout are all common authentication events, but in the scope of a compromised account tell the story of how and when a threat actor accessed the account. One notable event that will also appear with relative frequency but has more serious implications if the account is compromised is the Sensitive action allowed event.
More interesting are suspicious login events, which can fall into one of the following categories: Suspicious login, Suspicious login (less secure app), Suspicious programmatic login, and User signed out due to suspicious session cookie. The latter two are especially rare but any of these warrants investigation. The definition of “suspicious” is based on Google’s analysis so it may result in false positives, but it’s worth reviewing to determine the true nature of the activity.
Users have control over various account settings and changing those settings results in an event in the User Accounts audit log. The below settings are worth auditing in the scope of an investigation as they could be an attempt by a threat actor to gain persistence. Changes to these settings can either result in hindering a user’s access or ensuring the threat actor can regain access if they are removed:
- Out of domain email forwarding
- 2-step verification disable
- 2-step verification enroll
- Account password change
- Account recovery email change
- Account recovery phone change
The final category of user account events to look for that should not be common and could indicate an account compromise is user suspended events. There are three types of user suspended events, all indicating that one of the organization’s users is doing something that they should not: User suspended (spam through relay), User suspended (spam), and User suspended (suspicious activity).
Google Drive and Google Takeout
Another widely used Google Workspace service is Google Drive, Google’s file editing, storage, and sharing product. The primary risk here is that confidential business data stored within Google Drive can be exposed, either accidentally or intentionally. As with the other data sources, there’s a lot of events that are both widely observed with normal user activity but also highly relevant to investigators when viewed through the lens of an incident. These events include:
- Download
- Edit
- Delete
- Trash
On top of those, any permissions changes could be significant. It could indicate a user unintentionally over-permissioning a file or folder, or a threat actor attempting to gain access to files from outside the organization. There are many events that indicate such activity with the primary ones listed below:
- Owner changed
- Owner changed from parent folder
- Change document visibility
- Shared Drive Settings Change
- User Sharing Permissions Change
- Change access scope
- Change ACL editors
- Change document visibility
- Change shared drive membership
- Change user access from Parent Folder
The last event worth mentioning is the Script trigger created event. Google has a scripting engine built into Google Docs, which could hypothetically be abused by threat actors to run unauthorized code. In most environments, this event is likely to be rare so it should stand out if it is observed.
Google Takeout is a unique service that allows Google users, both personal and business accounts, to export copies of all data tied to their account. For Google Workspace, this service is on by default and provides threat actors or malicious insiders with an easy way to export mass amounts of potentially sensitive data. The audit log for Google Takeout has four event types, all worth monitoring for malicious usage:
- User completed a takeout
- User downloaded a takeout
- User initiated a takeout
- User scheduled takeout(s)
Ideally in enterprise environments this service should be disabled and only enabled on an as needed basis, but if it is enabled it’s usage needs to be closely monitored.
OAuth Usage
The final audit log we’ll look at in the scope of this blog post is the OAuth audit log. The amount of data in this log will vary depending on the organization. Often there is Google Chrome related activity in this log by users logging into the Chrome browser with their organization credentials, which is completely normal. Other activity will depend on both the organization’s third-party application policies and whether users leverage such applications, which are the most common source of OAuth traffic. When discussing the OAuth audit log, we can briefly cover each event type as there are only five possible event types:
- Request - a request was made for OAuth access
- Grant - OAuth access was granted
- Deny - a request for was denied
- Revoke - OAuth access was revoked
- API call - An API call was made with OAuth credentials
The most significant of these is API call as it shows exactly what action was taken with the credentials, which is the scope of OAuth abuse tells you what a threat actor did with their access. This event type, however, is only available to Enterprise and other premium licenses, so it may or may not be available depending on the organization’s subscription. The other events at minimum help audit granting of credentials and can tell you if access was obtained via OAuth, even if you can’t tell what API calls were made after access was obtained.
Summary
Although this blog post is not a comprehensive guide to incident response investigations in the cloud, it provides insights into some key events that could lead you in the direction of answers to investigative questions. As mentioned before, just because an event isn’t listed here doesn’t mean it cannot be noteworthy, but if you do not know where to start, this is a good place. To summarize the events highlighted in this blog post, we are releasing the “Google Workspace Artifact Reference Guide” cheat sheet, which can be downloaded here.
