Living off the Cloud

AWS Systems Manager and Azure Run Command are legitimate remote execution frameworks built into AWS and Azure infrastructure-as-a-service (IaaS) offerings. For attackers, they're also a reliable lateral movement and persistence path that bypasses traditional network controls entirely. Named threat groups including APT29 and Scattered Spider have used these mechanisms in confirmed intrusions. Both platforms create a detection challenge by redacting the actual command content from their primary audit logs, leaving responders to reconstruct attacker activity from secondary sources and host-based artifacts.

How These Mechanisms Work, and Why Attackers Use Them

AWS Systems Manager (SSM) runs a pre-installed agent on AWS-provided Amazon Linux and Windows Server AMIs that communicates outbound-only to the SSM service’s domain. No inbound firewall rules are required. An attacker with ssm:SendCommand permissions can issue aws ssm send-command targeting any EC2 instance reachable by the SSM service, without touching SSH or RDP. Commands execute as root on Linux and as SYSTEM on Windows. Session Manager (ssm:StartSession) provides a full interactive shell under the same privilege model.

Azure Run Command operates through the Azure VM Agent — WindowsAzureGuestAgent.exe on Windows and WALinuxAgent on Linux — invoked via portal, CLI, PowerShell, or REST API. The minimum RBAC permission required is Microsoft.Compute/virtualMachines/runCommand/action, attainable at the Virtual Machine Contributor role. Like SSM, execution is SYSTEM on Windows and root on Linux.

Both mechanisms are designed for legitimate remote administration, but threat actors are attracted by the ability to execute commands with the highest privileges on a host without needing host credentials.

Named Groups Exploiting Cloud Administration Commands

APT29, tracked by Microsoft as NOBELIUM during the SolarWinds campaign and subsequently renamed Midnight Blizzard, used Azure Run Command as part of post-compromise activity following the SolarWinds supply chain intrusion. In October 2021, MSTIC documented the group exploiting Azure Admin-on-Behalf-of (AOBO, a Microsoft partner delegation mechanism) to pivot from compromised managed service provider accounts into downstream customer environments. Once positioned in a customer tenant, APT29 used Run Command to conduct reconnaissance, steal credentials, and deploy Cobalt Strike BEACON via PowerShell.

Scattered Spider (UNC3944/Octo Tempest) used SSM aggressively across multiple phases of their operations, as documented by CISA in November 2023. Specific actions included running AWS-GatherSoftwareInventory across all EC2 instances for software reconnaissance, using SSM Send Command for lateral movement into EC2 instances, and deploying ETL tooling via SSM to centralize data before exfiltration. The group also enabled EC2 Serial Console Access (EnableSerialConsoleAccess) and used SendSerialConsoleSSHPublicKey via EC2 Instance Connect, extending persistence options beyond SSM alone. Scattered Spider conducts data exfiltration and extortion operations, with ransomware deployed in some intrusions through the ALPHV/BlackCat affiliate program.

The Logging Gap That Defines Detection Difficulty

Both platforms share the same fundamental gap: the actual command content is not logged in the primary audit trail.

In AWS CloudTrail, SendCommand events (source: ssm.amazonaws.com) capture the IAM principal, timestamp, target instanceIds, and documentName, but the parameters field, which contains the actual command text, is always redacted with HIDDEN_DUE_TO_SECURITY_REASONS. This is intentional AWS behavior, not a configuration failure. An analyst reviewing CloudTrail for AWS-RunShellScript invocations will see who ran a command and against which instances, but not what the command contained.

In Azure, the Activity Log records Microsoft.Compute/virtualMachines/runCommand/action for Action Run Commands, including useful fields: caller, CallerIpAddress, resourceId (identifying the specific VM), operationName, status, and authorization.scope. The actual script content is absent. Managed Run Commands introduce a second gap: creation events log as Microsoft.Compute/virtualMachines/runCommand/write which does not always also generate the /action event, meaning detections built solely on the action event will miss Managed Run Command activity. Azure Activity Log default retention is 90 days, and without export to Log Analytics or a Storage Account, older events are unrecoverable.

Understanding these gaps is prerequisite to building effective detection coverage. The log events establish context — who, when, and which target — but command content recovery requires separate data sources.

AWS CloudTrail Events of Interest

When investigating SSM activity or writing detections, look for the following on the following CloudTrail events from source ssm.amazonaws.com:

1. SendCommand — The primary execution event. Flag calls using AWS-RunShellScript, AWS-RunPowerShellScript, AWS-RunRemoteScript, or AWS-GatherSoftwareInventory.
2. DescribeInstanceInformation — Instance enumeration. Frequently observed in the reconnaissance phase before targeting specific instances.
3. StartSession — Session Manager interactive shell initiation.
4. ListCommandInvocations / GetCommandInvocation — Result retrieval. Useful for correlating with SendCommand events to establish full execution sequences.
5. CreateActivation — Hybrid activation creation. Legitimate use is narrow; this event appearing in an investigation context warrants close scrutiny, as it enables the Mitiga-documented SSM Agent-as-RAT technique.
6. EnableSerialConsoleAccess / SendSerialConsoleSSHPublicKey — EC2 serial console access, which is a Scattered Spider TTP. These are high-signal events with limited legitimate administrative use.

Azure Activity Log Events of Interest

From Azure Activity Log and Log Analytics (if configured):

1. Microsoft.Compute/virtualMachines/runCommand/action — Primary detection point for Action Run Commands.
2. Microsoft.Compute/virtualMachines/runCommand/write — Managed Run Command creation.
3. Microsoft.Compute/virtualMachines/extensions/write — Run Command Extension installation or update. Unexpected extension writes to production VMs are high-fidelity signals.

Host-Based Artifacts for Command Content Recovery

When command content is needed for attribution, scope determination, or chain-of-custody, host-based artifacts are the primary recovery path.

AWS SSM — Linux:

• Agent log: /var/log/amazon/ssm/amazon-ssm-agent.log
• Command stdout/stderr: /var/lib/amazon/ssm/<instance-id>/document/orchestration/<command-id>/<plugin>/<step>/stdout (and stderr)
• Session Manager temporary data: /var/lib/amazon/ssm/<target-id>/session/orchestration/<session-id>/Standard_Stream/ipcTempFile.log

AWS SSM — Windows:

• Agent log: %PROGRAMDATA%\Amazon\SSM\Logs\amazon-ssm-agent.log
• Command output: %PROGRAMDATA%\Amazon\SSM\InstanceData\<instance-id>\document\orchestration\<command-id>\<plugin>\stdout

Azure Run Command — Windows:

• Executed scripts: C:\Packages\Plugins\Microsoft.CPlat.Core.RunCommandWindows\<version>\Downloads\<#>\ (PS1 files)
• Status logs: C:\Packages\Plugins\Microsoft.CPlat.Core.RunCommandWindows\<version>\Status\<#>.status
• Expected process chain: exe → RunCommandExtension.exe → powershell.exe → [child process]
• Note: the plugin folder may be deleted by the Guest Agent post-execution; acquire these artifacts early in the investigation.

Azure Run Command — Linux:

• Script: /var/lib/waagent/run-command/download/<job-number>/script.sh
• stdout/stderr: same directory
• Handler log: /var/log/azure/run-command/handler.log
• Expected parent chain: waagent → /bin/sh -c /var/lib/waagent/run-command/download/<#>/script.sh

Endpoint Telemetry as a Complement

Cloud audit logs and host artifacts should be supplemented with endpoint telemetry, particularly where EDR or Sysmon is deployed on cloud-hosted VMs.

On Windows, Event ID 4688 (process creation) and Event ID 4104 (PowerShell Script Block Logging) capture child process execution and PowerShell content regardless of obfuscation. Sysmon Event ID 1 flags powershell.exe or cmd.exe spawned under WindowsAzureGuestAgent.exe or amazon-ssm-agent.exe — both parent processes with limited legitimate child process scope. Sysmon Event ID 11 captures PS1 file creation in plugin download directories. EDR process lineage from RunCommandExtension.exe or amazon-ssm-agent.exe provides the most direct signal, and multiple simultaneous SSM agent processes on a single host indicate the Mitiga-documented parallel agent technique.

On Linux, auditd or eBPF-based telemetry with execve syscall logging will surface commands executed with waagent or run-command-extension as the parent process.

Recommendations for Defenders and Responders

1. Enable SSM command output logging at the organizational level. Configure S3 output buckets and CloudWatch Logs (aws/ssm/<DocumentName>) as defaults for all SendCommand executions via Service Control Policies or organizational defaults. Without this, command content recovery depends entirely on host artifacts that may not survive.
2. Export Azure Activity Log to Log Analytics with retention exceeding 90 days. Build detection rules separately for /action and /WRITE Run Command event types. Alert on Run Command extensions written to production VMs outside approved change windows.
3. Build CloudTrail detections on CreateActivation and serial console events. These have narrow legitimate use cases; near-zero false-positive detections are achievable. Hybrid activation events (CreateActivation) warrant immediate investigation given their role in SSM Agent-as-RAT techniques.
4. Scope IAM and RBAC permissions using least privilege. ssm:SendCommand and Microsoft.Compute/virtualMachines/runCommand/action should not be broadly assigned. Audit current assignments and remove Virtual Machine Contributor from identities without operational need for remote execution.
5. During investigations, prioritize host artifact collection before agent cleanup. Azure's Guest Agent may delete Run Command plugin folders post-execution. Preserve logs from the paths documented above before initiating any remediation.
6. Deploy PowerShell Script Block Logging (Event ID 4104) on all Windows-based cloud VMs. This is the only reliable mechanism for capturing obfuscated PowerShell content that bypasses command-line argument logging, and it catches the APT29 and Scattered Spider PowerShell TTPs documented in confirmed intrusions.

The detection surface here is recoverable, but it requires deliberate configuration before an incident. The logs that don't exist at collection time cannot be retrieved during response.

If you’re looking to learn more about investigating cloud security threats, SANS FOR509: Enterprise Cloud Forensics and Incident Response is designed to prepare responders in cloud environments. Students learn what log sources are available, what threat actors do, and how those activities appear in the logs so that you can identify suspicious activity, reconstruct attacker actions despite logging gaps, and build effective detections and response strategies in real-world cloud environments.