Cloud Logging for Security and Beyond

This article aims to simplify cloud logging best practices for each major cloud service provider (CSP) while considering security, regulatory and business requirements.

As more organizations migrate their business operations to the cloud, a crucial question arises: “What logging should we enable in order to monitor and secure our cloud environment?” To answer that question holistically, organizations must consider many factors, including:

• Business needs
• Regulatory requirements
• Security use cases
• Cost optimization

Amazon Web Services (AWS), Azure and Google Cloud Platform (GCP) provide various unique logging configurations for visibility into cloud resources. These diverse logs, when aggregated, constitute the organization’s cloud logging framework.

Defining cloud logging requirements can be overwhelming. Organizations often use multiple CSPs, which have varying terminology, logging types and retention periods. This article explains the key components of a successful, customizable cloud logging framework.

Disclaimer: The information in this article regarding legal and regulatory requirements is being provided for general awareness only and does not constitute legal advice. Always consult a qualified lawyer on any specific legal problem or matter, including but not limited to logging and data requirements applicable to your organization.

To understand the types of cloud activities to monitor, cloud practitioners must first understand the difference between the data plane and the control plane. While the data plane supplies the base functionality of each cloud service provider, the control plane is a layer above that and manages the cloud resource operations themselves.

The data plane provides the underlying function of the CSP service, such as:

• Connecting into a virtual machine
• Placing items into object-level storage
• Creating a database table

The control plane, on the other hand, enables the administration and management of resources within each CSP. For example:

• Logging into a CSP console
• Making API calls to modify resources
• Creating new resources

While both data and control planes exist within every CSP, their native logging capabilities vary. Control plane logging exists for each CSP by default, but with different configurations available. This is in contrast to data plane logging, for which logging does not exist by default.

Understanding these distinctions clarifies the various logging components within each CSP and their relevance to business needs.

Organizations should identify applicable regulatory frameworks to prevent legal penalties, maintain customer trust and ensure responsible data handling. These frameworks often impose specific data retention and logging requirements, influencing the type of logs an organization must collect and store.

To navigate these complexities, collaborating with the organization’s compliance and legal teams can help clarify applicable regulatory requirements, which largely depend on the organization’s industry, location and types of data processed. A proactive approach to regulatory alignment helps ensure adherence to legal mandates while supporting secure and transparent business operations.

The following are some of the common regulatory frameworks that may apply to organizations:

This regulation applies to organizations processing personal data of European Union residents, regardless of the organization’s location. It mandates data privacy and protection, including but not limited to, what data can be logged and individuals’ rights to their data. For instance, GDPR’s Article 17 (e.g., “the right to be forgotten”) requires organizations to delete personal data upon request in certain circumstances, requiring an organization to accurately locate and remove data.

A U.S. regulation for protecting sensitive protected health information (PHI). HIPAA dictates strict requirements for logging access to and modification of PHI, along with data retention and security controls. For example, HIPAA’s audit control standards may require healthcare organizations to log every instance of access to a patient’s medical record, providing a comprehensive audit trail in the event of unauthorized access or a data breach.

This regulation applies to any organization handling payment card information, mandating specific logging and retention requirements for system components involved in card processing. For instance, an online retailer may be required to log every instance of database access where it has stored credit card information, including timestamps, user IDs and the specific action performed.

While not strictly regulatory frameworks, these provide valuable information security guidance and best practices for organizations across various sectors. They often influence the development of industry-specific regulations, as many industries (e.g., finance, energy, telecommunications) have their own specific logging and data protection regulations.

For example, a financial institution adopting these recommendations might log all instances of someone attempting to access account information outside of normal business hours. This could increase its ability to detect and respond to potential malicious activity by allowing the organization to maintain a more robust security posture even in the absence of a specific legal mandate.

Logging establishes a baseline of information necessary to equip security teams with the data to support security monitoring, threat hunting and incident response activities. These logs should provide:

• Detailed audit trails of user activity
• System changes
• Security-relevant events
• The complete context of an incident

For effective incident response, logging should cover a range of activity across the network perimeter, internal system interactions and user behaviors. This enables responders to determine the extent of unauthorized actions. Insufficient logging can obscure threat actor activity, potentially jeopardizing regulatory compliance and critical security decisions.

To meet regulatory and security requirements, organizations often default to logging as much data as possible “just in case.” This method increases costs and hinders log analysis by creating more extraneous data to sort through to understand suspicious activities.

For example, the AWS Simple Storage Service (S3), which helps organizations store objects, provides the ability to log data events either in AWS CloudTrail or via S3 server access logging. While enabling one of these features provides ample visibility into S3 data events, costs can escalate dramatically if not managed properly. Managing the costs of storing the data can include transitioning data to different S3 bucket storage types or shortening retention periods. Excessive data volume can also hinder efficient query and analysis.

Furthermore, excessive logging potentially violates privacy regulations, as certain regulatory frameworks (such as GDPR Article 5(1)(c)) emphasize collecting and retaining only necessary logs. Instead of indiscriminate logging, organizations should consider adopting a targeted logging strategy. This involves clearly defining business, regulatory and security requirements to optimize ingestion and retention costs. By doing this, organizations can optimize the ingestion and retention costs associated with logging.

For easier comparison across AWS, Azure and GCP, we’ve grouped common services into high-level categories. These categories include:

• Audit
• Compute
• Network
• Secrets
• Cloud-native storage
• Database
• Kubernetes

Each category typically requires both data and control plane level logging to gain comprehensive visibility into all activity. Consider these categories in light of your organization’s specific business, regulatory and security needs.

Now that we have an understanding of the background and purpose of the various log categories for enterprise environments, we can address the specific logs available for each category by CSP.

The following table (Table 1) breaks down the common AWS, Azure and GCP services that exist in each category, listing the different logging options. As previously mentioned, the creation, modification and deletion of each service resource resides within the different audit logs for control plane visibility, but the data plane logging does not exist by default.

Each service listed in the table has management level events recorded in the audit services, while the information in the parentheses represents the data plane logging available for each type.

Table 1. Cloud Service Provider logging options by category.

Understanding the nuances within each CSP’s logs forms the cornerstone to comprehending cloud logging holistically. We illustrate the complexities of cloud logging below with examples from AWS, Azure and GCP, showing where specific events appear across the different CSPs.

Each scenario shows a user:

• Logging in to the CSP’s interactive console
• Creating a new user
• Interacting with and creating resources specific to that CSP

The associated key in Figure 2 explains which logs exist by default and which logs require enablement.

Figure 3 shows the creation of a new identity and access management (IAM) user, Lambda function and S3 bucket that all result in artifacts present in the CloudTrail logs. In this case, the Lambda function execution does not exist by default and requires additional data plane logging for complete visibility.

Figure 4 shows similar administrative activities to those detailed in Figure 3, but this time for Azure. Due to the variety of Azure log types, the audit activities exist across multiple log types, but activities such as Key Vault key retrieval require additional data plane logging similar to the AWS Lambda function execution.

Figure 5 details a GCP example of creating a new user and failing to create a new virtual private cloud (VPC) network with the resulting logs present in the GCP audit logs. To view the retrieval of a secret from Secret Manager, data plane logging must be enabled akin to the secret retrieval action in Azure while AWS has Secret Manager retrieval enabled by default. Similar to AWS, fewer audit log types aggregate all the control plane activity.

Successfully navigating cloud logging requires a holistic approach that balances security, regulatory and business needs and cost optimization. The diverse logging services, tools and best practices of each CSP add to this challenge. Following the best practices laid out within this article will help provide organizations with the knowledge to better understand what logging and monitoring enables the best visibility into their unique cloud environments, while also providing them with better detection and ultimately prevention capabilities.

By implementing the best practices described in this article, organizations should be better positioned to effectively monitor, secure and manage their cloud environments.