Home > Sample chapters

Design and Implement an Azure Storage Strategy

Objective 4.5: Monitor storage

Azure Storage has a built-in analytics feature called Azure Storage Analytics used for collecting metrics and logging storage request activity. You enable Storage Analytics Metrics to collect aggregate transaction and capacity data, and you enable Storage Analytics Logging to capture successful and failed request attempts to your storage account. This section covers how to enable monitoring and logging, control logging levels, set retention policies, and analyze the logs.

To enable logging for read, write, and delete actions with retention of 30 days, run the following:

Set-AzureStorageServiceLoggingProperty –ServiceType Blob –LoggingOperations
read,write,delete –RetentionDays 30
Set-AzureStorageServiceLoggingProperty –ServiceType Table –LoggingOperations
read,write,delete –RetentionDays 30
Set-AzureStorageServiceLoggingProperty –ServiceType Queue –LoggingOperations
read,write,delete –RetentionDays 30

To disable the collection of metrics, run the following:

Set-AzureStorageServiceLoggingProperty –ServiceType Blob –LoggingOperations none
Set-AzureStorageServiceLoggingProperty –ServiceType Table –LoggingOperations none
Set-AzureStorageServiceLoggingProperty –ServiceType Queue –LoggingOperations none

Enabling client-side logging

You can enable client-side logging using Microsoft Azure storage libraries to log activity from client applications to your storage accounts. For information on the .NET Storage Client Library, see http://msdn.microsoft.com/en-us/library/azure/dn782839.aspx. For information on the Storage SDK for Java, see http://msdn.microsoft.com/en-us/library/azure/dn782844.aspx.

Analyzing storage logs

Logs are stored as block blobs in delimited text format. When you access the container, you can download logs for review and analysis using any tool compatible with that format. Within the logs, you’ll find entries for authenticated and anonymous requests, as listed in Table 4-3.

TABLE 4-3 Authenticated and anonymous logs

Request Type

Logged Requests

Authenticated requests

  • Successful requests
  • Failed requests such as timeouts, authorization, throttling issues, and other errors
  • Requests that use an SAS
  • Requests for analytics data

Anonymous requests

  • Successful requests
  • Server errors
  • Timeouts for client or server
  • Failed GET requests with error code 304 (Not Modified)

Logs include status messages and operation logs. Status message columns include those shown in Table 4-4. Some status messages are also reported with storage metrics data. There are many operation logs for the Blob, Table, and Queue services.

TABLE 4-4 Information included in logged status messages

Column

Description

Status Message

Indicates a value for the type of status message, indicating type of success or failure

Description

Describes the status, including any HTTP verbs or status codes

Billable

Indicates whether the request was billable

Availability

Indicates whether the request is included in the availability calculation for storage metrics

Finding your logs

When storage logging is configured, log data is saved to blobs in the $logs container created for your storage account. You can’t see this container by listing containers, but you can navigate directly to the container to access, view, or download the logs.

To view analytics logs produced for a storage account, do the following:

  1. Using a storage browsing tool, navigate to the $logs container within the storage account you have enabled Storage Analytics Logging for using this convention: https://<accountname>.blob.core.windows.net/$logs.
  2. View the list of log files with the convention <servicetype>/YYYY/MM/DD/HHMM/<counter>.log.
  3. Select the log file you want to review, and download it using the storage browsing tool.

Accessing logs with Windows PowerShell

Using Windows PowerShell, you can access logs with the Get-AzureStorageBlob cmdlet and then filter logs by filename and metadata. The following example illustrates how to filter a list of write logs for Blob storage entries on a particular date during the ninth hour:

Get-AzureStorageBlob -Container '$logs' |
where {
  $_.Name -match 'blob/2014/12/01/09' -and
  $_.ICloudBlob.Metadata.LogType -match 'write'
} |
foreach {
  "{0} {1} {2} {3}" –f $_.Name,
  $_.ICloudBlob.Metadata.StartTime,
  $_.ICloudBlob.Metadata.EndTime,
  $_.ICloudBlob.Metadata.LogType
}

Downloading logs

To review and analyze your logs, first download them to your local machine. You can do this with storage browsing tools, programmatically, or with AzCopy.

Viewing logs with Microsoft Excel

Storage logs are recorded in a delimited format so that you can use any compatible tool to view logs. To view logs data in Excel, follow these steps:

  1. Open Excel, and on the Data menu, click From Text.
  2. Find the log file and click Import.
  3. During import, select Delimited format. Select Semicolon as the only delimiter, and Double-Quote (“) as the text qualifier.

Analyzing logs

After you load your logs into a viewer like Excel, you can analyze and gather information such as the following:

  • Number of requests from a specific IP range
  • Which tables or containers are being accessed and the frequency of those requests
  • Which user issued a request, in particular, any requests of concern
  • Slow requests
  • How many times a particular blob is being accessed with an SAS URL
  • Details to assist in investigating network errors

Objective summary

  • Storage Analytics metrics provide the equivalent of Windows Performance Monitor counters for storage services.
  • You can determine which services to collect metrics for (Blob, Table, or Queue), whether to collect metrics for the service or API level, and whether to collect metrics by the minute or hour.
  • Capacity metrics are only applicable to the Blob service.
  • Storage Analytics Logging provides details about the success or failure of requests to storage services.
  • Storage logs are stored in blob services for the account, in the $logs container for the service.
  • You can specify up to 365 days for retention of storage metrics or logs, or you can set retention to 0 to retain metrics indefinitely. Metrics and logs are removed automatically from storage when the retention period expires.
  • Storage metrics can be viewed in the management portal. Storage logs can be downloaded and viewed in a reporting tool such as Excel.

Objective review

Answer the following questions to test your knowledge of the information in this objective. You can find the answers to these questions and explanations of why each answer choice is correct or incorrect in the “Answers” section at the end of this chapter.

  1. Which statements are true of Storage Analytics Metrics? (Choose all that apply.)

    1. Capacity metrics are recorded only for blobs.
    2. You can set hourly or by minute metrics through the management portal.
    3. By default, metrics are retained for one year.
    4. If you disable metrics, existing metrics are deleted from storage.
  2. Which statements are true of Storage Analytics Logging? (Choose all that apply.)

    1. Logs are stored in the same storage account where they are enabled and are measured as part of your storage quota.
    2. Logs can have duplicate entries.
    3. Logs cannot be deleted.
    4. You can log all read, write, and delete requests to blobs, queues, and tables in a storage account.
  3. Which of the following are captured by Storage Analytics Logging? (Choose all that apply.)

    1. Successful requests for authenticated calls only
    2. Failed requests for authenticated calls only
    3. Server errors
    4. Requests using SAS URIs