Azure Diagnostic Logging at Scale - Policy, Storage, and Cost Management

"Enable diagnostic logging on all resources" sounds simple. Then you see the Log Analytics bill.

Here's how to implement comprehensive diagnostic logging affordably using Azure Policy and Storage Accounts.

The Cost Problem

Log Analytics is expensive for long-term retention. At roughly £2.30/GB ingested plus retention costs, logging everything to Log Analytics for 7 years is prohibitive.

Storage Accounts are much cheaper: ~£0.02/GB/month for Hot tier, less for Cool and Archive.

The strategy: Log Analytics for operational data (30-90 days), Storage for compliance retention (7 years).

Built-In Policy Initiatives

Microsoft provides built-in initiatives for diagnostic settings. The key ones:

These initiatives contain individual policies for each resource type, all configured to send the "allLogs" category.

Setting Up the Storage Account

Create a dedicated storage account for diagnostics:

resource "azurerm_storage_account" "diagnostics" {
  name                          = "stdiagnosticsaudit"
  resource_group_name           = azurerm_resource_group.this.name
  location                      = azurerm_resource_group.this.location
  account_tier                  = "Standard"
  account_replication_type      = "GRS"
  public_network_access_enabled = false

  network_rules {
    default_action = "Deny"
    bypass         = ["AzureServices"]  # Critical!
  }
}

The bypass = ["AzureServices"] is essential. Without it, Azure resources can't write their diagnostic logs to the storage account.

Lifecycle Policy for Cost Management

Don't pay Hot tier prices for 7-year-old logs:

{
  "rules": [
    {
      "enabled": true,
      "name": "audit-log-lifecycle",
      "type": "Lifecycle",
      "definition": {
        "filters": {
          "blobTypes": ["blockBlob"],
          "prefixMatch": ["insights-logs-"]
        },
        "actions": {
          "baseBlob": {
            "tierToCool": {
              "daysAfterModificationGreaterThan": 30
            },
            "tierToArchive": {
              "daysAfterModificationGreaterThan": 90
            },
            "delete": {
              "daysAfterModificationGreaterThan": 2555
            }
          }
        }
      }
    }
  ]
}

This moves logs to Cool after 30 days, Archive after 90 days, and deletes after 7 years (2555 days).

Assigning the Policy Initiative

resource "azurerm_management_group_policy_assignment" "diag_to_storage" {
  name                 = "diag-to-storage"
  management_group_id  = azurerm_management_group.root.id
  policy_definition_id = "/providers/Microsoft.Authorization/policySetDefinitions/5cf56836-cd90-4a91-aaf4-df4b3b3f7c04"

  parameters = jsonencode({
    storageAccountId = {
      value = azurerm_storage_account.diagnostics.id
    }
  })

  identity {
    type = "SystemAssigned"
  }

  location = "uksouth"
}

The policy needs a managed identity to create diagnostic settings on your behalf.

What's NOT Included

Storage Accounts themselves are deliberately excluded from the initiative. Microsoft did this because storage account diagnostic logging generates massive volumes of data.

Enable storage account diagnostics manually and selectively - only on accounts where you need audit trails.

Running Remediation

Policies are audit-only by default. To apply to existing resources:

1. Go to Policy → Assignments → Your assignment
2. Click "Create remediation task"
3. Select the specific policy within the initiative
4. Run for each policy (there are many - one per resource type)

Or via CLI:

az policy remediation create \
  --name "remediate-kv-diag" \
  --policy-assignment "/providers/Microsoft.Management/managementGroups/root/providers/Microsoft.Authorization/policyAssignments/diag-to-storage" \
  --definition-reference-id "KeyVaultDiagnosticSettingsToStorage" \
  --management-group "root"

Cost Estimation

For a typical enterprise (1000 resources, 50GB logs/day):

Log Analytics only (7 year retention):

• Ingestion: 50GB × £2.30 × 365 × 7 = ~£294,000
• Plus retention costs

Storage Account with tiering:

• Year 1: ~£400 (mostly Cool/Archive)
• Years 2-7: ~£200/year (Archive)
• Total 7 years: ~£1,600

That's a 99% cost reduction for compliance logging.

Hybrid Approach

For operational needs, send to both:

1. Log Analytics - 30-90 day retention for operational queries
2. Storage Account - 7 year retention for compliance

Use two separate policy assignments, or create a custom initiative that sends to both destinations.

Need help implementing diagnostic logging at scale? Get in touch - we help organisations balance compliance requirements with cost efficiency.