Migrate Terraform State with AzCopy in Azure Containers.

Migrating Terraform State between Azure Containers with AzCopy

Introduction

Have you ever needed to migrate your Terraform state files from one Azure storage container to another? Perhaps you’re consolidating storage accounts, upgrading to a more secure configuration, or simply reorganizing your infrastructure. Moving Terraform state safely is critical to avoid disrupting your infrastructure management.

In this comprehensive guide, I’ll walk you through the exact process of reliably transferring Terraform state files between Azure Blob Storage containers using Microsoft’s AzCopy tool and PowerShell. Follow these steps to ensure your state migration goes smoothly without risking your infrastructure integrity.

Understanding Terraform State Files

Terraform state is a crucial component of infrastructure as code that allows users to define, manage and version infrastructure in a declarative way. The state file (typically named terraform.tfstate) is a JSON-formatted file that records the current state of all resources managed by Terraform.

Here’s why Terraform state is critical for your infrastructure:

  • It maps your configuration to real-world resources in your cloud provider
  • It enables Terraform to plan, apply, and destroy changes efficiently
  • It tracks resource dependencies for proper provisioning order
  • It stores sensitive information like resource IDs and access credentials

When you run terraform apply, Terraform:

  1. Reads your configuration files
  2. Creates or updates the required resources
  3. Records the resulting state in the state file

For team environments, remote state storage is essential, which is why Azure Blob Storage is a popular choice for hosting Terraform backend configurations.

Warning: Never manually edit your Terraform state files! This can cause corruption and lead to major infrastructure inconsistencies.

Why Use AzCopy for Terraform State Migration?

AzCopy is Microsoft’s command-line utility specifically designed for high-performance data transfers to, from, and between Azure storage accounts. Here’s why it’s ideal for Terraform state migration:

  • Speed: Optimized for fast transfer of files between Azure locations
  • Security: Supports encryption in transit with HTTPS and SAS token authentication
  • Reliability: Features like automatic retry and integrity validation ensure successful transfers
  • Recursive copying: Easily copies entire container structures with a single command
  • Cross-platform support: Works on Windows, Linux, and macOS

Compared to alternatives like Azure Storage Explorer or manual file transfers, AzCopy provides automation capabilities and superior performance when working with Azure Blob Storage.

Step-by-Step Terraform State Migration Process

Let’s break down the process into manageable steps for migrating your Terraform state between Azure storage containers.

1. Setting Up Variables

First, define variables for your resource group and storage accounts:

1$resourcegroupname = 'rg-resourcegroup-name'
2$account1name = 'storageaccount1'  # Source storage account
3$account2name = 'storageaccount2'  # Destination storage account

2. Inspecting Storage Resources

Verify your storage accounts and containers are accessible:

1# List storage accounts in the resource group
2az storage account list --resource-group $($resourcegroupname)
3
4# List containers in each storage account
5az storage container list --resource-group $resourcegroupname --account-name $account1name
6az storage container list --resource-group $resourcegroupname --account-name $account2name

3. Retrieving Storage Account Keys

Obtain the necessary authentication keys:

1# Get primary access keys for both accounts
2$account1key = $(az storage account keys list --account-name $account1name --resource-group $resourcegroupname --query [0].value --output tsv)
3$account2key = $(az storage account keys list --account-name $account2name --resource-group $resourcegroupname --query [0].value --output tsv)
4Write-Host 'Account Key for account1:' $account1key
5Write-Host 'Account Key for account2:' $account2key
6
7# Verify container access
8az storage container list --account-key $account1key --account-name $account1name
9az storage container list --account-key $account2key  --account-name $account2name

4. Generating SAS Tokens

Create Shared Access Signature (SAS) tokens for secure, time-limited access:

 1# Generate SAS tokens with appropriate permissions
 2$account1sas = $(az storage account generate-sas --account-key $account1key --account-name $account1name --expiry 2023-01-06 --https-only --permissions acdlrw --resource-types sco --services bfqt --output tsv)
 3
 4$account2sas = $(az storage account generate-sas --account-key $account2key --account-name $account2name --expiry 2023-01-06 --https-only --permissions acdlrw --resource-types sco --services bfqt --output tsv)
 5
 6# Display the tokens and URLs for verification
 7Write-Host "SAS Token for account1: $account1sas"
 8Write-Host "SAS Token for account2: $account2sas"
 9
10Write-Host "Source URL: https://$account1name.blob.core.windows.net?$account1sas"
11Write-Host "Destination URL: https://$account2name.blob.core.windows.net?$account2sas"

5. Downloading and Using AzCopy

Download the latest version of AzCopy and execute the transfer:

1# Download and extract AzCopy
2Invoke-WebRequest -Uri 'https://azcopyvnext.azureedge.net/release20220315/azcopy_windows_amd64_10.14.1.zip' -OutFile 'azcopyv10.zip'
3Expand-archive -Path '.\azcopyv10.zip' -Destinationpath '.\'
4$AzCopy = (Get-ChildItem -path '.\' -Recurse -File -Filter 'azcopy.exe').FullName
5
6# Execute the recursive copy operation
7& $AzCopy copy "https://$account1name.blob.core.windows.net?$account1sas" "https://$account2name.blob.core.windows.net?$account2sas" --recursive

6. Verifying the Migration

After the transfer is complete, verify that all files were copied correctly:

1# List blobs in source and destination to compare
2az storage blob list --account-name $account1name --account-key $account1key --container-name "terraform-state" --output table
3az storage blob list --account-name $account2name --account-key $account2key --container-name "terraform-state" --output table

Critical Best Practices for Terraform State Migration

To ensure a successful state migration without disrupting your infrastructure workflows, follow these essential best practices:

Before Migration

  1. Create a comprehensive backup: Before any migration, create a full backup of your state files

    1# Example: Create a backup locally
2az storage blob download-batch --source terraform-state --destination ./backup --account-name $account1name --account-key $account1key

  2. Apply state locking: Prevent concurrent modifications during migration

     1# Example backend configuration with locking enabled
 2terraform {
 3  backend "azurerm" {
 4    resource_group_name  = "rg-resourcegroup-name"
 5    storage_account_name = "storageaccount1"
 6    container_name       = "terraform-state"
 7    key                  = "prod.terraform.tfstate"
 8    use_azuread_auth     = true
 9    subscription_id      = "your-subscription-id"
10  }
11}

  3. Notify team members: Ensure all team members are aware of the migration window

During Migration

  1. Use appropriate permissions: The SAS tokens should have minimal required permissions
  2. Set reasonable token expiry: Choose an expiry time that allows for migration but limits exposure
  3. Always use HTTPS: Ensure secure transfer with the --https-only flag

After Migration

  1. Run terraform plan: Verify state integrity by running a plan against the new location

    1terraform init -reconfigure -backend-config="storage_account_name=storageaccount2"
2terraform plan

  2. Update backend configuration: Modify your Terraform backend to point to the new location

     1terraform {
 2  backend "azurerm" {
 3    resource_group_name  = "rg-resourcegroup-name"
 4    storage_account_name = "storageaccount2"  # Updated storage account
 5    container_name       = "terraform-state"
 6    key                  = "prod.terraform.tfstate"
 7    use_azuread_auth     = true
 8    subscription_id      = "your-subscription-id"
 9  }
10}

  3. Remove temporary access: Revoke SAS tokens after successful migration

Troubleshooting Common Migration Issues

Even with careful planning, you might encounter issues during migration. Here’s how to handle common problems:

Access Denied Errors

Problem: AuthorizationFailure or AuthenticationFailed errors during copy operation.

Solution:

  • Verify SAS token permissions are correct (need at least Read from source, Write to destination)
  • Check if token expiration has passed
  • Ensure the storage account has not been configured with network restrictions
1# Regenerate SAS token with longer expiry if needed
2$account1sas = $(az storage account generate-sas --account-key $account1key --account-name $account1name --expiry 2023-01-10 --https-only --permissions acdlrw --resource-types sco --services bfqt --output tsv)

Missing Files After Transfer

Problem: Some state files or folders are missing after transfer.

Solution:

  • Ensure the --recursive flag is used with AzCopy
  • Check for any error messages in the AzCopy output
  • Verify there are no special characters in blob names causing issues
1# Use the --include-pattern parameter for specific files if needed
2& $AzCopy copy "https://$account1name.blob.core.windows.net/terraform-state?$account1sas" "https://$account2name.blob.core.windows.net/terraform-state?$account2sas" --recursive --include-pattern="*.tfstate"

State File Corruption

Problem: Terraform shows errors about state corruption after migration.

Solution:

  • Restore from your backup
  • In extreme cases, use terraform import to rebuild state
1# Example of importing a resource back into state
2terraform import azurerm_resource_group.example /subscriptions/{subscription-id}/resourceGroups/example

Automating Regular State Backups

For ongoing protection, consider setting up automated backups of your Terraform state:

1# PowerShell script for automated daily backup
2$timestamp = Get-Date -Format "yyyy-MM-dd"
3$backupContainer = "terraform-state-backup"
4
5# Create backup container if it doesn't exist
6az storage container create --name $backupContainer --account-name $account2name --account-key $account2key
7
8# Copy all state files to backup with timestamp prefix
9az storage blob copy start-batch --destination-container $backupContainer --destination-blob-prefix "backup-$timestamp/" --source-container "terraform-state" --account-name $account2name --account-key $account2key

Conclusion

Moving Terraform state between Azure storage containers is a critical operation that requires careful planning and execution. By using AzCopy with PowerShell, you can automate this process and significantly reduce the risk of state file corruption or loss during migration.

The key takeaways from this guide:

  1. Always back up your state files before migration
  2. Use SAS tokens with appropriate permissions and expiry times
  3. Verify your migration with terraform plan before proceeding
  4. Update your backend configuration after successful migration
  5. Implement regular state backups as part of your DevOps practices

By following the steps outlined in this comprehensive guide, you’ll be able to confidently migrate your Terraform state files between Azure storage containers while maintaining the integrity of your infrastructure.

Have you implemented any additional safeguards for your Terraform state migrations? Share your experiences or questions in the comments below!