Use Bicep to manage FQDN on Azure IP addresses

This is a guide that we use to upgrade clusters by simply creating a new Azure Kubernetes Cluster and setting the cluster live by doing a simple DNS-switch. The safest way to do this is to point the FQDN to the resource we want to use. This allow us to have a very fast roll back technique, simply change the DNS back to the old cluster, in case of catastrophes. This post will take you through a Bicep script that will abstract this process; providing swift management of public IP addresses across different Kubernetes clusters in Azure. This technique is not unique to Kubernetes so feel free to use this in other Azure specific use-cases.

Why Use Bicep for Kubernetes Deployments?

Azure Bicep is a DSL to deploy Azure resources and is simpler in syntax than the standard ARM templates. This allows for easier readability, faster development, and more robust management of cloud resources. Kubernetes makes heavy use of Bicep for managing the lower infrastructure components-public IPs, DNS settings, and load balancers-which need to be in place for traffic routing to and from your cluster.

Below, I show the Bicep script which manages public IP addresses for various environments across different Kubernetes clusters. This is very useful in "lift-and-shift" deployments where one needs resources to be reassigned or moved with no downtime.

The Script

@description('Specifies the location for resources.')
param location string = 'westeurope'

@allowed([
  'dev'
  'test'
  'uat'
  'prod'
])
param env string = 'dev'

@allowed([
  'aks'
  'k8s'
])
param clusterPrefix string = 'aks'

var benchedCluster = clusterPrefix == 'aks' ? 'k8s' : 'aks'

var ipAddresses = {
  k8s: {
    dev: { 
      ip: '10.0.0.1' # Fake ip
      name: 'cluster-dev-ip'
      zones:[
        '1'
        '2'
        '3'
      ]
    }
		# add other environments here
    benchedDnsNameSuffix: 'inactive'
  } 
  aks: {
    dev: { 
      ip: '10.0.1.1'  # Fake ip
      name: 'cluster-dev-ip2'
      zones:[
        '1'
        '2'
        '3'
      ]
    }
		# add other environments here
    benchedDnsNameSuffix: 'previous'
  } 
}

var benchedDns = 'api-${env}-${ipAddresses[benchedCluster].benchedDnsNameSuffix}'
var activeDns = 'api-${env}'

resource putIpOnBench 'Microsoft.Network/publicIPAddresses@2024-01-01' = {
  name: ipAddresses[benchedCluster][env].name
  location: location
  sku: {
    name: 'Standard'
    tier: 'Regional'
  }
  zones: ipAddresses[benchedCluster][env].zones
  properties: {
    ipAddress: ipAddresses[benchedCluster][env].ip
    publicIPAddressVersion: 'IPv4'
    publicIPAllocationMethod: 'Static'
    idleTimeoutInMinutes: 4
    dnsSettings: {
      domainNameLabel: benchedDns
      fqdn: benchedDns
    }
    ipTags: []
  }
}

resource putNewIpOnPitch 'Microsoft.Network/publicIPAddresses@2024-01-01' = {
  name: ipAddresses[clusterPrefix][env].name
  location: location
  sku: {
    name: 'Standard'
    tier: 'Regional'
  }
  zones: ipAddresses[clusterPrefix][env].zones
  properties: {
    ipAddress: ipAddresses[clusterPrefix][env].ip
    publicIPAddressVersion: 'IPv4'
    publicIPAllocationMethod: 'Static'
    idleTimeoutInMinutes: 4
    dnsSettings: {
      domainNameLabel: activeDns
      fqdn: activeDns
    }
    ipTags: []
  }
  dependsOn:[
    putIpOnBench
  ]
}

Explanation of the Bicep Script

Let me go over the script line by line and explain some of the major components.

1. Parameters and Their Purpose

This script first defines three parameters.

@description('Specifies the location for resources.')
param location string = 'westeurope'

@allowed([
  'dev'
  'test'
  'prod'
])
param env string = 'dev'

@allowed([
  'aks'
  'k8s'
])
param clusterPrefix string = 'aks'

location: Defines the Azure region in which the resources should be created. The default is westeurope.
env: This denotes the environment on which this may be deployed, like dev, test, uat, prod. The advantage of having this variable here is that one may easily select to deploy to which environment.
clusterPrefix: This denotes whether the Kubernetes cluster is an AKS or some other form of Kubernetes cluster with k8s standing for a selfhosted Kubernetes cluster.

These parameters will make this script flexible to be able to specify different environments and clusters with minimal effort.

2. Dynamic Variables for Flexibility

The script uses variables that provide dynamic logic and flexibility within a given:

var benchedCluster = clusterPrefix == 'aks' ? 'k8s' : 'aks'

We switch between two clusters when upgrading, aptly named 'k8s' and 'aks'. The variable of benchedCluster is a simple ternary operation determining what cluster is currently "benched" or the one that is currently not in use. If the current active cluster is aks, then benchedCluster is set to k8s, and vice-versa.

The ipAddresses variable is a complex object that holds IP configuration details for each cluster type and environment:

var ipAddresses = {
  k8s: {
    dev: { 
      ip: '10.0.0.1'
      name: 'cluster-dev-ip'
    }
    // similar definitions for test, prod}
    benchedDnsNameSuffix: 'that-was'
  }
  aks: {
    dev: { 
      ip: '10.0.1.1'
      name: 'cluster-dev-ip2'
    }
    // similar definitions for test, prod
    benchedDnsNameSuffix: 'old'
  }
}

This variable does a detailed mapping between IPs, names, and name of the clusters. It will let the script dynamically decide which IP configuration to use depending on which environment and cluster is chosen.

3. Resource Definitions

It defines two main resources:

putIpOnBench: Takes the existing public IP of the inactive ("benched") cluster.

resource putIpOnBench 'Microsoft.Network/publicIPAddresses@2024-01-01' = {
  name: ipAddresses[benchedCluster][env].name
  location: location
  sku: {
    name: 'Standard'
    tier: 'Regional'
  }
  zones: ipAddresses[benchedCluster][env].zones
  properties: {
    ipAddress: ipAddresses[benchedCluster][env].ip
    publicIPAddressVersion: 'IPv4'
    publicIPAllocationMethod: 'Static'
    idleTimeoutInMinutes: 4
    dnsSettings: {
      domainNameLabel: benchedDns
      fqdn: benchedDns
    }
    ipTags: []
  }
}

putNewIpOnPitch: Sets up a new public IP for the active cluster.

resource putNewIpOnPitch 'Microsoft.Network/publicIPAddresses@2024-01-01' = {
  name: ipAddresses[clusterPrefix][env].name
  location: location
  sku: {
    name: 'Standard'
    tier: 'Regional'
  }
  zones: ipAddresses[clusterPrefix][env].zones
  properties: {
    ipAddress: ipAddresses[clusterPrefix][env].ip
    publicIPAddressVersion: 'IPv4'
    publicIPAllocationMethod: 'Static'
    idleTimeoutInMinutes: 4
    dnsSettings: {
      domainNameLabel: activeDns
      fqdn: activeDns
    }
    ipTags: []
  }
  dependsOn: [
    putIpOnBench
  ]
}

These resources manage the lifecycle of the two public IPs in order to rename the FQDN with a placeholder and replace the real FQDN the new IP for smooth transitions between clusters.

The dependsOn property on the putNewIpOnPitch resource ensures the "benching" operation is complete before the new IP is deployed. This will prevent any conflicts and ensure that resources are handled in the correct order.

Advantages of Using Bicep for DNS Settings

Using a Bicep script for public IP address and DNS settings management allows faster transitions between Kubernetes clusters, reducing downtime during transitions. This way, your applications will stay online during the migration process.

Simplicity and Readability: The syntax of the Bicep template is much simpler compared to ARM templates, hence less complex for the script to maintain.
Flexibility: The script can easily be manipulated on different environments and cluster types by using parameterization and dynamic variables.
Integration: Bicep scripts easily integrate with Azure DevOps in terms of automating the CI/CD pipeline.