AKS Azure Monitor Configuration Dashboard

Azure Monitor – Install AKS Monitoring Grafana Dashboard With Azure AD Integration Using Helm

Posted on Author stefanroth Comments(2)

image

In my last post I showed you how to configure Kubernetes to configure Azure Monitor scraping to collect Prometheus metrics from a GO application. This time I would like to show you how to present this data properly in a nice dashboard. Luckily Microsoft has just released an awesome looking Grafana template…

image

…which we will use to configure the necessary Azure Log Analytics plugin, import the dashboard and finally configure Azure AD integration automatically. Sounds good? Before we can start we need an AKS cluster, configure NGINX and cert-manager to publish Grafana outside the cluster. This setup is just for testing purposes and not to be used in production!

First we deploy an AKS cluster…

1

…default settings…

2

…once more default…

3

…and again…

4

…make sure you set the proper workspace…

5

…and then review and create the cluster. Congratulation, you just deployed an AKS cluster Smile.

Now we need to install NGINX ingress controller. NGINX is according to Kubernetes.io…

An API object that manages external access to the services in a cluster, typically HTTP.

Ingress can provide load balancing, SSL termination and name-based virtual hosting.

…to achieve this goal we follow the Microsoft documentation, switch to Azure Cloud Shell…

# Switch the default susbscription where you have aks installed
az account set –s [SUBSCRIPTIONID]

# Pull the AKS credentials from the AKS cluster
az aks get-credentials --resource-group aks3 --name aksnumberthree

#Next install Helm and tiller
kubectl -n kube-system create serviceaccount tiller
kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
helm init --service-account=tiller

# Create a namespace for ingress resources
kubectl create namespace ingress

# Use Helm to deploy an NGINX ingress controller
helm install stable/nginx-ingress --namespace ingress --set controller.replicaCount=2 --set controller.nodeSelector."beta\.kubernetes\.io/os"=linux --set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux

# Show Ingress services
kubectl get service -l app=nginx-ingress --namespace ingress

…after some time it should look similar to this output…

8

…and check the installed services…

9

…next we need to assign the DNS name, so we can reach the AKS cluster via FQDN. Therefore we get the public IP from the above output…

# Public IP address of your ingress controller
$IP="52.233.229.54"

# Name to associate with public IP address
$DNSNAME="aksnumberthree"

# Get the resource-id of the public ip
$PUBLICIPID=$(az network public-ip list --query "[?ipAddress!=null]|[?contains(ipAddress, '$IP')].[id]" --output tsv)

# Update public ip address with DNS name
az network public-ip update --ids $PUBLICIPID --dns-name $DNSNAME

…and the output should look something like the screenshot and the cluster FQDN should be resolvable like https//aksnumberthree.westeurope.cloudapp.azure.com or whatever name you named your cluster …

12

…because we want to secure our traffic, we need to install cert-manager, which handles the certification requests and issues the SSL certificates automatically via Let’s Encrypt…

# Install the CustomResourceDefinition resources separately
kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.8/deploy/manifests/00-crds.yaml

# Create the namespace for cert-manager
kubectl create namespace cert-manager

# Label the cert-manager namespace to disable resource validation
kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true

# Add the Jetstack Helm repository
helm repo add jetstack https://charts.jetstack.io

# Update your local Helm chart repository cache
helm repo update

# Install the cert-manager Helm chart
helm install --name cert-manager --namespace cert-manager --version v0.8.0 jetstack/cert-manager

…and the output looks like this…

13

…then we need to deploy the CA ca issuer which is…

Issuers (and ClusterIssuers) represent a certificate authority from which signed x509 certificates can be obtained, such as Let’s Encrypt. You will need at least one Issuer or ClusterIssuer in order to begin issuing certificates within your cluster.

…again create a ./cluster_issuer.yaml file and add the following configuration…

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
     name: letsencrypt-staging
     namespace: ingress
spec:
     acme:
       email: returnone@outlook.com
       http01: {}
       privateKeySecretRef:
         name: letsencrypt-staging
       server: https://acme-staging-v02.api.letsencrypt.org/directory

…this would go like this…

15

…wow, lot’s of stuff we had to install!

Just to repeat, we installed an AKS cluster, next deployed cert-manager which is a certificate management controller to deal with certificate requests and after that we installed the CA cluster issuer, which represents a certificate authority from which signed x509 certificates can be obtained. So we have built the foundation for the next steps.

At this point we are ready to deploy Grafana using Helm. There is a Helm chart available https://github.com/helm/charts/tree/master/stable/grafana having already a basic configuration available. What we need to do, is to override some of the values provided by the https://github.com/helm/charts/blob/master/stable/grafana/values.yaml file. Therefore we provide our own values file, which I will explain next.

Create a file called ./custom_values.yaml and then configure it like explained here. The first part defines some persistance storage, which Grafana will mount from Kubernetes to store data…

## PERSISTENCE STORAGE CONFIGURATION
persistence:
        type: pvc
        enabled: true
        accessModes:
          - ReadWriteOnce
        size: 10Gi
        finalizers:
          - kubernetes.io/pvc-protection

…the next section will tell Grafana to install the Azure Monitor plugin and configure it with the service principal configuration…

## LOAD PLUGIN
plugins:
          - grafana-azure-monitor-datasource
datasources:
        datasources.yaml:
          apiVersion: 1
          datasources:
          - name: AzureMonitor # !! DATA SOURCE NAME MUST MATCH DATA SOURCE NAME IN GRAFANA DASHBOARD !!
            type: grafana-azure-monitor-datasource
            orgId: 1
            typeLogoUrl: public/plugins/grafana-azure-monitor-datasource/img/logo.jpg
            access: proxy
            url: /api/datasources/proxy/2
            isDefault: true
            jsonData:
              cloudName: azuremonitor
              subscriptionId: [SUBSCRIPTION ID] # NEED TO CHANGE
              tenantId: [AZURE AD TENANT ID] # NEED TO CHANGE
              clientId: [CLIENT ID SERVICE PRINCIPAL] # NEED TO CHANGE
              logAnalyticsDefaultWorkspace: [WORKSPACE ID] #NEED TO CHANGE
              azureLogAnalyticsSameAs: true
            keepCookies: []
            secureJsonData:
              clientSecret: [CLIENT KEY SERVICE PRINCIPAL] #NEED TO CHANGE
            editable: true

…next we will automatically import the dashboard which can be stored in a repository…

##DASHBOARD IMPORT
dashboardProviders:
        dashboardproviders.yaml:
          apiVersion: 1
          providers:
          - name: 'default'
            orgId: 1
            folder: ''
            type: file
            disableDeletion: false
            editable: true
            options:
              path: /var/lib/grafana/dashboards/default
dashboards:
        default:
          local-dashboard:
            url: https://raw.githubusercontent.com/stefanrothnet/grafana-dashboards/master/aks/dashboard.json  # NEED TO CHANGE TO DASHBOARD REPO

…and next we will do the Azure AD integration…

grafana.ini:
## AZURE AD INTEGRATION
        auth.generic_oauth:
          name: "Azure AD"
          enabled: true
          allow_sign_up: true
          client_id: [CLIENT ID SERVICE PRINCIPAL] # NEED TO CHANGE
          client_secret: [CLIENT KEY SERVICE PRINCIPAL] # NEED TO CHANGE
          scopes: "openid email name"
          auth_url: https://login.microsoftonline.com/[TENANT ID]/oauth2/authorize # NEED TO CHANGE
          token_url: https://login.microsoftonline.com/[TENANT ID]/oauth2/token # NEED TO CHANGE

Finally we need to set the root URL of the Grafana server which is absolutely essential to work with OAuth authentication with Azure AD. This URL MUST match the service principal reply URL…

### ROOT URL SERVER
        server:
          root_url:  https://aksnumberthree.westeurope.cloudapp.azure.com/ # NEED TO CHANGE TO CUSTOM URL !! MUST MATCH REPLY URL ON SERVICE PRINCIPAL !!

…following this configuration we come to the conclusion, that we need to setup 2 service principals, one for the Azure Monitor plugin and one for the Azure AD integration. I am not going to explain how to create a service principal, please read the docs.

Give the first service principal “READER” permission on the subscription where Azure Monitor needs to monitor resources and in addition give “LOG ANALYTICS READER” permission on the Log Analytics workspace, which the AKS cluster is sending the data to. Then set the reply url like in the screenshot. Again, this is the service principal for the Azure Monitor plugin…

image

…the second service principal for Azure AD integration, does not need any special permissions in Azure, just create a service principal in the portal and set the reply URL like this…

image

…it is very important to match the FQDN of your server root url in the grafana.ini configuration. So the entire configuration file ./custom_values.yaml looks like this…

## PERSISTENCE STORAGE CONFIGURATION
persistence:
        type: pvc
        enabled: true
        accessModes:
          - ReadWriteOnce
        size: 10Gi
        finalizers:
          - kubernetes.io/pvc-protection
## LOAD PLUGIN
plugins:
          - grafana-azure-monitor-datasource
datasources:
        datasources.yaml:
          apiVersion: 1
          datasources:
          - name: AzureMonitor # !! DATA SOURCE NAME MUST MATCH DATA SOURCE NAME IN GRAFANA DASHBOARD !!
            type: grafana-azure-monitor-datasource
            orgId: 1
            typeLogoUrl: public/plugins/grafana-azure-monitor-datasource/img/logo.jpg
            access: proxy
            url: /api/datasources/proxy/2
            isDefault: true
            jsonData:
              cloudName: azuremonitor
              subscriptionId: [SUBSCRIPTION ID] # NEED TO CHANGE
              tenantId: [AZURE AD TENANT ID] # NEED TO CHANGE
              clientId: [CLIENT ID SERVICE PRINCIPAL] # NEED TO CHANGE
              logAnalyticsDefaultWorkspace: [WORKSPACE ID] #NEED TO CHANGE
              azureLogAnalyticsSameAs: true
            keepCookies: []
            secureJsonData:
              clientSecret: [CLIENT KEY SERVICE PRINCIPAL] #NEED TO CHANGE
            editable: true
##DASHBOARD IMPORT
dashboardProviders:
        dashboardproviders.yaml:
          apiVersion: 1
          providers:
          - name: 'default'
            orgId: 1
            folder: ''
            type: file
            disableDeletion: false
            editable: true
            options:
              path: /var/lib/grafana/dashboards/default
dashboards:
        default:
          local-dashboard:
            url: https://raw.githubusercontent.com/stefanrothnet/grafana-dashboards/master/aks/dashboard.json  # NEED TO CHANGE TO DASHBOARD REPO
grafana.ini:
## AZURE AD Integration https://community.grafana.com/t/some-questions-on-authentication-with-azure-ad/18133/7
        auth.generic_oauth:
          name: "Azure AD"
          enabled: true
          allow_sign_up: true
          client_id: [CLIENT ID SERVICE PRINCIPAL] # NEED TO CHANGE
          client_secret: [CLIENT KEY SERVICE PRINCIPAL] # NEED TO CHANGE
          scopes: "openid email name"
          auth_url: https://login.microsoftonline.com/[TENANT ID]/oauth2/authorize # NEED TO CHANGE
          token_url: https://login.microsoftonline.com/[TENANT ID]/oauth2/token # NEED TO CHANGE
### REPLY URL SERVER
        server:
          root_url:  https://aksnumberthree.westeurope.cloudapp.azure.com/ # NEED TO CHANGE TO CUSTOM URL !! MUST MATCH REPLY URL ON SERVICE PRINCIPAL !!

…save the file and apply it to the cluster to install Grafana like this…

#Install grafana
helm upgrade --install grafana --namespace ingress  stable/grafana -f ./custom_values.yaml

…this will take a short time and gives you an output like this…

17

…now we are almost done, the last step is to tell NGINX to route all traffic on port 80 from the cluster to Grafana. The configuration goes like that…

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
        name: grafana-ingress
        namespace: ingress
        annotations:
          kubernetes.io/ingress.class: nginx
          certmanager.k8s.io/cluster-issuer: letsencrypt-staging
          nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
        tls:
        - hosts:
          - aksnumberthree.westeurope.cloudapp.azure.com
          secretName: tls-secret
        rules:
        - host: aksnumberthree.westeurope.cloudapp.azure.com
          http:
            paths:
            - backend:
                serviceName: grafana
                servicePort: 80
              path: /(.*)

…save this configuration as ./ingress_route.yaml and apply it, like this…

18

Congratulation! Now you should be able to login into the dashboard using your Azure AD tenant. It looks like this…

image

…login with an Azure AD account, and you will get a very cool dashboard. Well, I slightly modified it in terms of size, but Microsoft has done an awesome job, creating this Grafana dashboard…

image

Well, I had to invest some time to figure out how to automatically configure the Azure Monitor plugin, import the dashboard from a repository and to configure the Azure AD integration. The goal was to have a complete configuration file which we could apply once and everything is deployed automatically. I published the dashboard I used for testing on GitHub, so try it out! I hope this provides you with some useful information and saves you some time!

stefanroth
http://www.stefanroth.net

Related Articles
ACS Configuration

SCOM 2012 SP1 – Part 2: Audit Collection Services (ACS) Cross Platform Setup

Posted on Author stefanroth

In part 1 we installed the ACS collector, ACS reporting and enabled one Windows server to forward the events to the collector server ACS01. In this part we will install the cross platform part of ACS to collect the events from Linux systems.
Dashboard

SCOM 2012 SharePoint Dashboard – Part 2

Posted on Author stefanroth

In SCOM 2012 SharePoint Dashboard – Part 1 I showed you how to add a SCOM Dashboard to SharePoint site. Because I am using the SharePoint Foundation 2010 so far only SCOM Admins and Operators would have access. Except you would configure shared credentials in SharePoint which is not supported in the SharePoint Foundation version. […]
Azure Operational Insights Configuration Recommended Testing Tool

Microsoft Azure Operational Insights – Past, Current, Future

Posted on Author stefanroth

This week Microsoft announced the Microsoft Azure Operational Insights. But what is Azure Operational Insights? Well, it is a cloud based tool which has grown over the past 3 years and it is a good example where you can see how technology evolved.

2 Replies to “Azure Monitor – Install AKS Monitoring Grafana Dashboard With Azure AD Integration Using Helm”

Leave a Reply to stefanroth Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.