OpsMx
HomeContact UsBlog
Search
⌃K
Links

Argo Rollouts analysis with ISD and Prometheus

This document explains the end-to-end flow for the analysis run with the Prometheus data source.
Follow the steps below for the end-to-end workflow for analysis run and to get an analysis report.

Step 1: Add Integration

Note: You must add all the integrations through ISD UI only.
To add a Prometheus integration as your data source to allow ISD to perform your metric analysis, follow the steps given below:
  1. 1.
    From the ISD dashboard, click "Setup" and click "Integrations" and then Click "+New Integration" button as shown in the image below:
  2. 2.
    The list of available integrators and their respective fields appears. Select the Prometheus integration and fill out the information about it that appears on the right pane of the screen.
    ​
    Update the following information on the above screen:
    • Account Name: User-defined account name for your Prometheus access.
    • End Point: Prometheus host address from which you access the Prometheus.
    • User Name(Optional): Prometheus User Name
    • Password(Optional): Prometheus Password
    • Permissions: To restrict permissions to this account, you can select the User Groups.
  3. 3.
    Once you have updated all the information, click the "Save" button. The newly created Prometheus integrator appears as shown below:
Create a metric template after adding the Prometheus integrator.

Step 2: Create Metric Template

We support the following two modes of template creation
  • Gitops mode template creation
  • Template creation in ISD UI

Gitops mode Metric Template creation:

You can create a metric template in a GitHub repository where your deployment manifest files are saved. The sample Metric Template for Prometheus data source is available here.
OpsMx provides the two types of sample Metric Templates for Prometheus s follows:
  1. 1.
    Minimal yaml file (prometheus-app-health-springboot-minimal.yaml)
  2. 2.
    Extended yaml file (prometheus-app-health-springboot-extended.yaml)
Minimal yaml file:
The following sample Prometheus minimal yaml file contains the mandatory parameters to create a metric template. You can use this sample yaml file instead of creating a new template.
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-app-health-springboot-min
data:
prometheus-app-health-generic-min: |
accountName: prometheus-account-name
metricType: ADVANCED
advancedProvider: PROMETHEUS
metricTemplateSetup:
groups:
- metrics:
- name: "avg(container_memory_usage_bytes{namespace=\"${namespace_key}\", pod=~\"${pod_key}\"})"
riskDirection: higher
group: "Memory Usage By Pod Name"
- metrics:
- name: "avg(rate(container_cpu_usage_seconds_total{namespace=\"${namespace_key}\", pod=~\"${pod_key}\"}[1m]) * 100)"
riskDirection: higher
group: "CPU Usage By Pod Name"
- metrics:
- name: "sum(rate(http_server_requests_seconds_sum{app=\"${app_name}\",kubernetes_pod_name=~\"${pod_key}\"}[1m])) / sum(rate(http_server_requests_seconds_count{app=\"${app_name}\",kubernetes_pod_name=~\"${pod_key}\"}[1m]))"
riskDirection: higher
group: "Application Latency"
- metrics:
- name: "avg(rate(http_server_requests_seconds_count{app=\"${app_name}\",kubernetes_pod_name=~\"${pod_key}\"}[1m]))"
riskDirection: higherOrLower
group: "Application Request Rate"
- metrics:
- name: "sum(rate(http_server_requests_seconds_count{app=\"${app_name}\",kubernetes_pod_name=~\"${pod_key}\",status=~\"^[4-5].*\"}[1m])) or vector(0)"
riskDirection: higher
group: "Application Error Rate"
Parameters details are as follows:
  • name: Give a name to the Metric Template. (This Metric Template name, must be used in OpsMx Provider configmap file)
  • accountName: Metric provider account name (Must be provided the same account name, which you have given while adding the Prometheus integrator in ISD UI)
  • metricType: Type of the metric.
  • advancedProvider: Provide the name of “PROMETHEUS”.
  • groups: Groups are the set of metrics to be configured for analysis. Each group can carry multiple metrics and has a group name associated with it, to be specific in the group field.
    • riskDirection: Direction in which the metric difference is allowed to expand. You can choose the value as Higher, Lower, ‘Higher or Lower’.
    • group: group name
Extended yaml file:
The following sample Prometheus extended yaml file contains all the available parameters even non-mandatory to create a metric template. You can use this sample yaml file instead of creating a new template.
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-app-health-springboot-ext
data:
prometheus-app-health-springboot-ext: |
accountName: isd312-saas-prom
metricType: ADVANCED
advancedProvider: PROMETHEUS
metricWeight: 1
criticality: normal
nanStrategy: remove
metricTemplateSetup:
groups:
- metrics:
- name: "avg(container_memory_usage_bytes{namespace=\"${namespace_key}\", pod=~\".*${pod_key}.*\"})"
riskDirection: higher
criticality: mustHave
customThresholdHigherPercentage: 50
nanStrategy: replaceWithZero
group: Memory Usage By Pod Name
- metrics:
- name: "avg(rate(container_cpu_usage_seconds_total{namespace=\"${namespace_key}\", pod=~\".*${pod_key}.*\"}[1m]) * 100)"
riskDirection: higher
criticality: mustHave
customThresholdHigherPercentage: 50
group: CPU Usage By Pod Name
- metrics:
- name: "sum(rate(http_server_requests_seconds_sum{app=\"${app_name}\",kubernetes_pod_name=~\"${pod_key}\"}[1m])) / sum(rate(http_server_requests_seconds_count{app=\"${app_name}\",kubernetes_pod_name=~\"${pod_key}\"}[1m]))"
riskDirection: higher
customThresholdHigherPercentage: 50
criticality: critical
watchlist: true
group: "Application Latency"
- metrics:
- name: "avg(rate(http_server_requests_seconds_count{app=\"${app_name}\",kubernetes_pod_name=~\"${pod_key}\"}[1m]))"
riskDirection: higherOrLower
customThresholdHigherPercentage: 50
customThresholdLowerPercentage: 50
group: "Application Request Rate"
- metrics:
- name: "sum(rate(http_server_requests_seconds_count{app=\"${app_name}\",kubernetes_pod_name=~\"${pod_key}\",status=~\"^[4-5].*\"}[1m])) or vector(0)"
riskDirection: higher
customThresholdHigherPercentage: 50
criticality: critical
watchlist: true
group: "Application Error Rate"
Parameters details are as follows:
  • name: Give a name to the Metric Template (This Metric Template name, must be used in OpsMx Provider configmap file )
  • accountName: Metric provider account name (Must be provided the same account name, which you have given while adding the Prometheus integrator in ISD UI)
  • metricType: Type of the metric.
  • advancedProvider: Provide the name of “PROMETHEUS”.
  • metricWeight: Numerical importance given to a metric. It can range from 0 as lowest and 1 as highest.
  • criticality:
    • Normal: To remove the metric from the metric group for score calculation if it has no data.
    • Critical: To fail the entire analysis if this metric fails or has no data.
    • MustHave: To fail a metric if data is missing.
  • nanStrategy: Handles NaN values which can occur if there is no data in a particular interval for metric data
    ​
    Note:
    • If you specify metricWeight, criticality and nanStrategy at global level, it is applicable to all the metric groups.
    • If you specify metricWeight, criticality and nanStrategy at local level, it is applicable to only that particular metric group and it will override the same parameters at the global level if already specified.
  • groups: Groups are the set of metrics to be configured for analysis. Each group can carry multiple metrics and has a group name associated with it, to be specific in the group field.
    • riskDirection: Direction in which the metric difference is allowed to expand. You can choose the value as Higher, Lower, ‘Higher or Lower’.
  • criticality:
    • Normal: To remove the metric from the metric group for score calculation if it has no data.
    • Critical: To fail the entire analysis if this metric fails or has no data.
    • MustHave: To fail a metric if data is missing.
  • groups: Groups are the set of metrics to be configured for analysis. Each group can carry multiple metrics and has a group name associated with it, to be specific in the group field.
  • customThresholdHigherPercentage: Percentage difference beyond which the metric is treated as fail
  • customThresholdLowerPercentage: Percentage difference beyond which the metric is treated as fail
  • nanStrategy: Handles NaN values which can occur if there is no data in a particular interval for metric data
  • watchlist: Metrics marked in watchlist will be shown first in the metric analysis report.
  • metricWeight: Numerical importance given to a metric. It can range from 0 as lowest and 1 as highest.
  • group: group name

Create Metric Template in ISD UI

You can create a Metric Template on the Setup → Analysis Templates page in ISD UI. To create a metric template in ISD UI, follow the steps below:
  1. 1.
    From the application dashboard, click "Setup" and click “Analysis Templates” and then click "+New Template" button. Refer to the image below.
  2. 2.
    After clicking the “+New Template” button, two options appear for you to choose the type of template you want to create. Select the “Metric Template” from the available options as shown in the image below.
  3. 3.
    The new Metric Template window appears. Update the necessary parameters and click “Save” button. Refer to the image below.
Update the following parameters in the above screen:
  • Metric Template Name: Give a name to the Metric Template. (This Metric Template name, must be used in OpsMx Provider configmap file)
  • Select Datasource: Select “PROMETHEUS” as data source from the drop-down.
  • Select Accounts: Select the account of interest in the configured data source from the drop-down. Metric provider account name (Must be provided the same account name, which you have given while adding the Prometheus integrator in ISD UI).
  • Metric Scope Placeholder: Metric Scope Placeholder will be replaced by Baseline and New Release values in the Metric Query.
  • Query Name: A meaningful name given to a query or a group of similar queries.
  • Query: Query to fetch the metric from the data source provider.
  • Risk Direction: Direction in which the metric difference is allowed to expand. You can choose the value as Higher, Lower, ‘Higher or Lower’.
  • Threshold percentage: Percentage difference beyond which the metric is treated as fail.
  • Criticality:
    • Normal: To remove the metric from the metric group for score calculation if it has no data.
    • Critical: To fail the entire analysis if this metric fails or has no data.
    • MustHave: To fail a metric if data is missing.
  • Watchlist: Metrics marked in watchlist will be shown first in the metric analysis report.
  • NaN Strategy: Handles NaN values which can occur if there is no data in a particular interval for metric data.
  • Weight: Numerical importance given to a metric. It can range from 0 as lowest and 1 as highest.
After creating Metric Template, it appears in the list for an application on the “Analysis Templates” page as shown below:

Step 3: Create Application

To create an application, refer here. This application must be specified in the “OpsMx Provider Configmap”.

Step 4: Create OpsMx Provider Configmap

Create “OpsMx Provider Configmap” with opsmx metric provider information, including “metric template name” and “application name”. Sample “OpsMx Provider Configmap” yaml file is available here. To create “OpsMx Provider Configmap” refer here.
Note: The user is given the flexibility to create metric and log templates in ISD and also, to maintain them as config maps via GitOps.
  • In the “OpsMx Provider Configmap”, If gitops is set to “true”, the metric provider prioritizes the presence of metric template as config map, and if not found, tries to load it from ISD.
    gitops: true
  • In the “OpsMx Provider Configmap”, If gitops is set to “false”, the provider only loads the metric template from ISD.
    Note: If you want to load the metric template from ISD, do not specify the following parameters in the “Analysis template”.
    ​
    volumeMounts:
    - name: metric-config-volume
    mountPath: /etc/config/templates
    ​
    volumes:
    - name: metric-config-volume
    configMap:
    name: metrixtemplates
Note: Must be provided the same Metric Template name in OpsMx Provider Configmap, which you specified in the Minimal yaml file\ Extended yaml file\ ISD UI.

Step 5: Create an Analysis template

Specify the “OpsMx Provider Configmap” under the job section in the Analysis template. Sample "Analysis Template" is available here. To create an Analysis Template, refer here.

Step 6: Modify rollout.yaml

Modify the rollout.yaml with the image and include the analysis step by specifying the Analysis template that you have already created as shown below. Sample rollout.yaml is available here.
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: rollout-app
spec:
replicas: 4
selector:
matchLabels:
app: rollout-app
revisionHistoryLimit: 2
template:
metadata:
annotations:
prometheus.io/scrape: 'true'
prometheus_io_path: /mgmt/prometheus
prometheus_io_port: '8088'
labels:
app: rollout-app
spec:
containers:
- name: rollout-app
image: quay.io/opsmxpublic/canary-issuegen:issue-canary-gen-1401
imagePullPolicy: Always
ports:
- containerPort: 8088
resources:
requests:
memory: 32Mi
cpu: 5m
strategy:
canary:
steps:
- setWeight: 25
- pause: { duration: 15s }
- analysis:
templates:
- templateName: opsmx-analysis
args:
- name: canary-hash
valueFrom:
podTemplateHashValue: Latest
- name: baseline-hash
valueFrom:
podTemplateHashValue: Stable
- setWeight: 75
- pause: { duration: 15s }
- analysis:
templates:
- templateName: opsmx-analysis
args:
- name: canary-hash
valueFrom:
podTemplateHashValue: Latest
- name: baseline-hash
valueFrom:
podTemplateHashValue: Stable

Step 7: Deploy application

To deploy the application, refer here.
Note: Please make sure all the above configuration files are stored in the folder where the rollout.yaml manifest file is stored.

Step 8: Trigger analysis run

If a newer version of the application is deployed, the Rollouts strategy will be invoked and an analysis run will be triggered. Update the image version in rollout.yaml as shown below and sync the application to trigger the analysis run. To sync the application, refer here.
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: rollout-app
spec:
replicas: 4
selector:
matchLabels:
app: rollout-app
revisionHistoryLimit: 2
template:
metadata:
annotations:
prometheus.io/scrape: 'true'
prometheus_io_path: /mgmt/prometheus
prometheus_io_port: '8088'
labels:
app: rollout-app
spec:
containers:
- name: rollout-app
image: quay.io/opsmxpublic/canary-issuegen:issue-canary-gen-1402
Note: For the first version of application deployment, Rollout strategy is not invoked. The Rollout strategy is followed only when a newer version of the application is deployed.

Step 9: View Analysis report

To view the Analysis report from Rollouts Dashboard, refer here.
Previous
Argo Rollouts analysis with ISD and NewRelic
Next
Argo Rollouts analysis with ISD and Stackdriver
Last modified 3mo ago