# Argo Rollouts analysis with ISD and Prometheus

This document explains the end-to-end flow for the analysis run with the **Prometheus** data source.

Follow the steps below for the end-to-end workflow for analysis run and to get an analysis report.

### Step 1: Add Integration

**Note: You must add all the integrations through ISD UI only.**

To add a **Prometheus** integration as your data source to allow ISD to perform your metric analysis, follow the steps given below:&#x20;

1.  From the ISD dashboard, click "**Setup**" and click "**Integrations**" and then Click "**+New Integration**" button as shown in the image below:\


    <figure><img src="https://lh3.googleusercontent.com/jXN_eBFmaqCyHRNe8zEbVbi9LMpwhee4erZApBbsnl0eAzp7d9Z4lwKSzYMwoOMQrw9lROImlyI4ANWO2z0kky-tfENZ9Fd8xA3KX6al7mxjtJWazium2e8p57uGbb8DOxLTVqDallO8WMQfGwbAQ7E" alt=""><figcaption></figcaption></figure>
2.  The list of available integrators and their respective fields appears. Select the **Prometheus** integration and fill out the information about it that appears on the right pane of the screen.



    <figure><img src="https://lh4.googleusercontent.com/QMACLFVYfDx499TvIwvoUDiYBITdPH68VCvKueG2HcnRs7bH3I3XfParWLn1l_4FPj7CUELYf1KkH8iRZIscc1cmeUYaiJULzt99dRcuVfKzY7OflveNvFNVjvYubQ4_wxvFgvSSh31msUr_1LNKDCo" alt=""><figcaption></figcaption></figure>

    Update the following information on the above screen:

    * **Account Name**: User-defined account name for your **Prometheus** access.
    * **End Point**: Prometheus host address from which you access the Prometheus.
    * **User Name(Optional)**: Prometheus User Name
    * **Password(Optional)**: Prometheus Password &#x20;
    * **Permissions**: To restrict permissions to this account, you can select the **User Groups**.
3.  Once you have updated all the information, click the "**Save**" button. The newly created **Prometheus** integrator appears as shown below:\


    <figure><img src="https://lh4.googleusercontent.com/Vp4ous5yZV6hk_Cniem0NwQCnt5WmGJyopuWQPFwILbO_nwmaYIaidjr17367db9xQK5zLr3CXPIsooHm0Kk5kzS3oJy__ga9Gm2w019kuQCEGp5rxbS-TVpnjUiTeqVfOwurMDhwZOJYOT99YanFJA" alt=""><figcaption></figcaption></figure>

Create a metric template after adding the **Prometheus** integrator.

### Step 2: Create Metric Template

We support the following two modes of template creation

* **Gitops mode template creation**
* **Template creation in ISD UI**

#### Gitops mode Metric Template creation:

You can create a metric template in a GitHub repository where your deployment manifest files are saved. The sample Metric Template for **Prometheus** data source is available [**here**](https://github.com/OpsMx/Delivery-Verification-Templates/tree/main/Prometheus)**.**&#x20;

OpsMx provides the two types of sample Metric Templates for **Prometheus** s follows:

1. **Minimal yaml file** (prometheus-app-health-springboot-minimal.yaml)
2. **Extended yaml file** (prometheus-app-health-springboot-extended.yaml)

**Minimal yaml file:**

The following sample **Prometheus minimal yaml** file contains the mandatory parameters to create a metric template. You can use this sample yaml file instead of creating a new template.

{% code overflow="wrap" %}
```
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-app-health-springboot-min
data:
  prometheus-app-health-generic-min: |
    accountName: prometheus-account-name
    metricType: ADVANCED
    advancedProvider: PROMETHEUS
    metricTemplateSetup:
      groups:
        - metrics:
            - name: "avg(container_memory_usage_bytes{namespace=\"${namespace_key}\", pod=~\"${pod_key}\"})"
              riskDirection: higher
          group: "Memory Usage By Pod Name"
        - metrics:
            - name: "avg(rate(container_cpu_usage_seconds_total{namespace=\"${namespace_key}\", pod=~\"${pod_key}\"}[1m]) * 100)"
              riskDirection: higher
          group: "CPU Usage By Pod Name"
        - metrics:
            - name: "sum(rate(http_server_requests_seconds_sum{app=\"${app_name}\",kubernetes_pod_name=~\"${pod_key}\"}[1m])) / sum(rate(http_server_requests_seconds_count{app=\"${app_name}\",kubernetes_pod_name=~\"${pod_key}\"}[1m]))"
              riskDirection: higher
          group: "Application Latency"
        - metrics:
            - name: "avg(rate(http_server_requests_seconds_count{app=\"${app_name}\",kubernetes_pod_name=~\"${pod_key}\"}[1m]))"
              riskDirection: higherOrLower
          group: "Application Request Rate"
        - metrics:
            - name: "sum(rate(http_server_requests_seconds_count{app=\"${app_name}\",kubernetes_pod_name=~\"${pod_key}\",status=~\"^[4-5].*\"}[1m])) or vector(0)"
              riskDirection: higher
          group: "Application Error Rate"
```
{% endcode %}

Parameters details are as follows:

* **name**: Give a name to the Metric Template. (This Metric Template name, must be used in OpsMx Provider configmap file)
* **accountName**: Metric provider account name (Must be provided the same account name, which you have given while adding the Prometheus integrator in ISD UI)
* **metricType**: Type of the metric.
* **advancedProvider**: Provide the name of “PROMETHEUS”.
* **groups**: Groups are the set of metrics to be configured for analysis. Each group can carry multiple metrics and has a group name associated with it, to be specific in the group field.
  * **riskDirection**: Direction in which the metric difference is allowed to expand. You can choose the value as Higher, Lower, ‘Higher or Lower’.
  * **group**: group name

**Extended yaml file:**

The following sample **Prometheus extended yaml** file contains all the available parameters even non-mandatory to create a metric template. You can use this sample yaml file instead of creating a new template.

{% code overflow="wrap" %}
```
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-app-health-springboot-ext
data:
  prometheus-app-health-springboot-ext: |
    accountName: isd312-saas-prom
    metricType: ADVANCED
    advancedProvider: PROMETHEUS
    metricWeight: 1
    criticality: normal
    nanStrategy: remove
    metricTemplateSetup:
      groups:
        - metrics:
            - name: "avg(container_memory_usage_bytes{namespace=\"${namespace_key}\", pod=~\".*${pod_key}.*\"})"
              riskDirection: higher
              criticality: mustHave
              customThresholdHigherPercentage: 50
              nanStrategy: replaceWithZero
          group: Memory Usage By Pod Name
        - metrics:
            - name: "avg(rate(container_cpu_usage_seconds_total{namespace=\"${namespace_key}\", pod=~\".*${pod_key}.*\"}[1m]) * 100)"
              riskDirection: higher
              criticality: mustHave
              customThresholdHigherPercentage: 50
          group: CPU Usage By Pod Name
        - metrics:
            - name: "sum(rate(http_server_requests_seconds_sum{app=\"${app_name}\",kubernetes_pod_name=~\"${pod_key}\"}[1m])) / sum(rate(http_server_requests_seconds_count{app=\"${app_name}\",kubernetes_pod_name=~\"${pod_key}\"}[1m]))"
              riskDirection: higher
              customThresholdHigherPercentage: 50
              criticality: critical
              watchlist: true
          group: "Application Latency"
        - metrics:
            - name: "avg(rate(http_server_requests_seconds_count{app=\"${app_name}\",kubernetes_pod_name=~\"${pod_key}\"}[1m]))"
              riskDirection: higherOrLower
              customThresholdHigherPercentage: 50
              customThresholdLowerPercentage: 50
          group: "Application Request Rate"
        - metrics:
            - name: "sum(rate(http_server_requests_seconds_count{app=\"${app_name}\",kubernetes_pod_name=~\"${pod_key}\",status=~\"^[4-5].*\"}[1m])) or vector(0)"
              riskDirection: higher
              customThresholdHigherPercentage: 50
              criticality: critical
              watchlist: true
          group: "Application Error Rate"
```
{% endcode %}

Parameters details are as follows:

* **name**: Give a name to the Metric Template (**This Metric Template name, must be used in OpsMx Provider configmap file** )
* **accountName**: Metric provider account name (**Must be provided the same account name, which you have given while adding the Prometheus integrator in ISD UI**)
* **metricType**: Type of the metric.
* **advancedProvider**: Provide the name of “**PROMETHEUS**”.
* **metricWeight**: Numerical importance given to a metric. It can range from 0 as lowest and 1 as highest.
* **criticality**:
  * **Normal**: To remove the metric from the metric group for score calculation if it has no data.
  * **Critical**: To fail the entire analysis if this metric fails or has no data.
  * **MustHave**: To fail a metric if data is missing.&#x20;
*   **nanStrategy**: Handles NaN values which can occur if there is no data in a particular interval for metric data



    {% hint style="info" %}
    **Note**:&#x20;

    * If you specify metricWeight, criticality and nanStrategy at global level, it is applicable to all the metric groups.&#x20;
    * If you specify metricWeight, criticality and nanStrategy at local level, it is applicable to only that particular metric group and it will override the same parameters at the global level if already specified.
    {% endhint %}
* **groups**: Groups are the set of metrics to be configured for analysis. Each group can carry multiple metrics and has a group name associated with it, to be specific in the group field.
  * **riskDirection**: Direction in which the metric difference is allowed to expand. You can choose the value as Higher, Lower, ‘Higher or Lower’.
* **criticality**:
  * **Normal**: To remove the metric from the metric group for score calculation if it has no data.
  * **Critical**: To fail the entire analysis if this metric fails or has no data.
  * &#x20;**MustHave**: To fail a metric if data is missing.&#x20;
* **groups**: Groups are the set of metrics to be configured for analysis. Each group can carry multiple metrics and has a group name associated with it, to be specific in the group field.
* **customThresholdHigherPercentage**: Percentage difference beyond which the metric is treated as fail&#x20;
* **customThresholdLowerPercentage**: Percentage difference beyond which the metric is treated as fail
* **nanStrategy**: Handles NaN values which can occur if there is no data in a particular interval for metric data
* **watchlist**: Metrics marked in watchlist will be shown first in the metric analysis report.
* **metricWeight**: Numerical importance given to a metric. It can range from 0 as lowest and 1 as highest.
* **group**: group name

### Create Metric Template in ISD UI

You can create a Metric Template on the **Setup** → **Analysis Templates** page in ISD UI. To create a metric template in ISD UI, follow the steps below:

1.  From the application dashboard, click "**Setup**" and click “**Analysis Templates**” and then click "**+New Template**" button. Refer to the image below.\


    <figure><img src="https://lh6.googleusercontent.com/TU_0oBRPamQScguvUImo6b6z0HRJXwzKiaGb6MmuA03czCYT6TJHgc45uL5z5ysSJtZXxI0h1J5oe-S8n17li_5Phv8Fb6n0GkwQ3NBbKlvIqDbUrvyX_ZTKzaBQhK1G5uEeRc11XREM7z9HinF3iSo" alt=""><figcaption></figcaption></figure>
2.  After clicking the “**+New Template**” button, two options appear for you to choose the type of template you want to create. Select the “**Metric Template**” from the available options as shown in the image below.\


    <figure><img src="https://lh6.googleusercontent.com/x16qsX8W1MiEsnkmEtuf6jVrLzQ0mE6WGOx3RcMfHMlUJP81ynBBW0kjIGqUROf3SS9Dn5ZZl_4pE596aD0R2Qc_0L6L6bQbiGstsibCvIflj7eSd_TJ7z_vz5mZ0W5SvObMM8UPyrMchnBnrZ1d6Vk" alt=""><figcaption></figcaption></figure>
3.  The new Metric Template window appears. Update the necessary parameters and click “**Save**” button. Refer to the image below. \


    <figure><img src="https://lh3.googleusercontent.com/i9lYLJtvKok8eqnUF91P14TQaBtZWOBsNwX3b2ubKWzPLfXrYovDoBQ_0vCYLAkIO3dO533vH-7Ve3oet1h1JBmSufdNSoVwznc6bKrv3Zo5aYjhkYyqZhPAmczJf0ODmasW6g9BlKP03GOgF4YR8JQ" alt=""><figcaption></figcaption></figure>

Update the following parameters in the above screen:

* **Metric Template Name**: Give a name to the Metric Template. (**This Metric Template name, must be used in OpsMx Provider configmap file**)
* **Select Datasource**: Select “**PROMETHEUS**” as data source from the drop-down.
* **Select Accounts**: Select the account of interest in the configured data source from the drop-down. Metric provider account name (**Must be provided the same account name, which you have given while adding the Prometheus integrator in ISD UI**).
* **Metric Scope Placeholder**: Metric Scope Placeholder will be replaced by Baseline and New Release values in the Metric Query.
* **Query Name**: A meaningful name given to a query or a group of similar queries.
* **Query**: Query to fetch the metric from the data source provider.
* **Risk Direction**: Direction in which the metric difference is allowed to expand. You can choose the value as Higher, Lower, ‘Higher or Lower’.
* **Threshold percentage**: Percentage difference beyond which the metric is treated as fail.&#x20;
* **Criticality**:
  * **Normal**: To remove the metric from the metric group for score calculation if it has no data.
  * **Critical**: To fail the entire analysis if this metric fails or has no data.
  * **MustHave**: To fail a metric if data is missing.&#x20;
* **Watchlist**: Metrics marked in watchlist will be shown first in the metric analysis report.
* **NaN Strategy**: Handles NaN values which can occur if there is no data in a particular interval for metric data.
* **Weight**: Numerical importance given to a metric. It can range from 0 as lowest and 1 as highest.

After creating Metric Template, it appears in the list for an application on the “**Analysis Templates**” page as shown below:

<figure><img src="https://lh6.googleusercontent.com/o8y8-4L5eKKNNh7moqsxrOxfvhQPhNR0DAOiBTupmh0Vf94qgTc90LndBBmvBPDWSA7VacNBXgd4-8tbmdQS8k6fnoWD27tij9cvJiVvoaK_oe0TEqhdsQxGNOSiqeMKocCjAIMrLJNczXC8Lm9F4sY" alt=""><figcaption></figcaption></figure>

### Step 3: Create Application

To create an application, refer[ **here**](https://docs.opsmx.com/opsmx-intelligent-software-delivery-isd-platform-argo/user-guide/manage-application/create-application). This application must be specified in the “OpsMx Provider Configmap”.

### Step 4: Create OpsMx Provider Configmap

Create “OpsMx Provider Configmap” with opsmx metric provider information, including “metric template name” and “application name”. Sample “OpsMx Provider Configmap” yaml file is available [**here**](https://github.com/OpsMx/Delivery-Verification-Examples/blob/main/Opsmx-Provider-Config/provider-config.yaml). To create “OpsMx Provider Configmap” refer [**here**](https://docs.opsmx.com/opsmx-intelligent-software-delivery-isd-platform-argo/user-guide/delivery-verification/opsmx-provider-configmap)**.**

{% hint style="info" %}
**Note**: The user is given the flexibility to create metric and log templates in ISD and also, to maintain them as config maps via GitOps.
{% endhint %}

*   In the “OpsMx Provider Configmap”, If **gitops** is set to “**true**”, the metric provider prioritizes the presence of metric template as config map, and if not found, tries to load it from ISD.\


    ```
    gitops: true
    ```
*   In the “OpsMx Provider Configmap”, If **gitops** is set to “**false**”, the provider only loads the metric template from ISD.\


    {% hint style="info" %}
    **Note**: If you want to load the metric template from ISD, do not specify the following parameters in the “Analysis template”.
    {% endhint %}



    ```
         volumeMounts:
          - name: metric-config-volume
            mountPath: /etc/config/templates

         volumes:
          - name: metric-config-volume
            configMap:
              name: metrixtemplates
    ```

{% hint style="info" %}
**Note**: Must be provided the same Metric Template name in OpsMx Provider Configmap, which you specified in the Minimal yaml file\ Extended yaml file\ ISD UI.
{% endhint %}

### Step 5: Create an Analysis template

Specify the “OpsMx Provider Configmap” under the job section in the Analysis template. Sample "Analysis Template" is available[ **here**](https://github.com/OpsMx/Delivery-Verification-Examples/blob/main/Opsmx-Provider-Config/analysis-template.yaml). To create an Analysis Template, refer[ **here**](https://docs.opsmx.com/opsmx-intelligent-software-delivery-isd-platform-argo/user-guide/delivery-verification/analysis-template).

### Step 6: Modify rollout.yaml

Modify the rollout.yaml with the image and include the analysis step by specifying the Analysis template that you have already created as shown below. Sample rollout.yaml is available[ **here**](https://github.com/OpsMx/Delivery-Verification-Examples/blob/main/Opsmx-Provider-Config/rollout.yaml)**.**

```
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: rollout-app
spec:
  replicas: 4
  selector:
    matchLabels:
      app: rollout-app
  revisionHistoryLimit: 2
  template:
    metadata:
      annotations:
        prometheus.io/scrape: 'true'
        prometheus_io_path: /mgmt/prometheus
        prometheus_io_port: '8088'
      labels:
        app: rollout-app
    spec:
      containers:
        - name: rollout-app
          image: quay.io/opsmxpublic/canary-issuegen:issue-canary-gen-1401
          imagePullPolicy: Always
          ports:
            - containerPort: 8088
          resources:
            requests:
              memory: 32Mi
              cpu: 5m
  strategy:
    canary:
      steps:
        - setWeight: 25
        - pause: { duration: 15s }
        - analysis:
            templates:
              - templateName: opsmx-analysis
            args:
              - name: canary-hash
                valueFrom:
                  podTemplateHashValue: Latest
              - name: baseline-hash
                valueFrom:
                  podTemplateHashValue: Stable
        - setWeight: 75
        - pause: { duration: 15s }
        - analysis:
            templates:
              - templateName: opsmx-analysis
            args:
              - name: canary-hash
                valueFrom:
                  podTemplateHashValue: Latest
              - name: baseline-hash
                valueFrom:
                  podTemplateHashValue: Stable
```

### Step 7: Deploy application

To deploy the application, refer[ **here**](https://docs.opsmx.com/opsmx-intelligent-software-delivery-isd-platform-argo/user-guide/manage-application/deploy-application).

{% hint style="info" %}
**Note**: Please make sure all the above configuration files are stored in the folder where the rollout.yaml manifest file is stored.
{% endhint %}

### Step 8: Trigger analysis run

If a newer version of the application is deployed, the Rollouts strategy will be invoked and an analysis run will be triggered. Update the image version in rollout.yaml as shown below and sync the application to trigger the analysis run. To sync the application, refer [**here**](https://docs.opsmx.com/opsmx-intelligent-software-delivery-isd-platform-argo/user-guide/manage-application/sync-application).

```
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: rollout-app
spec:
  replicas: 4
  selector:
    matchLabels:
      app: rollout-app
  revisionHistoryLimit: 2
  template:
    metadata:
      annotations:
        prometheus.io/scrape: 'true'
        prometheus_io_path: /mgmt/prometheus
        prometheus_io_port: '8088'
      labels:
        app: rollout-app
    spec:
      containers:
        - name: rollout-app
          image: quay.io/opsmxpublic/canary-issuegen:issue-canary-gen-1402
```

{% hint style="info" %}
**Note**: For the first version of application deployment, Rollout strategy is not invoked. The Rollout strategy is followed only when a newer version of the application is deployed.
{% endhint %}

### Step 9: View Analysis report

To view the Analysis report from Rollouts Dashboard, refer[ **here**](https://docs.opsmx.com/opsmx-intelligent-software-delivery-isd-platform-argo/user-guide/delivery-verification/analysis-history#view-analysis-report-from-rollouts-dashboard).
