Getting started with Automated Analysis
Argo Rollout enhances the deployment capabilities of Kubernetes with features such as blue-green, canary, experimentation, and progressive delivery. The OpsMx Intelligent Software Delivery (ISD) platform integrates with Argo Rollout to automate the verification of deployments and reduce the risk of releasing faulty updates to production through progressive delivery.
ISD uses machine learning algorithms, Natural Language Processing (NLP), and statistical analysis to evaluate new releases and identify potential risks in comparison to the base releases. It integrates with the widely used log and metrics monitoring tools to perform analysis during the rollout process.
ISD integrates with the following monitoring tools:
- Amazon Cloudwatch
- AppDynamics
- DataDog
- Dynatrace
- ElasticSearch
- Graphite
- Graylog
- New Relic
- Prometheus
- Splunk
- Stackdriver
- SumoLogic
- VMware Tanzu Observability
- Argo CD (v2.4.8 or later) with Argo Rollouts (v1.3 or later) Installed.
- Application configured for deployments using Argo Rollouts.
- A log or metrics monitoring tool for observing applications (Refer to the above list of monitoring tools that ISD supports).
- Application configured for collecting metrics and logs through the supported monitoring tools.
- 1.Connect the ISD to the existing Argo CD instances.
- 2.To monitor data, we need to integrate the ISD system with appropriate monitoring tools. This integration will allow us to access the data source for application logs, such as Elastic, and metrics, such as Prometheus. These integration accounts will be utilized in subsequent data analysis processes.
- 3.Choose a Deployment for which you want to perform Rollouts. If you are unfamiliar with Rollouts, please refer to the Migrating to Rollout document to convert your existing Deployment into a Rollout object. A sample application with all the necessary supporting files can be found here.
- 4.Generate configMaps and secrets in the namespace where the Application Rollout will be executed. These should include the updated Metric/Log templates and any additional information related to your installation. A sample Application with all the required supporting files is available here.
- 5.Choose a Log Template that is specific to your data source from here. The Log Template defines the application-specific log messages that are categorized. Follow the instructions provided in the file.
- OPTIONAL: Adjust the log selection strings and topic type (CRITICAL, ERROR, WARNING, IGNORE) as necessary based on the specific needs of your application.
- 6.Choose a Metric Template that is specific to your metric monitoring tool from here. The Metric Template defines the metrics that you want to compare between the new release and the baseline. Follow the instructions in the file.
- OPTIONAL: Adjust the metric selection as necessary based on the specific needs of your application
- 7.Modify the Argo Rollout AnalysisTemplate found here with your specific parameters, following the instructions in the file.
- 8.
- 9.Perform automated analysis during rollouts.
- 10.Evaluate analysis report.
To enable the Automated Analysis you need to perform a one-time platform setup and one-time setup for each application as stated below.
The following is the sequence of activities for one-time platform setup.
- 1.Connect to existing Argo CD
- 2.Integrate with monitoring tools - Data Sources
- 3.Set up Kubernetes profile Secret for ISD
- 4.Create ServiceAccount for ISD Analysis
Following are the sequence of activities for application setup for Automated Rollout analysis.
- 1.Create Log and Metric Template ConfigMaps
- 2.Create Analysis configuration
- 3.Setup Rollout AnalysisTemplate
- 4.Update rollout.yaml to enable Automated Analysis
To connect the ISD platform to one or more existing Argo CD installations, follow the steps below:
- 1.Access the CD Integration sub-menu under the top-level Setup menu in the ISD UI.
- 2.Deploy a secure agent in the same cluster as the Argo CD or in the same security domain as the Argo CD.
- 3.Enter a name for the Argo CD connection in the ISD platform.
Note: The secure agent ensures a secure connection between the ISD platform and the Argo CD installation. The name chosen for the Argo CD connection is used in later steps for identification purposes.
To integrate the ISD platform with the monitoring tools used to observe the applications during deployments, follow the steps below:
- 1.Access the Integrations sub-menu under the top-level Setup menu in the ISD UI.
- 2.Connect the ISD platform to the desired monitoring tools.
- 3.Create and configure "accounts" in the ISD platform to represent the connections to the monitoring tools.
- 4.Use the account names in the log and metrics templates during application setup.Note: The ISD platform pulls relevant data from the configured integrations (accounts) for analysis purposes during deployments. The account names are used in the log and metrics templates to identify the data source.
Create a Kubernetes profile secret in each Kubernetes target namespace as shown below. Sample kubernetes configuration file is available here. Customize the sample configuration file to match your specific scenario.
apiVersion: v1
kind: Secret
metadata:
name: opsmx-profile # Default Name. No change necessary. It is referred in the AnalysisTemplate
stringData:
opsmxIsdUrl: "https://ISD.REPLACE.THIS.WITH.YOURCOMPANY.COM" # URL of ISD platform
user: "admin" # Default User. Change to an ISD platform User who has access to the data source (monitoring tools)
sourceName: "argocd" # Argo CD name used in Setup->CDIntegration in ISD
# Default is True. Change to false if using Rollouts without Argo CD
cdIntegration: "true"
Note: The Kubernetes profile secret is used to securely store and access the credentials for your Kubernetes cluster. It is a crucial component in the process of deploying applications using Argo CD and ISD.
The ISD analysis is carried out by a Kubernetes job, and a service account with the necessary permissions should be set up in each namespace where the rollout is performed. A sample file is available here, and usually, no modifications are needed.
The following steps must be carried out for each application. Make sure that the application already has rollout enabled and that a canary or blue-green strategy is defined. If you need assistance with setting up the base rollout, refer to the Argo Rollout documentation.
The Log and Metrics templates differ depending on the type of application and the metrics or log monitoring tool provider. It is recommended to find the appropriate log and metrics default template from the OpsMx public repository.
The following example demonstrates a typical configuration for an ElasticSearch log provider. Create the Log and Metrics template in the application's git folder and update the account names for the log and monitoring, as defined in Step 2 of the platform setup section.
Sample Log Template (for ElasticSearch):
Api Version: v1
kind: ConfigMap
metadata:
name: elasticsearch-generic-minimal
data:
elasticsearch-generic-minimal: |
monitoringProvider: ELASTICSEARCH
accountName: elastic-account-name # THIS is the account name given Step 2 where configured the data source
index: kubernetes*
filterKey: kubernetes.pod_name
responseKeywords: log
This step involves defining the parameters for the rollout analysis such as the duration of the analysis and the minimum score required to pass. If the overall score falls below the defined pass score threshold, the rollout will automatically halt. The relevant log and metric templates defined in Step 1 should also be included in the analysis configuration. Sample analysis configurations for Prometheus and Elastic data sources are shown below and additional sample files are available on the OpsMx public repository.
apiVersion: v1
kind: ConfigMap
metadata:
name: opsmx-provider-config ## Default Name. Change if multiple analysis is needed in the same namespace
data:
providerConfig: |
application: ARGO-CD-APPLICATION-NAME ## Change this to your application name
lifetimeMinutes: 05 ## Change to Analysis Duration in minutes
gitops: true
passScore: 60 ## Change to Minimum Pass Score. Any score above this score will continue Rollout.
serviceList:
- metricScopeVariables: "${app_key},${host_key}"
baselineMetricScope: "{{env.STABLE_POD_HASH}}"
canaryMetricScope: "{{env.LATEST_POD_HASH}}"
metricTemplateName: "metrictemplate"
logScopeVariables: "kubernetes.pod_name"
baselineLogScope: ".*{{env.STABLE_POD_HASH}}.*"
canaryLogScope: ".*{{env.LATEST_POD_HASH}}.*"
logTemplateName: "logtemplate"
For the detailed information on how to specify analysis configuration in OpsMx-Provider ConfigMap, refer here.
The Argo Rollout AnalysisTemplate is used to specify the specifics of how to analyze the rollout data, using the provider-config and log and metrics templates from the previous steps. For general information about Analysis Templates in Argo Rollouts, please refer to the Argo documentation.
The ISD analysis is performed as a Kubernetes job and the AnalysisTemplate sample file can be found in the OpsMx public repository.
In most cases, there is no need to make any changes to the Analysis Template file, unless multiple applications and templates are being used in the same namespace. In that case, please follow the instructions provided in the sample file.
Now, we are ready to add the automated analysis as part of the rollout strategy.
The rollout.yaml file is the key configuration file for Argo Rollout and it specifies various details related to the rollout strategy. To enable automated analysis, update the rollout.yaml file with the following steps:
- 1.Specify the Rollout Strategy: Choose between Canary or Blue Green rollout strategy based on your requirements.
- 2.Configure Traffic Shaping: Specify the traffic shaping details such as the percentage of traffic to be redirected to the canary version.
- 3.Progressive Delivery Steps (optional): Define one or more progressive delivery steps if required.
- 4.Automated AnalysisTemplate specification: Add the automated analysis specification as defined in the previous steps in the AnalysisTemplate.
Sample rollout.yaml configuration with automated analysis specification is available below, which you can use as a reference. In this example, automated analysis is configured after 25% and 75% of traffic shaping to the canary version. More samples of rollout.yaml configuration for blue/green and canary is available here.
kind: Rollout
metadata:
name: opsmx-demo
spec:
replicas: 1
revisionHistoryLimit: 2
selector:
matchLabels:
app: opsmx-demo
workloadRef:
apiVersion: apps/v1
kind: Deployment
name: opsmx-demo-deployment
strategy:
canary:
canaryService: opsmx-demo-canary-svc
stableService: opsmx-demo-stable-svc
trafficRouting:
nginx:
stableIngress: opsmx-demo-ing
steps:
- setWeight: 25
- pause: { duration: 60s }
- analysis:
templates:
- templateName: opsmx-analysis-template
args:
- name: canary-hash
valueFrom:
podTemplateHashValue: Latest
- name: baseline-hash
valueFrom:
podTemplateHashValue: Stable
- setWeight: 75
- pause: { duration: 60s }
- analysis:
templates:
- templateName: opsmx-analysis-template
args:
- name: canary-hash
valueFrom:
podTemplateHashValue: Latest
- name: baseline-hash
valueFrom:
podTemplateHashValue: Stable
Finally, ensure that you have completed all the necessary platform setup steps, including creating the Kubernetes secret for the cluster and the service account per namespace. This will allow you to see the applications in the ISD UI and the integrations created in the setup page.
Make sure that all the following relevant files are present in the same Git folder.
- Rollout and supporting objects for your application.
- AnalysisTemplate that refers to log and/or metric template, profile-secret, provider-config
- Log and/or metric templates
- Profile secret
- Provider config
To check that everything is in place, you can use the sample output as a reference. In this example, the Kubernetes secret and the service account definitions required for the analysis job are located in the same folder as the application, but this is not a strict requirement.
During a rollout, automated analysis will be triggered automatically as part of the progressive delivery process by updating the rollout.yaml file for your service. The status of the rollout can be monitored using the Argo Rollout extension or the Rollout dashboard. Refer to the image below.
The analysis will be carried out based on the configuration specified in the AnalysisTemplate and a report will be generated after each analysis performed during the progressive rollout. Usually, the analysis is conducted after each progressive rollout step. If the rollout is successful, the new version will be deployed, and users can review the report for any identified risks.
The ISD platform produces one or multiple analysis reports per rollout, as defined in the rollout.yaml. The overall score determines whether the rollout continues or stops.
The report includes a detailed analysis of logs and metrics between the baseline and the new release. Refer to the images below. To understand the log and metric analysis and provide guided input, refer to the user guide here.
The ISD platform offers a sophisticated machine learning-based analysis tool for your Argo Rollout that significantly reduces the risk during the progressive release of new applications.
ISD automated analysis utilizes log and metric data sources to automatically evaluate and identify potential risks in new releases, streamlining the analysis process and reducing the potential for errors.
Last modified 3mo ago