Canary Scoring
Last updated
Last updated
ISD automates rollout verification with intelligence using machine learning algorithms and leverages Natural Language Processing (NLP) and Statistical analysis to identify risks in new releases compared to the base releases.
Canary scoring works by automating the comparison of log files of two different releases, the baseline release (or current release) & the canary release (or the new release). The outcome of the comparison is to capture the unexpected events that goes into the new release as compared to the current release, viz., Critial errors, warning etc. Typically a score of 100 means the log files of both the releases are the same & the new release is absolutely safe to proceed with. The more the difference in the log files the lesser the score will be. (Note: Scoring algorithms works the best with larger log files).
Canary scoring works with the principle of comparing the log files of two different releases. The log files of the v1 release also known as the baseline compared with the log files of the v2 release also known as the new release/canary. In this scoring method, the two log files are compared to figure out the unexpected events (CRITICAL ERROR, ERROR, WARNING) and determine the shift from v1 to v2. The lesser the shift in v2 the closer the score is to 100. Canary scoring works best when the size of v1 and v2 logs are almost the same and when log files are large in size.
Canary scoring is done based on the following factors:
For every unexpected cluster present in v2 and not in v1, score of the canary is reduced
For all unexpected clusters present in both v1 and v2, the score of the canary is reduced only when the number of logs of that cluster is higher in v2.
The Canary score also considers errors in v1 by introducing a normalization factor.
Score Reduction is higher at higher sensitivity levels.
Score Reduction is higher for clusters with higher criticality.
By default, canary scoring will give a score of 0 (fail) whenever there is a critical error.
To set the Canary Scoring Algorithm follow the steps below:
Users can select the Canary Scoring algorithm while creating a "Log Template". Users can set the Canary Scoring algorithm after creating an application.
If you haven’t created an application already, click here.
To know more about how to create a Log Template, refer here.
From the application dashboard, click "Setup" and click “Analysis Templates” and then click "+New Template" button.
After clicking “+New Template” button, two options appear for you to choose the type of template you want to create. Select the “Log Template” from the available options as shown in the below image.
Click the Scoring Algorithm drop-down and select the Canary as shown in the image below:
