Canary Scoring
Last updated
Last updated
Canary scoring works with the principle of comparing the log files of two different releases. The log files of the v1 release also known as the baseline is compared with the log files of the v2 release also known as the new release/canary. In this scoring method, the two log files are compared to figure out the unexpected events (CRITICAL ERROR,ERROR, WARNING) and determine the shift from v1 to v2. The lesser the shift in v2 the closer the score is to 100. Canary scoring works best when the size of v1 and v2 logs are almost the same and when log files are large in size.
Canary scoring is done based on the following factors:
For every unexpected cluster present in v2 and not in v1, score of the canary is reduced
For all unexpected clusters present in both v1 and v2, the score of the canary is reduced only when the number of logs of that cluster are higher in v2.
Canary score also takes into account errors in v1 by introducing a normalization factor.
Score Reduction is higher at higher sensitivity levels.
Score Reduction is higher for clusters with higher criticality.
By default, canary scoring will give a score of 0 (fail) whenever there is a critical error.
To set the Canary Scoring algorithm follow the steps below:
Click on "New Application" button to create a new application as shown below:
2. Following application details appears as shown below:
3. In the above form after adding the application details you need to add. To know more on how to add an application click here.
4. After you add the application details, you need to add the services to the application. To know more on how to add an application click here.
5. Next, you get the option to select the type of service you want to add. The options are:
Approval
Verification
Test Verification
Policy
Refer to the image below:
6. To add a log template select Verification or Test Verification type. The following options appear and click "+" Symbol:
7. Select Canary Scoring from the drop-down while creating new Log Template as shown below:
