Canary Scoring
Last updated
Last updated
Canary scoring works with the principle of comparing the log files of two different releases. The log files of the v1 release also known as the baseline is compared with the log files of the v2 release also known as the new release/canary. In this scoring method, the two log files are compared to figure out the unexpected events (CRITICAL ERROR,ERROR, WARNING) and determine the shift from v1 to v2. The lesser the shift in v2 the closer the score is to 100. Canary scoring works best when the size of v1 and v2 logs are almost the same and when log files are large in size.
Canary scoring is done based on the following factors:
For every unexpected cluster present in v2 and not in v1, score of the canary is reduced
For all unexpected clusters present in both v1 and v2, the score of the canary is reduced only when the number of logs of that cluster are higher in v2.
Canary score also takes into account errors in v1 by introducing a normalization factor.
Score Reduction is higher at higher sensitivity levels.
Score Reduction is higher for clusters with higher criticality.
By default, canary scoring will give a score of 0 (fail) whenever there is a critical error.
To set the Canary Scoring algorithm follow the steps below:
User can select the Canary Scoring algorithm while creating a "Log Template" for the verification gate. Users can set the Canary Scoring algorithm after creating a pipeline to the application.
If you haven’t created an application already, refer here.
If you haven’t created your pipelines yet, refer here.
To know more about how to create a Log Template, refer here.
After creating a pipeline, go to "Pipeline Builder" page and select your pipeline and then click "Add Stage".
Next, you get the option to select the type of stage. Select the "Verification Gate" from the drop down menu.
After selecting Verification gate, the following options appear and click "+" Symbol to create a log template:
The New Log Template form appears as shown below:
Select Canary Scoring from the drop-down while creating a new Log Template as shown below:
