Configuration changes for Kafka Addition

To make use of Kafka for internal communication among ISD services, please follow the steps given below:

1. Ensure that ISD consumes events from Spinnaker via webhook and not rabbitMQ.

2. If auto topic creation is not enabled then create the following topics in kafka-cluster.

1. Append the topic prefixes and suffixes before and after each topic. Final created topic name should be “TOPIC_PREFIX.topic_name.TOPIC_SUFFIX”. Notice the dot between prefix, suffix and the topic. Example - “dev-tools.dataclean.echo-events-preview-saas-audit.uw2”

2. Make sure to add Publish.id, Consumer.id and group for each of the events.

Topic Name	Publisher	Consumer	Type of events
echo-events-preview-saas-audit	Sapor service	Audit Service	Audit events
isd-events-isd-visibility-service	Sapor Service	Visibility service	Pipeline execution failure events
isd-events-cd-route-info	Sapor service	Visibility, audit and gate service	Apache Camel route events*
isd-events-sapor-service	Gate, visibility, audit and gate service	Sapor service	Apache Camel Route events*
isd-userLoginDetails	Gate service	Sapor service	User Login Details events

Service name	Consumer Group Name
Audit service	oes-audit-consumer-group
Visibility Service	oes-visibility-consumer-group
Gate service	oes-gate-consumer-group
Sapor service	oes-sapor-consumer-group

1. Do the following config changes under the message-broker section, for audit-service, oes-sapor, visibility-service and gate service in their respective ConfigMaps.

message-broker:
  enabled: true
  endpoint:
     name: kafka 
  bootstrap-address: <bootstrap-address>:9092
  security:
	protocol: <SASL_PLAINTEXT/SASL_SSL>
  sasl:
	mechanism: <SCRAM-SHA-256/PLAIN/AWS_MSK_IAM/OAUTHBEARER>
	username: <SASL username>
	password: <SASL password>
  access-key-id: <Access Key ID for AWS IAM and OAUTHBEARER>
  secret-key: <Secret key for AWS IAM and OAUTHBEARER>

1. ISD supports only custom use cases of SASL auth and AWS IAM auth for Kafka.

2. The above configuration varies depending on the type of authentication used.