Configuration changes for Kafka Addition

To make use of Kafka for internal communication among ISD services, please follow the steps given below:

1. Ensure that ISD consumes events from Spinnaker via webhook and not rabbitMQ.

2. If auto topic creation is not enabled then create the following topics in kafka-cluster.

1. Append the topic prefixes and suffixes before and after each topic. Final created topic name should be “TOPIC_PREFIX.topic_name.TOPIC_SUFFIX”. Notice the dot between prefix, suffix and the topic. Example - “dev-tools.dataclean.echo-events-preview-saas-audit.uw2”

2. Make sure to add Publish.id, Consumer.id and group for each of the events.

Topic Name	Publisher	Consumer	Type of events
echo-events-preview-saas-audit	Sapor service	Audit Service	Audit events
isd-events-isd-visibility-service	Sapor Service	Visibility service	Pipeline execution failure events
isd-events-cd-route-info	Sapor service	Visibility, audit and gate service	Apache Camel route events*
isd-events-sapor-service	Gate, visibility, audit and gate service	Sapor service	Apache Camel Route events*
isd-userLoginDetails	Gate service	Sapor service	User Login Details events

Service name	Consumer Group Name
Audit service	oes-audit-consumer-group
Visibility Service	oes-visibility-consumer-group
Gate service	oes-gate-consumer-group
Sapor service	oes-sapor-consumer-group

1. Do the following config changes under the message-broker section, for audit-service, oes-sapor, visibility-service and gate service in their respective ConfigMaps.

message-broker:
  enabled: true
  endpoint:
     name: kafka 
  bootstrap-address: <bootstrap-address>:9092
  security:
	protocol: <SASL_PLAINTEXT/SASL_SSL>
  sasl:
	mechanism: <SCRAM-SHA-256/PLAIN/AWS_MSK_IAM/OAUTHBEARER>
	username: <SASL username>
	password: <SASL password>
  access-key-id: <Access Key ID for AWS IAM and OAUTHBEARER>
  secret-key: <Secret key for AWS IAM and OAUTHBEARER>

1. ISD supports only custom use cases of SASL auth and AWS IAM auth for Kafka.

2. The above configuration varies depending on the type of authentication used.

AWS MSK Authentication

This section is specific to Amazon AWS usecase : https://docs.aws.amazon.com/msk/latest/developerguide/msk-password-tutorial-connect.html

For authentication to MSK in AWS, to set ‘sasl.mechanism’ as ‘SCRAM_SHA-512’ follow the steps given below:

1. Create the ‘jks’ file as mentioned in AWS documentation and mount that jks file to “/opsmx/conf/kafka.client.truststore.jks”.

2. Set JAVA_OPTS and KAFKA_OPTS as env variable in deployment of the services. Check out the AWS documentation as to what values we need to set.

3. Set the service’s config map as shown below:

. . .
message-broker:
  enabled: true
  endpoint:
     name: kafka 
  bootstrap-address: <bootstrap-address>:9092
  topic-prefix: <the common prefix to be appended to all topics>
  topic-suffix: <the common suffix to be appended to all topics>
  security:
	protocol: SASL_SSL
  sasl:
	mechanism: SCRAM-SHA-512
  ssl:
 	truststore-location: /opsmx/conf/kafka.client.truststore.jks
. . .

SASL username and password are mentioned in the above config. Internally the services are configured to use ‘ScramLoginModule’, so we don’t need to create ‘users_jaas.conf’ file as mentioned in AWS documentation for MSK auth.

Mounting JKS file

1. Prepare the certificate file of kafka server as a configMap for use by ISD.

2. Create a configMap yaml file and insert the certificate.

apiVersion: v1
kind: ConfigMap
metadata:
  name: oes-cacerts-cm
binaryData:
  cacerts: |
       <your certificate body here>
  oes-cacerts-cm.yaml

1. Apply the file in the ISD cluster:

kubectl -n <namespace> apply -f oes-cacerts-cm.yaml

1. Now mount a volume so the certificate becomes usable by the services connecting to kafka.

2. Edit deployment of the services to insert the below configs in volumes and volumeMounts section. The subPath parameter will have the value of the certificate's name as mentioned in the configMap.

....
volumeMounts:
           - name: oes-cacerts-volume
             mountPath: /opsmx/conf/kafka.client.truststore.jks
             subPath: kafka.client.truststore.jks
....
volumes:                               :
       - name: oes-cacerts-volume
         configMap:
           name: oes-cacerts-cm
           items:
             - key: cacerts
               path: kafka.client.truststore.jks
           defaultMode: 420
....

1. Save the deployments.