# Configuration changes for Kafka Addition

To make use of Kafka for internal communication among ISD services, please follow the steps given below:

1. Ensure that ISD consumes events from Spinnaker via webhook and not rabbitMQ. 
2. If auto topic creation is not enabled then create the following topics in kafka-cluster.

{% hint style="info" %}

1.  Append the topic prefixes and suffixes before and after each topic. Final created topic name should be “TOPIC\_PREFIX.topic\_name.TOPIC\_SUFFIX”. Notice the dot between prefix, suffix and the topic.\
   \
   Example - “dev-tools.dataclean.echo-events-preview-saas-audit.uw2”
2. Make sure to add Publish.id, Consumer.id and group for each of the events.
   {% endhint %}

| Topic Name                        | Publisher                                | Consumer                           | Type of events                    |
| --------------------------------- | ---------------------------------------- | ---------------------------------- | --------------------------------- |
| echo-events-preview-saas-audit    | Sapor service                            | Audit Service                      | Audit events                      |
| isd-events-isd-visibility-service | Sapor Service                            | Visibility service                 | Pipeline execution failure events |
| isd-events-cd-route-info          | Sapor service                            | Visibility, audit and gate service | Apache Camel route events\*       |
| isd-events-sapor-service          | Gate, visibility, audit and gate service | Sapor service                      | Apache Camel Route events\*       |
| isd-userLoginDetails              | Gate service                             | Sapor service                      | User Login Details events         |

| Service name       | Consumer Group Name           |
| ------------------ | ----------------------------- |
| Audit service      | oes-audit-consumer-group      |
| Visibility Service | oes-visibility-consumer-group |
| Gate service       | oes-gate-consumer-group       |
| Sapor service      | oes-sapor-consumer-group      |

3. Do the following config changes under the message-broker section, for audit-service, oes-sapor, visibility-service and gate service in their respective ConfigMaps.

```
message-broker:
  enabled: true
  endpoint:
     name: kafka 
  bootstrap-address: <bootstrap-address>:9092
  security:
	protocol: <SASL_PLAINTEXT/SASL_SSL>
  sasl:
	mechanism: <SCRAM-SHA-256/PLAIN/AWS_MSK_IAM/OAUTHBEARER>
	username: <SASL username>
	password: <SASL password>
  access-key-id: <Access Key ID for AWS IAM and OAUTHBEARER>
  secret-key: <Secret key for AWS IAM and OAUTHBEARER>

```

{% hint style="success" %}

1. ISD supports only custom use cases of SASL auth and AWS IAM auth for Kafka.
2. The above configuration varies depending on the type of authentication used.<br>
   {% endhint %}

### AWS MSK Authentication

{% hint style="info" %}
This section is specific to Amazon AWS usecase : <https://docs.aws.amazon.com/msk/latest/developerguide/msk-password-tutorial-connect.html>
{% endhint %}

For authentication to MSK in AWS, to set ‘sasl.mechanism’ as ‘SCRAM\_SHA-512’ follow the steps given below:<br>

1. Create the ‘jks’ file as mentioned in AWS documentation and mount that jks file to “/opsmx/conf/kafka.client.truststore.jks”.&#x20;
2. Set JAVA\_OPTS and KAFKA\_OPTS  as env variable in deployment of the services. Check out the AWS documentation as to what values we need to set.
3. Set the service’s config map as shown below:

```
. . .
message-broker:
  enabled: true
  endpoint:
     name: kafka 
  bootstrap-address: <bootstrap-address>:9092
  topic-prefix: <the common prefix to be appended to all topics>
  topic-suffix: <the common suffix to be appended to all topics>
  security:
	protocol: SASL_SSL
  sasl:
	mechanism: SCRAM-SHA-512
  ssl:
 	truststore-location: /opsmx/conf/kafka.client.truststore.jks
. . .

```

{% hint style="info" %}
SASL username and password are mentioned in the above config. Internally the services are configured to use ‘ScramLoginModule’, so we don’t need to create ‘users\_jaas.conf’ file as mentioned in AWS documentation for MSK auth.
{% endhint %}

### Mounting JKS file

1. Prepare the certificate file of kafka server as a configMap for use by ISD.&#x20;
2. Create a configMap yaml file and insert the certificate.

```
apiVersion: v1
kind: ConfigMap
metadata:
  name: oes-cacerts-cm
binaryData:
  cacerts: |
       <your certificate body here>
  oes-cacerts-cm.yaml
```

3. Apply the file in the ISD cluster:

```
kubectl -n <namespace> apply -f oes-cacerts-cm.yaml
```

4. Now mount a volume so the certificate becomes usable by the services connecting to kafka.&#x20;
5. Edit deployment of the services to insert the below configs in volumes and volumeMounts section. The subPath parameter will have the value of the certificate's name as mentioned in the configMap.

```
....
volumeMounts:
           - name: oes-cacerts-volume
             mountPath: /opsmx/conf/kafka.client.truststore.jks
             subPath: kafka.client.truststore.jks
....
volumes:                               :
       - name: oes-cacerts-volume
         configMap:
           name: oes-cacerts-cm
           items:
             - key: cacerts
               path: kafka.client.truststore.jks
           defaultMode: 420
....

```

6. Save the deployments.

<br>