Prepare for Apache Kafka integration

Before ingesting Apache Kafka metadata into Dawiso, prepare your account for authentication by configuring a technical Apache Kafka user and granting it the necessary permissions.

Dawiso Cloud currently supports the traditional authentication method using credentials.

Create a technical user with a JAAS file

To enable authentication, you need to create a JAAS configuration file and provide it to the Kafka broker as a JVM parameter when it starts. This file defines the credentials used by Kafka for login. For more details, refer to the official Kafka documentation.

1. Create a new JAAS configuration file with a descriptive name (for example, kafka_server_jaas.conf).

2. In the configuration file, include the following snippet:

   KafkaServer {
       org.apache.kafka.common.security.plain.PlainLoginModule required
       username="dawiso_technical_user"
       password="your_password";
   };

   • You can choose your own username.
   • Make sure to replace your_password with the actual password for your user.

3. Save the configuration file.

4. Pass the file path to Kafka as a JVM parameter when starting the broker:

   -Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf

Required permissions

Now, grant the necessary permissions to the newly created account (in this case, dawiso_technical_user).

1. Using your CLI, execute the following scripts from the location of your Kafka binaries.

   ./kafka-acls.sh \
   --bootstrap-server <broker-host>:<broker-port> \
   --add --allow-principal User:dawiso_technical_user \
   --operations Describe,Read \
   --topic '*' \
   --resource-pattern-type 'literal'

   • Make sure to replace the values of <broker-host> and <broker-port> to match your Kafka cluster deployment.

2. Then, execute this script to select which consumer groups you wish to grant permissions to:

   
   # Example consumer groups list
   consumer_groups=(
       "group1"
       "group2"
       "group3"
   )
   
   for group in "${consumer_groups[@]}"
   do
   
       # Grant Describe permission
       kafka-acls.sh --bootstrap-server <broker-host>:<broker-port> --add --allow-principal User:dawiso_technical_user --operation Describe --group "$group" --resource-pattern-type group
   
       # Grant DescribeConfigs permission
       kafka-acls.sh --bootstrap-server <broker-host>:<broker-port> --add --allow-principal User:dawiso_technical_user --operation DescribeConfigs --group "$group" --resource-pattern-type group
   done

   • Replace group1-3 with the real group names.
   • Make sure to replace the values of <broker-host> and <broker-port> to match your Kafka cluster deployment.