Confluent Kafka

Confluent Kafka is a fully managed, cloud-native event streaming platform based on Apache Kafka. It provides a robust, scalable, and highly available infrastructure for building event-driven applications and real-time data pipelines. Confluent Kafka offers enhanced features over Apache Kafka, including advanced security, multi-region clusters, and a comprehensive ecosystem of fully managed connectors and tools for easier data integration and stream processing.

Default hierarchy

In the chart below, you will find the organizational structure of Confluent Kafka objects. Objects are represented in green and folders are in yellow.

Object types

Dawiso ingests the following objects from Confluent Kafka:

Cluster

Source Attribute	Target Attribute in Dawiso
ClusterId	Cluster ID
ClusterName	Name
BootstrapServers	Bootstrap Servers

Topic

Source Attribute	Target Attribute in Dawiso
TopicName	Name
PartitionsCount	Number of Partitions
ReplicationFactor	Replication Factor
CleanupPolicy	Cleanup Policy
RetentionMs	Retention Time (ms)
RetentionBytes	Retention Size (bytes)
MaxMessageBytes	Max Message Size (bytes)
MinInsyncReplicas	Minimum In-Sync Replicas
SegmentBytes	Segment Size (bytes)
SegmentMs	Segment Time (ms)
MessageTimestampType	Message Timestamp Type
CompressionType	Compression Type

Consumer Group

Source Attribute	Target Attribute in Dawiso
GroupId	Group ID
State	State
MemberCount	Number of Members
SubscribedTopics	Subscribed Topics
TotalLag	Total Lag

Consumer Group-Topic Relationship

Source Attribute	Target Attribute in Dawiso
ConsumerGroupKey	Consumer Group Reference
TopicKey	Topic Reference
Partition	Partition
CurrentOffset	Current Offset
Lag	Lag

Schema Registry

Source Attribute	Target Attribute in Dawiso
RegistryName	Name
Endpoint	Endpoint URL

Schema

Source Attribute	Target Attribute in Dawiso
Subject	Name
SchemaType	Schema Type
Compatibility	Compatibility Mode

Schema Version

Source Attribute	Target Attribute in Dawiso
Version	Version Number
SchemaDefinition	Schema Definition

Schema Field

Source Attribute	Target Attribute in Dawiso
FieldName	Name
FieldType	Data Type
IsNullable	Is Nullable
DefaultValue	Default Value
Description	Description