Object types, hierarchy, and attribute mappings for the Confluent Kafka integration.
Confluent Kafka is a fully managed, cloud-native event streaming platform based on Apache Kafka. It provides a robust, scalable, and highly available infrastructure for building event-driven applications and real-time data pipelines. Confluent Kafka offers enhanced features over Apache Kafka, including advanced security, multi-region clusters, and a comprehensive ecosystem of fully managed connectors and tools for easier data integration and stream processing.
Default hierarchy
In the chart below, you will find the organizational structure of Confluent Kafka objects. Objects are represented in green and folders are in yellow.
Object types
Dawiso ingests the following objects from Confluent Kafka:
Cluster
| Source Attribute | Target Attribute in Dawiso |
|---|
| ClusterId | Cluster ID |
| ClusterName | Name |
| BootstrapServers | Bootstrap Servers |
Topic
| Source Attribute | Target Attribute in Dawiso |
|---|
| TopicName | Name |
| PartitionsCount | Number of Partitions |
| ReplicationFactor | Replication Factor |
| CleanupPolicy | Cleanup Policy |
| RetentionMs | Retention Time (ms) |
| RetentionBytes | Retention Size (bytes) |
| MaxMessageBytes | Max Message Size (bytes) |
| MinInsyncReplicas | Minimum In-Sync Replicas |
| SegmentBytes | Segment Size (bytes) |
| SegmentMs | Segment Time (ms) |
| MessageTimestampType | Message Timestamp Type |
| CompressionType | Compression Type |
Consumer Group
| Source Attribute | Target Attribute in Dawiso |
|---|
| GroupId | Group ID |
| State | State |
| MemberCount | Number of Members |
| SubscribedTopics | Subscribed Topics |
| TotalLag | Total Lag |
Consumer Group-Topic Relationship
| Source Attribute | Target Attribute in Dawiso |
|---|
| ConsumerGroupKey | Consumer Group Reference |
| TopicKey | Topic Reference |
| Partition | Partition |
| CurrentOffset | Current Offset |
| Lag | Lag |
Schema Registry
| Source Attribute | Target Attribute in Dawiso |
|---|
| RegistryName | Name |
| Endpoint | Endpoint URL |
Schema
| Source Attribute | Target Attribute in Dawiso |
|---|
| Subject | Name |
| SchemaType | Schema Type |
| Compatibility | Compatibility Mode |
Schema Version
| Source Attribute | Target Attribute in Dawiso |
|---|
| Version | Version Number |
| SchemaDefinition | Schema Definition |
Schema Field
| Source Attribute | Target Attribute in Dawiso |
|---|
| FieldName | Name |
| FieldType | Data Type |
| IsNullable | Is Nullable |
| DefaultValue | Default Value |
| Description | Description |