Apache Hive

Apache Hive is the SQL layer of a Hadoop data lake. It gives files in HDFS or object storage a schema, a name, and an owner. Dawiso connects to HiveServer2 and catalogs this layer: databases, tables, views including materialized views, their columns, partitions, and permanent user-defined functions.

Each table also carries the storage details that are otherwise visible only in a Beeline session, such as the SerDe, the input and output formats, the location, and the partitioning and bucketing scheme. View lineage shows which tables and views each view reads from.

Supported versions

Hive version	Support
4.x	Supported. Validated end to end against Hive 4.0.1, including every object type, attribute, and lineage relation on this page.
3.x	Supported. Verified against Hive 3.1.3. A few attributes behave differently, see Differences on Hive 3.x.
2.3.0 to 2.x	Supported, but not validated against a live Hive 2 server. Materialized views are found through the table listing.
2.2 and older	Not supported. The scan stops with an error.

Differences on Hive 3.x

Hive 3.x is supported, but four items differ from Hive 4.x. These differences were measured on a running Hive 3.1.3 server.

Item	Behavior on Hive 3.x
Managed Location of a database	Hive 3 doesn’t report it, so the attribute stays empty.
Lineage of materialized views	Hive 3 doesn’t publish the source tables of a materialized view. Dawiso derives them from the view’s query text instead.
Partition listing	Hive 3 can’t limit a partition listing on the server. Dawiso reads the full list and then applies your per-table cap, which is slower on tables with many partitions.
Transactional (ACID) tables and views over them	Cataloged, but not tested end to end on Hive 3.x.

Endpoints that are not Apache Hive

Some services accept HiveServer2 connections and identify themselves as HiveServer2, but they aren’t Apache Hive. Spark-based services answer DESCRIBE in a different format, so scanning them would fill your catalog with wrong metadata. Dawiso asks the server which product it is and stops with an error when the answer isn’t Apache Hive.

Endpoint	Supported	Reason
Spark Thrift Server	No	It’s Spark, not Hive.
Databricks in hive_metastore mode	No	It’s Spark, not Hive. Use the Databricks integration instead.
AWS Athena	No	Athena isn’t a HiveServer2 endpoint.
Cloudera CDH 6.x	No	CDH 6.x ships Hive 2.1, which is older than 2.3.0.
Cloudera CDP 7.1, Azure HDInsight 5.1	Not validated	Both run Hive 3.1. HDInsight exposes HiveServer2 only over HTTPS on port 443.

Default hierarchy

The chart below shows the organizational structure of Apache Hive objects. Objects are represented in green and folders are in yellow.

Each database holds three folders: Tables, Views, and Functions. Each table holds its columns and a Partitions folder. Each view holds its columns.

The integration also includes a Native hierarchy that mirrors Hive’s own structure without the folder level.

Materialized views are cataloged as views with the Materialized attribute set, not as a separate object type.

Data lineage

Dawiso derives lineage from view definitions. Every view gets a lineage relation from each table or view its query reads. The relations appear on the lineage diagram.

Lineage relation	Derived from
Table or view → view	The view’s expanded definition. Hive has already fully qualified every table reference in it, so each source resolves to one object.
Table → materialized view	On Hive 4, the source tables that Hive records for the materialized view. On Hive 3, the view’s query text.

Chains are followed: a view built on another view built on a table traces back to the table. A source outside the scan scope, such as a table in a database you didn’t include, is dropped instead of being drawn as a broken link.

Lineage that Hive can’t provide

• No column-level lineage. Lineage connects tables and views, not individual columns.
• No lineage from query history. Hive keeps no queryable query history: there is no history table, view, or API to read. The metastore notification log records that an object changed, one object per event, without a source and a target. It is also switched off by default.

If your platform runs Apache Atlas with the Hive hook, which is common on Cloudera and Hortonworks platforms, query-level lineage is stored in Atlas. Contact your Customer Success Account Manager to discuss it.

Object types

Dawiso ingests the following objects from Apache Hive. The Hive comment on a database, table, view, or column becomes its Scanned Description, including comments that contain tabs, line breaks, or quotes. Hive returns an empty comment and a missing comment the same way, so the catalog can’t tell them apart.

Database

A Hive database. Hive uses “database” and “schema” interchangeably.

Maps to Hive Database object type.

Attribute in Dawiso	Value
Name	Database name
Scanned Description	Database comment
Location	Storage location of the database
Owner	Database owner
Owner Type	USER, ROLE, or GROUP
Managed Location	Location used for managed tables. Hive 4 only

Table

A managed, external, or transactional (ACID) table.

Maps to Hive Table object type.

Attribute in Dawiso	Value
Name	Table name
Type	MANAGED_TABLE or EXTERNAL_TABLE. See Type and Raw Table Type
Scanned Description	Table comment
Location	Storage location of the table’s data
Input Format	Hadoop input format class
Output Format	Hadoop output format class
SerDe Library	Serializer/deserializer class
Owner	Table owner
Owner Type	USER, ROLE, or GROUP
Created at	Creation time
Last Used	Last access time recorded by Hive
Transactional (ACID)	Whether the table is transactional
Transactional Properties	default (full ACID) or insert_only
Partition Keys	Partitioning columns, in order
Partition Count	Number of partitions
Row Count	Row count from the last ANALYZE. See Row counts and sizes
Raw Data Size	Uncompressed data size from the last ANALYZE
Total Size (bytes)	Size on storage from the last ANALYZE
File Count	Number of files that store the table
Compressed	Whether the table is stored compressed
Bucket Count	Number of buckets
Bucket Columns	Bucketing columns
Raw Table Type	Table type exactly as Hive reports it
Translated to External	Whether Hive 4 converted the table to an external table

Type and Raw Table Type

Hive 4 changed what the table type means. Its metastore turns a plain CREATE TABLE into an external table and marks it as translated to external. After an upgrade from Hive 3 to Hive 4, every managed table therefore reports EXTERNAL_TABLE, without any DDL change. The same tables report MANAGED_TABLE on Hive 3.1.3 and EXTERNAL_TABLE on Hive 4.0.1.

Dawiso keeps the catalog stable across the upgrade:

• Type reports MANAGED_TABLE for the tables Hive converted.
• Raw Table Type keeps Hive’s literal value.
• Translated to External shows which tables Hive converted.

A table created with CREATE EXTERNAL TABLE reports EXTERNAL_TABLE in all three cases.

Table Column

A column of a table. Partitioning columns are cataloged as columns too, with Partition Key set.

Maps to Hive Table Column object type.

Attribute in Dawiso	Value
Name	Column name
Data Type	Full data type, including complex types
Scanned Description	Column comment
Ordinal position	Position of the column in the table, counting from 0
Partition Key	Whether the column is a partitioning column

A struct, array, or map column is one column whose Data Type holds the full type, for example struct<a:int,b:string>. Sub-fields aren’t cataloged as separate columns.

View

A virtual view or a materialized view.

Maps to Hive View object type.

Attribute in Dawiso	Value
Name	View name
Scanned Description	View comment
Definition (SQL)	The view’s query in the expanded form that Hive stores
Type	VIRTUAL_VIEW or MATERIALIZED_VIEW
Materialized	Whether the view is a materialized view
Rewrite Enabled	Whether Hive can rewrite queries to use the materialized view
Owner	View owner
Created at	Creation time
Column Count	Number of columns
Location	Storage location. Materialized views only
Row Count	Row count from the last ANALYZE. Materialized views only
Total Size (bytes)	Size on storage. Materialized views only

Hive stores two forms of a view’s query: the text you wrote, and an expanded text in which Hive fully qualifies every table reference. Dawiso stores the expanded form. It is less readable, but it is unambiguous, which makes the view lineage reliable.

View Column

A column of a view.

Maps to Hive View Column object type.

Attribute in Dawiso	Value
Name	Column name
Data Type	Full data type
Scanned Description	Column comment
Ordinal position	Position of the column in the view, counting from 0

Partition

One partition of a partitioned table. Partitions are scanned only when you turn on Scan partitions in the data source. See Create Apache Hive data source.

Maps to Hive Partition object type.

Attribute in Dawiso	Value
Name	The partition in readable form, for example year=2025/month=03
Partition Spec	The partition specification exactly as Hive reports it
Partition Values	The partition’s values, one per partitioning column

Function

A permanent user-defined function registered in a database.

Maps to Hive Function object type.

Attribute in Dawiso	Value
Name	Function name
Java Class	Class that implements the function
Function Type	PERSISTENT for permanent functions
Owner	Function owner

Hive’s built-in functions aren’t cataloged. They are the same on every Hive installation, 484 of them on Hive 4.0.1. Temporary functions exist only for the lifetime of one session and can’t be scanned.

Row counts and sizes stay 0 until ANALYZE runs

Hive doesn’t maintain table statistics automatically. Row Count, Raw Data Size, Total Size (bytes), and File Count come from statistics that Hive stores only after someone runs:

ANALYZE TABLE sales_dw.fact_orders COMPUTE STATISTICS;

Until then, these attributes are 0. After the data changes, they are out of date. This is how Hive works, and no Dawiso setting changes it. Read these attributes as the last computed statistics, not as a current row count.

To keep them populated and current, schedule ANALYZE in the pipeline that loads each table. Hive’s automatic statistics gathering covers only data written through Hive and doesn’t produce row counts for external data.

Limitations

• Hive 2.2 and older are refused. See Supported versions.
• Kerberos authentication isn’t supported. See Prepare for Apache Hive integration.
• No column-level lineage and no query-history lineage. See Lineage that Hive can’t provide.
• Partitions are off by default and capped per table when you turn them on, because one table can have more than 100,000 partitions.
• Only permanent user-defined functions are cataloged, not built-in or temporary functions.
• Primary key, foreign key, and NOT NULL constraints aren’t cataloged, even though Hive can declare them.
• Table DDL isn’t stored. View definitions are stored in full. Table definitions aren’t, because reading them would double the scan time on a large platform.
• Iceberg tables (Hive 4) aren’t covered and haven’t been validated.
• Hive 4 data connectors aren’t cataloged as such. Tables federated from another database through a data connector appear as ordinary external tables.
• Tables that Spark stores in its own format can show a placeholder column. In a metastore shared with Spark, a table that Spark saved in a format without a Hive SerDe, such as Delta or USING csv, can appear with a single column col of type array<string> instead of its real columns. Hive itself reports that placeholder. This case hasn’t been reproduced against a live server.
• No “Open in Hive” link. The HiveServer2 web interface has no page for an individual table, view, or column.
• A database the scanning account can’t see is missing from the catalog without an error. See Prepare for Apache Hive integration before the first scan.

Setup

1. Prepare for Apache Hive integration: version, transport mode, authentication, and the scanning account’s visibility
2. Create Apache Hive connection
3. Create Apache Hive data source