Before you create an Apache Hive connection in Dawiso, check four things: the HiveServer2 version, its transport mode, its authentication method, and what the scanning account is allowed to see. The first three decide whether the scan connects. The fourth decides whether the scan is complete.
Connection prerequisites
- HiveServer2 runs Apache Hive 2.3.0 or later.
- The endpoint is Apache Hive, not Spark Thrift Server or Databricks in
hive_metastoremode. - HiveServer2 uses the
NONE,LDAP, orCUSTOMauthentication method. Kerberos isn’t supported. - A dedicated scanning account exists and can see every database you want to catalog.
- The machine that runs the scan can reach the HiveServer2 port.
Check the Hive version
Run the following query in Beeline against the HiveServer2 you plan to connect to:
SELECT version();Dawiso supports Hive 2.3.0 and later and stops with an error on anything older. Older HiveServer2 versions don’t escape tabs and line breaks in table and column comments, which corrupts the metadata before it leaves the server. Hive 1.x and 2.x reached end of life in 2024.
Some services answer on the HiveServer2 port and identify themselves as HiveServer2 without being Apache Hive, for example Spark Thrift Server and Databricks in hive_metastore mode. Dawiso detects them and stops with an error. For the full list, see Endpoints that are not Apache Hive.
Check the transport mode
HiveServer2 serves its protocol in either binary mode or HTTP mode. One HiveServer2 process serves only one mode, and the Dawiso connection must use the same one. Dawiso supports both. The mode is set in hive-site.xml:
<property>
<name>hive.server2.transport.mode</name>
<value>binary</value>
</property>| Transport mode | Default port | Also needed |
|---|---|---|
binary (the Hive default) | 10000 | Nothing |
http | 10001 | The HTTP path, cliservice unless changed in hive.server2.thrift.http.path |
HTTP mode often comes with one of these setups:
| Setup | Status in Dawiso |
|---|---|
| Kerberos, carried as SPNEGO over HTTP | Not supported |
| Apache Knox gateway, HTTPS with username and password, typically on port 8443 | Uses HTTP transport. Not validated |
| Azure HDInsight, HTTPS on port 443 only | Uses HTTP transport. Not validated |
If your cluster uses Kerberos, Knox, or HDInsight, contact your Customer Success Account Manager before you plan the rollout.
Check the authentication method
The Dawiso connection offers three authentication methods. Each one matches a value of hive.server2.authentication on the server.
| Server setting | Select in Dawiso | What the server checks |
|---|---|---|
NONE | None (SASL PLAIN) | Nothing. Any username and password are accepted |
LDAP | LDAP | Username and password against your directory |
CUSTOM, including PAM | Custom / PAM | Username and password through the server’s own plug-in |
KERBEROS | Not supported | Not applicable |
With hive.server2.authentication=NONE, HiveServer2 accepts any username with any password. A deliberately wrong password connects successfully, because the server never checks it. On such a server, a successful connection test proves only that HiveServer2 is reachable. It proves nothing about the credentials or about what the account can see.
Validated configurations
| Configuration | Status |
|---|---|
| Hive 4.0.1, binary transport, username and password | Validated end to end |
| Hive 4.0.1, HTTP transport, username and password | Validated with a full scan |
| Hive 3.1.3 | Verified |
| LDAP and Custom / PAM authentication | Not validated against a live directory-backed HiveServer2. They use the same username and password exchange as the validated configurations; only the server side differs |
| SSL/TLS, on either transport | Not validated end to end |
| Apache Knox, Azure HDInsight | Not validated |
| Kerberos | Not supported |
Create a scanning account
Create a dedicated account for Dawiso, such as an LDAP user or whatever identity your HiveServer2 authenticates. A dedicated account makes the scan’s access easy to audit and to revoke.
The scan runs only the following read-only commands:
| Command | Purpose |
|---|---|
SHOW DATABASES | Find databases |
SHOW TABLES IN <db> | Find tables, views, and materialized views |
SHOW VIEWS, SHOW MATERIALIZED VIEWS | Confirm the object classification |
DESCRIBE FORMATTED <db>.<object> | Read table, view, and column metadata |
SHOW CREATE TABLE <db>.<view> | Read view definitions, the source of lineage |
SHOW PARTITIONS <db>.<table> | Read partitions, when partition scanning is on |
SHOW FUNCTIONS, DESCRIBE FUNCTION EXTENDED | Read permanent user-defined functions |
The scan never runs SELECT against your data and never writes anything.
Give the scanning account visibility
Apache Ranger and Hive’s SQL Standard Based Authorization filter the output of SHOW commands to what the connecting account is authorized to see. A database the scanning account can’t see is missing from the result, without an error or a warning. An under-privileged account therefore produces a successful scan with an incomplete catalog.
Use both of these defenses:
- Write down the databases and objects the account is expected to see before the first scan. After the scan, check the catalog against that list. A database-level check alone won’t catch an object skipped inside a visible database. See What happens when the account can’t read an object.
- Grant visibility under the authorization model your cluster uses:
| Authorization model | What to grant |
|---|---|
None (hive.security.authorization.enabled=false) | Nothing. Every database is visible |
| SQL Standard Based Authorization | SELECT on the tables and views you want to catalog |
| Apache Ranger | A Hive policy for the scanning account with at least select on database=*, table=*, column=*. You can narrow it to the databases in scope |
Dawiso hasn’t validated these grants against an authorization-enabled HiveServer2. Confirm the exact grants with your Hadoop administrator, then check the first scan against your list of expected databases.
Check whether doAs is enabled
The hive.server2.enable.doAs setting changes what else the scanning account needs:
hive.server2.enable.doAs | What the scanning account needs |
|---|---|
true (the Hive default) | Hive acts as the scanning account when it reads storage, so the account also needs read access in HDFS to the table locations. Without it, Hive can list a table but deny its details |
false | Nothing more. Hive reads storage as the hive service user |
Confirm this setting with your cluster administrator before the first scan. It is the most common reason a scan that connects successfully returns fewer objects than expected.
What happens when the account can’t read an object
Some setups let the account list an object but not describe it, for example a Ranger policy narrower than the database, or doAs without HDFS access. Dawiso handles the two kinds of failure differently:
| What Hive returns | What Dawiso does |
|---|---|
| Permission denied for an object | Leaves the object out of the catalog and continues. The skipped objects are recorded in the scanner’s technical log, not in the ingestion log in Dawiso |
| Any other error, such as a timeout, a dropped connection, or a server error | Retries once. If the retry fails, the scan stops with an error and the catalog keeps the result of the previous scan |
Each scan replaces the previous one. An object that a scan skipped without a reason would disappear from Dawiso together with its descriptions, owners, and relations. A failed scan changes nothing, which is why Dawiso stops instead.
Allow network access
| Requirement | Detail |
|---|---|
| Reachability | The scan must reach HiveServer2 on its port: 10000 for binary, 10001 for HTTP, or the port of your gateway, such as 8443 for Knox or 443 for HDInsight |
| Host name | Use the fully qualified domain name of HiveServer2, not an IP address or a short name |
| Firewall | Open the HiveServer2 port to the machine that runs the scan |
Hadoop clusters are rarely reachable from the internet. In that case, create a private connection and run the scan inside your network with Dawiso Integration Runtime (DIR). For more information, see Connection types.
To ingest metadata from your on-premises sources, whitelist Dawiso’s IP address so Dawiso can connect to your internal sources (pull method). Whitelist the same IP for Dawiso Integration Runtime (DIR) to send data from your on-premises infrastructure to Dawiso Cloud (push method).
Contact your Account Manager to get the Dawiso IP address.
Review what the scan reads
Use this list for your security review:
- Catalog metadata only: names, comments, data types, storage formats, locations, owners, partitioning and bucketing, view definitions, and function classes.
- No table contents. The only row-related values are the statistics Hive already stores from an
ANALYZE. - No writes. No
CREATE, noINSERT, no temporary objects. - Credentials in table properties are removed. Hive 4 JDBC data connectors store the source database’s username and password in clear text in the table properties, where
DESCRIBE FORMATTEDprints them. Dawiso removes them when it reads the output, so they never reach your catalog. - Only documented table properties are cataloged. For the list, see Apache Hive.
Row Count and the size attributes read 0 for every table on which nobody has run ANALYZE TABLE ... COMPUTE STATISTICS. Dawiso doesn’t run ANALYZE for you, because it is a write operation and a heavy workload on a large table. Decide whether to schedule it before users ask. For details, see Apache Hive.
Collect the connection details
Collect the following before you create the connection:
| Item | Where to find it |
|---|---|
| HiveServer2 host name and port | Fully qualified domain name, not an IP address |
| Transport mode | hive.server2.transport.mode: binary or http |
| HTTP path | HTTP mode only. hive.server2.thrift.http.path, usually cliservice |
| Authentication method | hive.server2.authentication: NONE, LDAP, or CUSTOM |
| Username and password | The scanning account. The password is required for LDAP and Custom / PAM |
| TLS | hive.server2.use.SSL |
| Expected databases | Your written list, to check the first scan against |
| doAs | hive.server2.enable.doAs, to know whether HDFS access is needed |