7

I thought Hive lineage was not available, but after some research I have found that it can be enable. Some of the things I found while searching was enabling its lineage via either Cloudera Manager or IBM Infosphere, which I am not interested in. Finally found a tag that was supposed to enable it:

<property>
    <name>hive.exec.post.hooks</name>
    <value>org.apache.hadoop.hive.ql.hooks.LineageLogger</value>
  </property>

The mentioned tag had to be written in the hive-site.xml file, and was supposed to write the lineage in the directory /var/log/hive/lineage. I said supposed because I haven't found anything in that directory, in fact, it wasn't even created. I am currently using Hive-server 3.12

Pablo Ochoa
  • 77
  • 1
  • 12
  • Did you enable it via Cloudera Manager: https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cn_admcfg_impala_lineage_log.html – Matt Andruff May 02 '22 at 14:09
  • I don't have a Cloudera Manager. Anyways, Hive capability of enabling lineage should be independent of it, right? – Pablo Ochoa May 03 '22 at 07:20
  • It is independent. When you enabled it and restarted were there any errors in the hive log? can you post the relevant portions in your question? – Matt Andruff May 03 '22 at 12:12
  • There are no errors. I have tried to create a table from another to see if the lineage would be created (create table x as select * from y;) but no errors have appeared neither the lineage. – Pablo Ochoa May 04 '22 at 08:19
  • @PabloOchoa my 2 cents, did you try to run the hive or create directory manually with admin user? may be it is suppressing since directory is not available ? – Girish501 May 09 '22 at 09:13
  • @Girish501 I have just tried creating the directory /var/log/hive/lineage as you have suggested. I entered beeline and created a table from another (create table x as select * from y;), but when I went back to the directory it was empty again. I have also tried adding the property "lineage_event_log_dir" to erase any possibility (although /var/log/hive/lineage is the default directory), but still got nothing. – Pablo Ochoa May 09 '22 at 09:39

0 Answers0