0

I have a use case where i have all my 4 tb of data in HBase tables that i have interrogated with HIVE tables .

Now i want to extract 5 k files out of this 30 tables that i have created in HIVE.

This 5K files will be created by predefined 5K queries. Can somebody suggest me what approach i should follow for this? Required time for this is 15 hrs .

Should i write java code to generate all this files .

File generation is fast .Out of 5k text files there are 50 files that takes around 35 minutes rest of all creates very fast .

I have to generate zipped file and have to send it to client using ftp.

Sudarshan kumar
  • 1,503
  • 4
  • 36
  • 83

1 Answers1

1

If I understand your question right, you can accomplish your task by first exporting the query results via one of methods from here : How to export a Hive table into a CSV file?, compressing the files in a zip archive and then FTP'ing them. You can write a shell script to automate the process.

Community
  • 1
  • 1
soulzcore
  • 71
  • 6