How to loop and pass parameter for long table list in Python

Question

I have a table that has more than 3,000+ tables and when I generate the DDLS for all of them, it gives me an error of " /user/hive Argument list to long".

I am trying to edit my code to scan through the first 1,000 first and then the next 1,000, and then the remaining. That way the argument can be split.

Here is my code:

 hiveDBName=$1;


 showcreate="show create table "
 showpartitions="show partitions "
 terminate=";"
 tables=`hive -e "use $hiveDBName;show tables;"`
 tab_list=`echo "${tables}"`

  for list in $tab_list
  do
         echo "Generating table script for " #${hiveDBName}.${list}
          showcreatetable=${showcreatetable}${showcreate}${hiveDBName}.${list}${terminate}

        done

        echo " ====== Create Tables ======= : "# $showcreatetable

 #Creates a filter ddls
 hive -e "use $hiveDBName; ${showcreatetable}"> a.sql
 #Removes the Warn: from the file
 grep -v "WARN" a.sql > /home/hive_ddls/${hiveDBName}_extract_all_tables.sql

 echo "Removing Filter File"
 #Remove Filter file
 rm -f a.sql

#Puts a ; after each create table statement in the document 
sed -i '/transient/s/$/;/' "/home/hive_ddls/${hiveDBName}_extract_all_tables.sql"

Is there any way to put a for loop in above script so that it splits the output by the thousands for it to run?

Possible duplicate of [Argument list too long error for rm, cp, mv commands](https://stackoverflow.com/questions/11289551/argument-list-too-long-error-for-rm-cp-mv-commands) — wjandrea, Jul 30 '19 at 15:57
after viewing that link, it did help a little but i still am confused on how to incorporate that inside my code — James Davinport, Jul 30 '19 at 16:15
"I am a table that has more than 3,000+ tables" what does it mean? — yoga, Jul 30 '19 at 17:38
If you want to do this in python you can easily split a list into 1000s by indexing, for example to get the first 1000 items `my_list[:1000]`. — CMMCD, Jul 30 '19 at 19:28

How to loop and pass parameter for long table list in Python

0 Answers0