I am trying out loading 8GB of CSV files (comprising approx 100 million records) into HDFS using Hive. I am doing this in batch by reading 10000 records at a time and loading them. It took approx 10 GB in HDFS.
This process did not succeed completely. It threw java.lang.OutOfMemoryError while loading last CSV file.
After this attempt to load file, I tried many times loading new data, but I consistently get many different exceptions with java.lang.OutOfMemoryError as its underlying subexception. Some of these exceptions are given below:
2015-07-30 18:57:11,887 ERROR [Thread-11]: server.TThreadPoolServer (TThreadPoolServer.java:serve(194)) - ExecutorService threw error: java.lang.OutOfMemoryError: unable to create new native thread
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:691)
:
:
at java.lang.Thread.run(Thread.java:722)
2015-07-30 18:57:11,888 FATAL [Thread-11]: thrift.ThriftCLIService (ThriftBinaryCLIService.java:run(101)) - Error starting HiveServer2: could not start ThriftBinaryCLIService
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
:
:
at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:99)
at java.lang.Thread.run(Thread.java:722)
2015-07-30 18:57:11,891 WARN [HiveServer2-Handler-Pool: Thread-3652]: thrift.ThriftCLIService (ThriftCLIService.java:FetchResults(681)) - Error fetching results:
org.apache.hive.service.cli.HiveSQLException: java.io.IOException: org.apache.hadoop.ipc.RemoteException(java.lang.OutOfMemoryError): unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:691)
:
:
at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException: org.apache.hadoop.ipc.RemoteException(java.lang.OutOfMemoryError): unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:691)
:
:
at org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
at org.apache.hadoop.security.Groups.getGroups(Groups.java:139)
I googled for this and found this thread.
I added ulimit -n 8192 -u 8192 in .bashrc and restarted hiveserver and metastore.
Now I did another attempt for loading data. This time it gave me new error but having same underlying java.lang.OutOfMemoryError:
2015-07-31 11:46:31,133 ERROR [HiveServer2-Handler-Pool: Thread-6368]: ql.Driver (SessionState.java:printError(960)) - FAILED: SemanticException Error creating temporary folder on: hdfs://mycluster/tmp/hive/anonymous/a3a45571-f359-4747-aaac-fad9e8f9c6ae/hive_2015-07-31_11-46-31_007_6758184846137099807-272/-mr-10000
org.apache.hadoop.hive.ql.parse.SemanticException: Error creating temporary folder on: hdfs://mycluster/tmp/hive/anonymous/a3a45571-f359-4747-aaac-fad9e8f9c6ae/hive_2015-07-31_11-46-31_007_6758184846137099807-272/-mr-10000
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFileSinkPlan(SemanticAnalyzer.java:6467)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:9020)
:
:
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.RuntimeException: Cannot create staging directory 'hdfs://mycluster/tmp/hive/anonymous/a3a45571-f359-4747-aaac-fad9e8f9c6ae/hive_2015-07-31_11-46-31_007_6758184846137099807-272/-mr-10000/.hive-staging_hive_2015-07-31_11-46-31_007_6758184846137099807-272': unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:691)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:521)
:
:
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFileSinkPlan(SemanticAnalyzer.java:6465)
... 40 more
Caused by: org.apache.hadoop.ipc.RemoteException(java.lang.OutOfMemoryError): unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:691)
It says Error creating temporary folder. We usually copy CSV files from local machine to current user's temporary folder in HDFS and then load those files in Hive tables. However, I checked through web browser, no files have been created in the HDFS even in the current user's temporary folder. No records are getting populated in the Hive table.
So I am guessing what might be wrong. Does number of open files and processes limits set by ulimit command really helped? We have restarted only hiveserver and metastore. Do we have to restart whole cluster?
Or its due to number of directories getting created in datanodes (HDFS) is reaching its limits (since we have already loaded 100 million records)?
We have total four datanodes with 4GB RAM. And cluster have total 30 GB of free HDFS space.
