Exception in cluster mode but works fine by running under eclipse

Question

I have no idea why my Hadoop project could not be executed under command line

hadoop jar class-examples-1-1.0.0-SNAPSHOT.jar stock.intro.MR2Screener1 "companylistNASDAQ.csv" output/mr2_screener1

The exception message shows no useful information to me =_=

Any direction or idea? Thanks

works perfectly under eclipse

inline

Part of the exception log by running with command line

2016-11-03 09:14:13,448 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 30206 for container-id container_1478145101159_0003_01_000001: 131.6 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-11-03 09:14:14,084 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1478145101159_0003_01_000001 is : 1
2016-11-03 09:14:14,085 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1478145101159_0003_01_000001 and exit code: 1
ExitCodeException exitCode=1: 
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:578)
    at org.apache.hadoop.util.Shell.run(Shell.java:481)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:763)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Full log

https://gist.github.com/anonymous/242e22801f289d38f30182ff44cd317d

https://gist.githubusercontent.com/anonymous/242e22801f289d38f30182ff44cd317d/raw/86ddf748700c0cc4cbc5d63dc3e6327833b308fb/hadoop_cluster.log

Update 2016-11-03 4pm

Tried the luanch command with HDFS absolute path

" hadoop jar class-examples-1-1.0.0-SNAPSHOT.jar stock.intro.MR2Screener1 "/user/cloudera/companylistNASDAQ.csv" "/user/cloudera/output/mr2_screener1""

Runner to remedy this.
16/11/03 16:24:42 INFO input.FileInputFormat: Total input paths to process : 1
16/11/03 16:24:42 INFO mapreduce.JobSubmitter: number of splits:1
16/11/03 16:24:42 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1478145101159_0007
16/11/03 16:24:42 INFO impl.YarnClientImpl: Submitted application application_1478145101159_0007
16/11/03 16:24:43 INFO mapreduce.Job: The url to track the job: http://quickstart.cloudera:8088/proxy/application_1478145101159_0007/
16/11/03 16:24:43 INFO mapreduce.Job: Running job: job_1478145101159_0007
16/11/03 16:24:52 INFO mapreduce.Job: Job job_1478145101159_0007 running in uber mode : false
16/11/03 16:24:52 INFO mapreduce.Job:  map 0% reduce 0%
16/11/03 16:24:52 INFO mapreduce.Job: Job job_1478145101159_0007 failed with state FAILED due to: Application application_1478145101159_0007 failed 2 times due to AM Container for appattempt_1478145101159_0007_000002 exited with  exitCode: 1
For more detailed output, check application tracking page:http://quickstart.cloudera:8088/proxy/application_1478145101159_0007/Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1478145101159_0007_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:578)
    at org.apache.hadoop.util.Shell.run(Shell.java:481)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:763)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

@RahulSharma it should be working, would you try again? https://i.imgur.com/UxnqXHx.png — newBike, Nov 03 '16 at 17:06
@RahulSharma it doesn't work as well, pls see my new update `Update 2016-11-03 4pm` on the original post. By the way, how did you make a judge for suspecting the problem was from the ABS path issue. I didn't see any clue about that. Thanks :D — newBike, Nov 03 '16 at 23:27
@RahulSharma could the JAVA version cause the problem? The jar could be only generate by JRE1.8. I saw someone said the cluster mode only works under JRE 1.7 — newBike, Nov 04 '16 at 04:11
I faced similar issue because of path so I suspected. I am using jdk1.8 for my project development, it runs without any issue. I am not getting any clue from logs :(. change --loglevel to debug then share logs -https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/YarnCommands.html — Rahul Sharma, Nov 04 '16 at 17:10

Beleagured · Answer 1 · 2016-11-06T20:59:12.623

You can only run the class examples in Eclipse, not on the cluster

While this question was asked and answered in class, maybe I wasn't entirely clear so let me try again.

The code I gave you will not run on the cluster because the cluster is using Hadoop 2.6.0 compiled with Java 1.7.

However it will run in Eclipse, because Eclipse is running Hadoop in standalone mode and you are compiling using Java 1.8.

How you might have debugged this

It's great that you provided log files but the ones that you provided were obtained when you followed the link in the job output to your terminal, which led you, eventually, to http://quickstart.cloudera:8042/logs/yarn-yarn-nodemanager-quickstart.cloudera.log

I applaud you for finding this rolling log, geared towards administrators. However, this is not the most useful log file.

You need to use the log files referenced in your homework (HW1, part 1).

When you are running on the cluster, the logs useful for debugging are stored in HDFS under

/var/log/hadoop-yarn/apps/cloudera/logs

It is easy to read the log files using Hue

Open Hue in a browser on your VM.
Select "FileBrowser"
In the FileBrowser, navigate /var/log/hadoop-yarn/apps/cloudera/logs/
Find the application that failed (the most recently run apps are at the bottom of the list)
Click to open the application's log - it will be in binary.
To make it legible, in the left panel, select "View as text"
Now you have a legible log; look for the first FATAL error that is thrown. It should be towards the top of the log.

I replicated the error and looked in the application log for the first error. Here it is:

2016-11-06 11:18:37,017 FATAL [main]     org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster

java.lang.UnsupportedClassVersionError:    stock/intro/SectorCountWithFilter$StockSectorMapper : Unsupported major.minor version 52.0

UnsupportedClassVersionError

An UnsupportedClassVersionError is thrown by a Java Virtual Machine (JVM) when it tries to read a class file whose Java version is supported. In our case, the Hadoop cluster JVMs can handle Java versions up to 1.7 and your code was compiled in Eclipse using Java 1.8.

How do we know what JVM is running? Well, looking at the error, we can see that this error appears when the MRApplicationMaster is starting - so it is the MRApplicationMaster JVM that is having trouble linking with your class files. As you know, this is the JVM started on Hadoop cluster when your job starts processing. Once it has given up, nothing else really matters. All kinds of errors may then appear because the job startup was aborted, but you should ignore those and concentrate on the first error that appears.

We can conclude that the main error that we are seeing is a mismatch is between the version of Java running on the cluster and the version of Java used to compile your code.

How do we fix it? The only thing to do, in our case, is reference a different JRE in Eclipse - and change the maven compile option in the pom.xml.

BTW, don't bother with the terrible advice propagated through nearly all the blogs about this error: you cannot just look at the version of java on your machine to find out what version a JVM is running. Using $ java -version will not solve our problem and, at best, provides a haphazard foundation for an unreliable fix.

The build for Hadoop that is running on the VM cluster uses java = 1.7.0_67.

What a horrible rabbit hole. Please just run in Eclipse.

Your teacher...

score 0 · Answer 2 · edited May 23 '17 at 12:19

0

Googling "java major minor version" will give you the exact java version. e.g.

java version numbers on stack overflow

It's a thorny problem for sure, but it's good to understand what these messages are a symptom of, as they are quite cryptic (for some unexplained reason ... why can't it just tell us in language we would understand?)

It DOES happen in real life!

edited May 23 '17 at 12:19

Community

1
1

answered Nov 07 '16 at 02:29

miles zarathustra

191
2
5