3

There is no guide on how to compile Hadoop code in the Hadoop: The Definitive Guide book. I am not using any Java IDE for this. Do I need to install any Java IDE? I made JAR file of the code and tried running it with Hadoop command but it is not working on Ubuntu. The book says to install and compile the examples using instructions on the book's website, but I cannot find it on O'Reilly's website.

If someone can show me how to compile simple code like this, that would be a great help to me.

import org.apache.hadoop.fs;
import java.io.IOException;
import java.net.URL;
import org.apache.hadoop.io.IOUtils;

public class URLCat{

    static{
        URL.setURLStreamHandlerFactory(new FSUrlStreamHandlerFactory());
    }


    public static void main(String[] args) throws Exception{
        InputStream in =null;
        try{
            in=new URL(args[0]).openStream();
            IOUtils.copyBytes(in,System.out,4096,false);

        }finally{
            IOUtils.closeStream(in);    
        }


    }
}
Faye
  • 127
  • 1
  • 1
  • 14
  • Well, certainly, an IDE will make the compile/debug process easier. However, SO is not a tutorial site. Sorry. There are many, many java tutorials online. See http://docs.oracle.com/javase/7/docs/technotes/tools/windows/javac.html – OldProgrammer Jan 30 '17 at 18:40
  • @OldProgrammer Why did you gave me Java program compile link? I know how to do that and I didn't asked for that. – Faye Jan 30 '17 at 18:46
  • "how to **compile** simple code." "it is not working" is not very helpful to a reader. If you have errors, please show them. Please take some time out to read [ask]. – OldProgrammer Jan 30 '17 at 19:02
  • I'd use eclipse if I were you. I just took a class on hadoop, and I used eclipse on my vm and it was very nice. – yalpsid eman Jan 30 '17 at 19:14
  • @Faye Did you get answer to this question. I am facing the same problem and the book does not use IDEs but asks to refer to the method that is mentioned in the website. But it is not there. – LearneriOS May 01 '17 at 04:44

2 Answers2

1

You can find the example code in the following link: Link to Github

To compile Hadoop code example you need:

  1. Need to have installed Maven and Java.

  2. Then navigate to the root folder where you have all the chapters folders in terminal and type:

    mvn package -DskipTests

  3. This command compile all the code.

pfRodenas
  • 326
  • 1
  • 2
  • 12
0

These are the steps that I followed to run the examples codes for the Book Hadoop The Definetive Guide 4 Ed (https://github.com/tomwhite/hadoop-book).

  1. Follow this instalation first https://stackoverflow.com/a/51617361/15576728, also this is define in the Read.md from the master code, so make you have done this first.

  2. Download the code from GIT

  3. Unzip it and then cd to the folder /home/hadoop-book-master/

  4. run mvn package -DskipTests, this will create some .jar files from which will have to the one that is suggested in the Book Chapter hadoop-examples.jar

Book Page 49.

% export HADOOP_CLASSPATH=hadoop-examples.jar

% hadoop MaxTemperature input/ncdc/sample.txt output

As you can see here (https://stackoverflow.com/a/31918625/15576728) the way you excecute a hadoop jar is:

"hadoop jar jarFileName mainClassname AnyCommandLineArguements"

Where "jarFileName" will be "hadoop-examples.jar" as this will have the examples compile code to run and mainClassname as MaxTemperature, finally "AnyCommandLineArguements" are the input path from HDFS and Output will be created in HDFS .

  1. this is how i run it:

hadoop jar $HADOOP_CLASSPATH MaxTemperature /data/sample.txt /outputTest

It is important to recall the any file you want to process have to be in the HDFS for your code to work.

hdfs dfs -mkdir /data

hadoop fs -copyFromLocal /home/hadoop-book-master/input/ncdc/sample.txt /data

Emmanuel Varela
  • 1
  • 1