3

Is it possible to override log4j properties per job?
I have a multi-step program (a pipeline) that runs some hadoop jobs
but before and after it also performs some other steps.

if I run my program using java command (java -jar my_program.jar) then it runs OK but I get warning:
Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

If I run my program using hadoop command (hadoop jar my_program.jar) then default hadoop log4j.properties are loaded

In my program I log information about steps execution to a specific file. I do not need logs from hadoop map/reduce tasks. I need logs from my program because it performs many other tasks.

Should I worry about that warning?
How to override log4j properties per job to keep logs from my pipeline with my custom log4j.properties?

I am using hadoop v1.0.4

EDIT:
I discovered that after exporting LD_LIBRARY_PATH the warning Unable to load native-hadoop library for your platform... no longer appears when running my program with command java -jar my_program.jar. Source description:
http://balanceandbreath.blogspot.ca/2013/01/utilnativecodeloader-unable-to-load.html
Does someone know how to How to override log4j properties per job?

fuggy_yama
  • 607
  • 6
  • 24

1 Answers1

2

Basing on this answer: https://stackoverflow.com/a/7390644/4496488
I am using folowing code to load my custom log4j.properties in the main code that launches hadoop jobs (in this case it actually overwrites the hadoop log4j config):

String log4jPropertiesFilePath = "path/to/my/custom/log4j.properties";
PropertyConfigurator.configure(log4jPropertiesFilePath);

It does not overwrite the log4j properties in datanodes - only in the program launching hadoop jobs - it is exactly what I wanted.

Community
  • 1
  • 1
fuggy_yama
  • 607
  • 6
  • 24