0

I want to import csv to hbase by org.apache.hadoop.hbase.mapreduce.ImportTsv.

2019-12-31 06:02:10,611 INFO  [main] mapreduce.JobSubmitter: Submitting tokens for job: job_1577781650355_0001
2019-12-31 06:02:10,611 INFO  [main] mapreduce.JobSubmitter: Executing with tokens: []
2019-12-31 06:02:10,982 INFO  [main] conf.Configuration: found resource resource-types.xml at file:/etc/hadoop/3.1.4.0-315/0/resource-types.xml
2019-12-31 06:02:11,532 INFO  [main] impl.YarnClientImpl: Submitted application application_1577781650355_0001
2019-12-31 06:02:11,668 INFO  [main] mapreduce.Job: The url to track the job: http://node1:8088/proxy/application_1577781650355_0001/
2019-12-31 06:02:11,670 INFO  [main] mapreduce.Job: Running job: job_1577781650355_0001
2019-12-31 06:02:34,983 INFO  [main] mapreduce.Job: Job job_1577781650355_0001 running in uber mode : false
2019-12-31 06:02:34,985 INFO  [main] mapreduce.Job:  map 0% reduce 0%
2019-12-31 06:02:47,134 INFO  [main] mapreduce.Job:  map 4% reduce 0%
2019-12-31 06:02:50,161 INFO  [main] mapreduce.Job:  map 7% reduce 0%
2019-12-31 06:02:53,184 INFO  [main] mapreduce.Job:  map 10% reduce 0%
2019-12-31 06:02:56,212 INFO  [main] mapreduce.Job:  map 14% reduce 0%
2019-12-31 06:02:59,247 INFO  [main] mapreduce.Job:  map 17% reduce 0%
2019-12-31 06:03:02,275 INFO  [main] mapreduce.Job:  map 20% reduce 0%
2019-12-31 06:03:05,296 INFO  [main] mapreduce.Job:  map 24% reduce 0%
2019-12-31 06:03:08,323 INFO  [main] mapreduce.Job:  map 27% reduce 0%
2019-12-31 06:03:11,349 INFO  [main] mapreduce.Job:  map 29% reduce 0%
2019-12-31 06:03:14,370 INFO  [main] mapreduce.Job:  map 33% reduce 0%
2019-12-31 06:03:18,402 INFO  [main] mapreduce.Job:  map 35% reduce 0%
2019-12-31 06:03:21,427 INFO  [main] mapreduce.Job:  map 39% reduce 0%
2019-12-31 06:03:24,455 INFO  [main] mapreduce.Job:  map 42% reduce 0%
2019-12-31 06:03:27,480 INFO  [main] mapreduce.Job:  map 45% reduce 0%
2019-12-31 06:03:30,502 INFO  [main] mapreduce.Job:  map 48% reduce 0%
2019-12-31 06:03:30,542 INFO  [main] mapreduce.Job: Task Id : attempt_1577781650355_0001_m_000000_0, Status : FAILED
2019-12-31 06:03:32,651 INFO  [main] mapreduce.Job:  map 50% reduce 0%
2019-12-31 06:04:25,000 INFO  [main] mapreduce.Job: Task Id : attempt_1577781650355_0001_m_000000_1, Status : FAILED
2019-12-31 06:05:18,339 INFO  [main] mapreduce.Job: Task Id : attempt_1577781650355_0001_m_000000_2, Status : FAILED
2019-12-31 06:05:31,481 INFO  [main] mapreduce.Job:  map 52% reduce 0%
2019-12-31 06:05:34,495 INFO  [main] mapreduce.Job:  map 54% reduce 0%
2019-12-31 06:05:37,515 INFO  [main] mapreduce.Job:  map 56% reduce 0%
2019-12-31 06:05:40,559 INFO  [main] mapreduce.Job:  map 57% reduce 0%
2019-12-31 06:05:43,580 INFO  [main] mapreduce.Job:  map 58% reduce 0%
2019-12-31 06:05:46,626 INFO  [main] mapreduce.Job:  map 59% reduce 0%
2019-12-31 06:05:49,657 INFO  [main] mapreduce.Job:  map 60% reduce 0%
2019-12-31 06:05:52,703 INFO  [main] mapreduce.Job:  map 62% reduce 0%
2019-12-31 06:05:55,732 INFO  [main] mapreduce.Job:  map 63% reduce 0%
2019-12-31 06:05:58,763 INFO  [main] mapreduce.Job:  map 65% reduce 0%
2019-12-31 06:06:01,783 INFO  [main] mapreduce.Job:  map 67% reduce 0%
2019-12-31 06:06:04,804 INFO  [main] mapreduce.Job:  map 68% reduce 0%
2019-12-31 06:06:07,827 INFO  [main] mapreduce.Job:  map 70% reduce 0%
2019-12-31 06:06:10,843 INFO  [main] mapreduce.Job:  map 71% reduce 0%
2019-12-31 06:06:13,861 INFO  [main] mapreduce.Job:  map 73% reduce 0%
2019-12-31 06:06:16,892 INFO  [main] mapreduce.Job:  map 74% reduce 0%
2019-12-31 06:06:25,977 INFO  [main] mapreduce.Job:  map 76% reduce 0%
2019-12-31 06:06:28,994 INFO  [main] mapreduce.Job:  map 77% reduce 0%
2019-12-31 06:06:32,027 INFO  [main] mapreduce.Job:  map 78% reduce 0%
2019-12-31 06:06:35,045 INFO  [main] mapreduce.Job:  map 79% reduce 0%
2019-12-31 06:06:38,071 INFO  [main] mapreduce.Job:  map 81% reduce 0%
2019-12-31 06:06:41,088 INFO  [main] mapreduce.Job:  map 82% reduce 0%
2019-12-31 06:06:44,112 INFO  [main] mapreduce.Job:  map 84% reduce 0%
2019-12-31 06:06:47,130 INFO  [main] mapreduce.Job:  map 86% reduce 0%
2019-12-31 06:06:50,147 INFO  [main] mapreduce.Job:  map 87% reduce 0%
2019-12-31 06:06:53,181 INFO  [main] mapreduce.Job:  map 89% reduce 0%
2019-12-31 06:06:56,201 INFO  [main] mapreduce.Job:  map 91% reduce 0%
2019-12-31 06:06:59,215 INFO  [main] mapreduce.Job:  map 93% reduce 0%
2019-12-31 06:07:02,230 INFO  [main] mapreduce.Job:  map 95% reduce 0%
2019-12-31 06:07:05,246 INFO  [main] mapreduce.Job:  map 97% reduce 0%
2019-12-31 06:07:08,267 INFO  [main] mapreduce.Job:  map 98% reduce 0%
2019-12-31 06:07:11,296 INFO  [main] mapreduce.Job:  map 100% reduce 0%
2019-12-31 06:07:11,306 INFO  [main] mapreduce.Job: Job job_1577781650355_0001 completed successfully
2019-12-31 06:07:11,565 INFO  [main] mapreduce.Job: Counters: 35
    File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=550894
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=209845961
        HDFS: Number of bytes written=0
        HDFS: Number of read operations=4
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=0
    Job Counters
        Failed map tasks=3
        Launched map tasks=5
        Other local map tasks=3
        Data-local map tasks=2
        Total time spent by all maps in occupied slots (ms)=965166
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=321722
        Total vcore-milliseconds taken by all map tasks=321722
        Total megabyte-milliseconds taken by all map tasks=988329984
    Map-Reduce Framework
        Map input records=1047386
        Map output records=1027529
        Input split bytes=206
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=665
        CPU time spent (ms)=47420
        Physical memory (bytes) snapshot=944381952
        Virtual memory (bytes) snapshot=8920907776
        Total committed heap usage (bytes)=627572736
        Peak Map Physical memory (bytes)=478003200
        Peak Map Virtual memory (bytes)=4466188288
    ImportTsv
        Bad Lines=19857
    File Input Format Counters
        Bytes Read=209845755
    File Output Format Counters
        Bytes Written=0

I want to know why the task fail but don't know where to find. The log not throws error.

wyx
  • 3,334
  • 6
  • 24
  • 44

1 Answers1

0

In the fifth line of the provided log, there is a Job Tracking url link to check the status attempts and logs of the job.

The url to track the job: http://node1:8088/proxy/application_1577781650355_0001/

As per the logs above only a particular task attempts got failed before it is finally successful(Job job_1577781650355_0001 completed successfully).

Refer : mapred-default.xml

mapreduce.map.maxattempts and mapreduce.reduce.maxattempts both will be set to 4 by default.

There is also a concept called as speculative execution. You may want to check it out as well.

Refer :

hadoop-speculative-task-execution

what-is-speculative-execution

yammanuruarun
  • 403
  • 3
  • 9
  • I want to know that if `attempt` task failed will affect my job(some data will lost) although it said successful and I found `mapreduce.map.speculative` has set `false` why the speculative task execute – wyx Jan 03 '20 at 02:30