2

I'm doing something with Spark-SQL and got error below:

YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 1 for reason Container marked as failed: container_1568946404896_0002_02_000002 on host: worker1. Exit status: -1000. Diagnostics: [2019-09-20 10:43:11.474]Task java.util.concurrent.ExecutorCompletionService$QueueingFuture@76430b7c rejected from org.apache.hadoop.util.concurrent.HadoopThreadPoolExecutor@16970b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]

I'm trying to figure it out by checking the meaning of Exit status: 1000, however, no valuable info returned by googling.

According to this thread, the -1000 is not even mentioned.

Any comment is welcomed, thanks.

DennisLi
  • 3,915
  • 6
  • 30
  • 66
Eugene
  • 10,627
  • 5
  • 49
  • 67
  • 2
    It looks like a yarn exception, not spark. Here is a list which includes -1000, but I don't think it will help much: https://www.cloudera.com/documentation/enterprise/5-8-x/topics/cdh_sg_yarn_container_exec_errors.html – Shaido Sep 20 '19 at 05:14
  • 1
    @Shaido It makes sense that the error is thrown from Yarn side. Thanks. I will try to check yarn log to figure out what happened. – Eugene Sep 20 '19 at 06:01

0 Answers0