3

I'm building a docker image to run zeppelin or spark-shell in local against a production Hadoop cluster with YARN. edit: the environment was macOS

I can execute jobs or a spark-shell well but when I try to access on Tracking URL on YARN meanwhile the job is running it hangs YARN-UI for exactly 10 minutes. YARN still working and if I connect via ssh I can execute yarn commands.

If I don't access SparkUI (directly or through YARN) nothing happens. Jobs are executed and YARN-UI is not hanged.

More info:

  • Local, on Docker: Spark 2.1.2, Hadoop 2.6.0-cdh5.4.3

  • Production: Spark 2.1.0, Hadoop 2.6.0-cdh5.4.3

  • If I execute it locally (--master local[*]) it works and I can connect to SparkUI though 4040.

  • Spark config:

      spark.driver.bindAddress           172.17.0.2 #docker_eth0_ip
  spark.driver.host                  192.168.XXX.XXX #local_ip 
  spark.driver.port                  5001
  spark.ui.port                      4040
  spark.blockManager.port            5003

  • Yes, ApplicationMaster and nodes have visibility over my local SparkUI or driver (telnet test)

  • As I said I can execute jobs then docker expose ports and its binding is working. Some logs proving it:

      INFO ApplicationMaster: Driver now available: 192.168.XXX.XXX:5001
  INFO TransportClientFactory: Successfully created connection to /192.168.XXX.XXX:5001 after 65 ms (0 ms spent in bootstraps)
  INFO ApplicationMaster$AMEndpoint: Add WebUI Filter. AddWebUIFilter(org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter,Map(PROXY_HOSTS -> jobtracker.hadoop, PROXY_URI_BASES -> http://jobtracker.hadoop:8088/proxy/application_000_000),/proxy/application_000_000)

Some ideas or where I can look to see what's happening?

Pau Trepat
  • 697
  • 1
  • 6
  • 24

1 Answers1

2

The problem was related with how docker manage IP incoming requests when it's executed on MacOS.

When YARN, which's running inside docker container, receives a request doesn't see original IP it sees the internal proxy docker IP (in my case 172.17.0.1).

When a request is send to my local container SparkUI, automatically redirects the request to hadoop master (is how YARN works) because it see that the request is not coming from hadoop master and it only accepts requests from this source.

When master receives the forwarded request it tries to send it to spark driver (my local docker container) which forward again the request to hadoop master because it see that the IP source is not the master, is the proxy IP.

It takes all threads reserved for UI. Until threads are not released YARN UI is hanged

I "solved" changing docker yarn configuration

<property>
  <name>yarn.web-proxy.address</name>
  <value>172.17.0.1</value> 
</property>

This allows sparkUI to handle any request made to docker container.

Pau Trepat
  • 697
  • 1
  • 6
  • 24