0

I have a cron job run on local server. I would like to use it to

  1. start a new EMR cluster
  2. trigger a Spark model training job on the cluster like this ssh -o "StrictHostKeyChecking no" -i xxx.pem hadoop@10.10.x.xxxx "bash ~/train_model.sh". Since the EMR cluster is new every time, use -o "StrictHostKeyChecking no" flag to avoid new host check.
  3. finally shut down the EMR cluster.

The problem is the the model training takes 10+hrs and the ssh connection in step2 timeout every time.

By searching around, I find it might be resolved by editing ssh config on EMR cluster master node, but since the EMR cluster is new every time so I have to also do the edit every time. I am interesting to find out if there is a more neat way?

Lamanus
  • 12,898
  • 4
  • 21
  • 47
user1269298
  • 717
  • 2
  • 8
  • 26
  • Run your script on the background, not directly run. – Lamanus Aug 19 '20 at 05:34
  • And this is pure ssh problem not emr related, so I deleted the tag of that. – Lamanus Aug 19 '20 at 05:34
  • Does this answer your question? [Getting ssh to execute a command in the background on target machine](https://stackoverflow.com/questions/29142/getting-ssh-to-execute-a-command-in-the-background-on-target-machine) – Lamanus Aug 19 '20 at 05:35

0 Answers0