0

I am trying to run simple SSHExecutorOperator in Airflow.

Here is my .py file:

from airflow.contrib.hooks.ssh_hook import SSHHook
from datetime import timedelta

default_args = {
'owner': 'airflow',
'start_date':airflow.utils.dates.days_ago(2),
'retries': 3

}

dag = DAG('Nas_Hdfs', description='Simple tutorial DAG',
      schedule_interval=None,default_args=default_args,
      catchup=False)

sshHook = SSHHook(conn_id='101')
sshHook.no_host_key_check = True

t2 = SSHExecuteOperator(task_id="NAS_TO_HDFS_FILE_COPY",
bash_command="hostname ",
ssh_hook=sshHook,
    dag=dag
    )

t2

The Connection id 101 looks like this: Connection configuration from UI

I am getting below error:

ERROR - Failed to create remote temp file

Here is the complete logs:

INFO - Subtask: --------------------------------------------------------------------------------
INFO - Subtask: Starting attempt 1 of 4
INFO - Subtask: --------------------------------------------------------------------------------
INFO - Subtask: 
INFO - Subtask: [2018-05-28 08:54:22,812] {models.py:1342} INFO - Executing <Task(SSHExecuteOperator): NAS_TO_HDFS_FILE_COPY> on 2018-05-28 08:54:12.876538
INFO - Subtask: [2018-05-28 08:54:23,303] {models.py:1417} ERROR - Failed to create remote temp file
INFO - Subtask: Traceback (most recent call last):
INFO - Subtask:   File "/opt/miniconda3/lib/python2.7/site-packages/airflow/models.py", line 1374, in run
INFO - Subtask:     result = task_copy.execute(context=context)
INFO - Subtask:   File "/opt/miniconda3/lib/python2.7/site-packages/airflow/contrib/operators/ssh_execute_operator.py", line 128, in execute
INFO - Subtask:     self.task_id) as remote_file_path:
INFO - Subtask:   File "/opt/miniconda3/lib/python2.7/site-packages/airflow/contrib/operators/ssh_execute_operator.py", line 64, in __enter__
INFO - Subtask:     raise AirflowException("Failed to create remote temp file")
INFO - Subtask: AirflowException: Failed to create remote temp file
INFO - Subtask: [2018-05-28 08:54:23,304] {models.py:1433} INFO - Marking task as UP_FOR_RETRY
INFO - Subtask: [2018-05-28 08:54:23,342] {models.py:1462} ERROR - Failed to create remote temp file
INFO - Subtask: Traceback (most recent call last):
INFO - Subtask:   File "/opt/miniconda3/bin/airflow", line 28, in <module>
INFO - Subtask:     args.func(args)
INFO - Subtask:   File "/opt/miniconda3/lib/python2.7/site-packages/airflow/bin/cli.py", line 422, in run
INFO - Subtask:     pool=args.pool,
INFO - Subtask:   File "/opt/miniconda3/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in wrapper
INFO - Subtask:     result = func(*args, **kwargs)
INFO - Subtask:   File "/opt/miniconda3/lib/python2.7/site-packages/airflow/models.py", line 1374, in run
INFO - Subtask:     result = task_copy.execute(context=context)
INFO - Subtask:   File "/opt/miniconda3/lib/python2.7/site-packages/airflow/contrib/operators/ssh_execute_operator.py", line 128, in execute
INFO - Subtask:     self.task_id) as remote_file_path:
INFO - Subtask:   File "/opt/miniconda3/lib/python2.7/site-packages/airflow/contrib/operators/ssh_execute_operator.py", line 64, in __enter__
INFO - Subtask:     raise AirflowException("Failed to create remote temp file")
INFO - Subtask: airflow.exceptions.AirflowException: Failed to create remote temp file
INFO - Task exited with return code 1

Any help is highly appreciated!

EDIT: I ran this in my airflow user python shell and this is the output:

from airflow.contrib.hooks.ssh_hook import SSHHook 
sshHook = SSHHook(conn_id='101') 
sshHook.no_host_key_check = True 
sshHook.Popen(["-q", "mktemp", "--tmpdir", "tmp_XXXXXX"])

Output: SshHookOutput

Deepesh Rehi
  • 883
  • 8
  • 31
  • This seems to be directly related: https://github.com/puckel/docker-airflow/issues/38 – tobi6 May 28 '18 at 14:14
  • @tobi6: I have already followed the link, before posting it to stackoverflow, that seems not to be working, I have tried- chmod 600 id_rsa, also included the extra, as per the image in the question. – Deepesh Rehi May 29 '18 at 06:13
  • Always helps to include your research in the question so we know what you've done before. – tobi6 May 29 '18 at 06:36
  • Thanks @tobi6, will keep that in mind, this is literally the last stop as I already have been to all the other links.. any help is much appreciated. – Deepesh Rehi May 29 '18 at 09:25
  • Enter a Python shell, mimicking your Airflow user and environment, and do the following: from airflow.contrib.hooks.ssh_hook import SSHHook sshHook = SSHHook(conn_id='101') sshHook.no_host_key_check = True sshHook.Popen(["-q", "mktemp", "--tmpdir", "tmp_XXXXXX"]) Please paste the output. – joebeeson Jun 01 '18 at 14:01
  • @joeb, I have attached the output in the question itself.Thanks – Deepesh Rehi Jun 07 '18 at 08:50
  • see here to check the output of the process in your python shell: https://stackoverflow.com/questions/2502833/store-output-of-subprocess-popen-call-in-a-string – stacksonstacks Sep 03 '18 at 04:22

1 Answers1

0

make sure follow the 3 steps below:

  1. use ssh key instead of password
  2. "key_file" use the id_rsa file not id_rsa.pub
  3. airflow need owner and permissions 0600 to touch the id_rsa and id_rsa.pub file
tancolat
  • 26
  • 1
  • Hi,,did all these steps .....but still the same Error "Airflow airflow.exceptions.AirflowException: Failed to create remote temp file SSHExecuteOperator" – naveen kumar Jul 26 '19 at 07:24