6

I need to use sshoperator in a DAG on AWS Airflow (Amazon MWAA), so I imported the following library in my DAG file

from airflow.contrib.operators.ssh_operator import SSHOperator

It seems sshoperator has been defined in paramiko library, so I have added the following modules to requiremets.txt file as well

  • paramiko>=2.6.0
  • pysftp>=0.2.9
  • sshtunnel>=0.1.4

But still, I'm getting the following error when I browse DAGs

Broken DAG: 
  File "/usr/local/lib/python3.7/site-packages/airflow/contrib/operators/ssh_operator.py", 
  line from airflow.providers.ssh.operators.ssh import SSHOperator
  ModuleNotFoundError: No module named 'airflow.providers.ssh'

My underestanding was sshoperator in an essential and built in operators in Airflow, so technically I shouldn't add any extra libraries or plugins to Airflow.

Sam
  • 426
  • 6
  • 16

2 Answers2

4

This command fixed it for me:

pip install apache-airflow-providers-ssh
Aramis NSR
  • 1,602
  • 16
  • 26
  • 1
    Works great. Note that for me, does not require a restart of any of the airflow components, but did require waiting several seconds for the change to kick in. – RyanQuey Apr 13 '22 at 13:57
  • Ah, just noticed this though: You'll have to restart airflow to create an ssh connection according to [this answer](https://stackoverflow.com/a/66471500/6952495) – RyanQuey Apr 13 '22 at 14:19
  • where do you input that command? do you have to ssh into the fargate instance for MWAA? – eljusticiero67 Dec 22 '22 at 18:38
  • 1
    @eljusticiero67 not sure if the self managed MWAA allows this, I believe you should have given it in requirement.txt when you were setting it up. – Aramis NSR Jan 07 '23 at 21:54
0

Was researching something similar and believe that I found the answer to both of our questions... According to Docker Questions...

Providers package is no longer included with Airflow, but you can separately install them with pip using the specific backport package

It looks like there is a backport for ssh here. So probably just have to add that to your requirements.txt?

ET Come Back
  • 198
  • 6
  • I have fixed the issue by adding apache-airflow-providers-ssh==1.3.0 to requirements.txt – Sam Jun 16 '21 at 19:56
  • the only disatvantage is that once you add extra modules via the requirements.txt, it then prevents you from running certain airflow cli commands. The worse being, you cant run a backfill – Emerson Jul 19 '21 at 21:32