I'm trying to schedule python script using apache airflow and my script requires extra packages. So I'm using PythonVirtualenvOperator, but script exits trying to install df2gspread package. Here is an example.
Content of DAG
import time
from pprint import pprint
from airflow import DAG
from airflow.operators.python_operator import PythonVirtualenvOperator
from airflow.utils.dates import days_ago
args = {
'owner': 'airflow',
'start_date': days_ago(2),
}
dag = DAG(
dag_id='test_example',
default_args=args,
schedule_interval=None,
tags=['example']
)
def callable_virtualenv():
from colorama import Fore
from time import sleep
import df2gspread
print(Fore.RED + 'some red text')
for i in range(10):
print(i)
sleep(1)
print('Finished')
virtualenv_task = PythonVirtualenvOperator(
task_id="virtualenv_python",
python_callable=callable_virtualenv,
requirements=[
"colorama==0.4.0",
"df2gspread==1.0.4",
],
system_site_packages=False,
python_version='3.7',
dag=dag,
)
And result of running this DAG is
ERROR: Failed building wheel for df2gspread
Colorama is installed without any problems and at the same time df2gspread raises the error
UPD:Here is complete output of install command
Building wheels for collected packages: df2gspread
Building wheel for df2gspread (setup.py): started
Building wheel for df2gspread (setup.py): finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /venv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/_z/6v0wj4dj7kldx67qgnw4m3640000gn/T/pip-install-2xx3cr8h/df2gspread/setup.py'"'"'; __file__='"'"'/private/var/folders/_z/6v0wj4dj7kldx67qgnw4m3640000gn/T/pip-install-2xx3cr8h/df2gspread/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/_z/6v0wj4dj7kldx67qgnw4m3640000gn/T/pip-wheel-yu4021ye
cwd: /private/var/folders/_z/6v0wj4dj7kldx67qgnw4m3640000gn/T/pip-install-2xx3cr8h/df2gspread/
Complete output (6 lines):
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help
error: invalid command 'bdist_wheel'
----------------------------------------
ERROR: Failed building wheel for df2gspread
Running setup.py clean for df2gspread
Failed to build df2gspread
Installing collected packages: colorama, argparse, uritemplate, httplib2, pyasn1, pyasn1-modules, rsa, six, oauth2client, google-api-python-client, chardet, urllib3, idna, certifi, requests, gspread, python-dateutil, pytz, numpy, pandas, df2gspread
WARNING: The scripts pyrsa-decrypt, pyrsa-encrypt, pyrsa-keygen, pyrsa-priv2pub, pyrsa-sign and pyrsa-verify are installed in '/var/folders/_z/6v0wj4dj7kldx67qgnw4m3640000gn/T/venvcmzzx47t/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The script chardetect is installed in '/var/folders/_z/6v0wj4dj7kldx67qgnw4m3640000gn/T/venvcmzzx47t/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts f2py, f2py3 and f2py3.7 are installed in '/var/folders/_z/6v0wj4dj7kldx67qgnw4m3640000gn/T/venvcmzzx47t/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Running setup.py install for df2gspread: started
Running setup.py install for df2gspread: finished with status 'done'
