0

I am writing an Airflow DAG to upload some CSV files to my google sheet. But somehow the DAG cannot find the json file that used to provide the credential. I tried to move the json file to many different places but never succeeded. What should I do to make my DAG find the json?

Here are the relating codes:

def upload():
scope = ['https://spreadsheets.google.com/feeds']
dicter = '~/airflow/credential/client_secret.json'
creds = ServiceAccountCredentials.from_json_keyfile_name(dicter, scope)
client = gspread.authorize(creds)

...

# 3. upload both CSV
task_upload = PythonOperator(
    task_id='tu',
    python_callable=upload
)

The command I used to test my task is:

airflow tasks test CVETaskGenerate tu 2022-8-24

And the error looks like this:

ERROR - Task failed with exception
Traceback (most recent call last):
  File "/opt/anaconda3/envs/airflow_env/lib/python3.9/site-packages/airflow/operators/python.py", line 171, in execute
    return_value = self.execute_callable()
  File "/opt/anaconda3/envs/airflow_env/lib/python3.9/site-packages/airflow/operators/python.py", line 189, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/Users/yifanhuang/airflow/dags/CVEtriageGenerator.py", line 97, in upload
    creds = ServiceAccountCredentials.from_json_keyfile_name(dicter, scope)
  File "/opt/anaconda3/envs/airflow_env/lib/python3.9/site-packages/oauth2client/service_account.py", line 219, in from_json_keyfile_name
    with open(filename, 'r') as file_obj:
FileNotFoundError: [Errno 2] No such file or directory: '~/airflow/credential/client_secret.json'
TryingToLearnSomething
  • 3
  • 3
  • Firstly your sample code has the path `dicter = '~/airflow/credential/client_secret.json'` but the error has `'~/airflow/client_secret.json'`. Do the paths just differ in this post or are your code changes not reflected in airflow for some reason? – unie Aug 24 '22 at 09:04
  • Sorry I paste the wrong error, the error above was for the second last run, I just updated it – TryingToLearnSomething Aug 24 '22 at 09:08
  • Then I'd try to debug the following: 1. Does it work if you run your `upload()` function locally? (changing the dicter-path accordingly if needed) 2. what happens if you use an absolute path to the creds file instead of `~`? – unie Aug 24 '22 at 09:17
  • The upload works fine locally and I tried to use the `dicter = os.path.expanduser('~/airflow/credential/client_secret.json')` but still doesn't work, I will try to check if I use the full path name – TryingToLearnSomething Aug 24 '22 at 09:21
  • it works, thank you! Just write down your answer and I will select it as the solution! – TryingToLearnSomething Aug 24 '22 at 09:24
  • Happy to hear, but absolute paths are kind of clumsy, so I recommended it mainly for debug purposes. I'd still figure out why it doesn't work with `~`, by for example creating a BashOperator task that just prints `ls ~` – unie Aug 24 '22 at 09:31
  • I tested it, the output is the just the output as we type `ls ~` in the terminal. And this means it can recognize ~ in DAG. – TryingToLearnSomething Aug 24 '22 at 09:35
  • There's a similar question: https://stackoverflow.com/questions/60850214/airfllow-cannot-find-json-file. However, their solution doesn't work for me. – TryingToLearnSomething Aug 24 '22 at 09:39
  • Then you could try to make sure that your python can get the path to ~ with os.path.expanduser, e.g. in python terminal: `>>> import os >>> print(os.path.expanduser("~"))` and you should get your home path in return. If you don't, that would explain why it doesn't work in your case. Note: if you use a virtual environment, use that environment's python, not the system python (i.e. activate the environment before testing) – unie Aug 24 '22 at 09:57
  • Sorry for the late reply. The output is /Users/username, which means it returns the correct answer. – TryingToLearnSomething Aug 25 '22 at 05:20
  • 1
    After nearly a month, I think I may find the solution: the airflow I ran was on the local virtual environment instead of a container, which is bad. And that should be the reason ~/ returned wired path. – TryingToLearnSomething Sep 27 '22 at 13:51

0 Answers0