how should I use the right owner task in airflow?

Question

I dont understand the "owner" in airflow. the comment of ower is "the owner of the task, using the unix username is recommended". I wrote some the following code.

   Default_args = {
'owner': 'max',
'depends_on_past': False,
'start_date': datetime(2016, 7, 14),
'email': ['max@test.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=5),

}

dag = DAG('dmp-annalect', default_args=default_args,
                schedule_interval='30 0 * * *')

pigjob_basedata = """{local_dir}/src/basedata/basedata.sh > 
{local_dir}/log/basedata/run_log &
""".format(local_dir=WORKSPACE)

task1_pigjob_basedata = BashOperator(
task_id='task1_pigjob_basedata',owner='max',
bash_command=pigjob_basedata ,
dag=dag)

But I used the command "airflow test dagid taskid 2016-07-20" , I got some error, ... {bash_operator.py:77} INFO - put: Permission denied: user=airflow, ....

I thought that my job ran with "max" user, but apperently , ran test using 'airflow' user .

I hope if I run my task using 'max' user, how should I do.

score 4 · Answer 1 · answered Jul 22 '16 at 08:57

4

I figured out this issue. Because I set the AIRFLOW_HOME in /home/airflow/, only airflow can access this file directory.

answered Jul 22 '16 at 08:57

Max.H

161
1
1
8

Hello Max, thanks for your investigation, I was having a hard time figuring it out as well. Currently it feels not very intuitive. Since I am working with docker and mapped the folder of airflow into a local folder it overwrote the folder with root permissions. Your question guided me into the right direction! – Pixelartist Apr 30 '17 at 23:05

score 1 · Answer 2 · answered Oct 16 '18 at 22:20

I've mitigated this by adding user airflow and all other users who own tasks into a group, then giving the entire group permission to read, write and execute files within airflow's home. Not sure if this is best practice, but it works and makes the owner field more useful than setting airflow as the owner of every DAG.

score 0 · Answer 3 · answered May 28 '22 at 04:45

You can filter the list of dags in webserver by owner name when authentication is turned on by setting webserver:filter_by_owner in your config. With this, a user will see only the dags which it is owner of, unless it is a superuser.

[webserver]
filter_by_owner = True

how should I use the right owner task in airflow?

3 Answers3

Linked