Stack Overflow
Stack Overflow

Questions tagged [airflow-api]

77 questions
4
votes
1 answer

Dynamic Task Mapping with Decorators in Airflow 2.3

I want a whole task group to run on the output of a single task, where both task and task group are defined via decorators - @task and @task_group respectively. Somewhat similar to For that, I updated one of the examples provided by Airflow. The…
Índio
  • 539
  • 5
  • 12
4
votes
2 answers

How to trigger airflow dag with REST API (I get "Property is read-only - 'state'", error)

I am trying to trigger airflow dags with REST API. It does not work. I get an ERROR 400 with response: { "detail": "Property is read-only - 'state'", "status": 400, "title": "Bad Request", "type":…
Jakub Pluta
  • 147
  • 5
  • 13
4
votes
2 answers

How can I enable the API in AWS Managed Workflows for Apache Airflow?

I'm testing the waters for running Apache Airflow on AWS through the Managed Workflows for Apache Airflow (MWAA). The version of Airflow that AWS have deployed and are managing for me is 1.10.12. When I try to access the v1 REST API at…
urig
  • 16,016
  • 26
  • 115
  • 184
3
votes
0 answers

How to connect MWAA CLI that is hosted in a public subnet

I am trying to invoke airflow Cli from my local terminal and I have followed these instruction https://dnx.solutions/how-to-use-apache-airflow-cli-with-amazon-mwaa/ and created cli_token and executed curl \ --request POST…
Xi12
  • 939
  • 2
  • 14
  • 27
3
votes
1 answer

Accessing Airflow REST API in AWS Managed Workflows?

I have Airflow running in AWS MWAA, I would like to access REST API and there are 2 ways to do this but doesn't seem to work for me. Overriding api.auth_backend. This used to work and now AWS MWAA won't allow you to add this, it is consider as…
user15364819
  • 61
  • 5
2
votes
0 answers

Airflow - How to handle tasks relationship for the tasks inside dynamic task group mapping

I have implemented dynamic task group mapping with a Python operator and a deferrable operator inside the task group. I got stuck with controlling the relationship between mapped instance value passed during runtime i.e when the deferrable operator…
saravana kumar
  • 255
  • 1
  • 3
  • 10
2
votes
0 answers

In Airflow GUI - ModuleNotFoundError: No module named 'airflow.providers.cncf

Apache Airflow version 2.5.2 What happened Not able to create new (Admin - > Connection -> When I press + sign it gives below error regarding airflow.providers.cncf Error: Ooops! Something bad has happened. Airflow is used by many users, and it is…
Austin Jackson
  • 153
  • 6
2
votes
1 answer

Airflow set dag run note

How to use the note present in DAG runs panel from the ui? I would want to programmatically fill it. For example changing the content depending the on the params passed to the DAG run
Justin
  • 67
  • 1
  • 5
2
votes
0 answers

How to pass a conf to a scheduled DAG

When the DAG is triggered manually there are multiple ways to pass the config. It could be done from the UI, via the airflow CLI using --conf argument & using the REST API. But when a DAG is scheduled using a cron expression, the DAG always fails…
user3583252
  • 493
  • 1
  • 5
  • 15
2
votes
2 answers

Triggering Airflow DAG via API

I have installed Airflow 2.0.1 on EC2 with PostgreSQL RDS as metadata db. I want to trigger DAG from Lambda so tried to test the code with curl but am receiving Unauthorized as response. What if anything should I be doing…
Ronak Shah
  • 51
  • 1
  • 4
1
vote
1 answer

How to get the initial start-time and end-time for Airflow DAG, including all tasks

How can I get the start and end time of the DAG in overall, which includes all the tasks (which is the initial task start time and end time of the last task)?
Austin Jackson
  • 153
  • 6
1
vote
0 answers

How to identify airflow dependent DAGS and Tasks through airflow rest api

i have some dependent dags (parent and child dags). How to i identify those dags are dependent , How to classify whether dag is dependent or not. im using airflow version 2.4.0 How to get dependent dags details through airflow rest api. Ref : How to…
vbala vbala
  • 15
  • 3
1
vote
0 answers

Why airflow marks kubernetes pod operator tasks in airflow as failed on the UI while the containers are still running on k8s?

I am having a for loop in my DAG which creates some tasks using kubernetes pod operator. These tasks run in parallel and there are enough resources available on the k8s nodes to let them run in parallel. These containers share an EFS volume as a PVC…
Naxi
  • 1,504
  • 5
  • 33
  • 72
1
vote
0 answers

Using Airflow, How to handle asynchronous API calls with Airflow Tasks?

I'm quite new to Airflow, I need to make asynchronous POST API calls to start the execution of the external service. And Need to make GET API calls to check the status of the execution and have to make that call until the execution gets completed. I…
saravana kumar
  • 255
  • 1
  • 3
  • 10
1
vote
2 answers

Accessing task_instance or ti via simpleHttpOperator to do an xcom push

TLDR In the python callable for a simpleHttpOperator response function, I am trying to push an xcom that has combined information from two sources to a specificied key (a hash of the filename/path and an object lookup from a DB) Longer Tale I have a…
Daryl
  • 23
  • 4
1
2 3 4 5 6