Stack Overflow
Stack Overflow

Questions tagged [astronomer]

12 questions
1
vote
1 answer

Airflow Docker import error of Astronomer provider

I have docker with airflow to operate on DAGs. Want to import SnowflakeOperatorAsync from astronomer.providers.snowflake. Every time I reinstall python, change environment settings etc. my airflow gets an error Broken DAG:…
Nielm
  • 11
  • 1
1
vote
0 answers

Access XCOM values outside operator

I need to access XCOM values outside the operator, it can be done using variables, but i want to avoid it. I am pushing the xcom values from one operator and then a funcction to fetch xcom values using getone/getmany function, but it is returning…
Ankit Gupta
  • 11
  • 2
1
vote
1 answer

Schedule airflow dag with delay

I’m trying to create an airflow dag that runs an sql query to get all of yesterday’s data, but I want the execution date to be delayed from the data_interval_end. So the data interval is ending at midnight, but it takes few hours for the data itself…
Wave
  • 11
  • 1
0
votes
1 answer

Pull xcom from KubernetesPodOperator

I have a DAG that uses KubernetesPodOperator and the task get_train_test_model_task_count in the DAG pushes an xcom variable and I want to use it in the following tasks. run_this = BashOperator( task_id="also_run_this", bash_command='echo…
Tom J Muthirenthi
  • 3,028
  • 7
  • 40
  • 60
0
votes
1 answer

Adding Custom Airflow Provider to Airflow Running in Docker

I have Airflow running in Docker. I now want to add my custom airflow provider to it. How can I do it? Thanks
Naga Vijayapuram
  • 845
  • 7
  • 11
0
votes
0 answers

Trying BranchDateTimeOperator in Airflow, But getting Naive Aware datetime issue

I am trying to use BranchDatetimeOperator in Airflow, for a special use case. But I am getting error. cond3 =…
Mani Shankar.S
  • 39
  • 6
0
votes
1 answer

Bash Operator to access GCS bucket file in GCP Astronomer

I need to run some bash operation on a file that has been present in GCS Bucket. bash_operator = BashOperator( task_id='mani_bash', bash_command="""if [ `awk -F: '/^[^HDR][^TRL]/ { print }' gs:////filename.txt | awk -F…
Mani Shankar.S
  • 39
  • 6
0
votes
0 answers

Mounted host's aws credential in docker container error: Unable to locate credential

I'd like to mount host's aws credential to astro airflow container, but it didn't recognize it. Astro CLI Version: 1.11.0 Host OS: Ubuntu 20.04 How to reproduce: $ mkdir temp && cd temp && astro dev init Update packages.txt awscli Create file…
user3595632
  • 5,380
  • 10
  • 55
  • 111
0
votes
1 answer

File not found in airflow

I'm using astro-cli for running airflow I have folder structure as follows: astro-cli: dags(dir) include(dir) test.yaml airflow_settings.yaml Dockerfile when I fire this command in my airflow dag cmd_dataprep = az ml job create -f test.yaml…
Prateek Singh
  • 33
  • 5
0
votes
0 answers

how to get the Airfow log in a Task using context?

I want to obtain all the logs generated by a task within an operator, in order to send it to Slack when an error occurs with the on_failure_callback method. I want to get and send the Task INFO/ERROR/WARNING log using on_failure_callback. Can I…
Danieledu
  • 391
  • 1
  • 4
  • 19
0
votes
0 answers

How to run Docker in AWS VPN?

I have previous experience of running Astronomer (Airflow - through Astro CLI ) in my local Docker. But now, I have been provided with AWS VPN. I also Astro cloud UI. Do I need Docker to be installed in AWS VPN ? If so how? Thanks Sabarisri
sabari
  • 2,595
  • 5
  • 28
  • 44
-1
votes
1 answer

GCP - how to identify the no of lines in a file has more than specific delimiter count, by ignoring header & trailer - Python/Bash operator

In GCP - how to identify the no of lines in a file has more than specific delimiter count, by ignoring header & trailer - Python/Bash…
Mani Shankar.S
  • 39
  • 6