GCP Bigquery exception handling in Python operator

Question

I am trying to catch any error, when I ran query in Bigquery through Python Operator in airflow. But I am not able to get the expected behaviour. Its not throwing any exception.

This is what I have tried.

from google.cloud import bigquery
client = bigquery.Client(project='project_id')
query = "select 1 as cnt from gudb.TEST001 limit 1"

    try:
        client.query(query)
    except Exception as e:
        print(e)

its not throwing any exception, even the table is not available in bigquery. But when I tried to store the variable

var=client.query(query) # in try     
for x in var:
    print(x)

I can see the exception stored in that var. But I am expecting to get the exception to be prompted in the stdout when I try to run the try except block.

Can someone please let me know, how to get the exception

Hi @Mani Shankar.S,I have posted an answer. I hope it will help you. Do consider upvoting and accepting if it helps, else let me know so that I can improve my answer. — kiran mathew, Jun 15 '23 at 12:33
thanks Kiran.. I got the answer through this link.. but forgot to update here.. https://stackoverflow.com/questions/57039771/how-to-get-detailed-big-query-error-by-using-pythonhttps://stackoverflow.com/questions/57039771/how-to-get-detailed-big-query-error-by-using-pythonhttps://stackoverflow.com/questions/57039771/how-to-get-detailed-big-query-error-by-using-python but anyway thanks for the answer.. it helps.. in the mentioned link, badrequest was used, here exception is used.. Badrequest has lot of verbose, exception has straight error message..any other differences you want to include — Mani Shankar.S, Jun 15 '23 at 13:31

kiran mathew · Accepted Answer · 2023-06-15T13:15:36.047

You can find the error status of the query by calling the result function.For your requirement you can consider the following code:

try:
        client.query(query).result()

except Exception as e:
        print(e)

Example:

def bigquerytest():
    client = bigquery.Client()
    query = "SELECT * FROM `my-project-36069-km.demo12.vvvvvvv`"

    try:
        query_job = client.query(query)
        query_job.result()

    except Exception as s:
        print(f'The error is {s}')

with DAG(dag_id='ne',start_date=datetime(2021, 4, 5, 15, 0),schedule_interval='@daily',catchup=False) as dag:
    t1=BashOperator(task_id='print',bash_command='echo finished')
    t2=PythonOperator(task_id='bigquery',python_callable=bigquerytest)

    t2>>t1

Error:

GCP Bigquery exception handling in Python operator

1 Answers1