0

I have a variable that I must generate value every time I perform the function below and its response is used to compose a query that I need to execute next. In this example that I am sharing I believe it is not the correct way to get the information as it is generated before executing the task "bq_diff_id". Does anyone have any suggestions of getting this variable after the result of the previous task?

def get_data_from_bigquery():
    """query bigquery to get data to import to PSQL"""
    bq = bigquery.Client()
    #IDs
    query = """SELECT ID FROM dataset.table1"""
    query_job = bq.query(query)
    data = query_job.result()
    rows = list(data)
    diff1 = str(tuple(np.array(rows).T.tolist()))
    diff = diff1.replace("[", "").replace("]", "").replace(",)",")")
    #Count 
    count_query = """SELECT count(*) as qtt FROM dataset.table1"""
    count_query_job = bq.query(count_query)
    count_data = count_query_job.result()
    count_rows = list(count_data)
    count_end = str(count_rows[0][0])
    if int(count_end) <= 0:
        query = 'id is null'
        return query
    else:
        query = 'id in ' + diff
        return query

Python_1 = PythonOperator(task_id='bq_diff_id',
    python_callable=get_data_from_bigquery,
    dag=dag)

query = get_data_from_bigquery()

sql_query = """select id as id, \
value \
from table2 where """ + query + """ """ #Query for extract

MsSql = MsSqlToGoogleCloudStorageOperator(
    task_id='import_orders',
    mssql_conn_id=mssql_connection,
    google_cloud_storage_conn_id='gcp',
    sql=sql_query,
    bucket=nm_bucket,
    filename=nm_arquivo,
    schema_filename=sc_arquivo,
    dag=dag)
Felipe FB
  • 1,212
  • 6
  • 22
  • 55
  • Any suggestion? – Felipe FB Sep 26 '19 at 19:03
  • I didn't get your question, but if you intend to just **programmatically create a `Variable`**, then you can refer to [this](https://stackoverflow.com/a/54048104/3679900). For even fine-grained code, example, see [this](https://stackoverflow.com/a/57765115/3679900) answer that concerns programmatically creating a `Connection` – y2k-shubham Sep 27 '19 at 14:52

0 Answers0