Task Instance is not defined

Question

I have an object I've passed to xcom that I want to read from an operator.

Here is my operator:

load_csv = GCSToBigQueryOperator(
    task_id='gcs_to_bigquery',
    bucket='test',
    source_objects=['{{ execution_date.strftime("%Y-%m") }}'],
    providers=True,
    destination_project_dataset_table=f'{stg_dataset_name}' + '.' + '{{ execution_date.strftime("%Y_%m") }}',
    schema_fields={{ti.xcom_pull(task_ids='print_the_context')}},
    write_disposition='WRITE_TRUNCATE',
    provide_context=True,
    dag=dag)

I want to pass the value from xcom to the schema_fields variable.

I'm trying to access the object using the following template {{ti.xcom_pull(task_ids='print_the_context')}} but I have it is not defined...

What's wrong here?

score 1 · Accepted Answer · answered Aug 22 '20 at 23:21

Unfortunately this is not possible at the moment

To be able to leverage macros in operator arguments, the corresponding field must be defined as template_fields in operator's source code

But here in the source of GCSToBigQueryOperator, I see that schema_fields is missing from template_fields

template_fields = ('bucket', 'source_objects',
                    'schema_object', 'destination_project_dataset_table')

Therefore you can't supply value for schema_fields via XCOM template

That said, while I'm not aware of internals of GCSToBigQueryOperator, I can see 2 possible solutions

(straightforward) use schema_object field instead

:param schema_object: If set, a GCS object path pointing to a .json file that
     contains the schema for the table. (templated)
     Parameter must be defined if 'schema_fields' is null and autodetect is False.
 :type schema_object: str

you can try subclassing it and including schema_fields in template_fields

Interesting reads

Task Instance is not defined

1 Answers1