How python bigquery library DB-API interface supports WHERE IN or WHERE ANY clause

Question

I'm using the DB-API interface of the python bigquery library https://googleapis.dev/python/bigquery/latest/index.html. It throws errors like below when I pass parameters to the Cursor.execute() for WHERE IN or WHERE ANY clause

google-cloud-bigquery version: 1.19.0

from google.cloud import bigquery

from google.cloud.bigquery import dbapi

client = bigquery.Client()
conn = dbapi.Connection(client)
curr = conn.cursor()

query = """
  SELECT name, state
  FROM `bigquery-public-data.usa_names.usa_1910_2013`
  WHERE state = %s
  LIMIT 2
"""
curr.execute(query, ('NY', ))
result = curr.fetchall()
print(result)

query = """
  SELECT name, state
  FROM `bigquery-public-data.usa_names.usa_1910_2013`
  WHERE state IN %s
  LIMIT 2
"""
curr.execute(query, (('NY', 'TX'), ))
result = curr.fetchall()
print(result)

Output

[Row(('Mildred', 'NY'), {'name': 0, 'state': 1}), Row(('Irene', 'NY'), {'name': 0, 'state': 1})]
Traceback (most recent call last):
  File "hello_bq.py", line 25, in <module>
    curr.execute(query, (('NY', 'TX'), ))
  File "/home/haibin/.local/share/virtualenvs/python-6nCS1ipk/lib/python3.6/site-packages/google/cloud/bigquery/dbapi/cursor.py", line 159, in execute
    query_parameters = _helpers.to_query_parameters(parameters)
  File "/home/haibin/.local/share/virtualenvs/python-6nCS1ipk/lib/python3.6/site-packages/google/cloud/bigquery/dbapi/_helpers.py", line 117, in to_query_parameters
    return to_query_parameters_list(parameters)
  File "/home/haibin/.local/share/virtualenvs/python-6nCS1ipk/lib/python3.6/site-packages/google/cloud/bigquery/dbapi/_helpers.py", line 84, in to_query_parameters_list
    return [scalar_to_query_parameter(value) for value in parameters]
  File "/home/haibin/.local/share/virtualenvs/python-6nCS1ipk/lib/python3.6/site-packages/google/cloud/bigquery/dbapi/_helpers.py", line 84, in <listcomp>
    return [scalar_to_query_parameter(value) for value in parameters]
  File "/home/haibin/.local/share/virtualenvs/python-6nCS1ipk/lib/python3.6/site-packages/google/cloud/bigquery/dbapi/_helpers.py", line 69, in scalar_to_query_parameter
    name, value
google.cloud.bigquery.dbapi.exceptions.ProgrammingError: encountered parameter None with value ('NY', 'TX') of unexpected type

Any help is appreciated.

BTW, filtering by array elements should use the `IN UNNEST(...)` construct ([BigQuery docs](https://cloud.google.com/bigquery/docs/reference/standard-sql/arrays#scanning-arrays)). With that said, BigQuery `1.19.0` indeed lacks support for array-like parameters, but will likely be added in the next release (I just opened a PR for this). — plamut, Sep 09 '19 at 16:44

score 2 · Accepted Answer · answered Sep 06 '19 at 05:44

I don't see support for array types on the parameter conversion code, but this alternative approach works:

from google.cloud import bigquery

from google.cloud.bigquery import dbapi

client = bigquery.Client()
conn = dbapi.Connection(client)
curr = conn.cursor()

query = """
  SELECT name, state
  FROM `bigquery-public-data.usa_names.usa_1910_2013`
  WHERE state IN UNNEST(SPLIT(%s))
  LIMIT 2
"""
curr.execute(query, ('NY,TX', ))
result = curr.fetchall()
print(result)

There's an open GitHub issue to track progress for native support:

https://github.com/googleapis/google-cloud-python/issues/9177

FWIW, I opened a [PR](https://github.com/googleapis/google-cloud-python/pull/9189) to add support for array-like parameters, and the workaround might not be necessary anymore in BigQuery `1.20.0+`. — plamut, Sep 09 '19 at 16:48

How python bigquery library DB-API interface supports WHERE IN or WHERE ANY clause

1 Answers1