Questions tagged [python-bigquery]
27 questions
9
votes
1 answer
How to get detailed Big Query error by using PYTHON
I'm looking to see this information as python message:
But currently, I can see only first/second
This is what I'm currently using
from google.api_core.exceptions import BadRequest
if __name__ == '__main__':
try:
…
Oksana Ok
- 515
- 3
- 7
- 19
5
votes
2 answers
How to use Except clause in Bigquery?
I am trying to use the existing Except clause in Bigquery. Please find my query below
select * EXCEPT (b.hosp_id, b.person_id,c.hosp_id) from
person a
inner join hospital b
on a.hosp_id= b.hosp_id
inner join reading c
on a.hosp_id= c.hosp_id
As…
The Great
- 7,215
- 7
- 40
- 128
3
votes
1 answer
How python bigquery library DB-API interface supports WHERE IN or WHERE ANY clause
I'm using the DB-API interface of the python bigquery library https://googleapis.dev/python/bigquery/latest/index.html. It throws errors like below when I pass parameters to the Cursor.execute() for WHERE IN or WHERE ANY clause
google-cloud-bigquery…
Haibin Liu
- 610
- 1
- 7
- 18
1
vote
1 answer
Appending CSV to BigQuery table with Python client
I have a new CSV file each week in the same format, which I need to append to a BigQuery table using the Python client. I successfully created the table using the first CSV, but I am unsure how to append subsequent CSVs going forward. The only way I…
Max888
- 3,089
- 24
- 55
1
vote
1 answer
Apache Beam + Big Query Table Read
I have dataset in big query in project:
Project: project-x
Table: table01
Dataset: dataset01
I would like to connect to it from Apache Beam and read value of one column-column01 for example...
This is what I have:
import apache_beam as beam
from…
shabak
- 11
- 1
1
vote
2 answers
How to fix: compairing result of a bigquery query to a list
I am a newbie in Python, I appreciate all help.
I want a list of activities of a group, who purchased for 4$ this way: purchase_date(1,1,0,1,1,0,1) where purchase_date is the date of the purchase, and the arrays indeces+1 gives the days after the…
Akos
- 49
- 2
- 8
1
vote
1 answer
Arrays not supported in Bigquery Python API
The support for python Bigquery API indicates that arrays are possible, however, when passing from a pandas dataframe to bigquery there is a pyarrow struct issue.
The only way round it seems its to drop columns then use JSON Normalise for a separate…
David Draper
- 79
- 9
1
vote
1 answer
how to fix "field units already exist in schema" for pandas gpq
Versions:
Mac OS Mojave 10.14.5
Python 3.6.5
Pandas 0.24.2
pandas-gbq 0.10.0
I am trying to pull data from the shipstation api and load it into bigquery to use in our BI platform (tableau). I have successfully made the api call, which returned a…
Hunter Maxfield
- 109
- 1
- 2
- 10
0
votes
1 answer
Insert timestamp into bigquery table using pandas
I have a table in Google bigquery, where a column is set to datatype timestamp.
I have to insert data using to_gbq function of pandas.
If I set the datatype to string instead of timestamp the data is loaded into the table.
But I want the column to…
Aman Khandelwal
- 148
- 1
- 13
0
votes
1 answer
How to interpret query process GB in Bigquery?
I am using a free trial of Google bigquery. This is the query that I am using.
select * from `test`.events where subject_id = 124 and id = 256064 and time >= '2166-01-15T14:00:00' and time <='2166-01-15T14:15:00' and id_1 in…
The Great
- 7,215
- 7
- 40
- 128
0
votes
1 answer
How can I use multiple bigquery projects together in python
Suppose I have a big query project A and B. I need to join a table of project A with project B. How I can we join two projects using python.
project_id = 'xyz'
query_job = """SELECT * from test limit 10"""
results_df =…
Arvind Pant
- 65
- 1
- 1
- 8
0
votes
1 answer
Import XLS file from GCS to BigQuery
I have some .xls datas in my Google Cloud Storage and want to use airflow to store it to GCP. Can I export it directly to BigQuery or can i use additional library (such a pandas and xlrd) to convert the files and store it into BigQuery?
Thanks
0
votes
1 answer
Handle BigQuery error from the querying by using Python
To handle the errors from bigquery I have used - job.errors[0]['message'] and everything was fine so far.
The issue is when you querying and for example, when you divide the value by 0, BigQuery will show you:
QUESTION: By using PYTHON - HOW to…
Oksana Ok
- 515
- 3
- 7
- 19
0
votes
2 answers
Increase Count Sequentially
I have a dataset which tracks when a user reads a website. A user can read a website and at anytime therefore the user will appear multiple time. I want to create a column which tracks the number of times a user reads a specific website. But since…
Bryce Ramgovind
- 3,127
- 10
- 41
- 72
0
votes
1 answer
BigQuery only shows first letter of value in streaming buffer
I am streaming data into BigQuery using the Python client library. The row of data lands in the BQ streaming buffer just fine, but when I run a query to view it I can only see the first letter of the value I have inserted.
Specifically, I run a…
Balkan
- 691
- 1
- 8
- 22