Stack Overflow
Stack Overflow

Questions tagged [bq]

bq is a python-based, command-line tool for BigQuery. It allows the creation of datasets, tables, query jobs; and listing and exporting data among other utilities using the command-line.

bq is a python-based, command-line tool for BigQuery. It allows the creation of datasets, tables, query jobs; and listing and exporting data among other utilities using the command-line.

104 questions
3
votes
1 answer

Bigquery | Result of ORDER BY queries cannot be partitioned by field

I am using Bigquery Operator for my Airflow Task. I am passing an Sql query to operator and trying to insert result of BQ query into a BQ table. While doing this, getting below error google.api_core.exceptions.BadRequest: 400 Result of ORDER BY…
Amol
  • 93
  • 7
3
votes
2 answers

Error while reading data, error message: CSV table references column position 174, but line starting at position:136868 contains only 94 columns

I'm trying to unload data from Snowflake to GCS and then GCS to bq table. Here is the Code for unloading data from snowflakes. ```copy into @unload_stage/FF_TBL_UNLOAD20200906.csv.gz from ( select * from SF_DB.SF_TBLS.FF_TBL_UNLOAD ) file_format =…
Saivarma Manthena
  • 59
  • 3
  • 5
2
votes
0 answers

Loading Sharepoint files into Bigquery

I want to create a BigQuery table out of some files that are present in our organizations secured space of sharepoint. The files will be added on weekly basis and I need to setup a pipeline that I can use to ingest data in bigquery. I have been…
Amit Sharma
  • 21
  • 3
2
votes
1 answer

String becomes float in BiqQuery table after import from pandas dataframe

I have a pandas dataframe with the following dtypes: Int64Index: 579585 entries, 0 to 579613 Data columns (total 7 columns): # Column Non-Null Count Dtype --- ------ -------------- …
Serge de Gosson de Varennes
  • 7,162
  • 3
  • 18
  • 39
2
votes
1 answer

Expected type '_SpecialForm[str]', got 'str' instead

I'm trying to create a BQ table schema, as seen on this page But I get a compilation error for all the mode="REQUIRED" I didn't see anything special to import but the bq module. Expected type '_SpecialForm[str]', got 'str' instead The code: …
Elad Benda
  • 35,076
  • 87
  • 265
  • 471
2
votes
2 answers

List all objects in a dataset using bq ls command in BigQuery

I am trying to list all the objects present in a dataset in BigQuery. I tried using bq ls projectID:dataset_name command in Google SDK shell. However this returned only the list of tables present in the dataset. I am interested in listing all the…
Rajalakshmi
  • 541
  • 5
  • 21
2
votes
1 answer

BigQuery use conditions to create a table from other tables (manage big number of columns)

I am facing an issue related to a project of mine. Here is the summary of what i would like to do : I have a big daily file (100 Go) with the following extract (no header)…
muslash
  • 39
  • 6
2
votes
1 answer

How can I extract a single partition from a partitioned BigQuery table?

According to the BigQuery docs, I should be able to export a single partition of a partitioned table: Exporting all data from a partitioned table is the same process as exporting data from a non-partitioned table. For more information, see…
Graham
  • 71
  • 7
2
votes
0 answers

How to catch errors in BigQuery CLI?

I write simple bash script and run it: #!/bin/bash set -e bq load --replace ... and I get this error: ERROR: permission denied for table message WRITING LINES: 0 As we can see, we got an obvious error related to access. Why did the script end…
Arslanbekov Denis
  • 1,674
  • 12
  • 26
2
votes
1 answer

Google Cloud Platform - Use Multiple Service Account

I am trying to use bq cli to export data using big Query to GCS. Currently there are two projects and each project is having its own service account. I have authenticated the services account using gcloud auth active-service-account by passing the…
Berry Jenson
  • 21
  • 2
1
vote
2 answers

Count the number of times a string appeared in a delimited field In BIGQUERY

I have a dataset with a delimiter "->" like: ROW 1- "Q -> Res -> tes -> Res -> twet" ROW 2- "rw -> gewg -> tes -> Res -> twet" ROW 3- "Y -> Res -> Res -> Res -> twet" I just want to count the number of "Res" in every row Output would be: ROW 1-…
HAIDER TAYYAB
  • 47
  • 4
1
vote
1 answer

Big Query: Query doesn't allow us to get the min of a struct

I recently moved from using apache hive to GCP big query as my SQL database. I have a set of data as…
shiv_alagar
  • 21
  • 2
1
vote
1 answer

How to pass URIs as variable in BQ stored procedure

I am new to BQ and trying to load table via file from GCS bucket and getting error for uris . Query error: Found unsupported function call 'ARRAY[...]'; failed to set 'uris' in OPTIONS() Code Snippet : begin declare filename STRING; declare…
Mudgal
  • 35
  • 4
1
vote
0 answers

Parquet file bq load is failing with Resources exceeded during query execution: UDF out of memory. error

I am trying to load a parquet file to bigquery using bq comand line utility with option --parquet_enable_list_inference=true. I have confirmed that the number of columns is less than 20 in the file and the row size is not more than 50 mb, maximum it…
ted
  • 3,911
  • 3
  • 26
  • 49
1
vote
0 answers

--preserve_ascii_control_characters not working in bq load command line

I am trying to load the data into BQ using bq load command line with the argument '--preserve_ascii_control_characters=true' . First few lines are given here bq load --source_format=CSV --preserve_ascii_control_characters=true \ But I keep getting…
Selvalakshmi Praveen
  • 11
  • 1
1
2 3 4 5 6 7