Collecting data from BigQuery via R is failing with an Internal Error message.
For example, the following query (with specific table / project names hidden), attempts to download 2.9 million rows across 299 pages.
connection <- DBI::dbConnect(bigrquery::bigquery(), project = project,
billing = billing, page_size = page_size)
tbl <- dplyr::tbl(connection, paste0(dataset, ".", schema, ".", table))
data <- tbl %>% dplyr::collect()
This fails with the following error:
Complete Billed: 0 B Downloading 2,982,643 rows in 299 pages. Downloading data [=====>-------------------------------------------------------------------------------------------------------------------] 5% ETA: 5mError: An internal error occurred and the request could not be completed. [internalError]
When I look at BigQuery UI jobs, the query id itself is marked as successfully complete. Similarly when I look at the Stackdriver logging for the BigQuery API call, the job is also marked as successful.
Jobs of this nature have been failing when querying with a service token and individual user credentials. The error is intermittent, but seem most common when returning large numbers of rows. Running the same query directly in the BigQuery UI works succesfully, suggesting there is something problem with the final data collection / download step.
