How do we import large datasets from Google BigQuery to R?

Question

How do we import large datasets from Google BigQuery to R ?

I tried using the BigQuery package and tried importing. I am able to succeed in importing smaller datasets but not the huge ones

sql <- "SELECT * FROM Table name"

todo_copies <- query_exec(sql, project = 'data-warehouse', dataset = 'name', useLegacySql = FALSE)

bytes processed Error: Requested Resource Too Large to Return [responseTooLarge]

If you just want the whole table, use an export instead of a query — Felipe Hoffa, May 30 '19 at 20:42
exporting will cause memory problems on local PC so i would like to get the data on R from big query as dataframe. This query is written in R to get the data from big query to R — Nitesh Shannhag, May 31 '19 at 06:40

score 1 · Answer 1 · answered Sep 19 '19 at 00:02

The message refers to a limit related to the Maximum response size for query jobs in BQ. It can be managed if you write to a detination table; however, most likely this option don't meet your needs.

Please note that R documentation indicates that query_exec is being deprecated in favor of bq_project_query and from other threads like How to load large datasets to R from BigQuery?, it is suggested to adjust the page_size property:

"It should be adjusted when an error responseTooLarge appears"

How do we import large datasets from Google BigQuery to R?

1 Answers1