Data source on GCP BigQuery

Question

I tried to look for any existing intake components such as Driver, Plugin that can support GCP BigQuery. Given that if it cannot support, please advise on how to implement subclassing of intake.source.base.DataSource

Answer 1

Pandas can read from BigQuery with the function read_gbq. If you are only interested in reading whole results in a single shot, then this is all you need. You would need to do something like the sql source, which calls pandas to load the data in _get_schema method.

There is currently no GBQ reader for dask, so you cannot load out-of-core or in parralel, but see the discussion in this thread.