0

I have been exploring google cloud dataplex data profiling feature here

I followed the steps given in creating the data profile scans and noticed the profile results are shown in the Profile section as per this documentation

If I want to retrieve those profile results to BigQuery, what options does gcp provides. I did check some of the apis, but did not notice any details to export profile scans to bigquery.

I am fairly new to using google cloud, therefore I may be missing something.

PK_gcp
  • 29
  • 5
  • It's not possible, for now, to export Profiling data to BigQuery automatically. If you want that, you have to write a piece of code (Cloud Run/Cloud Functions for instance) that request the API and sink the data in BigQUery. – guillaume blaquiere May 26 '23 at 11:23
  • thanks, but can you help me why would google do this? generally, the consumer of such information are generally data governance personas, why wouldn't google enable direct load to BQ when they already do with the other feature auto dq results – PK_gcp May 27 '23 at 10:38
  • I'm sure it's in the roadmap (don't have the detail right now)! – guillaume blaquiere May 27 '23 at 11:33
  • curious to know if the roadmap available for viewing for everyone? – Bihag Kashikar May 29 '23 at 07:50

1 Answers1

0

Exporting dataplex data profile results to BigQuery is supported now.

Please check the public doc at Step Optional: Export the scan results to a BigQuery standard table for more details.

You can either bring your own table (which schema needs to be compatible with the required schema) or let the dataplex created the table for you if the table doesn't exist.

Please make sure the export BigQuery dataset exists and assign the dataplex service account corresponding permissions.

rongqi
  • 1
  • 2