0

I have created a Google Cloud Logging sink to BigQuery. When a log is first sent to the sink, a schema for the BQ table is generated as described here. So for example if I log JSON like this:

{
  "foo": "bar",
  "baz": 10
}

Then a schema like this (omitting much) is created:

jsonPayload RECORD
  foo STRING
  baz FLOAT

What I would like though is a schema like this:

jsonPayload JSON

because I would also like to have log messages with different jsonPayloads (different and currently unknown keys) in the same table.

I tried to change what I am logging to look like this thinking it might not parse the jsonPayload into a schema but it did not work.

{
  "@type": "type.googleapis.com/google.protobuf.Struct",
  "foo": "bar",
  "baz": 10
}

It didn't treat the @type field specially as I hoped it might but rather just renamed @type to _type and gave it the STRING type.

mab
  • 760
  • 4
  • 18
  • Perhaps I should define the table schema explicitly before the first log comes in setting the jsonPayload type to JSON? – mab Nov 29 '22 at 18:37
  • I tried predefining the table schema with jsonPayload being a JSON type but then got export errors saying "This field: jsonpayload is not a record." – mab Nov 29 '22 at 21:35
  • I guess I can log the json as text and then treat it as json in queries. – mab Nov 29 '22 at 21:59

0 Answers0