1

We have a api project written in JAVA using spring hibernate and reading data from MySQL. Recently, we added another data source as BigQuery, therefore we want to allow user to call certain apis to query data from BigQuery - order count from orders table in BigQuery using JAVA api client. Looked at the github sample, but not clear about how to setup and access.

Also found this link but not sure about registering a web application, etc. Please advise.

Update: Imagine this as a web application which shows count of orders in the last 5 days if I select a merchant. but the orders table is in BigQuery. My web application calls-> java api layer which calls -> bigquery using client library -> populates the response as a json -> web application receives the count of orders.

I feel the hiccup is in authenticating using GoogleCredential. Generated a new Oauth Client Id which provided a client_id and client_secret. But still it is not able to authenticate to the project and give results.

Thanks.

Dragon
  • 423
  • 9
  • 18
  • Looks the link, you mentioned, is missing. – Rao Jan 28 '16 at 07:09
  • Since your quest in generalized, i have few questions Are you aware that the data needs to be moved to Google's cloud inorder to access the data using BigQuery ? If so then I hope you referred this [link](https://cloud.google.com/bigquery/what-is-bigquery) – Adheep Jan 28 '16 at 07:35
  • You have a low rate. Important on SO, you have to mark accepted answers by using the tick on the left of the posted answer, below the voting. This will increase your rate. See how this works by visinting this link: http://meta.stackoverflow.com/questions/5234/how-does-accepting-an-answer-work#5235 – Pentium10 Jan 28 '16 at 07:36
  • Thanks Pentium10 will keep that in mind. – Dragon Jan 28 '16 at 07:48
  • updated the link. sorry it did not take the insert link, not it is present. thanks for pointing out. – Dragon Jan 28 '16 at 07:50
  • updated the description with more details. hope it helps. – Dragon Jan 28 '16 at 07:57

2 Answers2

2

Since your question is generalized, I believe what you need is to understand Google's BigQuery on how it works, how to setup the data etc.

Once you setup the data in BigQuery, you can access BigQuery by using a web UI or a command-line tool, or by making calls to the BigQuery REST API using a variety of client libraries such as Java, .NET or Python

You also haven't mentioned that whether you have gone through the basics.

I hope this link will be helpful in understanding how to import data to BigQuery and setting up the data, querying etc.

Adheep
  • 1,585
  • 1
  • 14
  • 27
  • Hi Adheep, I have idea about Big Query and have been using BigQuery as well. But now the use case was to integrate access to the data in BigQuery using API. Since we have a JAVA api layer we are trying to read data using client api. We referred to the github bigquery-appengine sample but it is unable to show a good sign, probably not sure how api client authenticates. In aws sdk it is pretty straightforward using access key and secret key. – Dragon Jan 28 '16 at 07:52
  • Thanks for the clarity Dragaon. Yes Java API for BigQuery could be very confusing and there are no direct tutorials to it. In that case is it possible for you to use a REST Client api like RESTClient or Jersey client and access the data using BigQuery's REST? – Adheep Jan 28 '16 at 08:02
  • Agreed Google Cloud should simplify this process in order to allow users to migrate from AWS to their platform. How does BigQuery REST authenticate? – Dragon Jan 28 '16 at 08:08
  • 1
    BigQuery REST uses a token to authenticate, which is generated from O Auth 2.0. Pentium10 has shared a [link](https://developers.google.com/identity/protocols/OAuth2ServiceAccount) about Service Accounts which will be helpful for you. – Adheep Jan 28 '16 at 08:36
1

Use Service Accounts to connect to your BQ.

And please be aware that the response time will be 2-3 seconds as this is a big data tool not a real time db for web use. Not sure if that's how you want your web application to work. You may need to cache the number in your local database.

Pentium10
  • 204,586
  • 122
  • 423
  • 502
  • We used the credentials tab to generate a client id and client client_secret. Now we are using this example https://cloud.google.com/bigquery/bigquery-api-quickstart#insertqueryjob Still no luck. It says the user does not have permission to access the bigquery – Dragon Jan 28 '16 at 08:09
  • 1
    You need to use service account and setup the p12 file or the JSON. – Pentium10 Jan 28 '16 at 08:13
  • and this: http://stackoverflow.com/questions/12123219/does-bigquery-instance-in-java-api-refresh-its-oauth-2-0-authorization-automati – Pentium10 Jan 28 '16 at 08:16
  • Hi Pentium10, we had generated a service account JSON key earlier but looks like that account did not have permission. Could not figure out how to provide permission. But on using default account, it worked. Thanks a lot. – Dragon Jan 28 '16 at 08:19