0

Background

I have a sensor "A" which connects to my mobile app over Bluetooth. This sensor generates 512 data points per seconds.

What I need

I need to send send this data to Google BigTable via Cloud pub/sub. I have already created a topic and I can push message to the topic.

What is the problem

  1. This answer says that I need to create an app engine and connect my app to that app engine. The app engine will receive the data and send it to Bigtable via pub/sub and dataflow.I get this, but now I have two sub problem here -

    a. How can stream the data from my mobile app to app engine? Websocket? Because I think I can't do simple post request to post the data to a route (/publishdata), Am I correct?

    b. I am authenticating my user on mobile app via firebasea authentication. Can I validate my user on app engine via firebase-admin sdk? I can do that by passing the tokenId but is it the correct method? or can I use firebase to authenticate my users on app engine?

  2. I have seen projects where people are suggesting to create a service account and add it into the app. The problem is I don't want to expose my credentials to client. Another talk

Can anyone suggest me tackle this problem?

Thoughts that I have

I can send data to firebase realtime database and then thinking of moving that data to Bigtable via pubsub or cloud function. I couldn't find any example of an app engine connecting to a firebase and reading data from realtime database. I thought in this example this guys sending data to Realtime database and then using pubsub to move data into Bigtable. If this is possible then can I authenticate my user on the app engine via firebase-admin sdk?

B L Λ C K
  • 600
  • 1
  • 8
  • 24

1 Answers1

0

Firstly, you should take a look for this Build an Android App Using Firebase and the App Engine Flexible Environment article, which demonstrates how to write a mobile app with backend data storage, real-time synchronization, and user-event logging using Firebase.

Regarding to your first question about sending data to App Engine, you can send it as an HTTP POST. Please, take a look to following questions on Stackoverflow: 1, 2.

With reference to create a service account and add it into the app, it's not secure option. In this case, you can use Callable Cloud Functions. Refer to this Stackoverflow thread.

You can get data out of the Firebase Realtime Database via backup service which allows for exporting data to GCS Bucket. Then, you can move it to Bigtable, by using open-source Dataflow templates, for instance Cloud Storage Avro to Bigtable. Please, refer to official documentation, to find more about exporting data to Bigtable.

I hope it helps.

aga
  • 3,790
  • 3
  • 11
  • 18
  • thank you for your reply. I just saw your message. Let me go through the links that you have provided and get back to you. and thank you again :) be safe. – B L Λ C K Apr 02 '20 at 13:13
  • I went through the entire link that you have sent for App engine flexible environment with firebase. It seems promising, but I have few questions - (1) How app engine would recognize user and related database in firebase, if I have multiple users. (2) Do I have set servlet identifier manually? (3) Right now example uses java-servlets, can it be done with just NodeJS? – B L Λ C K Apr 02 '20 at 19:42
  • I did make an app using app engine with NodeJS and used firebase-admin sdk. It can read all the database in the given project with service account. It has full read and write access to the database. Right now, I can not make a listener to per user but I do see the entire dataset I can do looping but that doesn't look like a right way to do it when you have million of user. Where as java-servlet was taking care of it. Can you guide me here? Now my another step is to incorporate pubsub into the app to send the data to bigTable. Can you guide me a little bit? Thank you :) – B L Λ C K Apr 02 '20 at 20:48
  • You should take a look here: https://firebase.google.com/docs/database/admin/start -> "Retrieve data". Another interesting thing, you can use is Database Triggers https://firebase.google.com/docs/functions/database-events , where you receive a data object that contains a snapshot of the data stored in the specified document. – aga Apr 03 '20 at 09:45
  • can you give me some idea to design schema for Bigtabel? Yes I have gone through the documentation and I am confused. What kind of schema I should go ahead with storing time series data per user basis in Bigtable? – B L Λ C K Apr 06 '20 at 15:56
  • Have you checked this one? :) https://cloud.google.com/bigtable/docs/schema-design-time-series – aga Apr 07 '20 at 08:50
  • Hey, yes I have seen that. And I was exactly talking about that docs in my last comment. I have gone through but I got so confused. To be precise, I couldn't think of how storing everything under single table is efficient. Why can't we design per user based data logging? Another one is, Since you know what I am doing, How would you go about designing schema for the Bigtable? Thank you :) – B L Λ C K Apr 07 '20 at 16:25
  • I've done some research and found really interesting topics on Stackoverflow: [1](https://stackoverflow.com/questions/60948647), [2](https://stackoverflow.com/questions/52392904/big-table-vs-big-query-usecase-for-timeseries-data), you should take a look, because it's similar to your problem. – aga Apr 10 '20 at 09:39
  • thanks man but I have already found them and read it. :) :) I am trying out right now some ideas. Let see – B L Λ C K Apr 10 '20 at 15:20