2

I am using GAE to set up a simple REST API which allows me to process data in MongoDB.

Additionally, I have a forever script (like a daemon process) which is running a redis based queue.

I need to run my queue separately from my server instance... I am confused as to how to set this up in google app engine. Should I create a second project with a new server instance?

My package.json has this:

 "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "start": "node server.js"
  },

I suppose I will need either a new project or a second app engine instance. Any help in the right direction is appreciated.

Thank you for your time.

Patrick Meyer
  • 75
  • 1
  • 9
  • You could just make it another [service](https://cloud.google.com/appengine/docs/standard/python/microservices-on-app-engine) inside your existing application. Long running tasks are possible, but they're not really a good fit for GAE. Maybe you could break it up in a (possibly endless) sequence of smaller tasks, GAE is quite good at these. – Dan Cornilescu Sep 26 '18 at 02:22
  • I am trying to use google cloud tasks for my solution with an "execute in seconds" option – Patrick Meyer Sep 26 '18 at 02:36
  • I didn't play with those yet, I just used regular GAE push tasks. – Dan Cornilescu Sep 26 '18 at 02:39
  • Yeah they're in beta.. still doesn't work that well as there isnt much documentation and it didnt work for me. Do push tasks have any limits? Can I set up infinite push tasks? cloud tasks seemed to be a good choice for me because they let you "delay" your job in seconds... Would my app get out of control if i did a sleep or setTimeout in a regular push task? – Patrick Meyer Sep 26 '18 at 03:03
  • You don't need to sleep in tasks - that'd be a waste. You can just schedule them "later", using a relative delay/countdown or an absolute ETA when you enqueue them. See https://stackoverflow.com/questions/40475174/do-google-app-engine-pull-queues-return-tasks-in-fifo-order/40478036#40478036. There are some quotas, but pretty generous, my app didn't even make a dent in them. – Dan Cornilescu Sep 26 '18 at 03:08
  • Oh that's great. I am using node.js do you know a tutorial to get started? – Patrick Meyer Sep 26 '18 at 03:13
  • Oh my push notifications are generated by the user and randomly sent out by the server. I will have to create them programatically. Cron jobs wont be the best for this – Patrick Meyer Sep 26 '18 at 03:16
  • Apologies, I forgot that node.js doesn't have support for task queues... :( – Dan Cornilescu Sep 26 '18 at 03:24

1 Answers1

0

To decompose functionality of your app it is possible to deploy separate services in one project, that will serve different tasks, and communicate one to each other. Thank to micro services you get benefits on various levels:

  1. Code isolation
  2. Data isolation
  3. Performance overhead
  4. Cost accounting
  5. Permissions and security
  6. Request tracing

Please check documentation to know more how Contracts, Addressing, and APIs for Microservices works.

To communicate between App Engine services and products in the Google Cloud, you can use Pub/Sub. Thank to this you can trigger background processing, and deal with life cycle events.

To understand better the error in the package.json please check another StarckOverflow thread.

Pawel Czuczwara
  • 1,442
  • 9
  • 20