1

I have setup a message queue with Google Pubsub. Everything works fine. The only issue is that all tasks are run simoultanously (ok, currently it's only 3 tasks). As they are fairly heavy weight on the target server it has some issues with it.

My solution would be to process one task, wait until it's completed and the run it again. Unfortunately I haven't found anything in the Python library to set a MaxProcessing.

How would I do this?

John Hanley
  • 74,467
  • 6
  • 95
  • 159
rapsli
  • 749
  • 1
  • 8
  • 23
  • Hi rapsli. IMHO I believe that controlling messages so that they are pulled "one at a time" is not possible using the Google Python PubSub API (rather irritatingly). I tried to do this some time ago and tested several different ways to do it but couldn't get it to work, though I should caution that I am strictly amateur with Python. I have expanded on this and provided links in a response [here](https://stackoverflow.com/questions/50053968/is-it-possible-to-perform-real-time-communication-with-a-google-compute-engine-i/50057983#50057983) that may help. Good luck! – Paul Oct 24 '18 at 13:37

1 Answers1

1

You are able to limit the amount of outstanding messages to your client as described here in the "Message Flow Control" subsection. I've copied the minimal example for python below. https://cloud.google.com/pubsub/docs/pull#subscriber-flow-control-python

from google.cloud import pubsub_v1

# TODO project_id = "Your Google Cloud Project ID"

# TODO subscription_name = "Your Pub/Sub subscription name"

subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(
    project_id, subscription_name)

def callback(message):
    print('Received message: {}'.format(message.data))
    message.ack()

# TODO max_messages_outstanding = 1

# Limit the subscriber to only have ten outstanding messages at a time.
flow_control = pubsub_v1.types.FlowControl(
    max_messages=max_messages_outstanding)
subscriber.subscribe(
    subscription_path, callback=callback, flow_control=flow_control)

-Daniel

Daniel Collins
  • 752
  • 3
  • 14