Stack Overflow
Stack Overflow

Questions tagged [google-cloud-pubsub]

For programming/API questions related to Google Cloud Pub/Sub, which provides asynchronous messaging and ingestion services for event-driven systems and streaming analytics on the Google Cloud Platform infrastructure.

The official documentation can be found here: https://cloud.google.com/pubsub/docs/overview

2920 questions
129
votes
3 answers

I am evaluating Google Pub/Sub vs Kafka. What are the differences?

I have not worked on kafka much but wanted to build data pipeline in GCE. So we wanted to know Kafka vs PUB/Sub. Basically I want to know how message consistency, message availability, message reliability is maintained in both Kafka and…
Naresh
  • 1,301
  • 2
  • 9
  • 9
40
votes
1 answer

Best practices for draining or clearing a Google Cloud pubsub topic

For pubsub topics with number of messages in the range of ~100k, what is the best practice for draining/dropping/clearing/deleting all messages using gcloud-java SDK? Possible solutions: Deleting and recreating the subscribers and then the…
DrMarshall
  • 1,034
  • 2
  • 9
  • 14
33
votes
10 answers

Pip install on Mac OS gets error: command '/usr/bin/clang' failed with exit code 1

I want to install google-cloud-pubsub via pip installation on Mac OS but I get an error: distutils.errors.CompileError: command '/usr/bin/clang' failed with exit code 1. The command I run: pip install google-cloud-pubsub==2.1.0. Here the complete…
Fausto Fusaro
  • 506
  • 1
  • 4
  • 10
29
votes
4 answers

"pubsub error INVALID_ARGUMENT" when creating a subscription to URL endpoint

Created a test topic on GC pubsub and can create pull subscriptions no problem but as soon as I try to create a URL endpoint subscription I get a "pubsub error INVALID_ARGUMENT" Params Subscription Name: testingsub Endpoint URL:…
Adam McCrory
  • 391
  • 3
  • 7
29
votes
6 answers

Google PubSub - Counting messages in topic

I've looked over the documentation for Google's PubSub, and also tried looking in Google Cloud Monitoring, but couldn't find any means of figuring out what's the queue size in my topics. Since I plan on using PubSub for analytics, it's important for…
Gil Adirim
  • 1,834
  • 2
  • 21
  • 33
24
votes
2 answers

Equivalent for Kafka / AWS Kinesis Stream on Google Cloud Platform

I'm building an app that is constantly appending to a buffer while many readers consume from this buffer independently (write-once-read-many / WORM). At first I thought of using Apache Kafka, but as I prefer an as-a-service option I started…
Renan
  • 1,705
  • 2
  • 15
  • 32
21
votes
2 answers

Benefits with Dataflow over cloud functions when moving data?

I'm relatively new to GCP and just starting to setup/evaluate my organizations architecture on GCP. Scenario: Data will flow into a pub/sub topic (high frequency, low amount of data). The goal is to move that data into Big Table. From my…
Tsuni
  • 5,348
  • 2
  • 13
  • 20
21
votes
3 answers

Pros/cons of streaming into BigQuery directly vs through Google Pub/Sub + Dataflow

We have a NodeJS API hosted on Google Kubernetes Engine, and we'd like to start logging events into BigQuery. I can see 3 different ways of doing that : Insert each event directly into BigQuery using the Node BigQuery SDK in the API (as described…
renaudg
  • 662
  • 6
  • 11
20
votes
3 answers

Delay message processing and delete before processing

I need this ability to send push notifications for an action in a mobile app but wait for the user to undo the action until say 10 seconds. Is it possible to delay the processing of a message published in a topic by 10 seconds ? And then (sometimes,…
iPhoney
  • 771
  • 1
  • 10
  • 14
19
votes
2 answers

What is the best practise concerning topic names

What is the best practise concerning the topic name using google-pubsub. If I have theses events : customer email updated customer address updated customer created customer deleted article created article deleted ... What is the best practise…
Julien TASSIN
  • 5,004
  • 1
  • 25
  • 40
18
votes
2 answers

Is there a GCP equivalent to AWS SQS?

Im curious to understand the implementation of GCP's PubSub. Although Pubsub seems to point to follow a Publish-Subscribe design pattern, it seems more close to AWS's SQS (queue) than AWS SNS (that use publish-subscribe model). Why is think this is,…
user1384205
  • 1,231
  • 3
  • 20
  • 39
18
votes
4 answers

How To Rate-Limit Google Cloud Pub/Sub Queue

I'm using Google's Pub/Sub queue to handle messages between services. Some of the subscribers connect to rate-limit APIs. For example, I'm pushing street addresses onto a pub/sub topic. I have a Cloud function which subscribes (via push) to that…
emma ray
  • 13,336
  • 1
  • 24
  • 50
17
votes
2 answers

Google Cloud Functions Cron Job Not Working

I am trying to set up a scheduled function in Firebase Cloud Functions. As a simple test, I have tried to recreate the sample shown on the documentation page: const functions = require('firebase-functions') exports.scheduledFunction =…
Moshe
  • 6,011
  • 16
  • 60
  • 112
17
votes
3 answers

Testing Google Cloud PubSub push endpoints locally

Trying to figure out the best way to test PubSub push endpoints locally. We tried with ngrok.io, but you must own the domain in order to whitelist (the tool for doing so is also broken… resulting in an infinite redirect loop). We also tried…
hampusohlsson
  • 10,109
  • 5
  • 33
  • 50
17
votes
4 answers

Cloud Pub/Sub Demo : 403 User not authorized to perform this action. when try to push notification

I am learning Google Cloud Pub/Sub and following this official document : Writing and Responding to Pub/Sub Messages - Python When I deploy it to cloud, and try to submit the message, I get the following error : An internal error occurred: 403…
yusong
  • 479
  • 2
  • 6
  • 21
1
2 3
…
99 100