What is difference between Apache Kafka and GCP PubSub?

Question

What is difference between Apache Kafka and GCP PubSub? when to use kafka and when to use pubsub.

Answer 1

Since you did not provide your use case, I will state below the main characteristics of each tool.

1. PubSub: It is a cloud asynchronous messaging service that decouples senders and receivers provided by Google Cloud. It offers high availability and consistent performance at scale.

• No Ops: in PubSub you do not need to worry about partitions and shards.
• Scalability: is built-in without any required operation, it handles scalability automatically.
• Monitoring: you can monitor your process at a Topic and Subscription level within StackDriver.
• Access management: you can configure access at a project, Topic and Subscriber level.
• Reliability: it guarantees the the message will be delivered at least once. Although, it does not guarantee ordering (which can be handled in Dataflow).
• Message retention in PubSub: the minimum is 10 minutes and the maximum is 7 days.

2. Kafka: It is an open-source distributed publish-subscribe messaging ecosystem. It can be used on-prem or deployed in cloud environments.

• Scalability: it does not support automatic scalability. Thus, you need to increase partitions, replications etc, manually.
• Ordering: it can support ordered messages in the partition level.
• Reliability: it guarantees no data loss.
• Monitoring: it offers various types of built-in monitoring systems.

Notice that I just shared the main characteristics of each tool. Although there are many others which can be more relevant for your use case. Here are some links where you can find other information about each one's aspects: 1, 2, 3.