7

This is similar to this other question but with a bit of a twist: I read in the specification that the message-id for AMQP messages should be set by the application itself, so in theory I could use that to guarantee a certain degree of uniqueness, right?

My main question is now: In what scope is that message-id garantueed to be unique? For the messages currently enqueued inside a specific queue? Over all queues? Over the universe? :-)

And is this behaviour standardized? I plan to use RabbitMQ here, but it would be nice to have something not vendor specifc :-)

Thanks.

Community
  • 1
  • 1
Horst Gutmann
  • 10,910
  • 2
  • 28
  • 31
  • _Why_ do you want messages to be unique? If you give us more info about what you are trying to achieve, we will be able to help you better. – cdeszaq Feb 14 '12 at 15:04
  • We use RabbitMQ a lot and do not specifically set the message-id and have never had problems. We have 6 application, 20 exchanges, 40 odd queues and about 6000 messages a month. So does not answer you question but does suggest that it does not need to be unique ? – Simon Thompson Aug 25 '11 at 21:27

2 Answers2

2

Another suggestion is according to the dump pipes - smart endpoints school of thought.

You could handle uniqueness in your application, using some sort of shared state.

We had the same problem when switching from Gearman to RabbitMQ. We use memcached to keep track of unique message ID's posted and consumers drop messages for which the message ID is already stored in memcache (duplicates). You could also check memcache before putting it on the queue altogether.

This frees you from using this feature in your message bus layer (so you can more easily switch between brokers, also those who do not guarantee uniqueness)

Community
  • 1
  • 1
Joost Pastoor
  • 253
  • 1
  • 7
  • 1
    What command do you use in Memcached there? From what I've seen of the commands available there it kind of sounds like a good entry for race-conditions. Wouldn't Redis' SETNX be more useful here? – Horst Gutmann Jul 23 '15 at 20:52
  • 1
    You can use [add()](http://php.net/manual/en/memcached.add.php) which ensures only 1 process can set the key. (Rest gets Memcached::RES_NOTSTORED). This works for a single memcached instance, not sure how it works in a cluster. – Joost Pastoor Jul 24 '15 at 07:07
1

Message Id is application-specific only and may be not unique at all. You have to take care of uniqueness by yourself.

pinepain
  • 12,453
  • 3
  • 60
  • 65