1

I have to implement Message Queue on site which do 20K/Day write operation in Database. We are willing to introduce message queues to take load off these write operations. I dont know much about Message Queue Implementation. If we implement it on one server only will it be helpful? or suggest some better impentation.

Also which MQ technology would be useful? We are php Shop with Apache

Volatil3
  • 14,253
  • 38
  • 134
  • 263
  • 2
    *(related)* [What is the best format messages for queing](http://stackoverflow.com/questions/2444157/what-is-the-best-to-format-messages-for-queueing/2444275#2444275 "Links to an overview of available Message Queues that work with PHP") – Gordon Oct 21 '10 at 10:40
  • Thanks for detailed answer. I am a bit inclined towards beanstalkd. Its not clear whether,like Krestal,it was also written in Ruby or written in C – Volatil3 Oct 21 '10 at 11:48
  • @Volatil3 you're welcome. feel free to upvote the linked answers to indicate they have been helpful. Judging by the [beanstalkd source files on Git](http://github.com/kr/beanstalkd) I'd say it's written in C. – Gordon Oct 21 '10 at 12:00
  • Can you explain a little what type of app this is and how you think offloading to a queue would help. I'm interested too. Based on that, I'd also give you a suggesting. – Till Oct 21 '10 at 14:36
  • @Till: It's actually a portal having 7 million members with 2 million pageviews/day. we are using MySQL as database – Volatil3 Oct 22 '10 at 05:17
  • @Volatil3: And if I may ask, how do you use mysql? And what is the bottleneck that you are trying to overcome? – Till Oct 23 '10 at 11:50
  • 20K operation per day is not a lot. When that is performance problem, you have to find out the original problem. Trying to use queing will probably not solve your problem. – Paco Oct 24 '10 at 12:20

2 Answers2

1

There's lots to choose from (nobody's mentioned rabbitmq so far) however if you are really only processing 20,000 transactions a day then using message queueing, particularly on a single server is not the right way to solve the problem.

symcbean
  • 47,736
  • 6
  • 59
  • 94
  • @symcbean: Then what would you – Volatil3 Oct 21 '10 at 12:51
  • 1
    Look at why the system can't currently cope with this volume. What database? What tuning has been done already? How is it configured? – symcbean Oct 21 '10 at 13:51
  • I think it's a suitable pattern. But I'm interested what you would suggest. – Till Oct 21 '10 at 14:35
  • What about kestrel,made by Twitter? – Volatil3 Oct 22 '10 at 05:18
  • 1
    So instead of writing data to (real) database, you are proposing writing data to (queue) database, retrieving data from (queue) database, writing data to (real) database. And you expect this to be faster when running on a single machine? – symcbean Oct 22 '10 at 08:22
  • There's a replication setup of One master and multiple slaves neither I said that only single Queue instance would be initiated. – Volatil3 Oct 22 '10 at 11:15
  • 1
    @symcbean: I agree that performance on a single server wouldn't really improve, but performance and scalability are two different concerns. I guess offloading to a queue is really faster regardless than writing to a database were locks are a natural thing. – Till Oct 23 '10 at 11:48
  • @Till: You're damn right. We are experiencing Db locks which often cause query to get stuck. – Volatil3 Oct 24 '10 at 08:55
0

So breaking it down, it sounds like you also have some MySQL issues.

Maybe we can help you optimize this part of the stack first before you setup a queue? Especially because I'm not yet sure how your application uses MySQL and how or where you plan to queue items to offload from a potential write-lock.

I just answered a similar question a few days ago:

Let me know if this helps or how we can help further.

Community
  • 1
  • 1
Till
  • 22,236
  • 4
  • 59
  • 89