1

I'm completely stuck. I am using SQS to communicate between server to server. I add a message "init_scrape" to my queue once at 3am. The queue is polled by a script on my server every 5 mins to check for messages on the queue and run tasks. This is my email log:

2013-07-29 03:05:01,857 INFO  Queue Messages Found: run_scrape init_scrape
2013-07-29 03:05:01,858 INFO  Init_Scrape Running
2013-07-29 03:22:45,055 INFO  init_scrape deleted from queue
2013-07-29 03:22:45,211 INFO  Queue Messages Found: run_scrape

At this point, init_scrape has been removed from the queue leaving just run_scrape. Absolutely nothing else adds the init_scrape message to the queue, and yet this is the next log:

2013-07-29 03:25:02,054 INFO  Queue Messages Found: run_scrape init_scrape
2013-07-29 03:25:02,055 INFO  Init_Scrape Running
2013-07-29 03:42:44,739 INFO  init_scrape deleted from queue
2013-07-29 03:42:44,879 INFO  Queue Messages Found: run_scrape

Why is it that I only send the message to the queue once, and yet it seems cropping up even when my script deletes it from the queue?

EDIT: Is this something to do with visibility_timeout?

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
J.Zil
  • 2,397
  • 7
  • 44
  • 78
  • 1
    Might be a dumb question, but are you calling DeleteMessage on the received message (init_scrape) when it prints "init_scrape deleted from queue"? – jjm Aug 06 '13 at 00:40

1 Answers1

1

SQS guarantees that the message will be delivered at least once. They make no assurance that it won't be delivered a whole bunch of times. SQS is really neat in that it scales, but you actually pick up some odd behaviors due to how they made it scale, like this and messages being delivered out of order that make it a little bit of a hassle for projects that don't really need scale.

Jeff
  • 6,646
  • 5
  • 27
  • 33
  • http://stackoverflow.com/questions/13484845/what-is-a-good-practice-to-achieve-the-exactly-once-delivery-behavior-with-ama/13490008#13490008 – Jeff Jul 29 '13 at 20:33