0

My Objective:

I have a very simple workflow I would like to implement. Given an ID, create a file in a s3 bucket called ID.txt only if that file does not already exist.

What I have tried so far:

I have read the doco and walked through some examples using aws-ruby-flow.
Makes total sense so far.

What I still don't get:

My desire is to have the workflow process SQS messages that contain s. What I don't grasp is how to kick off the workflow when queued messages appear.

Question

Do I need to implement my own polling service somewhere that triggers the workflow whenever it finds messages in SQS?

mconlin
  • 8,169
  • 5
  • 31
  • 37
  • See http://stackoverflow.com/questions/20548814/triggering-a-swf-workflow-based-on-sqs-messages. But maybe you could just trigger the workflow directly in SWF rather than doing it indirectly via posting a message to an SQS queue. – jarmod Aug 08 '15 at 14:28
  • yeah, I guess I was thinking that a SWF should be configurable in a way that says 'every N minutes, check the SQS and process messages if they exist' or similar. As of now I think I need to write me own job somewhere that polls the SQS queue, and then starts the workflow if messages are present... which seems like I am not using these capabilities correctly. – mconlin Aug 08 '15 at 14:39
  • Maybe you could use alestic's UTC (https://alestic.com/2015/05/aws-lambda-recurring-schedule/) as a cron to trigger a small Lambda event that polls your SQS queue and triggers the SWF workflow. That way you wouldn't be managing servers. – jarmod Aug 08 '15 at 15:02
  • @mconlin, try looking at AWS Lambda. You can create a Lambda job that kicks off the workflow when the SQS queue gets a message. – devonlazarus Aug 08 '15 at 15:05
  • What is your use-case for creating these files? If you provide more information about your larger goal, we might be able to provide a more appropriate implementation method. – John Rotenstein Aug 09 '15 at 06:37
  • File example is made up for simplicity and is not my actual use case. – mconlin Aug 09 '15 at 10:47

1 Answers1

1

The Amazon Simple Workflow Service can orchestrate work across distributed components. However, each component must "call into" SWF to check whether any work is waiting to be performed.

A very recent update now also allows Amazon SWF to trigger an AWS Lambda function rather than waiting for a component to request work.

However, Amazon SWF cannot be triggered by an Amazon Simple Queue Service (SQS) message.

It's worth stepping-back to look at your total workflow. For example, instead of pushing a message into SQS, you could:

  • Send to Amazon Simple Notification Service (SNS), which can trigger an AWS Lambda function, or
  • Trigger an AWS Lambda function directly, or
  • Simply upload to Amazon S3 at all times (if you're just creating an empty file, what's the harm in overwriting a file if it already exists?), or
  • Rethink why you are creating such a file in Amazon S3 and potentially use a database instead
John Rotenstein
  • 241,921
  • 22
  • 380
  • 470