Stack Overflow
Stack Overflow

Questions tagged [aws-step-functions]

AWS Step Functions is a visual workflow service that helps developers use AWS services to build distributed applications, automate processes, orchestrate microservices, and create data and machine learning (ML) pipelines.

AWS Step Functions

AWS Step Functions is a visual workflow service that helps developers use AWS services to build distributed applications, automate processes, orchestrate microservices, and create data and machine learning (ML) pipelines.

Workflows

A workflow describes how people get work done from start to finish. A workflow consists of the steps and states in a process. Workflows can appear as a series of steps in a checklist, or as a diagram that visualizes those steps. Workflows promote logical separation between the control flow of your job's stepwise logic and the actual units of work that contain your unique business logic. With this division, you can manage, maintain, and scale the state machinery of your application separately from the core business logic. As your business requirements change, you can easily change application logic without changing the underlying state machinery, task dispatch, and flow control. Workflows

Amazon States Language

Step Functions implements the open source Amazon States Language (ASL), which is a JSON-based workflow definition language. ASL models workflows as a state machine, where you define a set of states with transitions between them. ASL includes several state types that support the flow of data through your workflow and control of the logic (e.g., Pass State, Choice State, Parallel State, and Fail State). Task states are used to execute work.

The state machine model for ASL has the advantage of making these workflows inherently visual. This is true for creation, editing, and sharing of workflow definitions but also for interpreting and observing historical behavior of workflow executions.

Step Functions Service Integrations

Step Functions includes direct integration with over 220 AWS Services. Optimized Service Integrations provide simplification to make it easier to integrate with services such as AWS Lambda, AWS Glue, Amazon SQS, and Amazon EMR. In addition, AWS SDK Service Integrations provide direct integration with over 11,000 API Actions from over 220 AWS Services.

Workflow Studio

Workflow Studio is a low-code visual workflow designer for Step Functions. Workflow Studio is available in the AWS Console and makes it easy build, edit, visualize, and share your workflows.

When to use this tag?

Use this tag when you have a question about the Step Functions service, about using the Step Functions service along with other AWS Services, or if you have questions about how to build cloud workflows in AWS. Also use this tag for questions you have about the Amazon States Language

Other Step Functions Resources

1505 questions
58
votes
3 answers

Airflow versus AWS Step Functions for workflow

I am working on a project that grabs a set of input data from AWS S3, pre-processes and divvies it up, spins up 10K batch containers to process the divvied data in parallel on AWS Batch, post-aggregates the data, and pushes it to S3. I already have…
sanjayr
  • 1,679
  • 2
  • 20
  • 41
43
votes
2 answers

AWS SQS trigger Step Functions

Quick question: Is it possible to trigger the execution of a Step Function after an SQS message was sent?, if so, how would you specify it into the cloudformation yaml file? Thanks in advance.
Carlos
  • 855
  • 2
  • 9
  • 18
37
votes
6 answers

AWS step functions and optional parameters

I would like to have a default value for a parameter passed into a step function e.g., "Parameters": { "foo.$": "$.foo" OR "bar" if "$.foo" not specified } is there a way to do this natively with JSONPath or do I have to use a choice + pass…
wrschneider
  • 17,913
  • 16
  • 96
  • 176
37
votes
2 answers

AWS Lambda vs AWS step function

I am designing an application for which input is a large text file (size ranges from 1-30 GB) uploaded to S3 bucket every 15 min. It splits the file into n no of small ones and copy these files to 3 different S3 buckets in 3 different aws regions.…
dhamu
  • 605
  • 1
  • 7
  • 17
36
votes
7 answers

AWS Step Function - Wait until an event

I have a use case where I have a AWS Step function that is triggered when a file is uploaded to S3, from there the first step runs an ffprobe to get the duration of the file from an external service such as transloadit where the output is written…
khinester
  • 3,398
  • 9
  • 45
  • 88
35
votes
6 answers

Nested Step Function in a Step Function: Unknown Error: "...not authorized to create managed-rule"

I have a Step Function (Parent) created in a SAM/CloudFormation template that, among other things, calls another Step Function (Child). I'm following the instructions on calling Child, from Parent, using the service integration pattern. But I'm…
Matt
  • 907
  • 1
  • 8
  • 17
34
votes
4 answers

How to concatenate constant string with jsonpath

I have AWS step machine and one of the step is used to notify failure using SNS service. I want to select some metadata from input json into outgoing message. So i am trying to concatenate constant string with jsonpath like below "Notify Failure":…
LP13
  • 30,567
  • 53
  • 217
  • 400
34
votes
5 answers

Api gateway get output results from step function?

I followed tutorial on creating and invoking step functions I'm getting output in my GET request of api as { "executionArn": "arn:aws:states:ap-northeast-1:123456789012:execution:HelloWorld:MyExecution", "startDate": 1.486772644911E9 } But,…
Jagdish Idhate
  • 7,513
  • 9
  • 35
  • 51
33
votes
1 answer

AWS Data Pipeline vs Step Functions

I am working on a problem where we intend to perform multiple transformations on data using EMR (SparkSQL). After going through the documentation of AWS Data Pipelines and AWS Step Functions, I am slightly confused as to what is the use-case each…
archilius
  • 621
  • 6
  • 16
27
votes
4 answers

How to specify multiple result path values in AWS Step Functions

I have a AWS Step Function State formatted as follows: "MyState": { "Type": "Task", "Resource": "", "ResultPath": "$.value1" "Next": "NextState" } I want to add a second value but can't find out how anywhere. None of the AWS…
Jake T.
  • 4,308
  • 2
  • 20
  • 48
24
votes
5 answers

Parallel States Merge the output in Step Function

Is it possible to have following kind of Step Function graph, i.e. from 2 parallel state output, one combined state: If yes, what would json for this looks like? If not, why?
hatellla
  • 4,796
  • 8
  • 49
  • 101
24
votes
3 answers

Why would a step function cancels itself when there are no errors

I have a step function (illustrated below) and I was working out the few remaining issues when all of a sudden I hit a rather unexpected result: All tasks that have completed, completed successfully without error All remaining tasks simulateously…
ken
  • 8,763
  • 11
  • 72
  • 133
23
votes
1 answer

Pass multiple inputs into Map State in AWS Step Function

I am trying to use AWS Step Functions to trigger operations many S3 files via Lambda. To do this I am invoking a step function with an input that has a base S3 key of the file and part numbers each file (each parallel iteration would operate on a…
sedavidw
  • 11,116
  • 13
  • 61
  • 95
22
votes
1 answer

Step function exceeding the maximum number of characters service limit

My state in a step function flow returns an error of state/task returned a result with a size exceeding the maximum number of characters service limit.. In the step function documentation, the limit for characters for input/output is 32,768…
Abdullah
  • 603
  • 3
  • 8
  • 19
22
votes
4 answers

Pass and use input (parameters) to a lambda task from a step function

I have a simple step function launching a lambda and I am looking for a way to pass parameters (event / context) to each of several consequent tasks. My step function looks like this: { "Comment": "A Hello World example of the Amazon States…
Ilia Gagarin
  • 361
  • 1
  • 2
  • 6
1
2 3
…
99 100