State Machines in AWS
I love state machines. It is the abstract level of a flow. Recently I am obsessed with AWS Step Functions and so my approach to solve infrastructural problems changed with Step Functions.
AWS Step Functions, basically, coordinates multiple AWS services (well mostly used with Lambda functions) into serverless workflows. Workflows are made up of steps, with the output of one step acting as input into the next.
The great part with Step Functions is visualisation of serverless workflow.
You can also see every execution’s graph.
The language to define the workflow is Amazon State Language which is very easy to learn.
Here is an example
---
Comment: An example
StartAt: FirstState
States:
FirstState:
Type: Task
Resource: FUNCTION_ARN
Next: LastState
LastState:
Type: Task
Resource: OTHER_FUNCTION_NAME
End: trueThis needs to be json, but I used yaml to show simpler.
The language specification documentation is very short and easy, yet you can create complex enough workflows for your serverless design. Check out the docs
We’ve faced some challenges and solved many of them with AWS best practices. I will explain some of the designs to you are probably going to need during an implementation.
SQS
You should consider AWS Step Functions when you need to coordinate service components in the development of highly scalable and auditable applications. You should consider using Amazon Simple Queue Service (Amazon SQS), when you need a reliable, highly scalable, hosted queue for sending, storing, and receiving messages between services. Step Functions keeps track of all tasks and events in an application. Amazon SQS requires you to implement your own application-level tracking, especially if your application uses multiple queues. The Step Functions Console and visibility APIs provide an application-centric view that lets you search for executions, drill down into an execution’s details, and administer executions. Amazon SQS requires implementing such additional functionality. Step Functions offers several features that facilitate application development, such as passing data between tasks and flexibility in distributing tasks. Amazon SQS requires you to implement some application-level functionality. While you can use Amazon SQS to build basic workflows to coordinate your distributed application, you can get this facility out-of-the-box with Step Functions, alongside other application-level capabilities.
Sub-Minute Frequency
SNS
You can create Step Function subscriptions for SQS, and you can create SQS subscription for SNS. So indirectly your step function can subscribe to your SNS topic. But there is no way to subscribe a FIFO SQS queue. You should create a poller step function for that. Personally I don’t like to design a poller on lambda functions or step functions. It doesn’t feel scalable and also not cheap. There a re some sample projects here in https://serverlessrepo.aws.amazon.com/applications
Old Executions’ Graphs
You can always fetch the old executions and create diagrams of them like you see on AWS console. I am creating an open source library for it, soon I will publish it.
Limits
There is a concurrency limit for lambda functions and that you can always change your account’s limits via AWS Support Centre. but you can not change the hard limits of step functions https://docs.aws.amazon.com/step-functions/latest/dg/limits.html , so beware of the limits of step functions if you are going to build a massive application on step functions.
Framework for Development
Serverless is an amazing framework if you are using Nodejs. You can use almost all CloudFormation features with serverless also you can import many plugins written by the community. It also supports Azure and . google Cloud.
Deployment and management of Lambda functions are already easy but it can be messy. With serverless you will keep your configuration clean, which is very important if you are designing your application decoupled. Serverless is a very light framework. That is why I like it. https://serverless.com/
Further Reading and Related Resources
- AWS Step Functions API Reference
- AWS Step Functions Command Line Reference
- Product information for Step Functions
