SQS & Lambda: teaming up
(This post has been updated to reflect the release of native support for SQS in Lambda in July 2018)
I like SQS. It’s one of my favorite AWS services. It’s got a clear purpose, it handles unexpected errors well with a decent config and dead letter queues, and the API is simple enough.
We use currently us SQS for 2 systems at IOpipe:
Our weekly email pipeline, which you can see a snippet of our own production version below (yes we run lots of lambda too!)
We started out with SNS but we ran into some strange situations that seemed hard to debug — to each their own I guess?
We also use SQS for our alerts pipeline. We run 1 lambda (aka Scheduler) on a CloudWatch event to gather up all of the alerts we need to evaluate for the time period. Then we create separate SQS messages for each. Seconds later, lambda #2 (aka Executor) evaluates the logic it needs to for an SQS message. This is a scalable solution that gives us some options if one of the components is broken or running slow.
But you might be thinking — how does the Executor know about the individual SQS message?
When we originally wrote this in December 2017, no native solution for SQS triggers in Lambda existed, but here in July 2018, AWS has released the ability to implement SQS custom triggers to AWS Lambda functions. You can learn more about it in their documentation.
So at the time, we needed to build something ourselves. You might come across this nice library by the BBC and implement that as a long-running service you maintain. There’s even an SQS-to-Lambda package built on top of that here. You would probably end up using the new native trigger that AWS now provides.
While those may work for you, we had one extra requirement that those configurations didn’t seem to quite address — we wanted the Executor to decide if it was time to delete the SQS message. That way, we can retry later if our SQS config allows it. The AWS native solution just deletes the message for you when the Lambda completes successfully, which still doesn’t cover what we needed. In other words we need the power of…async
So we created a library that addressed that functionality, and added a few sprinkles of sugar like a pre-invoke message formatter and the ability to handle multiple queues.
https://github.com/iopipe/sqs-to-lambda-async
We have been successfully running an implementation of this on an Elastic Beanstalk setup for months that handles both of our SQS use cases at the same time.
But you say, enlightened:
EC2 Instance? Are you crazy! We don’t run servers anymore!
Great job, you’re awesome. Depending on your SQS needs — namely how fast the SQS message needs to be processed — you could run a lambda every 5 minutes or more that executes this library. Cool. 😎 Maybe check out Servers.lol to see if it’s cost or speed that will ultimately win the day for you.
As always, we welcome feedback over at GitHub.
If you’d like to try IOpipe, we’ve got a free trial with no commitment and you can sign up here. If you want to connect with the team, participate in feedback sessions, or have general questions — peep us in our community slack channel.
