IOpipe Alerts 🚨 v1.1.0

Since we launched IOpipe alerts nearly 7 months ago, we’ve seen how powerful they can be as a first line of defense against erroring, slow, or runaway lambda functions. We’ve triaged several incidents internally where we were dogfooding our own product (surprisingly easy). So after you launch a new function in your environment, ensure it’s covered by an alert!

Launching a lambda with no alert coverage

Let’s go over the current alert methods and how they have helped:

Errors

This one is pretty easy. If a function has errors, you probably don’t want that. And you’re probably going to want to know when that happens. Alerting on the number of errors within a time window is a simple, effective method to get eyes on a potential issue. Is a dependent service down? Did the team deploy a syntax error (try eslint for js)? Was an event malformed (perhaps try middy)?

Invocation Count

The number of invocations per function over time is obviously contextual. Are you running a cron-like lambda once a day? Is it a signup form funnel with spikes during the afternoon? Or maybe it’s a data pipeline that has millions of invocations each week (or day!). An alert on an invocation count that is above or below a threshold your team cares about is an easy win for visibility without the need to stare at a dashboard all day.

Invocation Duration

Duration is similar to errors in that it can signify that reliant services (apis, databases, etc) might be having issues. But maybe those services are just a little slow instead of completely bailing on you. Perhaps you have high concurrency on a function, overloading connection count to your database. Duration alerting can help you recognize the issue as it’s happening. And IOpipe tracing can help you identify which service or code path is at fault 😎.

So, that all sounds pretty nice, right? Here’s where we talk about the 🆕 stuff!

Custom Metric Alerting

We don’t want to limit your use cases for IOpipe alerting to the metrics we collect by default — so bring your own! Use custom metrics to output values that are meaningful to the context of your function, and now you can alert on those.

Alert if the number of coins collected is less than 100

Trace Measure Alerting

It turns out that tracing code paths or requests is pretty powerful, so we’ve unlocked the power of alerting on durations inside your lambda code (so meta). Just include the IOpipe trace plugin, set up a new rule, and bang! You won’t even have to guess which dependent service is at fault.

Alert if the record-high-score trace is greater than 200ms

Along with new metrics, we’ve made some updates that make the user experience that much sweeter.

Global 🌏 and Multi-function Alerts

With IOpipe alerts version 1.0, we took an iterative approach and began with the limitation that an alert only applied to a single function. Since then we’ve seen compelling reasons for multi-function and project-wide alerts. We are happy to announce both today. Global alerts are particularly special because they apply not only to the functions currently inside of the project, but any that are launched in the future too 🙌.

Using the “Global” option when creating an alert

More informed alert creation

We’ve also provided extra feedback mechanisms for the alert workflows. You’ll see which rules are matched and how many invocations would currently trigger the alert, or an all-clear if you’re good to go.

A rule signifying a “match” if invocations are found with the rule criteria

A warning that the current alert will be created in ALARM state due to 45 matched invocations

You can also specify if you’d like the alert to change status based on every rule matching, or any of them.

A new alert setting specifying if the alert should trigger only if every rule matches

Looking Forward

Our ultimate goal with IOpipe alerts is to help you and your team identify issues within your serverless architectures quickly with as much context as we can provide. You can also use them as smoke tests or even as part of your end-to-end testing paradigms to give you extra piece of mind (we do).

We have a running list of enticing ideas that we believe can further enhance IOpipe alerts — but due the nature of our small team we can’t just ship from the hip™. Your feedback is extremely important to us, so let us know how we are doing — and how we can help.

If you’d like to try IOpipe, we’ve got a free trial with no commitment and you can sign up here. If you want to connect with the team, participate in feedback sessions, or have general questions — peep us in our community slack channel.