Serverless out of Control

Inspiration for this post comes from Kevin Vandenborne’s article, and the Serverless Failure Stories repository

We love using serverless whenever we can. It makes a ton of sense for many of our client-fat projects that only need to interact with a REST API or PaaS (see our typical cloud setup here) .  In our experience, we’ve found the following to be two of the best parts of serverless computing:

• Each request spins up a new function, gracefully scaling up
• You only pay for what you use

Ironically, these features combined to create a horrible monster on one of our projects.

We were prototyping a file sharing app using the Parse Server library (a favorite of ours). Parse does most of the legwork needed to save persistent data from the frontend, while scaling up gracefully in security and complexity (example project here). We decided to throw a Parse Server onto a serverless express app.  This setup virtually gave us a server for free to showcase our project.

Because the object is overwritten, it triggers the event again because it's seen as a new object - which in turn triggers the Lambda function again - Kevin Vandenborne's serverless nightmare

An event triggering a Lambda repeatedly is bad. Potentially even worse is a Lambda triggering itself.

Here’s a code snippet taken directly from the Parse Dashboard repo:

// actions.types.js
var express = require("express");
var ParseServer = require("parse-server").ParseServer;
var ParseDashboard = require("parse-dashboard");

var api = new ParseServer({
  // Parse Server settings
});

var options = { allowInsecureHTTP: false };

var dashboard = new ParseDashboard(
  {
    // Parse Dashboard settings
  },
  options
);

var app = express();

// make the Parse Server available at /parse
app.use("/parse", api);

// make the Parse Dashboard available at /dashboard
app.use("/dashboard", dashboard);

var httpServer = require("http").createServer(app);
httpServer.listen(4040);

A few modifications, and we tossed this directly into a serverless express application. If you know Parse really well, you may recognize the problem:

The Parse Dashboard package makes a request TO the Parse API to retrieve all data to display. In other words, this Lambda is calling itself!

The Damage

In a normal server environment, no problem. At worst, an endless feedback loop like this will cause the server to crash. It’s also less likely to happen because of the way servers handle requests. In our case, more and more Lambdas spawned until they consumed all of our resources. This went on for roughly three full days before we became aware of the monthly AWS bill:

The result was a 100,000 percent increase in hosting costs, a tough pill to swallow.

Main takeaway: a physical server can cushion a lot of what might go wrong. Mainly,

• You control the number of instances
• A server can crash and restart if things go awry

In short, take full advantage of all the fruits of serverless. But always exercise a bit more caution. There’s no fallback if your code spirals out of control.