Serverless dynamic scheduler

One of the critical parts of a serverless application is the operations scheduling. Sometimes a simple cron expression defined in a SAM template may be too simplistic as a solution.

In a complex environment the application cloud requires to schedule an event in the future, maybe to send an abandoned cart notification or unpublished draft content.

Fixed recurring

A simple solution for this challenge is to schedule a Lambda function every minutes or every hour, retrieve the operations list (abandoned carts, draft contents) and evaluate the target notification date with the current one.

This solution works if there are a lot of events to process every cron schedule tick although there will be a lot of unnecessary Lambda executions.

Fixed recurring schema

Lambda executions are not so expensive, for example a Lambda execution every 15 minutes will cost $0.01 every month.

EventBridge rules

A more fancy solution could be using the AWS EventBridge API creating and deleting Rules in order to schedule a Lambda execution in a single future date.

Before the Lambda function exits the EventBridge rule should be deleted to avoid duplication and avoid reaching rules count quota.

event bridge rules schema

This solution can scale up to the rules quota, it requires some logic to clean-up executed rules.

AWS APIs are free but we can incur in throttling and quota limitations.

SQS Queue

Using an SQS queue the message can hidden for a certain amount of time, sadly, only for standard, can be only up to 15 minutes.

SQS schema

Useful for short scheduling but not suitable for solution that requires a date in the future of days or even months.

StepFunction

There is a StepFunction state type called Wait that can pause the states transition for a specific amount of seconds or until a certain date!

The standard StepFunction execution price is based on state transitions, this means the “schedule tick” will be charged.

stepfunction schema

The standard StepFunction execution can stay active until a year, this means it can wait up to one year since the execution start date.

There is a quota on StepFunction concurrent executions (10.000) that needs to be increased in case the scheduled events begin to proliferate.

Scheduler service

I tried to implement the most pluggable solution in order to re-use this solution as a service for other projects.

I used an EventBridge bus to send schedule commands and receive the trigger for the scheduled operation.

{
  "Source": "test",
  "DetailType": "Schedule Start",
  "Detail": {
    "id": "test:schedule", 
    "cron": "30 * * * *"
  }
}

When the Start event is received a Lambda function will start a StepFunction execution that will wait until the next scheduled date.

StartAt: WaitUntilDate
States:

  WaitUntilDate:
    Type: Wait
    TimestampPath: $.timestamp
    Next: SendCallbackEvent

  SendCallbackEvent:
    Type: Task
    Resource: 'arn:aws:states:::events:putEvents'
    Parameters:
      Entries:
        - EventBusName: "${EventBusName}"
          Source: "${EventSource}" 
          DetailType: "Schedule Tick"
          Detail.$: $.schedule
    End: true

When the StepFunction ends the wait operation will send an event back to the EventBridge bus. This callback event (called “tick”) can easily trigger a Lambda function to perform the scheduled operations.

{
  "Source": "com.my-stack-name",
  "DetailType": "Schedule Tick",
  "Detail": {
    "id": "test:schedule"
  }
}

When the “tick” event is emitted also a second Lambda function will be executed to elaborate the next scheduled date and start the StepFunction execution again.

recurring scheduling schema

If the event is a one-shot execution that specify just a duration:

{
  "Source": "test",
  "DetailType": "Schedule Start",
  "Detail": {
    "id": "test:duration-5m", 
    "duration": "5m"
  }
}

the Lambda function connected to Tick event will ends without starting a new StepFunction execution, this will break the loop.

Credits: LibreOffice Draw.

Repository: daaru00/serverless-dynamic-scheduler