Building common SaaS features à la serverless: Scheduled Tasks
Premise
More often than not, computer systems' architectures involve scheduled tasks that, for the context of this post, one could liberally distinct into two broad categories:
- either need to run once at some point in time in the future,
- or on a timed interval.
Historically, the implementation of such tasks was OS-specific. CRON jobs on Unix-based servers or Windows Services on Windows-based servers. Both involved some kind of expression language that represented a scheduling mechanism with which one could define when their task or service should run.
In more recent times, container orchestration platforms, such as Google's Borg and it's spiritual successor Kubernetes, have made the distinction between those two categories of executable code explicit; indeed Kubernetes has different workload resource types: Job and CronJob define tasks that run to completion and then stop. Jobs represent one-off tasks, whereas CronJobs recur according to a schedule. Please see the References section for a detailed read on those.
Usual examples of such jobs include, but are not limited to:
- usage reports automatically generated on a weekly/monthly/quarterly basis,
- E-mail/SMS batch jobs or
- CRON jobs checking for current billing or imminent subscription renewals.
The list goes on and on really, the SaaS business use-cases are endless.
By the end of this post, you'll know how to build AWS Lambda functions that will run either once, based on some predicate, or on a schedule using Amazon EventBridge.
Design
This is the high level overview of the proposed system architecture for Scheduled Tasks in AWS:
Create a Lambda function
- Sign in to the AWS Lambda console and choose Create function.
- In the Create function section, do the following:
- Choose Author from scratch.
- For Function name, enter Scheduler-with-EventBridge.
- For Runtime, choose .NET Core 3.1.
- Under Permissions, choose Create a new role with basic Lambda permissions.
3. Choose Create function.
Create an Amazon EventBridge Rule
1. Sign in to the Amazon EventBridge console and choose Create rule.
For this example, we invoke the Scheduler-with-EventBridge Lambda function every 5 minutes.
1. In the Create rule section, do the following:
- Under Name and description, enter EventbridgeScheduler.
- Under Define pattern, choose Schedule.
- For Fixed rate of, enter 5 minutes.
- Under Select event bus, choose AWS default event bus.
- Under Select targets, choose Lambda function and then choose the Scheduler-with-EventBridge Lambda function.
2. Choose Create.
.NET Implementation
Building a reusable Library
Eventually, we'll end up creating numerous EventBridge Lambdas over the course of our SaaS business' lifetime, trust me 🤞. We start by creating a .NET Standard 2.1 library project, which will be the dependency which all Lambda functions that need to process EventBridge events will reference to streamline & standardize the processing of EventBridge events with a connected Lambda function trigger.
Run dotnet new classlib in your command line of choice to create the library.
First, make sure you reference the Amazon.Lambda.CloudWatchEvents NuGet package. The savvy reader will notice how EventBridge is curiously missing from the name of this package but there's some, ahem, unfortunate product naming history there; originally the "Rules" UI was part of CloudWatch and EventBridge wasn't even a thing up until 2 years ago so 🤷♂️.
Building this library on top of the generic event handler function that I talked about two posts ago, we need a contract, one that an actual Lambda function's entry point should be able to call:
public interface
IScheduledEventHandler<TEventBridgeEvent> where TEventBridgeEvent : class
{
Task HandleAsync(TEventBridgeEvent eventBridgeEvent,
ILambdaContext context);
}There's something going on here beyond the provided method; notice the interface's generic constraint: TEventBridgeEvent needs to be a class (hint: the ScheduledEvent class from the Amazon.Lambda.CloudWatchEvents.ScheduledEvents namespace, but also, potentially, any other class that represents a concrete implementation of a scheduled event interface).
Next, we'll need to add a reference to our existing generic EventHandler library from the previous post in order to implement the EventBridgeEventHandler class that inherits from the IEventHandler<T> interface defined there, where <T> is the ScheduledEvent.
public class EventBridgeEventHandler<TEventBridgeEvent>
: IEventHandler<ScheduledEvent> where TEventBridgeEvent : class
{
private readonly ILogger _logger;
private readonly IServiceProvider _serviceProvider;
public EventBridgeEventHandler(ILoggerFactory loggerFactory,
IServiceProvider serviceProvider)
{
_logger = loggerFactory?.CreateLogger("EventBridgeEventHandler")
?? throw new ArgumentNullException(nameof(loggerFactory));
_serviceProvider = serviceProvider
?? throw new ArgumentNullException(nameof(serviceProvider));
}
public async Task HandleAsync(ScheduledEvent input,
ILambdaContext context)
{
using (_serviceProvider.CreateScope())
{
var handler = _serviceProvider
.GetService<IScheduledEventHandler<TEventBridgeEvent>>();
if (handler == null)
{
_logger.LogCritical(
$"No INotificationHandler<{typeof(TEventBridgeEvent).Name}> could be found.");
throw new InvalidOperationException(
$"No INotificationHandler<{typeof(TEventBridgeEvent).Name}> could be found.");
}
_logger.LogInformation("Invoking EventBridge handler");
await handler.HandleAsync(input, context);
}
}
}Pretty straightforward, we just wire up the logging factory & dependency injection services here, add some rudimentary exception handling and asynchronously process each of the available records that constitute the invocation event.
That's all it is! Before we move on to using our new library in a Lambda function .NET project though, it's worth discussing the DI part briefly.
Microsoft has provided it's own implementation of a dependency injection container in .NET (in the form of a NuGet package) since .NET Core 2.1, called Microsoft.Extensions.DependencyInjection.
If you're looking to do DI as part of any library, you'll need to implement the IServiceCollection interface in a static class, so that the framework is able to collect the necessary service descriptors.
For our library, this will look like this 👇🏻
public static class ServiceCollectionExtensions
{
public static IServiceCollection
UseScheduledEventHandler<TEventBridgeEvent, THandler>
(this IServiceCollection services)
where TEventBridgeEvent : class
where THandler : class, IScheduledEventHandler<
TEventBridgeEvent>
{
services.AddTransient<
IEventHandler<ScheduledEvent>, EventBridgeEventHandler<
TEventBridgeEvent>>();
services.AddTransient<
IScheduledEventHandler<TEventBridgeEvent>,
THandler>();
return services;
}
}Using the Library with an AWS Lambda .NET project template
Let's create an AWS .NET Lambda project: dotnet new lambda.EmptyFunction. If you've missed the part about installing the AWS project templates for the dotnet CLI, please check my previous post.
Next, reference both the generic Event function library as well as the EventBridge specific one you just created.
Modify Function.cs, which is the entry point for Lambda function invocations as follows:
public class Function : EventFunction<ScheduledEvent>
{
protected override void Configure(
IConfigurationBuilder builder)
{
builder.Build();
}
protected override void ConfigureLogging(
ILoggerFactory loggerFactory,
IExecutionEnvironment executionEnvironment)
{
loggerFactory.AddLambdaLogger(new LambdaLoggerOptions
{
IncludeCategory = true,
IncludeLogLevel = true,
IncludeNewline = true
});
}
protected override void ConfigureServices(
IServiceCollection services)
{
// TODO: services registration (DI) &
// environment configuration/secrets
services.UseNotificationHandler<
ScheduledEvent,
YourAwesomeImplEventBridgeEventHandler>();
}
}Startup.csY'all know exactly what's up above already. All that's left at this point is to create a class to implement your business logic, YourAwesomeImplEventBridgeEventHandler.cs.
public class YourAwesomeImplEventBridgeEventHandler : IScheduledEventHandler<
ScheduledEvent>
{
private readonly ILogger<
YourAwesomeImplEventBridgeEventHandler> _logger;
public YourAwesomeImplEventBridgeEventHandler(
ILogger<YourAwesomeImplEventBridgeEventHandler> logger)
{
logger = _logger ??
throw new ArgumentNullException(nameof(logger));
}
public async Task HandleAsync(
ScheduledEvent eventBridgeEvent,
ILambdaContext context)
{
// TODO: your business logic goes here
}
}Lastly, deploy the Lambda function using the Amazon.Lambda.Tools CLI by running dotnet lambda deploy-function Scheduler-with-EventBridge.
If you've missed the part about the installation & usage of this .NET global tool, please check the end of the ".NET Implementation" section of my previous post.
Conclusion
The implementation part of this post turned out remarkably similar to the previous one, wouldn't you say so?
That's the whole point of thinking AWS Lambda triggers with this mental model! I'm sure that at this point you can go on to build a .NET library for whatever AWS service acts as a Lambda trigger for your business.
What's next
In the next post, we'll go over data replication & consistency from a NoSQL, typically serverless-first database like AWS DynamoDB to a SQL store like MySQL, using DynamoDB Event Streams (event-driven architecture 'n all) & AWS Lambda.
The goal will be to exhibit how one might go on about building a SQL database for reporting or business intelligence purposes (and all the tooling that can be used on top of SQL like Power BI), when their application's source of truth is a NoSQL database. I will also go over alternative approaches (read: existing managed AWS services that will cost you $$$ compared to doing this using Lambda) in achieving the same end result in the AWS cloud.
References
